A research paper was published this summer by Dr Jingxuan Yang of UCL that most likely will be conveniently ignored or quickly forgotten by most people in our industry, but which reveals quite starkly how broken building performance modelling is in practice.
We have talked at length in recent years about the performance gap between the results we get from energy modelling and real-life energy consumption in buildings. And most people have been assuming, or buying the idea, that the problem was just with the modelling methods. That models used for regulation compliance are not detailed or bespoke enough for estimating real energy consumption. Due to this, some energy efficiency standards have also been shifting to measured energy use targets instead of meeting theoretical, model-based targets.
But, is that really the only problem? That the standard modelling methods and tools used for regulations compliance are not accurate enough and if we shift to more detailed models the performance gap problem will be solved?
My experience tells me that there is a lot more to improve in this sector than just the approach and detail of the modelling tools. And this study demonstrates it quite starkly.
The stark results
The title of the paper is “Assessing the impact of modelling approach, modeller and inter-model variability on building performance simulation results”. And, as a quick summary, it demonstrates how different the results obtained by different energy modellers can be for the same building, having received the same information on specifications and even the building use profiles to apply.
It also analyses the differences between different software (IES VE and DesignBuilder) and between detailed HVAC and template HVAC system approaches. And discusses the reasons for the differences, providing insights to support improvements. It is a great research work and not an easy one to put together, I imagine. Worth a read by everyone working in this discipline.
We will discuss the reasons and insights later but, first of all, I want to focus on the stark numbers:
The total energy consumption of the building predicted by the 20 energy modellers participating in the study ranged from 24 kWh/m² to 186 kWh/m².
Yes, you read that right and there is no typo. The result from one of the modellers was more than 6 times higher than another.
And, in general, the standard deviation was around 40 kWh/m² from the mean result of 92 kWh/m². I am no expert in statistics, but looks like a huge spread of results and very little consistency for technical calculations using the same input data. And this is for the total energy result, the results for the different energy uses were even more inconsistent.
The study also lists the misunderstandings and input errors in the models that drive some of the inconsistency in results (some of them quite blatant for modellers with years of experience by the way). But I am not so surprised, really, by the results and findings of this study. When I review models or results by other modellers, I also quite often find significant misunderstandings in inputs, wrong or unnecessary assumptions being used, and suspicious results in general.
However, my aim here is not criticizing other modellers (nor the aim of the authors of that study) but bringing to light a serious problem with serious implications in this discipline and the wider industry of energy efficiency and sustainability in the built environment.
We should not forget that these models are used (or should be) to inform design decisions that affect large investments in terms of money, resources and carbon emissions. And how can we optimise design options when the same alternative is estimated by different modellers with such differences?
I think we need to acknowledge this modelling performance problem first. Then, find and address the reasons for it. And only then we will be ready to solve the other performance gap between models and real-life energy use. Otherwise, we’ll be just going around in circles, wasting time and money.
Reasons for modelling misperformance
The first issue to consider is quality assurance. This research paper identified misunderstandings, omissions, or errors in almost all areas of the model:
- Fabric elements definition or assignment
- Assignment of room activities and profiles
- Main HVAC system configuration
- Auxiliary energy inputs
- Lighting settings
- Natural ventilation and infiltration settings
And, in this case, geometry modelling errors where excluded since the geometry was provided. But errors or omissions are also quite common in geometry creation, especially in buildings with special features.
In my view, the main reason for modelling errors in this case and in many others is clear: the lack of a reviewing process.
In building performance modelling, it is quite easy to make errors, omissions, or misunderstandings. Information comes from different sources and can be in very different formats. And inputs to the software can also be tricky since there are many parameters and different uses of spaces in a building.
Even the most experienced professionals can make mistakes in the first ‘round’ of a modelling exercise, but for less experienced modellers they are almost guaranteed. And if quality-assurance reviews are not in the budget, the errors will remain in the final results.
A well-defined and structured quality review process is essential in building performance modelling. I will insist in any occasion.
However, as the paper also points out, not all the variability in results should be attributed to modelling errors or misunderstandings of the information provided. Even in this case, where instructions on the information to use were clear, there was some room for interpretation from the modeller. Especially in the selection of the most adequate activity templates to use in certain rooms. And in real-life modelling, where the templates to use are not pre-selected, the variability in the interpretation of how the different spaces will be used can be much higher.
Furthermore, as discussed in other articles, there is a big gap in understanding how buildings are used in real life, and there is significant room for improving our assumptions in this aspect. The outputs of a model can only be as good as the inputs used, and in the aspect of the use of the building by their occupants, the information available for modellers is minimal.
Regarding the availability of technical data sheets of mechanical systems and fabric, there is also significant room for improvement in how specifications information is provided, as it is also highlighted by Dr Yang’s paper. There is no standardisation of the information provided for detailed energy modelling, leaving significant room for interpretation and requiring modellers to have the necessary experience to make good interpretations and translate it to realistic inputs. And simplified template modelling inputs lack in many cases the capacity to model important characteristics of the systems and controls that determine their performance.
On the other hand, there is no performance modelling training that covers all the potential formats of technical data that can be found in real-life projects.
Potential solutions
Now, what can we do to reduce this other performance gap?
Regarding avoiding modelling errors, omissions and misinterpretations, the solution is very clear in my view. Energy models should be reviewed by an experienced professional always. Not as a routine. Not as tick box that looks good on reports. And not just the results and the model itself, but the whole modelling process. From the interpretation of the information received and the assumptions made, to the approach taken and the conclusions drawn.
For this to happen, there needs to be a specific allowance in time and cost estimates when preparing quotations. Which should also be appropriate to the size and complexity of the project, as well as to the experience of the modeller (a less experienced modeller requires more thorough reviews at key steps).
Regarding reducing the effect of interpretation and improving our understanding of how buildings are used, there is not much that individual modellers and companies can do. This problem should be tackled a at a higher level, in my view, given its importance. By professional bodies, research organisations, or even the governments’ departments for energy and sustainability.
Regarding the availability of technical data and its interpretation, it would also probably require the application of new specific regulations, since ‘market forces’ have proven ineffective in both aspects. On the one hand, manufacturers don’t seem to feel compelled to provide more detailed information about the energy efficiency features of their products, as it is unclear that would help them with sales or be worth the effort. On the other hand, neither training providers nor traineesor their employers currently seem to value the additional effort, time and cost involved in teaching/learning how to better interpret technical data for performance modelling.
It should be clear, though, that things need to change for this discipline to be more reliable and fulfil its essential role in delivering more efficient, better-performing and more sustainable buildings.
You can find the research paper mentioned on this link: https://journals.sagepub.com/doi/10.1177/01436244261469050
If you think an independent review of your building performance models could be useful, feel free to get in touch.