Add an emissions 1990 query validation spec

[louispt1]

Context

There was a desire within the direct emissions enhancements to check if the queried total emissions equals the sum of the emissions in the emissions.csv with separate sums for co2, other_ghg and total_ghg for 1990.

Implemented changes

Added another spec to lib/graph_data_validation/spec/validation/emissions.rb to validate the totals queries align with actual csv data to ensure no queries are missed when adding aggregates

@noracato I'm not sure how much to abstract here? The spec looks a little over the top, but it also feels extra to make another class to hold the logic. If you think that's the right approach I'm happy to extract an emissions_csv_reconciler class in the lib to clean up the spec.

Related

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[kndehaan]

The functionality of the test works well! I have one suggestion regarding the nr of validated datasets.

The purpose of the test is to check whether all EMISSIONS() rows are included in the co2, other_ghg and total_ghg queries. It is currently tested for all country datasets, but just checking a few datasets would already suffice and will be faster.

We could define a couple of datasets in config.yml for which the queries should be tested with the emissions.csv and only do the test for those datasets:

emissions_1990_validation:
- nl2023
- nl2019
- DE_germany
- UKNI01_northern_ireland

Would it be possible to setup the spec up in a way to only test these few datasets?

[noracato]

If been looking and playing with it a bit.

This is adding query and atlas data validation to this lib. That is why it does not fit.

This lib was quickly designed for modellers to be able to themselves add tests for nodes. I don't think it is a bad idea to abstract the complicated looking logic out of the spec to make it more readable for modellers. Try to keep modeller logic in the spec, like:

dataset.query("present:Q(direct_emissions_total_ghg_1990)") -
   dataset.query("present:Q(direct_emissions_bunkers_total_ghg_1990)")

[louispt1]

If been looking and playing with it a bit.

This is adding query and atlas data validation to this lib. That is why it does not fit.

This lib was quickly designed for modellers to be able to themselves add tests for nodes. I don't think it is a bad idea to abstract the complicated looking logic out of the spec to make it more readable for modellers. Try to keep modeller logic in the spec, like:

dataset.query("present:Q(direct_emissions_total_ghg_1990)") -
   dataset.query("present:Q(direct_emissions_bunkers_total_ghg_1990)")

I had a go - let me know what you think.