New tranching approach

[tsmbland]

Description

Please include a summary of the change and which issue is fixed (if any). Please also
include relevant motivation and context. List any dependencies that are required for
this change.

Fixes #1358

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[tsmbland]

Failing for missing_commodity, two_outputs and circularity. Might be related to #1347, although I wasn't expecting that to crop up with the example models. @alexdewar Let's discuss tomorrow

[tsmbland]

I've had a look at the failing models, and these are all due to commodities having zero/questionable shadow prices in one year, which makes activity coefficients negative the following year:

• circularity: GASNAT in 2030 has a shadow price of zero in summer because there's no demand. In 2040 there is demand for GASNAT in summer (in fact, there's only demand in summer) but the activity coefficients are negative so this doesn't get met

• two_outputs: GASPRD in 2020 has a shadow price of zero in summer because there's excess supply (it's a side product of OAGRSV). Activity coefficients in 2030 are negative.

• missing_commodity: BIOPEL in 2030 has no consumption/production in summer. The shadow price, while not zero (should it be?) is very small, so the coefficients in 2040 are negative.

These all seem like fairly normal scenarios so makes me less convinced that #1347 should be a special feature hidden behind a warning, unless there's a deeper issue here that we can fix.

In the latter two cases, it invests in assets to meet demand in the other seasons, but then exits when it has leftover demand in summer that it's chosen not to meet with these assets. I think an issue with the approach suggested in #1347 is that it would allow the system to invest in extra capacity to meet the demand in summer, whereas really it would have been better to utilise the existing assets in summer all along.

[tsmbland]

@ahawkes I've had a go at implementing the new capacity selection approach as described in #1358.

Two questions/discussion points:

• What do you think would be a "reasonable" value(s) for this parameter? Currently, since I didn't want to manually define it for every process in every model, I've gone for a default of 10 / capacity_to_activity, as I think generally it would scale inversely with capacity_to_activity. The 10 is arbitrary, although we could make this a configurable parameter. Alternatively, we could force users to define capacity_granularity for every process.

• Even with this fix, a few models are still failing, which seems to be related to "weird" shadow prices. See comment above. The solution proposed in Investment fails when only remaining options fail to dispatch #1347 would no doubt allow these models to run to completion, but I think there are some bigger issues we need to think about.

[ahawkes]

@ahawkes I've had a go at implementing the new capacity selection approach as described in #1358.

Two questions/discussion points:

• What do you think would be a "reasonable" value(s) for this parameter? Currently, since I didn't want to manually define it for every process in every model, I've gone for a default of 10 / capacity_to_activity, as I think generally it would scale inversely with capacity_to_activity. The 10 is arbitrary, although we could make this a configurable parameter. Alternatively, we could force users to define capacity_granularity for every process.

Not sure about 10/capacity_to_activity. How about enough capacity to meet about 1/10 of the capacity requirements of the original demand profile - but instead of the DLC approach which focused on the worst time slice (and got capacities that are too large) - we focus on the best (highest availability) time slice - and choose a capacity that can serve 1/10 of the capacity needs in that time slice. There are probably pitfalls to this approach - feel free to refine.

• Even with this fix, a few models are still failing, which seems to be related to "weird" shadow prices. See comment above. The solution proposed in Investment fails when only remaining options fail to dispatch #1347 would no doubt allow these models to run to completion, but I think there are some bigger issues we need to think about.

For this issue, remember we were adding epsilon to make sure things dispatch for the old NPV formulation? I guess this is not similar to that? If yes, then maybe I could take a look at one that is failing to see what's going on?

[tsmbland]

@ahawkes I've had a go at implementing the new capacity selection approach as described in #1358.
Two questions/discussion points:

• What do you think would be a "reasonable" value(s) for this parameter? Currently, since I didn't want to manually define it for every process in every model, I've gone for a default of 10 / capacity_to_activity, as I think generally it would scale inversely with capacity_to_activity. The 10 is arbitrary, although we could make this a configurable parameter. Alternatively, we could force users to define capacity_granularity for every process.

Not sure about 10/capacity_to_activity. How about enough capacity to meet about 1/10 of the capacity requirements of the original demand profile - but instead of the DLC approach which focused on the worst time slice (and got capacities that are too large) - we focus on the best (highest availability) time slice - and choose a capacity that can serve 1/10 of the capacity needs in that time slice. There are probably pitfalls to this approach - feel free to refine.

Ok... that would take us closer to where we were before. The key difference is whether capacities are defined upfront for all years (i.e. in the input data) or whether they depend on run-time info. I assumed you wanted the former so that's how I've done it. In any case, I'd probably lean into one direction or the other, rather than a mixed approach.

• Even with this fix, a few models are still failing, which seems to be related to "weird" shadow prices. See comment above. The solution proposed in Investment fails when only remaining options fail to dispatch #1347 would no doubt allow these models to run to completion, but I think there are some bigger issues we need to think about.

For this issue, remember we were adding epsilon to make sure things dispatch for the old NPV formulation? I guess this is not similar to that? If yes, then maybe I could take a look at one that is failing to see what's going on?

That helps when activity coefficients are zero. In this case, coefficients are negative, so this is a different problem.

[tsmbland]

How about enough capacity to meet about 1/10 of the capacity requirements of the original demand profile - but instead of the DLC approach which focused on the worst time slice (and got capacities that are too large) - we focus on the best (highest availability) time slice - and choose a capacity that can serve 1/10 of the capacity needs in that time slice. There are probably pitfalls to this approach - feel free to refine.

Not sure about this. What if demand is zero (or extremely low) in timeslices with high availability (but may be higher elsewhere). Capacities could end up extremely small, which could mean a huge number of appraisal rounds if there aren't other more feasible technologies. Feels like there could be equally many pitfalls with this approach compared to the DLC approach.

[ahawkes]

How about enough capacity to meet about 1/10 of the capacity requirements of the original demand profile - but instead of the DLC approach which focused on the worst time slice (and got capacities that are too large) - we focus on the best (highest availability) time slice - and choose a capacity that can serve 1/10 of the capacity needs in that time slice. There are probably pitfalls to this approach - feel free to refine.

Not sure about this. What if demand is zero (or extremely low) in timeslices with high availability (but may be higher elsewhere). Capacities could end up extremely small, which could mean a huge number of appraisal rounds if there aren't other more feasible technologies. Feels like there could be equally many pitfalls with this approach compared to the DLC approach.

Yes, fair enough. Though it seems there should be a way to choose based on availabilities, time sliced demands, etc - just needs to be more nuanced than DLC or this alternative demand-based approach. Anyway, in order to have something working, perhaps let's try with 1/cap2act as you suggest and see how it goes? Should we also make it a configurable parameter? If yes, would it be associated with process or commodity? I think I prefer the former.

[tsmbland]

How about enough capacity to meet about 1/10 of the capacity requirements of the original demand profile - but instead of the DLC approach which focused on the worst time slice (and got capacities that are too large) - we focus on the best (highest availability) time slice - and choose a capacity that can serve 1/10 of the capacity needs in that time slice. There are probably pitfalls to this approach - feel free to refine.

Not sure about this. What if demand is zero (or extremely low) in timeslices with high availability (but may be higher elsewhere). Capacities could end up extremely small, which could mean a huge number of appraisal rounds if there aren't other more feasible technologies. Feels like there could be equally many pitfalls with this approach compared to the DLC approach.

Yes, fair enough. Though it seems there should be a way to choose based on availabilities, time sliced demands, etc - just needs to be more nuanced than DLC or this alternative demand-based approach. Anyway, in order to have something working, perhaps let's try with 1/cap2act as you suggest and see how it goes? Should we also make it a configurable parameter? If yes, would it be associated with process or commodity? I think I prefer the former.

As it currently is in this PR:

• users can manually specify capacities at the process level in processes.csv. These are in Capacity units
• if this is unspecified for a process, it will default to x / capacity_to_activity. x in this case has Activity units, currently hardcoded to 10 (completely arbitrary).

I suggest we make the default x configurable in settings.toml (and give this a sensible name!). We can then keep the option to override this at the process level in processes.csv. At the risk of utterly confusing users, it may be better for the process-level field to also take capacities in Activity units (i.e. to then be scaled by capacity_to_activity), rather than absolute Capacity units. This way it's consistent with x (i.e. you define a global default for x, then override it at the process level). Could be really confusing though...