diff --git a/benches/assets.rs b/benches/assets.rs index 5387c2b70..1a48bbd16 100644 --- a/benches/assets.rs +++ b/benches/assets.rs @@ -247,7 +247,7 @@ fn criterion_benchmark(c: &mut Criterion) { black_box(&agent), black_box(region_id), black_box(&prices), - demand, + &demand, black_box(YEAR), writer.debug_writer(), ) diff --git a/docs/glossary.md b/docs/glossary.md index f9fcbc879..9e9729bed 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -110,7 +110,7 @@ time is valued at approximately £95.24 today (100 / (1 + 0.05)). **Dispatch:** The way in which *Assets* are operated to serve demand. Within the limits imposed by *Availability*, *Commodity* balances, and other user-defined constraints, MUSE2 effectively dispatches *Assets* in *Merit Order*: cheaper *Assets* are used first, up to their available capacity or until -demand is met. The resulting operation is calculated by a least-cost linear programme. +demand is met. The resulting operation is calculated by a least-cost linear program. **End Year:** The final year in the model *Time Horizon*. @@ -135,7 +135,7 @@ commodities. **Investment Appraisal:** The process by which a competitive *Agent* evaluates each existing and *Candidate Asset* to determine which investments it should make in a particular *Commodity Market* and *Milestone Year*. Appraisal considers the *Agent Objective*, expected costs, commodity prices, -and the asset's expected operation. +and the asset's expected operation alongside any assets the agent has already selected. **Investment Order:** The sequence in which *Commodity* markets are considered for investment. MUSE2 generally processes downstream markets before upstream markets, so that demand for input diff --git a/docs/model/agents.md b/docs/model/agents.md index 7bbf4c7fb..3176504aa 100644 --- a/docs/model/agents.md +++ b/docs/model/agents.md @@ -31,12 +31,17 @@ competitive agents (`competitive`) and portfolio agents (`portfolio`) ## Competitive agents +Competitive agents are experimental. To use them, set +`please_give_me_broken_results = true` in [`model.toml`][model-toml]. + Competitive agents select assets in a series of rounds. In each round, every remaining option (a -tranche of an existing asset or of a candidate asset) is appraised against the agent's remaining -demand. A small [dispatch optimisation](#mini-dispatch-optimisation) determines how the option -would operate, and an [investment metric](#metric-calculation) is then calculated according to the -agent's objective. The best option is selected, its production is subtracted from the remaining -demand, and the process repeats until demand is met or no feasible options remain. +tranche of an existing asset or of a candidate asset) is appraised alongside the assets already +selected in previous rounds. A small [dispatch optimisation](#mini-dispatch-optimisation) of the +already-selected assets plus the option determines how the option would operate, and an +[investment metric](#metric-calculation) is then calculated for the option's +[marginal activity](#marginal-activity) according to the agent's objective. The best option is +selected and added to the set of selected assets, and the process repeats until demand is met or no +feasible options remain. ### Commodity prices used in appraisal @@ -101,10 +106,16 @@ remaining limit. ### Mini Dispatch Optimisation For each supply option being appraised, a small linear program (LP) is solved to determine the -optimal activity profile given the current remaining demand. +optimal activity profile. The LP includes every asset selected so far in the current market +(whether existing or candidate) as well as the option being appraised, and is solved against the +market's full demand rather than the demand remaining after earlier selections. The option is +therefore appraised on how it would operate alongside the assets already selected, rather than in +isolation. #### Activity coefficients +Each asset in the LP has its own activity coefficients, calculated as described below. + The mini dispatch optimisation implicitly frames each time slice as a choice: the asset can either produce the commodity of interest, or it can be treated as procured from an alternative source at the **fallback price** \\( \phi_{c,r,t} \\). Each unit of activity produces @@ -149,30 +160,53 @@ in [`model.toml`][model-toml]. #### Constraints -- **Activity bounds**: the sum of activity within each time-slice selection is bounded by the - asset's availability limits multiplied by its capacity. +- **Activity bounds**: for each asset, the sum of activity within each time-slice selection is + bounded by the asset's availability limits multiplied by its capacity. - **Demand constraints**: demand for a commodity is balanced at the commodity's defined *time-slice level* (e.g. annual, seasonal, or time-slice). The total supply (activity × flow - coefficient) within each balance bucket must not exceed the remaining demand for that bucket. + coefficient) of all assets in the LP within each balance bucket must not exceed the demand for + that bucket. +- **Equal utilisation of equivalent assets**: assets with equivalent dispatch properties are + constrained to have the same utilisation in every time slice, as in the full-system dispatch (see + [Equal Utilisation of Equivalent Assets][dispatch-equal-utilisation]). #### Objective -The optimisation maximises the total net revenue across all time slices, subject to the above -constraints: +The optimisation maximises the total net revenue across all assets and time slices, subject to the +above constraints: \\[ - \max \sum_t \alpha_t \cdot \mathrm{Activity}_t + \max \sum_{a} \sum_t \alpha_{a,t} \cdot \mathrm{Activity}_{a,t} \\] In addition, seasonal/annual utilisation penalties are included to encourage even spread of utilisation within and across seasons. These are equivalent to the [Seasonal/Annual Utilisation Penalties][dispatch-utilisation-penalties] used in the full-system -dispatch optimisation. +dispatch optimisation, and are applied to each asset separately. ### Metric Calculation -After the dispatch LP is solved, an investment metric is calculated from the resulting activity -profile using **market prices**. +After the dispatch LP is solved, an investment metric is calculated for the option using **market +prices**. The metric is based on the option's marginal activity, while the fixed costs and market +costs are those of the option itself. + +#### Marginal activity + +Because the LP includes previously selected assets, the option's activity cannot be read directly +from the solution. Instead, the metric uses the *marginal activity* +\\( \Delta\mathrm{Activity}_t \\): the change in activity, relative to the previous round's +solution, of all assets equivalent to the option (including the option itself): + +\\[ + \Delta\mathrm{Activity}_t = \sum_{a \in G} \left(\mathrm{Activity}_{a,t}^{(k)} - + \mathrm{Activity}_{a,t}^{(k-1)}\right) +\\] + +where \\( G \\) is the set of assets in the LP that are equivalent to the option (see +[Equal Utilisation of Equivalent Assets][dispatch-equal-utilisation]), and \\( k \\) is the current +round. Activity in the previous solution is zero for any asset not present in it, so in the first +round the marginal activity is simply the option's activity. All of the marginal activity is +attributed to the option being appraised. #### Market costs per time slice @@ -196,9 +230,9 @@ The LCOX metric is calculated as the total annualised cost divided by total annu the above defined market costs which *exclude* the primary output commodity: \\[ - \mathrm{LCOXMetric} = \frac{\mathrm{AFC} \times \mathrm{Capacity} + \sum_t \mathrm{Activity}_t - \times \mu_t^{\mathrm{LCOX}}} - {\sum_t \mathrm{Activity}_t} + \mathrm{LCOXMetric} = \frac{\mathrm{AFC} \times \mathrm{Capacity} + \sum_t + \Delta\mathrm{Activity}_t \times \mu_t^{\mathrm{LCOX}}} + {\sum_t \Delta\mathrm{Activity}_t} \\] Lower values indicate lower-cost investments. @@ -209,16 +243,16 @@ The NPV metric is based on the Specific Net Annualised Surplus (SNAS). This the unit of activity, using market costs that *include* the primary output commodity: \\[ - \mathrm{SNAS} = \frac{-\left(\mathrm{AFC} \times \mathrm{Capacity} + \sum_t \mathrm{Activity}_t \times - \mu_t^{\mathrm{NPV}}\right)}{\sum_t \mathrm{Activity}_t} + \mathrm{SNAS} = \frac{-\left(\mathrm{AFC} \times \mathrm{Capacity} + \sum_t + \Delta\mathrm{Activity}_t \times \mu_t^{\mathrm{NPV}}\right)}{\sum_t \Delta\mathrm{Activity}_t} \\] Higher values indicate more profitable investments. -> For both metrics, any option with zero total activity after the mini dispatch LP is excluded from -> consideration, as it cannot contribute to meeting demand. This will generally happen if all -> time slices have negative activity coefficients, unless the process has lower-bound activity -> constraints that force activity. +> For both metrics, any option with zero total marginal activity after the mini dispatch LP is +> excluded from consideration, as it cannot contribute to meeting demand. This will generally +> happen if all time slices have negative activity coefficients, unless the process has lower-bound +> activity constraints that force activity. ### Asset selection @@ -234,9 +268,11 @@ equal metrics, the following tie-breaking rules are applied in order: #### Selection loop -The best-ranked asset is committed. Its production profile from the mini dispatch optimisation is -subtracted from the remaining demand, and the loop repeats with the updated demand profile. This -continues until: +The best-ranked asset is committed and added to the set of selected assets. The solution of its mini +dispatch optimisation is kept as the baseline for calculating +[marginal activity](#marginal-activity) in the next round. The remaining demand is calculated as the +market's demand minus the supply from all selected assets in that solution, and the loop repeats +with the updated set of selected assets. This continues until: - The remaining demand falls below `remaining_demand_absolute_tolerance` (in [`model.toml`][model-toml]), or @@ -301,8 +337,13 @@ The optimiser maximises \\( 10 \cdot \mathrm{Activity}_{t_0} + (-10) \cdot \math so it prefers to dispatch during \\( t_0 \\) and minimise activity during \\( t_1 \\), subject to demand and availability constraints. -Suppose the optimiser determines \\( \mathrm{Activity}_{t_0} = 80 \\) MWh and -\\( \mathrm{Activity}_{t_1} = 20 \\) MWh. +For simplicity, assume this is the first round of selection, so that no assets have been selected +yet and the plant is the only asset in the LP. Suppose the optimiser determines +\\( \mathrm{Activity}_{t_0} = 80 \\) MWh and \\( \mathrm{Activity}_{t_1} = 20 \\) MWh. As there is +no previous solution, the marginal activity equals the activity: +\\( \Delta\mathrm{Activity}_{t_0} = 80 \\) MWh and \\( \Delta\mathrm{Activity}_{t_1} = 20 \\) MWh. +In later rounds, the marginal activity would instead be the change in activity relative to the +previous round's solution. #### LCOX Metric @@ -464,3 +505,4 @@ Once the LP has been solved, the selected assets are constructed from the capaci [agent-search-spaces-csv]: ../file_formats/input_files.md#agent_search_spacescsv [agent-objectives-csv]: ../file_formats/input_files.md#agent_objectivescsv [dispatch-utilisation-penalties]: ./dispatch_optimisation.md#seasonalannual-utilisation-penalties +[dispatch-equal-utilisation]: ./dispatch_optimisation.md#equal-utilisation-of-equivalent-assets diff --git a/schemas/input/agents.yaml b/schemas/input/agents.yaml index e3c4bf0e6..34afb2437 100644 --- a/schemas/input/agents.yaml +++ b/schemas/input/agents.yaml @@ -19,8 +19,9 @@ fields: Optional (defaults to `portfolio` if omitted). With `portfolio`, the capacities of all existing and candidate assets are chosen together in a single optimisation. With `competitive`, existing and candidate assets compete in rounds, with the best option according - to the agent's objective selected in each round. See the [Agents](../model/agents.md) page for - details. + to the agent's objective selected in each round. Competitive agents are experimental and + require `please_give_me_broken_results = true` in `model.toml`. See the + [Agents](../model/agents.md) page for details. - name: decision_rule type: string description: The decision rule applied to objectives diff --git a/src/input.rs b/src/input.rs index 21b92361c..2c47ddfd0 100644 --- a/src/input.rs +++ b/src/input.rs @@ -281,6 +281,7 @@ pub fn load_model>(model_dir: P) -> Result { &processes, ®ion_ids, years, + model_params.allow_dangerous_options, )?; let agent_ids = agents.keys().cloned().collect(); let user_assets = read_user_assets(model_dir.as_ref(), &agent_ids, &processes, ®ion_ids)?; diff --git a/src/input/agent.rs b/src/input/agent.rs index c36c999f7..b6ac929ec 100644 --- a/src/input/agent.rs +++ b/src/input/agent.rs @@ -5,6 +5,7 @@ use crate::agent::{ AgentType, DecisionRule, }; use crate::commodity::CommodityMap; +use crate::model::ALLOW_DANGEROUS_OPTION_NAME; use crate::process::ProcessMap; use crate::region::{RegionID, parse_region_str}; use anyhow::{Context, Result, bail, ensure}; @@ -58,8 +59,17 @@ pub fn read_agents( processes: &ProcessMap, region_ids: &IndexSet, milestone_years: &[u32], + allow_dangerous_options: bool, ) -> Result { let mut agents = read_agents_file(model_dir, region_ids)?; + ensure!( + allow_dangerous_options + || agents + .values() + .all(|agent| agent.agent_type != AgentType::Competitive), + "Competitive agents are experimental. To use them, set the \ + {ALLOW_DANGEROUS_OPTION_NAME} option to true." + ); // We read commodity portions first as they are required by `read_agent_search_spaces` let mut agent_commodities = read_agent_commodity_portions( @@ -165,9 +175,6 @@ where mod tests { use super::*; use crate::agent::DecisionRule; - use crate::fixture::assert_patched_runs_ok_simple; - use crate::patch::FilePatch; - use itertools::Itertools; use rstest::rstest; use std::{fs, iter}; use tempfile::tempdir; @@ -257,24 +264,4 @@ mod tests { } } } - - #[rstest] - #[case::all_competitive(["competitive"; 4])] - #[case::all_portfolio(["portfolio"; 4])] - #[case::mixed(["competitive", "portfolio", "competitive", "portfolio"])] - fn model_runs_with_agent_types(#[case] agent_types: [&str; 4]) { - let mut lines = vec![ - "id,description,regions,decision_rule,decision_lexico_tolerance,agent_type".to_string(), - ]; - for (id, agent_type) in ["A0_GEX", "A0_GPR", "A0_ELC", "A0_RES"] - .iter() - .zip(agent_types) - { - lines.push(format!("{id},,all,single,,{agent_type}")); - } - let lines = lines.iter().map(String::as_str).collect_vec(); - assert_patched_runs_ok_simple!(vec![ - FilePatch::new(AGENT_FILE_NAME).with_replacement(&lines) - ]); - } } diff --git a/src/simulation/investment/competitive.rs b/src/simulation/investment/competitive.rs index dd04a1edf..6844f9c13 100644 --- a/src/simulation/investment/competitive.rs +++ b/src/simulation/investment/competitive.rs @@ -22,6 +22,7 @@ use crate::time_slice::TimeSliceSelection; use crate::units::{Capacity, Dimensionless, Flow, FlowPerCapacity}; use anyhow::{Result, ensure}; use coefficients::calculate_coefficients_for_assets; +use indexmap::IndexMap; use indexmap::IndexSet; use itertools::{Itertools, chain}; use log::{debug, warn}; @@ -95,7 +96,7 @@ pub fn select_assets( agent, region_id, prices, - demand.clone(), + demand, year, debug_writer, ) @@ -245,7 +246,7 @@ pub fn select_best_assets( agent: &Agent, region_id: &RegionID, prices: &Prices, - mut demand: DemandMap, + demand: &DemandMap, year: u32, mut debug_writer: Option<&mut DebugDataWriter>, ) -> Result> { @@ -269,9 +270,11 @@ pub fn select_best_assets( // Iteratively select the best asset until demand is met let mut round = 0; - let mut best_assets: Vec = Vec::new(); + let mut selected_assets: Vec = Vec::new(); + let mut unmet_demand = demand.clone(); + let mut previous_solution = IndexMap::new(); while is_any_remaining_demand( - &demand, + &unmet_demand, model.parameters.remaining_demand_absolute_tolerance, ) { // Remove assets that would exceed the remaining limits for their processes from the options @@ -289,6 +292,7 @@ pub fn select_best_assets( commodity.id, region_id ); + // Appraise all options in parallel: each asset's appraisal is independent (all shared // state is read-only within this block), so we can safely use Rayon here. // Each HiGHS solve inside `appraise_investment` is configured to use only one thread @@ -303,11 +307,13 @@ pub fn select_best_assets( Ok(Some(appraise_investment( model, + &selected_assets, asset, commodity, objective_type, - &coefficients[asset], - &demand, + &coefficients, + demand, + &previous_solution, )?)) }) .collect::>>()? // propagate any solver error @@ -322,7 +328,7 @@ pub fn select_best_assets( year, &format!("{} {} round {}", commodity.id, agent.id, round), &outputs, - &demand, + demand, commodity.time_slice_level, )?; } @@ -366,15 +372,16 @@ pub fn select_best_assets( ); // Record the selected asset - record_asset_selection(best_output.asset, &mut best_assets); + record_asset_selection(best_output.asset, &mut selected_assets); + previous_solution = best_output.solution; - demand = best_output.unmet_demand; + unmet_demand = best_output.unmet_demand; round += 1; } // Convert Candidate assets to Ready // At this point we also assign the agent ID to the asset - for asset in &mut best_assets { + for asset in &mut selected_assets { if asset.is_candidate() { asset .make_mut() @@ -382,7 +389,7 @@ pub fn select_best_assets( } } - Ok(best_assets) + Ok(selected_assets) } /// Prepare existing assets for reappraisal. diff --git a/src/simulation/investment/competitive/constraints.rs b/src/simulation/investment/competitive/constraints.rs index 924df5bdc..5f293cf01 100644 --- a/src/simulation/investment/competitive/constraints.rs +++ b/src/simulation/investment/competitive/constraints.rs @@ -4,6 +4,7 @@ use crate::asset::AssetRef; use crate::commodity::Commodity; use crate::model::Model; use crate::simulation::investment::DemandMap; +use crate::simulation::optimisation::group_dispatch_equivalent_assets; use crate::time_slice::{Season, TimeSliceID, TimeSliceInfo, TimeSliceSelection}; use crate::units::{MoneyPerCapacityPerYear, Year}; use highs::RowProblem as Problem; @@ -18,21 +19,24 @@ use indexmap::IndexMap; /// sum of activity in that selection. pub fn add_activity_constraints( problem: &mut Problem, - asset: &AssetRef, - activity_vars: &IndexMap, + assets: &[AssetRef], + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, time_slice_info: &TimeSliceInfo, ) { - for (ts_selection, limits) in asset.iter_activity_limits() { - let limits = limits.start().value()..=limits.end().value(); + // Create constraints for each asset + for asset in assets { + for (ts_selection, limits) in asset.iter_activity_limits() { + let limits = limits.start().value()..=limits.end().value(); - // Collect activity terms for the time slices in this selection - let terms = ts_selection - .iter(time_slice_info) - .map(|(time_slice, _)| (activity_vars[time_slice], 1.0)) - .collect::>(); + // Collect activity terms for the time slices in this selection + let terms = ts_selection + .iter(time_slice_info) + .map(|(time_slice, _)| (activity_vars[&(asset.clone(), time_slice.clone())], 1.0)) + .collect::>(); - // Constraint: sum of activities in selection within limits - problem.add_row(limits, &terms); + // Constraint: sum of activities in selection within limits + problem.add_row(limits, &terms); + } } } @@ -43,20 +47,33 @@ pub fn add_activity_constraints( /// flow coefficients) to the total demand for that selection. pub fn add_demand_constraints( problem: &mut Problem, - asset: &AssetRef, + assets: &[AssetRef], commodity: &Commodity, time_slice_info: &TimeSliceInfo, demand: &DemandMap, - activity_vars: &IndexMap, + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, ) { - let flow_coeff = asset.get_flow(&commodity.id).unwrap().coeff; + // Create a constraint for each ts-selection for ts_selection in time_slice_info.iter_selections_at_level(commodity.time_slice_level) { - let demand_for_ts_selection = demand[&ts_selection]; - let terms: Vec<_> = ts_selection - .iter(time_slice_info) - .map(|(time_slice, _)| (activity_vars[time_slice], flow_coeff.value())) - .collect(); - problem.add_row(0.0..=demand_for_ts_selection.value(), terms); + // Collect terms: all assets in all time slices in the selection + let terms = assets + .iter() + .flat_map(|asset| { + let coefficient = asset.get_flow(&commodity.id).unwrap().coeff.value(); + + ts_selection + .iter(time_slice_info) + .map(move |(time_slice, _)| { + ( + activity_vars[&(asset.clone(), time_slice.clone())], + coefficient, + ) + }) + }) + .collect::>(); + + // Create the constraint + problem.add_row(0.0..=demand[&ts_selection].value(), terms); } } @@ -68,8 +85,8 @@ pub fn add_demand_constraints( pub fn add_utilisation_peak_constraints( problem: &mut Problem, model: &Model, - asset: &AssetRef, - activity_vars: &IndexMap, + assets: &[AssetRef], + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, ) { let has_seasonal_penalty = model.parameters.seasonal_utilisation_penalty > MoneyPerCapacityPerYear(0.0); @@ -82,64 +99,74 @@ pub fn add_utilisation_peak_constraints( } // So long as either penalty is applied, we need to add seasonal peak variables and constraints - let seasonal_peak_vars = add_seasonal_peak_variables(problem, model); + let seasonal_peak_vars = add_seasonal_peak_variables(problem, model, assets); add_seasonal_peak_constraints( problem, - asset, activity_vars, &model.time_slice_info, &seasonal_peak_vars, ); - // If the annual penalty is applied, we also need to add an annual peak variable and constraints + // If the annual penalty is applied, we also need to add annual peak variables and constraints if has_annual_penalty { - let annual_peak_var = add_annual_peak_variable(problem, model); + let annual_peak_vars = add_annual_peak_variables(problem, model, assets); add_annual_peak_constraints( problem, &model.time_slice_info, - annual_peak_var, + &annual_peak_vars, &seasonal_peak_vars, ); } } -/// Add seasonal peak variables to the problem. +/// Add seasonal peak variables to the problem for each (asset, season) pair. fn add_seasonal_peak_variables( problem: &mut Problem, model: &Model, -) -> IndexMap { + assets: &[AssetRef], +) -> IndexMap<(AssetRef, Season), Variable> { let mut seasonal_peak_vars = IndexMap::new(); - for (season, duration) in &model.time_slice_info.seasons { - // Scale penalty by season duration - // We make the penalty negative as we're doing a max optimisation - let col_factor = -(model.parameters.seasonal_utilisation_penalty * *duration); - let variable = problem.add_column(col_factor.value(), 0.0..); - seasonal_peak_vars.insert(season.clone(), variable); + for asset in assets { + for (season, duration) in &model.time_slice_info.seasons { + // Scale penalty by season duration + // We make the penalty negative as we're doing a max optimisation + let col_factor = -(model.parameters.seasonal_utilisation_penalty * *duration); + let variable = problem.add_column(col_factor.value(), 0.0..); + seasonal_peak_vars.insert((asset.clone(), season.clone()), variable); + } } seasonal_peak_vars } -/// Add annual peak variable to the problem. -fn add_annual_peak_variable(problem: &mut Problem, model: &Model) -> highs::Col { +/// Add annual peak variables to the problem for each asset. +fn add_annual_peak_variables( + problem: &mut Problem, + model: &Model, + assets: &[AssetRef], +) -> IndexMap { // Penalty is applied over the whole year, so scale by 1 year // We make the penalty negative as we're doing a max optimisation let col_factor = -(model.parameters.annual_utilisation_penalty * Year(1.0)); - problem.add_column(col_factor.value(), 0.0..) + assets + .iter() + .map(|asset| { + let variable = problem.add_column(col_factor.value(), 0.0..); + (asset.clone(), variable) + }) + .collect() } /// Add constraints linking seasonal peak variables to activity variables for each (asset, season) pair. fn add_seasonal_peak_constraints( problem: &mut Problem, - asset: &AssetRef, - activity_vars: &IndexMap, + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, time_slice_info: &TimeSliceInfo, - seasonal_peak_vars: &IndexMap, + seasonal_peak_vars: &IndexMap<(AssetRef, Season), Variable>, ) { - let activity_per_capacity = asset.process().capacity_to_activity; - for (season, &peak_variable) in seasonal_peak_vars { - for (time_slice, ts_length) in - TimeSliceSelection::Season(season.clone()).iter(time_slice_info) - { + for ((asset, season), &peak_variable) in seasonal_peak_vars { + let activity_per_capacity = asset.process().capacity_to_activity; + let season_selection = TimeSliceSelection::Season(season.clone()); + for (time_slice, ts_length) in season_selection.iter(time_slice_info) { let time_slice_fraction = ts_length / Year(1.0); let activity_per_capacity_in_time_slice = activity_per_capacity * time_slice_fraction; let capacity_required_per_activity = 1.0 / activity_per_capacity_in_time_slice.value(); @@ -151,7 +178,10 @@ fn add_seasonal_peak_constraints( 0.0.., [ (peak_variable, 1.0), - (activity_vars[time_slice], -capacity_required_per_activity), + ( + activity_vars[&(asset.clone(), time_slice.clone())], + -capacity_required_per_activity, + ), ], ); } @@ -162,13 +192,64 @@ fn add_seasonal_peak_constraints( fn add_annual_peak_constraints( problem: &mut Problem, time_slice_info: &TimeSliceInfo, - annual_peak_var: highs::Col, - seasonal_peak_vars: &IndexMap, + annual_peak_vars: &IndexMap, + seasonal_peak_vars: &IndexMap<(AssetRef, Season), Variable>, ) { - for season in time_slice_info.seasons.keys() { - problem.add_row( - 0.0.., - [(annual_peak_var, 1.0), (seasonal_peak_vars[season], -1.0)], - ); + for (asset, &annual_peak_variable) in annual_peak_vars { + for season in time_slice_info.seasons.keys() { + let seasonal_peak_variable = seasonal_peak_vars + .get(&(asset.clone(), season.clone())) + .expect("Missing seasonal peak variable for annual peak constraint"); + problem.add_row( + 0.0.., + [(annual_peak_variable, 1.0), (*seasonal_peak_variable, -1.0)], + ); + } + } +} + +/// Add constraints requiring dispatch-equivalent assets to have equal utilisation in each time +/// slice. +/// +/// Flexible-capacity assets are excluded because their maximum activity depends on a decision +/// variable. +pub fn add_equal_utilisation_constraints( + problem: &mut Problem, + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, + time_slice_info: &TimeSliceInfo, + assets: &[AssetRef], +) { + let asset_groups = group_dispatch_equivalent_assets(assets.iter()); + + // For each group of assets, add constraints to force equal utilisation in each time slice + // This is done by anchoring each asset to the first asset in the group (-> (n-1) constraints + // for a group of n assets) + for assets in asset_groups { + let Some((reference_asset, others)) = assets.split_first() else { + continue; + }; + + let reference_max = reference_asset.max_activity().value(); + + for asset in others { + let asset_max = asset.max_activity().value(); + + for time_slice in time_slice_info.iter_ids() { + // Constraint: (act_a * max_b) - (act_b * max_a) = 0 + problem.add_row( + 0.0..=0.0, + [ + ( + activity_vars[&((*asset).clone(), time_slice.clone())], + reference_max, + ), + ( + activity_vars[&((*reference_asset).clone(), time_slice.clone())], + -asset_max, + ), + ], + ); + } + } } } diff --git a/src/simulation/investment/competitive/metrics.rs b/src/simulation/investment/competitive/metrics.rs index 21aac7d3c..d4bc96532 100644 --- a/src/simulation/investment/competitive/metrics.rs +++ b/src/simulation/investment/competitive/metrics.rs @@ -17,6 +17,7 @@ use indexmap::IndexMap; use serde::Serialize; use std::any::Any; use std::cmp::Ordering; +use std::collections::HashMap; use std::sync::Arc; /// Compares two values with approximate equality checking. @@ -54,6 +55,8 @@ pub struct AppraisalOutput { pub metric: Option>, /// Activity coefficients and market costs used in the appraisal pub coefficients: Arc, + /// Full solution to the optimisation + pub solution: IndexMap<(AssetRef, TimeSliceID), Activity>, } impl AppraisalOutput { @@ -66,10 +69,11 @@ impl AppraisalOutput { ) -> Self { Self { asset, - activity: results.activity, + activity: results.asset_activity, unmet_demand: results.unmet_demand, metric: metric.map(|m| Box::new(m) as Box), coefficients, + solution: results.solution, } } /// Compare this appraisal to another on the basis of the comparison metric. @@ -208,25 +212,35 @@ impl MetricTrait for NPVMetric {} /// The returned `metric` is the LCOX value (lower is better). fn calculate_lcox( model: &Model, + selected_assets: &[AssetRef], asset: &AssetRef, commodity: &Commodity, - coefficients: &Arc, + coefficients: &HashMap>, demand: &DemandMap, + previous_solution: &IndexMap<(AssetRef, TimeSliceID), Activity>, ) -> Result { - let results = perform_optimisation(model, asset, commodity, coefficients, demand)?; + let results = perform_optimisation( + model, + selected_assets, + asset, + commodity, + coefficients, + demand, + previous_solution, + )?; let cost_index = lcox( asset.total_capacity(), annual_fixed_cost(asset), - &results.activity, - &coefficients.market_costs, + &results.asset_activity, + &coefficients[asset].market_costs, ); Ok(AppraisalOutput::new( asset.clone(), results, cost_index.map(LCOXMetric::new), - coefficients.clone(), + coefficients[asset].clone(), )) } @@ -237,12 +251,22 @@ fn calculate_lcox( /// An `AppraisalOutput` containing the hypothetical capacity, activity profile and unmet demand. fn calculate_npv( model: &Model, + selected_assets: &[AssetRef], asset: &AssetRef, commodity: &Commodity, - coefficients: &Arc, + coefficients: &HashMap>, demand: &DemandMap, + previous_solution: &IndexMap<(AssetRef, TimeSliceID), Activity>, ) -> Result { - let results = perform_optimisation(model, asset, commodity, coefficients, demand)?; + let results = perform_optimisation( + model, + selected_assets, + asset, + commodity, + coefficients, + demand, + previous_solution, + )?; let annual_fixed_cost = annual_fixed_cost(asset); assert!( @@ -253,15 +277,15 @@ fn calculate_npv( let snas = snas( asset.total_capacity(), annual_fixed_cost, - &results.activity, - &coefficients.market_costs, + &results.asset_activity, + &coefficients[asset].market_costs, ); Ok(AppraisalOutput::new( asset.clone(), results, snas.map(NPVMetric::new), - coefficients.clone(), + coefficients[asset].clone(), )) } @@ -271,19 +295,30 @@ fn calculate_npv( /// /// The `AppraisalOutput` produced by the selected appraisal method. The `metric` field is /// comparable with other appraisals of the same type (npv/lcox). +#[allow(clippy::too_many_arguments)] pub fn appraise_investment( model: &Model, + selected_assets: &[AssetRef], asset: &AssetRef, commodity: &Commodity, objective_type: ObjectiveType, - coefficients: &Arc, + coefficients: &HashMap>, demand: &DemandMap, + previous_solution: &IndexMap<(AssetRef, TimeSliceID), Activity>, ) -> Result { let appraisal_method = match objective_type { ObjectiveType::LevelisedCostOfX => calculate_lcox, ObjectiveType::NetPresentValue => calculate_npv, }; - appraisal_method(model, asset, commodity, coefficients, demand) + appraisal_method( + model, + selected_assets, + asset, + commodity, + coefficients, + demand, + previous_solution, + ) } /// Compare assets as a fallback if metrics are equal. @@ -458,6 +493,7 @@ mod tests { activity: IndexMap::new(), unmet_demand: IndexMap::new(), metric: Some(metric), + solution: IndexMap::new(), }) .collect() } @@ -732,6 +768,7 @@ mod tests { activity: IndexMap::new(), unmet_demand: IndexMap::new(), metric: None, + solution: IndexMap::new(), }; let mut outputs = vec![output]; diff --git a/src/simulation/investment/competitive/optimisation.rs b/src/simulation/investment/competitive/optimisation.rs index f20861e43..1349968b3 100644 --- a/src/simulation/investment/competitive/optimisation.rs +++ b/src/simulation/investment/competitive/optimisation.rs @@ -1,20 +1,25 @@ //! Optimisation problem for investment tools. use super::coefficients::ObjectiveCoefficients; use super::constraints::{ - add_activity_constraints, add_demand_constraints, add_utilisation_peak_constraints, + add_activity_constraints, add_demand_constraints, add_equal_utilisation_constraints, + add_utilisation_peak_constraints, }; +use super::record_asset_selection; use crate::asset::AssetRef; use crate::commodity::Commodity; use crate::model::Model; use crate::simulation::investment::DemandMap; use crate::simulation::optimisation::ModelError; use crate::simulation::optimisation::apply_highs_options_from_toml; +use crate::simulation::optimisation::group_dispatch_equivalent_assets; use crate::simulation::optimisation::solve_optimal; use crate::time_slice::{TimeSliceID, TimeSliceInfo}; use crate::units::{Activity, Flow}; use anyhow::{Context, Result}; use highs::{RowProblem as Problem, Sense}; use indexmap::IndexMap; +use std::collections::HashMap; +use std::sync::Arc; /// A decision variable in the optimisation /// @@ -24,10 +29,12 @@ pub type Variable = highs::Col; /// Map containing optimisation results and coefficients pub struct ResultsMap { - /// Activity variables in each time slice - pub activity: IndexMap, + /// Activity variables in each time slice for the asset of interest + pub asset_activity: IndexMap, /// Remaining unmet demand per time slice, computed post-solve pub unmet_demand: DemandMap, + /// Full solution to the optimisation (activities of all assets) + pub solution: IndexMap<(AssetRef, TimeSliceID), Activity>, } /// Adds activity variables to the problem, one per time slice. @@ -35,38 +42,40 @@ pub struct ResultsMap { /// Returns a map from time slice to the corresponding decision variable. fn add_activity_vars( problem: &mut Problem, - cost_coefficients: &ObjectiveCoefficients, -) -> IndexMap { - cost_coefficients - .activity_coefficients - .iter() - .map(|(time_slice, cost)| { + assets: &[AssetRef], + coefficients: &HashMap>, +) -> IndexMap<(AssetRef, TimeSliceID), Variable> { + let mut activity_vars = IndexMap::new(); + for asset in assets { + for (time_slice, cost) in &coefficients[asset].activity_coefficients { let var = problem.add_column(cost.value(), 0.0..); - (time_slice.clone(), var) - }) - .collect() + activity_vars.insert((asset.clone(), time_slice.clone()), var); + } + } + activity_vars } /// Adds constraints to the problem. fn add_constraints( problem: &mut Problem, model: &Model, - asset: &AssetRef, + assets: &[AssetRef], commodity: &Commodity, - activity_vars: &IndexMap, + activity_vars: &IndexMap<(AssetRef, TimeSliceID), Variable>, demand: &DemandMap, time_slice_info: &TimeSliceInfo, ) { - add_activity_constraints(problem, asset, activity_vars, time_slice_info); + add_activity_constraints(problem, assets, activity_vars, time_slice_info); + add_utilisation_peak_constraints(problem, model, assets, activity_vars); add_demand_constraints( problem, - asset, + assets, commodity, time_slice_info, demand, activity_vars, ); - add_utilisation_peak_constraints(problem, model, asset, activity_vars); + add_equal_utilisation_constraints(problem, activity_vars, time_slice_info, assets); } /// Computes remaining unmet demand per time-slice selection after a solve. @@ -77,20 +86,32 @@ fn add_constraints( /// there is no need to distribute values across individual time slices. fn compute_unmet_demand( demand: &DemandMap, - activity: &IndexMap, + assets: &[AssetRef], + activity: &IndexMap<(AssetRef, TimeSliceID), Activity>, commodity: &Commodity, - asset: &AssetRef, time_slice_info: &TimeSliceInfo, ) -> DemandMap { - let flow_coeff = asset.get_flow(&commodity.id).unwrap().coeff; time_slice_info .iter_selections_at_level(commodity.time_slice_level) .map(|ts_selection| { let demand_for_selection = demand[&ts_selection]; - let supply_for_selection: Flow = ts_selection - .iter(time_slice_info) - .map(|(ts, _)| activity[ts] * flow_coeff) + + // Calculate supply for selection: sum output over all assets over all time slices in + // the selection + let supply_for_selection: Flow = assets + .iter() + .flat_map(|asset| { + let coefficient = asset.get_flow(&commodity.id).unwrap().coeff; + + ts_selection + .iter(time_slice_info) + .map(move |(time_slice, _)| { + let value = activity[&(asset.clone(), time_slice.clone())]; + value * coefficient + }) + }) .sum(); + let unmet = (demand_for_selection - supply_for_selection).max(Flow(0.0)); (ts_selection, unmet) }) @@ -100,20 +121,26 @@ fn compute_unmet_demand( /// Performs optimisation for an asset, given the coefficients and demand. pub fn perform_optimisation( model: &Model, + selected_assets: &[AssetRef], asset: &AssetRef, commodity: &Commodity, - coefficients: &ObjectiveCoefficients, + coefficients: &HashMap>, demand: &DemandMap, + previous_solution: &IndexMap<(AssetRef, TimeSliceID), Activity>, ) -> Result { - // Create problem and add variables + // Record hypothetical asset selection + let mut assets = selected_assets.to_vec(); + record_asset_selection(asset.clone(), &mut assets); + + // Create problem let mut problem = Problem::default(); - let activity_vars = add_activity_vars(&mut problem, coefficients); + let activity_vars = add_activity_vars(&mut problem, &assets, coefficients); // Add constraints add_constraints( &mut problem, model, - asset, + &assets, commodity, &activity_vars, demand, @@ -133,16 +160,83 @@ pub fn perform_optimisation( let solution = solve_optimal(highs_model) .map_err(ModelError::into_anyhow)? .get_solution(); + + // Extract asset activities and calculate unmet demand let solution_values = solution.columns(); - let activity = activity_vars + let all_activity = activity_vars .keys() .zip(solution_values.iter()) - .map(|(time_slice, &value)| (time_slice.clone(), Activity::new(value))) + .map(|(key, &value)| (key.clone(), Activity::new(value))) .collect(); - let unmet_demand = - compute_unmet_demand(demand, &activity, commodity, asset, &model.time_slice_info); + let unmet_demand = compute_unmet_demand( + demand, + &assets, + &all_activity, + commodity, + &model.time_slice_info, + ); + + // Collect equivalence group for the asset of interest + let equivalence_groups = group_dispatch_equivalent_assets(assets.iter()); + let asset_group = equivalence_groups + .iter() + .find(|group| group.contains(&asset)) + .expect("Asset of interest must be present in an equivalence group"); + + // Calculate marginal activity for this asset equivalence group - we attribute this all to the + // candidate tranche + let marginal_activity: IndexMap<_, _> = all_activity + .iter() + .filter(|((asset, _ts), _value)| asset_group.contains(&asset)) + .fold( + IndexMap::new(), + |mut marginal, ((candidate, ts), current_value)| { + let previous_value = previous_solution + .get(&(candidate.clone(), ts.clone())) + .copied() + .unwrap_or(Activity(0.0)); + + *marginal.entry(ts.clone()).or_insert(Activity(0.0)) += + *current_value - previous_value; + marginal + }, + ); + + // Note, there are some cases where marginal activity in certain timeslices (and overall) can be + // negative (i.e. where adding a tranche to the equivalence group actually decreases activity). + // Two known cases where this can happen: + // + // For annually/seasonally balanced commodities, where increasing capacity hits a binding demand + // constraint which then gives the asset freedom to choose the best timeslices to meet demand. + // Before the demand constraint is met, the asset will serve using all profitable timeslices. + // Once the demand constraint is met, not all timeslices need to be active to meet demand, and + // the asset will prioritise the _most_ profitable timeslices, at the expense of less profitable + // timeslices which will see their activities fall. In these circumstances individual timeslices + // can have negative marginal activity, but marginal activity in each balanced time slice + // selection (and the year as a whole) cannot be negative (but may be zero). If marginal + // activity for the year is zero then LCOX is undefined and the tranche will not be considered, + // even if it would be beneficial for the asset as a whole to add that tranche. This is + // something we should consider. + // + // A second scenario is where there are two different (non-equivalent) assets with identical + // marginal cost (despite having different VOC and/or flows), and where moving activity between + // the assets isn't prevented by constraints (e.g. activity constraints or utilisation peak + // constraints). In this case there will be two identical corner solutions, and adding a tranche + // to asset A may inadvertently cause it to switch to the equally good solution where asset B + // wins, hence resulting in a decrease in activity (the opposite can also happen where adding a + // tranche to asset B switches it to a solution where asset B wins, resulting in artificially + // inflated marginal activity). For example, in the simple example for 2021 investments RELCHP + // and RGASPB have identical marginal cost in autumn.night and autumn.evening. This is a bigger + // problem, going back to the issues of degeneracy and non-deterministic results, and something + // we should try to understand better and address at the dispatch stage (which ultimately sets + // the shadow prices that determine marginal costs). In the case of investment appraisal, this + // can artificially decrease/increase marginal activities resulting in nonsensical LCOX. This is + // a problem. It may help to seed each model with the previous solution, to reduce the risk of + // flip-flopping, but the underlying issue also needs to be addressed. + Ok(ResultsMap { - activity, + asset_activity: marginal_activity, unmet_demand, + solution: all_activity, }) } diff --git a/src/simulation/investment/competitive/output.rs b/src/simulation/investment/competitive/output.rs index 9d6d319f7..121837f5c 100644 --- a/src/simulation/investment/competitive/output.rs +++ b/src/simulation/investment/competitive/output.rs @@ -110,7 +110,7 @@ mod tests { use crate::asset::{Asset, AssetRef}; use crate::fixture::{asset, time_slice}; use crate::output::DataWriter; - use indexmap::indexmap; + use indexmap::{IndexMap, indexmap}; use itertools::{Itertools, assert_equal}; use rstest::{fixture, rstest}; use std::iter; @@ -134,6 +134,7 @@ mod tests { activity, unmet_demand, metric: Some(Box::new(LCOXMetric::new(MoneyPerActivity(4.14)))), + solution: IndexMap::new(), } } diff --git a/src/simulation/optimisation.rs b/src/simulation/optimisation.rs index 32ba2b28c..fd5baf237 100644 --- a/src/simulation/optimisation.rs +++ b/src/simulation/optimisation.rs @@ -20,6 +20,7 @@ use std::error::Error; use std::ops::Range; mod constraints; +pub(crate) use constraints::group_dispatch_equivalent_assets; use constraints::{ConstraintKeys, add_model_constraints}; /// A map of commodity flows calculated during the optimisation diff --git a/src/simulation/optimisation/constraints.rs b/src/simulation/optimisation/constraints.rs index b33d32b69..912f54af3 100644 --- a/src/simulation/optimisation/constraints.rs +++ b/src/simulation/optimisation/constraints.rs @@ -511,7 +511,7 @@ where /// /// The caller must ensure that `assets` contains only assets eligible for equal-utilisation /// constraints. -fn group_dispatch_equivalent_assets<'a, I>(assets: I) -> Vec> +pub fn group_dispatch_equivalent_assets<'a, I>(assets: I) -> Vec> where I: Iterator, {