Core configuration
Table of Contents
CLI
Infilling database
- climate_assessment.cli.create_infiller_database(*args: Any, **kwargs: Any) Any
Creates infiller database by harmonizing data in
input_emissions_file
which is followed by some common-sense vetting and then saved inoutdir
input_emissions_file
should be a path to a file of emissionsoutdir
should be a path a directory which already exists
Infilling
- climate_assessment.cli.infill(*args: Any, **kwargs: Any) Any
Infill harmonized data in
input_emissions_file
, saving output inoutdir
input_emissions_file
should be a path to a file of harmonized emissionsoutdir
should be a path a directory which already exists
Harmonisation
- climate_assessment.cli.harmonize(*args: Any, **kwargs: Any) Any
Harmonise data in
input_emissions_file
, saving output inoutdir
input_emissions_file
should be a path to a file of emissionsoutdir
should be a path a directory which already exists
Harmonisation and infilling
- climate_assessment.cli._harmonize_and_infill(input_df, inputcheck, key_string, outdir, infilling_database, harmonize, prefix, harmonization_instance)
Thin wrapper function for running both harmonization and infilling.
Returns True if there are scenarios that can be run by a climate emulator. Those scenarios are not returned by this function, but rather written to the outdir. Returns False if there are no complete scenarios to be run.
For more information, see the code description under
climate_assessment.harmonization_and_infilling.harmonization_and_infilling()
.
- climate_assessment.cli.harmonize_and_infill(*args: Any, **kwargs: Any) Any
Harmonise and infill data in
input_emissions_file
, saving output inoutdir
input_emissions_file
should be a path to a file of emissionsoutdir
should be a path a directory which already exists
- climate_assessment.cli.harmonization_and_infilling(df, key_string, infilling_database, prefix='AR6 climate diagnostics', instance='ar6', outdir='output', do_harmonization=True)
- Parameters
input_df (
pyam.IamDataFrame
) – Input native emissions to harmonize and infill.inputcheck (bool) – Perform checks to remove unsuitable emissisons pathways. [Default: True]
key_string (str) – Identifier string for writing out results. By default derived from the input_emissions_file string in a CLI command.
outdir (str) – Path to output folder.
infilling_database (str) – Path to the infiller database emissions file.
harmonize (bool) – Perform harmonization (with False, only infilling is run). [Default: True]
prefix (str) – Prefix used to identify the new variable names of results produced by the workflow.
harmonization_instance (str) – Config string required by aneris.
- Returns
Returns True if there are scenarios that can be run by a climate emulator. Those scenarios are not returned by this function, but rather written to the outdir. Returns False if there are no complete scenarios to be run.
- Return type
bool
Workflow
- climate_assessment.cli.workflow(*args: Any, **kwargs: Any) Any
Run the full IPCC AR6 climate asessment workflow.
Example usage:
python scripts/run_workflow.py tests/test-data/ex2.csv output --model "FaIR" --model-version "1.6.2" --num-cfgs 2237 --probabilistic-file tests/test-data/fair-1.6.2-wg3-params-slim.json --fair-extra-config tests/test-data/fair-1.6.2-wg3-params-common.json --infilling-database src/climate_assessment/infilling/cmip6-ssps-workflow-emissions.csv
See further documentation at Description of the workflow.
Climate
- climate_assessment.cli.clim_cli(*args: Any, **kwargs: Any) Any
Run the climate emulator step of the IPCC AR6 climate asessment workflow.
Example usage:
python scripts/run_clim.py tests/test-data/workflow-fair/ex2_harmonized_infilled.csv output --model "fair" --model-version "1.6.2" --num-cfgs 2237 --probabilistic-file tests/test-data/fair-1.6.2-wg3-params-slim.json --fair-extra-config tests/test-data/fair-1.6.2-wg3-params-common.json --scenario-batch-size 4
For more information, see the code description under
climate_assessment.climate.climate_assessment()
.
Postprocess
- climate_assessment.cli._postprocess_worker(fname, outdir, **kwargs)
Helper function which takes a file that ends with “_rawoutput.xlsx” and the output location, and then calls the function do_postprocess(output, outdir, key_string, prefix).
- climate_assessment.cli.postprocess(*args: Any, **kwargs: Any) Any
Merge and postprocess a collection of rawoutput into a single set of output files
This script should be run after run_workflow.py if –no-postprocess was supplied. This is particularly useful for a clusted run where a run_workflow is run multiple times with batches of scenario data.
Each file is processed individually, but the metadata from all the runs is merged into a single output file.
Infilling
- climate_assessment.infilling.run_infilling(harmonised_df, prefix, database_filepath=None, start_year=2015, end_year=2100)
Run infilling
- Parameters
harmonised_df (
pyam.IamDataFrame
) – Input harmonised emissions to infillprefix (str) – Prefix used for the variable names
database_filepath (str) – Path to the file which contains the infilling database
start_year (int) – First year which should be reported in output
end_ear (int) – Last year which should be reported in output
- Returns
Infilled emissions. Output also includes the input emissions. The output is interpolated onto an annual timestep.
- Return type
pyam.IamDataFrame
- climate_assessment.infilling._infill_variables(cruncher, variables, to_infill, db, lead, output_timesteps, old_prefix)
Start core infilling run, using silicone.multiple_infillers.infill_all_required_variables
- Parameters
cruncher (str) – Infilling method
variables (list) – List of required variables to infill
to_infill (
pyam.IamDataFrame
) – Emissions data to be infilleddb (
pyam.IamDataFrame
) – The infilling databaselead (str) – Lead infiller variable
output_timesteps (list) – List of required output timesteps
old_prefix (str) – The prefix of harmonized emissions, which will be replaced by a prefix for the infilled emissions variables.
- Returns
Infilled emissions.
- Return type
pyam.IamDataFrame
- climate_assessment.infilling._add_to_infilled(infilled, infilled_variables)
Helper function which takes two dataframes, and binds them together.
- Parameters
infilled (
pyam.IamDataFrame
) – An IamDataFrame.infilled_variables (str, None) – An IamDataFrame.
- Returns
An IamDataFrame.
- Return type
pyam.IamDataFrame
- climate_assessment.infilling.load_csv_or_xlsx_for_one_region(file, sheet)
This loads data from either a csv or an xls(x) file and checks that it has only a single region.
- Parameters
file (str, None) – The (relative or absolute) file address of the file to load
sheet (str, None) – If file is a csv, use None. Otherwise, the sheet name to read from the file.
- Returns
The information in the file.
- Return type
pyam.IamDataFrame
- climate_assessment.infilling.postprocess_infilled_for_climate(df_infilled, prefix, start_year=2015)
Helper function that takes a set of infilled emissions data, adds a provided prefix, and filters out the scenarios that are appropriately infilled by checking both for a minimum set of years of data and checking that there are no non-co2 negative emissions caused by the infilling method.
- Parameters
df_infilled (
pyam.IamDataFrame
) – Infilled data.prefix (str) – Prefix used for the variable names
- Returns
Checked and reformatted infilled data.
- Return type
pyam.IamDataFrame
Harmonization
- climate_assessment.harmonization.postprocessing(harmonized_results, prefix)
Helper function that makes sure that the harmonization prefix is added.
- climate_assessment.harmonization.add_year_historical_percentage_offset(df, dfhist, yr=2015, low_yr=2010)
In the case that the chosen harmonization year (yr, default=2015) is not reported, this function uses an earlier year (low_yr, default=2010) to derive the historical offset for harmonization. It takes the relative difference in low_yr, and assumes that this relative difference stays the same until yr.
- climate_assessment.harmonization.run_harmonization(df, instance, prefix)
Run harmonization. Hamronization method overrides by specific species are set within this function too.
- Parameters
df (
pyam.IamDataFrame
) – Input native emisisons in IAMC format.instance (str) – String used to choose what historical data to use.
prefix (str) – Prefix used for the variable names
- Returns
Harmonized scenarios.
- Return type
pyam.IamDataFrame
Harmonisation and infilling
- climate_assessment.harmonization_and_infilling.harmonization_and_infilling(df, key_string, infilling_database, prefix='AR6 climate diagnostics', instance='ar6', outdir='output', do_harmonization=True)
- Parameters
input_df (
pyam.IamDataFrame
) – Input native emissions to harmonize and infill.inputcheck (bool) – Perform checks to remove unsuitable emissisons pathways. [Default: True]
key_string (str) – Identifier string for writing out results. By default derived from the input_emissions_file string in a CLI command.
outdir (str) – Path to output folder.
infilling_database (str) – Path to the infiller database emissions file.
harmonize (bool) – Perform harmonization (with False, only infilling is run). [Default: True]
prefix (str) – Prefix used to identify the new variable names of results produced by the workflow.
harmonization_instance (str) – Config string required by aneris.
- Returns
Returns True if there are scenarios that can be run by a climate emulator. Those scenarios are not returned by this function, but rather written to the outdir. Returns False if there are no complete scenarios to be run.
- Return type
bool
Climate
- climate_assessment.climate.climate_assessment(df, key_string, outdir, model='magicc', model_version=None, num_cfgs=600, historical_warming=0.85, historical_warming_reference_period='1850-1900', historical_warming_evaluation_period='1995-2014', test_run=False, scenario_batch_size=20, save_raw_output=False, probabilistic_file='data/magicc/0fd0f62-derived-metrics-id-f023edb-drawnset.json', magicc_extra_config=None, fair_extra_config=None, co2_and_non_co2_warming=False, prefix='AR6 climate diagnostics')
Run the climate assessment
- Parameters
df (
pyam.IamDataFrame
) –pyam.IamDataFrame
with data to assesskey_string (str) – String to use to identify output files and find input files
outdir (str) – Directory in which to save the output
model (str) – Reduced-complexity climate model to run assessment with
model_version (str, None) – Version of the climate model. If None, use default
num_cfgs (int) – Number of model configs to run. Multiply this number by the number of scenarios to get total runs. The full drawnset is 600 so if you’re just testing it’s probably worth using a smaller number unless you want to wait for 58 000 runs (of course by using a smaller subset the temperature numbers won’t make sense but if all you’re checking is that things run you also don’t really care).
historical_warming (float) – Historical warming to match the climate model output to
historical_warming_reference_period (str) – Reference period to use for the historical warming (e.g. “1850-1900”)
historical_warming_evaluation_period (str) – Evaluation period to use for the historical warming (e.g. “1995-2014”)
test_run (bool) – Is this a test run? If it is, we won’t raise an error if the historical temperatures don’t match the assessment perfectly.
scenario_batch_size (int) – How many scenarios do you want to run at once? This should be adjusted to balance the amount of RAM and number of cores available on your system. The limit depends mostly on the RAM available.
save_raw_output (bool) – Should we also save the raw climate output (i.e. every ensemble member) to disk?
probabilistic_file (str) –
- Path to file containing parameters to run climate model with (will be passed to
openscm-runner). Currently, only json files are supported.
co2_and_non_co2_warming (bool) – Include assessment of CO2 and non-CO2 warming (requires 3x as many runs)?
prefix (str) – Prefix for all variable names
- Returns
pyam.IamDataFrame
containing climate assessment- Return type
pyam.IamDataFrame
- climate_assessment.climate.run_and_post_process(scenarios, climate_models_cfgs, climate_models_out_config, historical_warming, historical_warming_reference_period, historical_warming_evaluation_period, outdir, test_run, save_raw_output, co2_and_non_co2_warming)
Run the climate models probabilistically
Uses
openscm-runner
to parallise the model runs. The results are then post processed to calculate exceedence probabilities- Parameters
scenarios ([
scmdata.ScmRun
,pyam.IamDataFrame
]) – Emissions for the scenarios of interestclimate_models_cfgs (dict) – Configuration as expected by openscm-runner
climate_models_out_config (dict) – Climate models output config as expected by OpenSCM-Runner
historical_warming (float) – Historical warming to match the climate model output to
historical_warming_reference_period (str) – Reference period to use for the historical warming (e.g. “1850-1900”)
historical_warming_evaluation_period (str) – Evaluation period to use for the historical warming (e.g. “1995-2014”)
outdir (str) – Output directory for the raw output
test_run (bool) – If true, check model output
save_raw_output (bool) – If True, save all the raw climate model output for later analysis.
co2_and_non_co2_warming (bool) – Include assessment of CO2 and non-CO2 warming?
- climate_assessment.climate._get_model_configs_and_out_configs(model, model_version, probabilistic_file, magicc_extra_config, fair_extra_config, num_cfgs, co2_and_non_co2_warming)
Postprocess
- climate_assessment.climate.post_process.check_hist_warming_period(period)
Check period for historical warming calculations
- Parameters
period (str) – Input period, must be formatted as “YYYY-YYYY” where the first year is less than or equal to the last year. The last year is included in the period.
- Returns
All years in the period
- Return type
range
- Raises
ValueError –
period
is formatted incorrectly
- climate_assessment.climate.post_process.calculate_exceedance_probability_timeseries(res, exceedance_probability_calculation_var, test_run=False, historical_warming=0.85, historical_warming_reference_period='1850-1900', historical_warming_evaluation_period='1995-2014')
Calculate the timeseries with which we should determine exceedance probabilities.
Please note that calculating the statistical properties like the exceedance probability is done on the count of the raw, full ensemble data. It cannot be derived from from the time series of the final statistical indicator afterwards.
- climate_assessment.climate.post_process.calculate_co2_and_nonco2_warming_and_remove_extras(res)
Calculate non-CO2 warming. Currently only implemented for MAGICC, which can be run twice to allow for this calculation, using (the CLI option) co2_and_non_co2_warming.
- climate_assessment.climate.post_process.post_process(res, outdir, test_run=False, save_raw_output=False, co2_and_non_co2_warming=False, temp_thresholds=(1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0), peak_percentiles=(5, 10, 17, 25, 33, 50, 66, 67, 75, 83, 90, 95), percentiles=(5, 10, 16.666666666666664, 17, 25, 33, 50, 66, 67, 75, 83, 83.33333333333334, 90, 95), historical_warming=0.85, historical_warming_reference_period='1850-1900', historical_warming_evaluation_period='1995-2014')
Postprocess
- climate_assessment.postprocess.do_postprocess(output, outdir, key_string, prefix, model_version='v7.5.3', categorisation=True, reporting_completeness_categorisation=True, gwp=True, model='magicc')
Runs any required postprocessing steps
- Namely:
Categorisation
Completeness checks
Add GWP100 Kyoto Gases sums
Adding additional metadata
Plots
Checks on input and output scenario data
- climate_assessment.checks.add_completeness_category(df, filename, delete_no_confidence=False, output_csv=False, outdir='output', prefix='AR6 climate diagnostics|Harmonized|')
Add a meta column that specified the reporting completeness based on qualitative categories.
- climate_assessment.checks.check_against_historical(df, filename, instance, output_csv=False, outdir='output')
Check against historical data. Currently requires 2015 to already be in the dataframe.
- climate_assessment.checks.check_negatives(df, filename=None, negativethreshold=-0.1, outdir='output', prefix='')
Check for negative emissions and remove any timeseries which has negative non-CO2 values.
- climate_assessment.checks.check_reported_co2(df, filename, output_csv=False, outdir='output')
Check CO2 reporting.
- climate_assessment.checks.co2_energyandindustrialprocesses(df)
Auxiliary function for
climate_assessment.checks.check_reported_co2()
. Check if either CO2|Energy and Industrial are reported. Add the aggregate Energy and Industrial, if only subcomponents reported
- climate_assessment.checks.count_variables_very_high(df, vars, num=9, prefix='AR6 climate diagnostics|Harmonized|')
Auxiliary function for
climate_assessment.checks.add_completeness_category()
. Performs check of variables + number of variables
- climate_assessment.checks.infiller_vetting(df, prefix='AR6 climate diagnostics')
Filters out a set of harmonized emissions pathways
df
to then return an infiller_database that does not have emissions pathways that might skew infilling due to what is likely a model reporting error.
- climate_assessment.checks.perform_input_checks(df, output_csv_files=True, output_filename='checks', lead_variable_check=False, historical_check=False, reporting_completeness_check=False, outdir='output')
Perform several selected checks on the native emissions input to the climate assessment workflow.
- Parameters
input_df (
pyam.IamDataFrame
) – Input native emissions before checks.inputcheck (bool) – Perform checks to remove unsuitable emissisons pathways. [Default: True]
key_string (str) – Identifier string for writing out results. By default derived from the input_emissions_file string in a CLI command.
outdir (str) – Path to output folder.
- Returns
IAM dataframe with only model-scenario that will be without emissions. Output also includes the input emissions. The output is interpolated onto an annual timestep.
- Return type
pyam.IamDataFrame
- climate_assessment.checks.reclassify_waste_and_other_co2_ar6(df)
Reclassify waste and other CO2 into the energy and industrial processes category
Reclassify CO2 emissions reported under Emissions|CO2|Other and Emissions|CO2|Waste, instead putting them under Emissions|CO2|Energy and Industrial Processes.
- Parameters
df (
pyam.IamDataFrame
) – The original set of reported emissions- Returns
Reclassified set of emissions.
- Return type
pyam.IamDataFrame
- climate_assessment.checks.remove_rows_with_only_zero(df, filename=None, outdir='output')
Check and remove rows with zero emissions.
- climate_assessment.checks.remove_rows_with_zero_in_harmonization_year(df, filename=None, harmonization_year=2015, outdir='output')
Check for zeros in harmonization year emissions and remove rows.
- climate_assessment.checks.require_allyears(df, filename='test', output_csv=False, outdir='output', required_years=[2020, 2030, 2040, 2050, 2060, 2070, 2080, 2090, 2100], base_yr=2015, low_yr=2010)
Check if some variables are reported for the required years (drops per variable)
- climate_assessment.checks.require_allyears_and_drop_scenarios(df, filename='test', output_csv=False, outdir='output', required_years=[2020, 2030, 2040, 2050, 2060, 2070, 2080, 2090, 2100])
Filters out full scenarios if some year is not reported (using pyam)
- climate_assessment.checks.sanity_check_bounds_kyoto_emissions(output_postprocess, out_kyoto_infilled)
Check that the calculated Kyoto gases of the infilled emissions data are within certain bounds
- climate_assessment.checks.sanity_check_comparison_kyoto_gases(output_postprocess, out_kyoto_harmonized, out_kyoto_infilled)
Check that the calculated Kyoto gases of the infilled emissions data is in every year smaller than the calculated Kyoto gases of the harmonized emission data
- climate_assessment.checks.sanity_check_hierarchy(co2_inf_db, harmonized, infilled, out_afolu, out_fossil)
Check that hierarchy of variables is internally consistent (in this case check that Emissions|CO2 is the sum of AFOLU and Energy and Industrial Processes emissions)