A Hackers Guide to AEMO & NEM Data
A simple guide to data provided by AEMO for the Australia’s National Electricity Market (NEM).
This is a short guide to the electricity grid & market data supplied by the Australian Energy Market Operator (AEMO) for the Australian National Electricity Market (NEM).
The NEM is Australia’s electricity grid in Queensland, New South Wales, Victoria, South Australia, and Tasmania.
Participant Infomation & Carbon Intensities
Market participant information in the NEM is given in the NEM Registration and Exemption List:
The carbon intensities for generators are given in the Available Generators CDEII file:
Both of these files are linked by a Dispatchable Unit Identifier (DUID), which identifies generating unit.
Interval data for the NEM is provided in two sources the NEM Dispatch Engine (NEMDE) and the Market Management System Data Model (MMSDM).
The NEMDE dataset provides infomation about how the grid is dispatched and price are set (including infomation about the marginal generator) in the
NemPriceSetter XML files.
Data for each day is provided in a single ZIP file (NemPriceSetter_20220101_xml.zip), which contains many XML files:
# NemPriceSetter_20220101_xml/NEMPriceSetter_2022010100100.xml <PriceSetting PeriodID="2022-01-01T04:05:00+10:00" RegionID="NSW1" Market="Energy" Price="87.69011" Unit="LBBG1" DispatchedMarket="R5RE" BandNo="6" Increase="1" RRNBandPrice="23.7" BandCost="23.7" /> <PriceSetting PeriodID="2022-01-01T04:05:00+10:00" RegionID="NSW1" Market="Energy" Price="87.69011" Unit="BW04" DispatchedMarket="R5RE" BandNo="1" Increase="-0.47368" RRNBandPrice="1" BandCost="-0.473684" /> <PriceSetting PeriodID="2022-01-01T04:05:00+10:00" RegionID="NSW1" Market="Energy" Price="87.69011" Unit="BW03" DispatchedMarket="R5RE" BandNo="1" Increase="-0.52632" RRNBandPrice="1" BandCost="-0.526316" />
The MMSDM provides both actual data and forecasts for a range of variables - including prices, demand and electricity flows.
Data in the MMSDM is supplied from three different, overlapping sources:
Some report names can be different across sources - for example
The settlement price in the NEM is known as the trading price - it is the price that matters for what generators get paid and what customers pay.
Historically (before October 2021) it was settled on a 30 minute basis, as the average of the six 5 minute dispatch prices for the same interval.
AEMO timestamp with the time at the end of the interval. This means that
01/01/2018 14:00 refers to the time period
01/01/2018 13:30 - 01/01/2018 14:00. This will be true for columns like
SETTLEMENTDATE, which refer to an interval. Columns like
LASTCHANGED, which refer to a single instant in time are not affected by this.
I prefer shifting the AEMO time stamp backwards by one step of the index frequency (i.e. 5 minutes). This allows the following to be true:
dispatch_prices.loc['01/01/2018 13:30': '01/01/2018 14:00'].mean() == trading_price.loc['01/01/2018 13:30']
The shifting also allows easier alignment with external data sources such as weather, which is usually stamped with the timestamp at the beginning of the interval.
If the AEMO timestamp is not shifted, then the following is true:
dispatch_prices.loc['01/01/2018 13:35': '01/01/2018 14:05'].mean() == trading_price.loc['01/01/2018 14:00']
Useful MMSDM Reports
All examples below are for MMSDM May 2018:
- trading price (30 & 5 min electricity price) - TRADINGPRICE - PUBLIC_DVD_TRADINGPRICE_201805010000.zip,
- dispatch price (5 min electricity price) - DISPATCHPRICE - PUBLIC_DVD_DISPATCHPRICE_201805010000.zip,
- generation of market participants - UNIT_SCADA - PUBLIC_DVD_DISPATCH_UNIT_SCADA_201805010000.zip,
- market participant bid volumes - BIDPEROFFER - PUBLIC_DVD_BIDPEROFFER_201805010000.zip,
- market participant bid prices - BIDAYOFFER - PUBLIC_DVD_BIDDAYOFFER_201805010000.zip,
- demand - DISPATCHREGIONSUM - PUBLIC_DVD_DISPATCHREGIONSUM_201805010000.zip,
- interconnectors - INTERCONNECTORRES - PUBLIC_DVD_DISPATCHINTERCONNECTORRES_201805010000.zip.
- trading price forecast - PUBLIC_DVD_PREDISPATCHPRICE_201805010000.zip,
- dispatch price forecast - PUBLIC_DVD_P5MIN_REGIONSOLUTION_201805010000.zip.
A major benefit of the large & open dataset shared by AEMO is the ecosystem tools built on top of it.
A simple CLI for downloading NEMDE & MMSDM data - created & maintained by yours-truly:
$ pip install nem-data $ nemdata --table trading-price --start 2020-01 --end 2020-12
A Python package for downloading historical data published by the Australian Energy Market Operator (AEMO):
$ pip install nemosis
Use in Python:
from nemosis import dynamic_data_compiler start_time = '2017/01/01 00:00:00' end_time = '2017/01/01 00:05:00' table = 'DISPATCHPRICE' raw_data_cache = 'C:/Users/your_data_storage' price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache)
AEMO Dashboard - interactive map
Gas & Coal Watch
- NEM on the AEMO website,
- Winds of change: An analysis of recent changes in the South Australian electricity market - University of Melbourne,
- Li, Zili (2016) Topics in deregulated electricity markets. PhD thesis, Queensland University of Technology,
- Dungey et. al (2018) Strategic Bidding of Electric Power Generating Companies: Evidence from the Australian National Energy Market.
Thanks for reading!