Overview
What NESO is, who it serves, and how this knowledge base is organised.
Overview
NESO is the National Energy System Operator's data portal — a stack that catalogues, ingests, and publishes datasets used internally and by external consumers.
What it does
- Catalogues every published dataset with metadata and access controls.
- Lets approved publishers upload, edit, and version datasets through an admin UI.
- Serves search, browsing, and previews to end users via a public-facing portal.
- Exposes the same data through a read-only HTTP API for programmatic consumers.
- Ingests data on a schedule or on dataset events via Airflow pipelines.
Components
| Component | Role |
|---|---|
| Servers | GKE cluster, networking, object storage |
| PortalJS Admin | Admin UI for publishers and sysadmins |
| CKAN | Source of truth for dataset metadata |
| Data API | Read-only HTTP API for consumers |
| Datastore | BigQuery (or DuckLake) — holds the rows of tabular data |
| Ingestion | Airflow DAGs that load data into the datastore |
| DXP Frontend | Public Drupal portal for end users |
| Monitoring | Prometheus + Grafana dashboards and alerts |
Hard rule: consumers read through the Data API, never CKAN directly. Writes happen only through PortalJS Admin → CKAN, or through ingestion DAGs.
Who this knowledge base is for
| Audience | Where to start |
|---|---|
| End users | User Guide |
| Portal admins | Admin Guide |
| Developers | Architecture overview · Development · Getting Started |
Last reviewed: 2026-05-04
