Lidar Data Formats and Archives: Making Sense of Atmospheric Measurements
We still remember the afternoon vividly. Two datasets, both claiming to hold ozone profiles over the same European site, both produced by competent teams, and yet the files refused to line up. One used a time axis in seconds since midnight; the other used hours since the start of the campaign. One stored altitude in metres above sea level, the other in kilometres above ground. We spent more time reconciling metadata than analysing the atmosphere itself. That afternoon taught us a lesson we have never forgotten: data formats are not a housekeeping detail. They are the difference between measurements that travel and measurements that languish on a local drive.
Why Lidar Data Formats Matter for Atmospheric Science
Lidar is one of the most powerful tools we have for probing the stratosphere. A single system can retrieve ozone concentration, aerosol backscatter, temperature, and water vapour by firing laser pulses upward and timing the faint returns. But the value of those measurements multiplies when they can be compared with data from other instruments, other stations, and other decades. That comparison depends entirely on whether the files can be read, understood, and trusted by someone who was not in the room when they were written.
The cost of incompatible files
Inconsistent formats slow down everything. A researcher trying to assemble a multi-station ozone record may spend weeks writing bespoke parsers for each provider. Calibration coefficients hide in PDFs rather than in the file headers. Units drift between datasets. In ozone monitoring, where a small systematic bias can distort a trend that only becomes visible over fifteen years, these frictions are not merely annoying — they undermine the science. We have seen campaigns where a promising intercomparison stalled simply because no one could agree on which column held the uncertainty estimate.
How NDACC set a common baseline
The Network for the Detection of Atmospheric Composition Change (NDACC) recognised this problem early. As a global network spanning more than twenty countries, NDACC needed its member stations to deliver data that could be ingested by centralised analysis tools without manual intervention. The network therefore established standardised file structures, mandatory metadata fields, and clear guidance on units and conventions. The result is that an ozone lidar profile from a station in the Alps can be plotted alongside one from the Arctic with minimal effort. That harmonisation is the quiet infrastructure behind a great deal of stratospheric research.
Common Lidar Data Formats We Encounter
In practice, three families of formats dominate the lidar world, and each has its place.
NetCDF and HDF5: strengths and trade-offs
NetCDF and HDF5 are both self-describing, array-oriented formats built on similar underlying concepts. They store data alongside metadata, support compression, and handle large multidimensional arrays gracefully — ideal for a time series of backscatter profiles at thousands of range gates. NDACC recommends NetCDF for lidar data archiving, and this is the format we default to for final products. NetCDF’s climate and forecast conventions are widely understood, and tools such as Python’s xarray make reading and writing straightforward.
HDF5 is equally capable and is common in the satellite community. Here in the UK, the Chilbolton Observatory in Hampshire often uses HDF5 for raw signals, partly because its radar and lidar instruments share a common acquisition heritage. The trade-off is mainly one of ecosystem: NetCDF tends to dominate atmospheric chemistry archives, while HDF5 dominates remote sensing and Earth observation. Converting between them is technically simple but easy to get wrong if metadata attributes are not carried across carefully.
When ASCII still makes sense
Plain ASCII files remain surprisingly persistent. They are human-readable, trivially portable, and easy to inspect with a text editor. For quick-look plots, small calibration tables, or legacy datasets that predate modern archiving, ASCII is perfectly reasonable. What it lacks is structure: without a strict header convention, every ASCII file risks becoming its own dialect. We treat ASCII as a convenience format for exchange and debugging, never as the archival master.
NDACC Archives and Data Access
The NDACC data archive is the network’s collective memory. It holds lidar, FTIR, UV-visible, and microwave measurements from stations worldwide, each accompanied by the metadata needed to interpret it. Access is open to registered users, and the archive is designed so that a newcomer can find, understand, and cite a dataset without emailing the principal investigator.
Navigating the NDACC archive
The archive is organised by instrument type and station, with search tools that let you filter by species, altitude range, and time period. Each station submits data in the agreed format, and the archive team performs quality checks before ingestion. The Reading site contributes ozone lidar measurements to NDACC, and those profiles sit alongside data from partner stations across Europe, making them directly comparable within the archive interface.
Metadata that makes or breaks usability
If there is one lesson we repeat to every new team member, it is this: metadata is the data. A profile without a documented wavelength, a stated integration time, and a clear uncertainty budget is nearly useless. NDACC’s metadata requirements cover instrument configuration, calibration history, geolocation, and processing version. When those fields are complete, a dataset can be reused years later by someone with no connection to the original campaign. When they are missing, even the original team may struggle to reconstruct what was measured.
UK Lidar Networks and Their Archives
The United Kingdom has a long and productive history in atmospheric lidar, and its infrastructure is well integrated with international efforts.
NCAS and Chilbolton data portals
The UK’s National Centre for Atmospheric Science (NCAS) manages atmospheric data archives that serve universities, research institutes, and government bodies. Its portals provide access to instrument data from across the national capability, including lidar systems. The Chilbolton Observatory in Hampshire operates lidar systems contributing to NDACC, and its facility data are a key part of the national record. For anyone working on stratospheric ozone or aerosol profiles over northern Europe, these archives are a first stop.
Linking UK archives to NDACC
British data do not sit in isolation. The British Atmospheric Data Centre (BADC) hosts UK atmospheric science data and provides long-term stewardship for many datasets generated by NCAS facilities. From there, quality-controlled products are submitted to NDACC, where they become part of the global record. This chain — instrument to national archive to international network — is what allows a measurement made on a Hampshire hillside to inform a global assessment of stratospheric change.
Best Practices for Archiving Lidar Data
Over the years we have distilled our archiving habits into a checklist. It is not glamorous, but it saves careers.
- Choose the archival format deliberately. For NDACC submissions, that means NetCDF with documented conventions.
- Version everything. Every processing change gets a new version number and a note explaining what changed and why.
- Document calibration procedures. Wavelengths, reference cells, Rayleigh fits, and dead-time corrections all belong in the metadata, not in a notebook.
- Encode uncertainty. Statistical and systematic errors should travel with the data, not be reconstructed later.
- Follow the FAIR principles. Data should be Findable, Accessible, Interoperable, and Reusable — a framework that maps neatly onto good lidar practice.
- Deposit in a trusted repository. In the UK, that often means the British Atmospheric Data Centre (BADC), which provides persistent identifiers and long-term curation.
Documentation and versioning
We treat documentation as part of the measurement, not an administrative afterthought. A short processing log, stored alongside the data, answers the questions that future users will inevitably ask. Version control prevents the quiet catastrophe of two analysts using different revisions of the same dataset and reaching different conclusions.
Long-term preservation strategies
Hard drives fail, servers are decommissioned, and staff move on. Long-term preservation means handing data to an institution whose mission is stewardship. National facilities such as BADC and international archives such as NDACC’s provide that continuity. Our rule is simple: if the only copy of a dataset lives on a laptop, it does not yet exist.
FAQ
Which format does NDACC recommend for lidar data archiving?
NDACC recommends NetCDF for lidar data archiving. The format’s self-describing structure and broad tool support make it well suited to long-term atmospheric data records.
Why does the Chilbolton Observatory use HDF5?
The Chilbolton Observatory in Hampshire operates lidar systems contributing to NDACC and often uses HDF5 for raw signals, largely because its instruments share acquisition systems with other remote sensing equipment. Final products are typically converted to NetCDF for archiving.
Where can I find UK atmospheric lidar data?
The UK’s National Centre for Atmospheric Science (NCAS) manages atmospheric data archives, and the British Atmospheric Data Centre (BADC) hosts UK atmospheric science data for long-term access. The Reading site contributes ozone lidar measurements to NDACC, which are available through the NDACC archive.
Is ASCII ever acceptable for lidar data?
Yes, for small exchange files, calibration tables, and quick-look products. We would not use it as the archival master because it lacks the structured metadata that makes datasets reusable.
What are the FAIR principles and why do they matter?
FAIR stands for Findable, Accessible, Interoperable, and Reusable. Applying them to lidar data means using standard formats, complete metadata, persistent identifiers, and trusted repositories — exactly the practices that keep atmospheric records useful for decades.
Data formats will never be the most exciting part of atmospheric science, but they are a shared responsibility that determines whether our measurements outlive the instruments that produced them. By aligning with NDACC conventions, depositing data in national archives such as BADC, and documenting our calibration work properly, we give future researchers the best possible chance of understanding how the stratosphere has changed. Interoperable archives are not a technical luxury — they are how a network of lidar stations becomes something greater than the sum of its parts.