Of hierarchies, chunking, and HEALPix

The key goal of the nextGEMS project is to produce the first-ever km-scale simulations that span a 30-year period, i.e., “one climate”. This produces an immense amount of output data even for single variables, in fact, the number of grid points in a single world map exceeds the number of pixels of a screen by far. To keep this dataset manageable, hierarchical output, chunking, and the HEALPix grid can help.

Hierarchical output involves generating multiple copies of output at different resolutions, just as the functionality of online map services. This spatial hierarchy enables users to access data at resolutions appropriate for their analysis needs, optimizing data loading and hence analysis speed. By storing data in a hierarchical structure, storage requirements remain efficient, because coarser levels occupying significantly less space compared to finer resolutions.

The hierarchical output is accompanied by horizontal chunking of data, which facilitates regional access without the need to load global fields. This approach enables targeted analysis and reduces processing time, particularly for regional-scale investigations, contributing to enhanced workflow efficiency in climate model output analysis.

Both hierarchical output and horizontal chunking can only be applied accurately and efficiently, if the data is stored on an appropriate grid. The HEALPix grid offers a structured pixelation of a sphere based on quadrilaterals with equal area. The structure of the grid ensures that data-locality maps to regional locality, which is necessary for efficient chunking. The equal-area property reduces rounding errors and eliminates numerical smoothing when computing the different levels of the output hierarchy.

Further information and examples on how to efficiently work with the new output hierarchy can be found on easy.gems. In the meantime, check out the short movie by M. Veerman (WUR) showing the surface longwave irradiance with different HEALPix zoom levels.