Hierarchical Data Format for Water-related Big Geodata (HDF4Water)

Project Description

The HDF4Water project is funded by DFG (project no. 460036893) as an 6-month incubator project via the NFDI Consortium Earth System Sciences (NFDI4Earth).

In this project, we developed a big data framework based on the modern HDF5 technology, called AtlasHDF (Werner & Li, 2022), in which we designed lossless data mappings (i.e., immediate mapping and analysis-ready mapping) from OpenStreetMap (OSM) vector data into a single HDF5 data container to facilitate fast and flexible GeoAI applications, for instance, surface water mapping with OSM and Sentinel-2 data. This document describes concisely all decisions that we took towards adapting the flexible AtlasHDF data storage framework to the requirements of a sustainable, simple, (almost) dependency-free data representation for GeoAI across multiple modalities.

Related Resources

• GitHub repository at: https://github.com/tum-bgd/atlashdf
• How to install at: https://github.com/tum-bgd/atlashdf/blob/main/Dockerfile
• How to use at: https://github.com/tum-bgd/atlashdf/blob/main/HDF4Water-Tutorial.ipynb
• Promotion video at: https://youtu.be/An-qhB30CLU

Publications

1. Werner, M., & Li, H. (2022). AtlasHDF: An Efficient Big Data Framework for GeoAI. 1–7. https://doi.org/10.1145/3557917.3567615 [PDF] [Online] [BibTeX]

   @inproceedings{2022_atlashdf_Werner,
     author = {Werner, Martin and Li, Hao},
     title = {AtlasHDF: An Efficient Big Data Framework for GeoAI},
     year = {2022},
     isbn = {9781450395311},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3557917.3567615},
     doi = {10.1145/3557917.3567615},
     pages = {1–7},
     numpages = {7},
     keywords = {GeoAI, OpenStreetMap, big data, hierarchical data format, immediate mapping},
     location = {Seattle, Washington},
     series = {BigSpatial '22}
   }