This toolset automates downloads of global, multi-year Moderate Resolution Imaging Spectroradiometer (MODIS) and related data necessary for performing net radiation and evapotranspiration (Rn/ET) modeling, and provides an application programming interface (API) for simple and reliable access of these data from Python or MATLAB applications. Several useful utilities for validating and curating/indexing the downloaded data are also included.
The MODIS and National Centers for Environmental Prediction (NCEP) datasets used by Rn/ET have been sourced from several providers, including the U.S. Geological Survey (USGS), Lawrence Berkeley Lab (LBL), and the National Oceanic and Atmospheric Administration’s Earth System Research Laboratory (ESRL).
Given the number of data providers, data sizes, formats, and ranges, the software provides necessary functionality in three key areas: data acquisition, data validation, and data access. The process of downloading full, multi-year collections of MODIS and NCEP data often takes days or weeks at a time, and is complicated by the high probability of a server or data connection failure at some point during the process. It is essential, then, that the procedures be highly automatic and that, should a failure (invariably) occur, restarts be made as straightforward and complete as possible.
Even after a successful download, it is highly likely that some files will have been truncated or skipped altogether. If not detected in advance, such data gaps can lead to runtime application failure or implausible analysis results. Auto matic validation of the downloaded data is essential and, in the case of the software described here, is based on log file scans for typical signatures of success or failure, and checksum tests for file completeness.
Once the downloaded data are complete and validated, data access at the application program level is generally still complicated by the non-uniform file and directory organization schemas adopted by the various providers. Rather than attempting to enforce consistency by renaming files and directories (thus introducing a host of other data curation and administrative issues), the toolset described here implements a simple Python-based indexing layer and functional interface that, together, enable uniform access to the underlying directories and geospatial data files.