Introduction
This post illustrates how to search, download and organize remote sensing data from the Moderate Resolution Imaging Spectroradiometer (MODIS) sensor using Karttur’s GeoImagine Framework.
Prerequisites
You must have the complete SPIDE installed as described in the post Install and setup spatial data IDE. You must have setup Karttur’s GeoImagine Framework, either by importing or by copying (drag and drop). The Framework postgres database must be setup and the processes defined. The Framework MODIS python package must be installed, and the MODIS tiling system added to the postgres database.
MODIS
The Moderate Resolution Imaging Spectroradiometer (MODIS) is an optical and thermal sensor carried on two satellites, Terra (launched in 1999) and Aqua (launched in 2002). The MODIS sensor has a spectral resolution of 36 bands and a wide swath that allows daily imaging of the entire globe. The spatial resolution varies from 250 m (or 232 to be exact) for red and near infra-red, 500 m for five (5) additional optical wavelengths designated for land observation, and 1 km for 29 bands mainly intended for atmospheric and oceanic observation, including thermal emissivity. The MODIS raw data is processed into a series of products, and both the raw data and the products are freely available. The products are stored in a wall-to-wall defined tile system using a sinusoidal projection. Each tile is equally large and the Earth is divided into 36 horizontal and 18 vertical tiles.
In this post you will use the reflectance corrected product MCD43A4 (version 006) that combine observations from both MODIS instruments into 16-day (16D) composites. The 16D composites are available for any consecutive days, but here we will use every 8th or 16th day. This means that we will use Day Of Year (DOY) 1, 9, 17 … etc, where the day number refers to the central (9th) day of each 16D period (note that if you use MODIS product version 005 or earlier, the day number refers to the first day of the 16D period).
Python Package
The GeoImagine Framework includes a package for specific MODIS processing: geoimagine-modis. However, also several other packages in the Framework are needed for repeating the steps below.
Project module
The project module file (projMODIS.py) is available in the Project package projects.
Process chain
The project file links to an ASCII text file that contains a list of the xml files to execute.
projFN ='doc/MODIS/modis_YYYYMMDD.txt'
As the path to the project file does not start with a slash “/”, the path is relative to the project module itself. The project package available on Karttur’s GitHub page contains the path and the files required for running the process chain. Both the text file and the xml files are available under the subfolder doc/MODIS.
Search and download
The MODIS data is freely available from the Land Processes Distributed Active Archive Center (LP DAAC) Data Pool. Users are required to log in with their Earthdata login credentials to obtain data. Once you have registered for an Earthdata account, you can use the Data Pool to search and download MODIS data, as well as other data. Karttur’s GeoImagine Framework have special processes for bulk downloading and managing MODIS data.
Searching the MODIS online repository
The way Karttur´s GeoImagine Framework is organized, you first have to search the online repository for available MODIS tiles, then register the search results in the Framework postgres database. Once the data is registered you can download and extract the actual MODIS data.
Search
I have tried to find some library or database that lists the data available in the Data Pool, but have failed to find any. Instead I created a solution where I use wget (“web get”) for downloading an html coded list of available data. If you do not have wget, this post includes the installation instructions
The Framework process for searching the online repository for MODIS data using wget is searchdatapool. As the MODIS tiles are stored per date in the Data Pool that is how the data search is done. This means that the globally available tiles are always captured for each product and date that are included in the search.
Before running the process searchdatapool you must have the credentials for accessing the Data Pool in a .netrc file, with the username corresponding to the one given in the xml file (‘YourEarthDataUser’). How to use .netrc for handling connection is detailed in this post.
The process will drill into the Data Pool and load all the available data as html coded files. By default the process saves all the html files under the path ../modis/source/yyyy.mm.dd/ under the volume identified in the xml file. The files are ordinary html files, but with the .html extension omitted.
Transfer search to database
To transfer the search results to the GeoImagine Framework database you must run the process ModisSearchToDB. This process reads the html files, extracts the required information and inserts the information in the database. When finished it moves the html file to a sub-folder called done. If, for some reason, you delete your database all you need to do is to take all the html files under the done sub-folder and move them one level up and then re-run ModisSearchToDB.
Downloading MODIS data
With the available MODIS Data Pool holdings registered in the database you can download any of the registered data using either downloadModisRegion or downloadModisSingleTile.
When downloading you can either download the files on the fly, or write the download commands to a shell script file. The latter is the default. To change it you need to set the parameter asscript to False. The example below downloads single tiles (h20v10).
If you did not add the parameter asscript, including setting it to False, the process produces a script file that you must run manually. You can also copy the script to another machine (with better internet connection) and run the script from there. The machine you run from must have a .netrc file with your EarthData credentials. And the volume indicated in the xml must either exists on the machine from which you download, or you need to edit the script to reflect a volume that is available on the machine from which you download. To run the shell script you must first make it executable, and then execute it:
$ chmod 777 /path/to/script.sh
$ /path/to/script.sh
If you choose to use batch scripting you have to run a second process that checks the exploded files and transfers the results to the Framework database. You can either re-run the process that you used for generating the shell script file, making sure the overwrite option is set to False (default). Alternatively you can run the corresponding special process CheckMODISRegion or CheckMODISSingleTile.
Check MODIS tiles
If you have used shell script batch processing for either downloading (pervious section) or extracting (next section) MODIS data, you can run one of the two versions of the CheckMODIS process to update the database. CheckMODIS will check the consistency of your local MODIS files and the database, and harmonise the database with the local data holdings.
Extracting MODIS layers
The layers included in each MODIS HDF file, as well as metadata, can be accessed using the GDAL command gdalinfo. (GDAL is part of the GeoImainge Framework, and must be installed in order to extract the MODIS data.)
gdalinfo path/to/modis/modisfile.HDF
In Karttur’s GeoImagine Framework the layers to extract have to be defined in the database, in the table templates under the modis schema. The template table also define the celltype, cellnull, projection and folder where to store the extracted layer. At present the modis.templates table is pre-defined and to change it you either need to change the xml file defining the table, or manually change the table using postgres SQL (this post shows how to update the template table for SMAP data).
In this post you are going to explode the combined MODIS reflectance product, MCD43A4, version 006. By default all the seven bands included in the product are extracted (nir = near infra-red, mir = mid infra-red):
| Band | Wavelenght (nm) | Name | code |
|---|---|---|---|
| 1 | 620–670 | red | rl |
| 2 | 841–876 | nir | na |
| 3 | 459–479 | blue | bl |
| 4 | 545–565 | green | gl |
| 5 | 1230–1250 | mir A | ma |
| 6 | 1628–1652 | mir B | mb |
| 7 | 2105–2155 | mir C | mc |
To extract the MODIS layers from the HDF files, either run ExplodeMODISRegion or ExplodeMODISSingleTile.
By default these processes generate shell script files that must be manually executed for extracting the requested layers. As explained above, if you choose to use batch scripting you have to run a second process that checks the exploded files and transfers the results to the Framework database. You can either re-run the process that you used for generating the shell script file, making sure the overwrite option is set to False (default). Alternatively you can run the corresponding special process CheckMODISRegion or CheckMODISSingleTile (see above).