This post outlines a process chain for downloading and organizing Moderate Resolution Imaging Spectroradiometer (MODIS) reflectance data (MODIS product MCD43A4) for a region covering Sub Saharan Africa using Karttur´s GeoImagine Framework. The processing of the MODIS data is covered in the next post, and then follows posts with the results.

Prerequisites

To follow the process steps in this post you must set up a Spatial Data Integrated Development Environment (SPIDE) and download and install Karttur’s GeoImagine Framework from Github.

The GeoImagine blog contains a post on MODIS processing that is more detailed compared to this post.

MODIS

MODIS is an optical and thermal sensor carried on two satellites, Terra (launched in 1999) and Aqua (launched in 2002). In this project you are going to use MODIS product MCD43A4 for estimating soil moisture dynamics at 465 m spatial resolution and 16 day intervals.

Process chain

The principal steps for downloading and organizing MODIS data using Karttur’s GeoImagine Framework include:

Search online library for available tiles
Register available tiles
Download selected tiles
Explode selected bands

In the Framework a process chain can be built as a series of calls to xml coded instructions. This section contains the calls and the remaining parts of the post details each called xml.

Search online library

The way Karttur´s GeoImagine Framework is organized, you first have to search the online repository for available MODIS tiles, then register the search results in the Framework postgres database. Once the data is registered you can download and extract the actual MODIS data.

The MODIS online library is hosted by the Land Processes Distributed Active Archive Center (LP DAAC) Data Pool. Users are required to log in with their Earthdata login credentials to obtain data. The default solution within the Framework is to register your credentials in a .netrc file as described in this post.

The Framework solution for finding available tiles is to trawl the LP DDAC Data Pool using the commandline tool wget (“web get”); if you need to install wget, please consult this post. The process for searching the Data Pool holdings of MODIS products is searchdatapool.


  
<?xml version='1.0' encoding='utf-8'?>
<africasubsahara>

<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
<period startyear = "1998" endyear = "2018" endmonth ='7' endday = '31' timestep='M'></period>

<!-- tile the original (monthly) TRMM data to the region (karttur-africasubsahara).
The TRMM data must be downloaded and organized
If you set the parameter "asscript" to True (= default),
you have to execute the shell script file as reported by the process,
and then rerun the xml with overwrite set to False to add the layers to the database
-->

<!-- Tile the TRMM data to the MODIS tile standard spatial resolution (463 m) -->
<processx processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '463.313'  yres = '463.313' resample='near' asscript='False'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</processx>

<!-- Tile the same data, but to a spatial resolution of 27798.78 m -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '27798.78'  yres = '27798.78' resample='near' asscript='False' suffix = 'v7-f-m-30km'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

<!-- Tile the same data, but to a , but to a spatial resolution of approx 1 deg (111195.12)  -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '111195.12'  yres = '111195.12' resample='average' asscript='False' suffix = 'v7-f-m-1deg'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

</africasubsahara>

Register available tiles

To transfer or register the search results to the GeoImagine Framework database you must run the process ModisSearchToDB. This process reads the html files created by wget, extracts the required information and inserts the information in the database.


  
<?xml version='1.0' encoding='utf-8'?>
<africasubsahara>

<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
<period startyear = "1998" endyear = "2018" endmonth ='7' endday = '31' timestep='M'></period>

<!-- tile the original (monthly) TRMM data to the region (karttur-africasubsahara).
The TRMM data must be downloaded and organized
If you set the parameter "asscript" to True (= default),
you have to execute the shell script file as reported by the process,
and then rerun the xml with overwrite set to False to add the layers to the database
-->

<!-- Tile the TRMM data to the MODIS tile standard spatial resolution (463 m) -->
<processx processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '463.313'  yres = '463.313' resample='near' asscript='False'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</processx>

<!-- Tile the same data, but to a spatial resolution of 27798.78 m -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '27798.78'  yres = '27798.78' resample='near' asscript='False' suffix = 'v7-f-m-30km'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

<!-- Tile the same data, but to a , but to a spatial resolution of approx 1 deg (111195.12)  -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '111195.12'  yres = '111195.12' resample='average' asscript='False' suffix = 'v7-f-m-1deg'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

</africasubsahara>

Download selected tiles

With the available MODIS Data Pool holdings registered in the database you can download any of the registered data using downloadModisRegion. When running the process you can either download the files on the fly, or write the download commands to a shell script file. The latter is the default. To change it you need to set the parameter asscript to False.


  
<?xml version='1.0' encoding='utf-8'?>
<africasubsahara>

<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
<period startyear = "1998" endyear = "2018" endmonth ='7' endday = '31' timestep='M'></period>

<!-- tile the original (monthly) TRMM data to the region (karttur-africasubsahara).
The TRMM data must be downloaded and organized
If you set the parameter "asscript" to True (= default),
you have to execute the shell script file as reported by the process,
and then rerun the xml with overwrite set to False to add the layers to the database
-->

<!-- Tile the TRMM data to the MODIS tile standard spatial resolution (463 m) -->
<processx processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '463.313'  yres = '463.313' resample='near' asscript='False'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</processx>

<!-- Tile the same data, but to a spatial resolution of 27798.78 m -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '27798.78'  yres = '27798.78' resample='near' asscript='False' suffix = 'v7-f-m-30km'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

<!-- Tile the same data, but to a , but to a spatial resolution of approx 1 deg (111195.12)  -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '111195.12'  yres = '111195.12' resample='average' asscript='False' suffix = 'v7-f-m-1deg'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m">
</trmm-3b43v7-precip>
</srccomp>
</process>

</africasubsahara>

If you did not add the parameter asscript, including setting it to False, the process produces a script file that you must run manually. To run the shell script you must first make it executable, and then execute it:

$ chmod 777 /path/to/script.sh

$ /path/to/script.sh

If you choose to use batch scripting you have to re-run the same downloadModisRegion process to identify the downloaded tiles. Alternatively you can instead run the process CheckMODISRegion but it does a more thorough database control and takes longer time.

Explode selected bands

Each MODIS product contains several bands; which bands to explode are defined in the postgres table modis.template. When you installed Karttur’s GeoImagine Framework some default bands were added to the table (see the GeoImagine MODIS post). The default for the product used here (MCD43A4) is to extract seven (7) spectral bands, which are the bands we are interested in. To eplode the MODIS hdf file, use the process ExplodeMODISRegion.


  
<?xml version='1.0' encoding='utf-8'?>
<africasubsahara>

<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
<period startyear = "1998" endyear = "2017"  timestep='A'></period>

<!-- tile the original (annaul) TRMM data to the region (karttur-africasubsahara).
The TRMM data must be downloaded and organized
If you set the parameter "asscript" to True (= default),
you have to execute the shell script file as reported by the process,
and then rerun the xml with overwrite set to False to add the layers to the database
-->

<!-- Tile the TRMM data to the MODIS tile standard spatial resolution (463 m) -->
<processx processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '463.313'  yres = '463.313' resample='near' asscript='False'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall-A" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m-A">
</trmm-3b43v7-precip>
</srccomp>
</processx>

<!-- Tile the same data, but to a spatial resolution of 27798.78 m -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '27798.78'  yres = '27798.78' resample='near' asscript='False' suffix = 'v7-f-m-A-30km'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall-A" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m-A">
</trmm-3b43v7-precip>
</srccomp>
</process>

<!-- Tile the same data, but to a , but to a spatial resolution of approx 1 deg (111195.12)  -->
<process processid = 'tileRegionToModisAncillary' version = '1.3'>
<overwrite>False</overwrite>
<parameters src_defregid = 'trmm' epsg = '6842'  xres = '111195.12'  yres = '111195.12' resample='average' asscript='False' suffix = 'v7-f-m-A-1deg'></parameters>
<srcpath volume = "travel"></srcpath>
<dstpath volume = "travel"></dstpath>
<srccomp>
<trmm-3b43v7-precip source = "trmm" product = "3b43" folder = "rainfall-A" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m-A">
</trmm-3b43v7-precip>
</srccomp>
</process>

</africasubsahara>

Next steps

If you followed the tutorial you should now have a complete set of spectral images covering your region (Sub-Saharan Africa) for the defined period. The Next step is to process the spectral bands and derive estimates of soil moisture.