Introduction

Online resource

The project code and links, including the xml files are available under GitHub geoimagine-project.

Create region

There is no initial default region that covers Africa south of the Sahara. To create a default region, login as superuser and define the corners of the default region in longitude and latitude. The parent region for Sub-Saharan Africa should be the African continent.

At present you have to create the default region from the setup_processes package. To access the xml file above, either create a direct link to the xml file, or add it to the existing text file linking other xml command files.

Link MODIS tiles

The xml below identifies all MODIS SIN tiles that overlap with the African Sub-Sahara region defined in the xml above. The process LinkDefaultRegionsToMODIS by default identifies all MODIS tiles for all regions. But if you set overwrite = False (the default setting) only default regions that do not have any identified MODIS tiles are tested. The source composition (<srccomp>) defined in the xml is for the shape file with the MODIS tiles as polygons. If your Framework lack the MODIS tiles, you need to setup the regions, including the MODIS tiling system as described in this post.

You can check directly after the script finishes for the MODIS tiles linked to the region africasubsahara. Open the postgreSQL Graphical User Interface (GUI) and the production database, and then type the SQL command:

SELECT * FROM modis.regions where regionid = 'africasubsahara'

If you followed followed Karttur´s installation guide for postgres, you should have a GUI for postgres installed.

Create user tract for Sub-Saharan Africa

The Framework is built in such a way that no user, including the super users, can use default regions for actual data processing. All processing can only be done vis-a-vis regions that are owned by the logged in user. Thus you have to create user regions (called “tracts”) that correspond to the default regions that you created above.

Explore MODIS Data Pool

Before downloading and processing any MODIS data you can update the Framework database on MODIS tiles available for download by running the process searchDataPool.

In the example above the MODIS data holdings are updated for 2018 and 2019. If any additional data is found, it will be under the sub-folder /Volumes/”volume”/DataPoolModis/ (where “volume” is the path given in the xml file).

Transfer MDOIS tile to local db

If you identified any additional MODIS tiles in the previous step, you need to add them to the local postgres database. The process ModisSearchToDB will do that:

Download MODIS tiles

The process for actually downloading regional MODIS tiles from the DataPool is downloadModisRegionTiles. By default the process only downloads tiles that are not registered as downloaded in the local database. If, ofr some reason, you want to download also tiles registered as already downloaded you set the parameter downloaded to True. If the tile is already in the <dstpath> the process will not download the tile (even if downloaded is set to True), unless <overwrite> is set to True.

In the above example the parameter asscript is set to True (= default, only set for clarity) and you can run the download script (reported when the script ends) from any machine. Just copy the script and change the local paths in the script. Remember that you must convert the shell script file to be executable before you can actually run it:

$ chmod 777 /path/to/MCD43A4-karttur-africasubsahara.sh

You need to make sure that your login credentials to the DataPool are correctly given on the machine you are using for the download (as explained in this post).

Register downloaded tiles

If you set the download parameter asscript to True the MODIS tiles are both downloaded and registered in the local database on the fly. By default, however, the download is done via a shell script file and you must register the downloaded tiles by rerunning the process downloadModisRegionTiles.

Alternatively you can run the process CheckMODISRegion that by default checks both for downloaded and exploded tiles. You can turn off the checking of either by setting the parameters checkdownloaded or checkexploded to False. If you set both to False the script will report an error as there is nothing to check.

While the process downloadModisRegionTiles identifies tiles (.hdf) from the database, the process downloadModisRegionTiles searches the folder tree for .hdf files. The former is process is thus faster as it i) retrieves the tiles from the database, and ii) does not have to clean the identified tiles to fit the set timestep.

Explode tile layers

The downloaded MODIS tiles come as .hdf files. The Framework can access this format, but only as a complete file (not as sequences of smaller buffers), and to processes MODIS data you need to explode (or extract) the bands (or layers) that you wish to use. Which bands to extract from each MODIS product is hardcoded in the postgres database, in the table modis.template. To see which bands to exctrat, and the default folder and layer name, use the postgreSQL GUI, e.g.:

SELECT * FROM modis.template WHERE source = 'MCD43A4v006';

The xml file command below explodes all tiles from the MODIS product MCD34A4 (version 006) over Sub-Saharan Africa fulfilling the period criteria. The default is again that the explosion is done asscript (but I also added it explicitly for clarity in the example below):

Register exploded files

If you set the parameter asscript to False, the process ExplodeMODISRegion explodes and registers the layers on the fly, and then also updates the tiles table to state that the tile is exploded (after all requested layers belonging to that tile are exploded). If you do not set the parameter asscript, or set it to False, you have to either rerun the process ExplodeMODISRegion, or run the process CheckMODISRegion.

In the latter case you should set the parameter checkdownload to False. If you leave both checkdownloaded and checkexploded to True, the process will first look for downloaded (.hdf) files, and then look for corresponding layer files (i.e. .tif files). If you set checkdownloaded to False the process will instead search for all .tif files falling inside the region and period defined. If you think that you have unregistered tiles (.hdf) as well as unregistered layers (.hdf) you should run the process twice, with checkdownloaded set to to True the first time and False the second time. It is, however, much faster to rerun the process ExplodeMODISRegion, and if your database and datafile organization is flawless, that is the preferred option.

Tile Ancillary data to Sub Saharan Africa

With the MODIS derived data organized into tiles, you need to convert other data into the same tiling system.

TRMM

The Tropical Rainfall Measurement Mission (TRMM) data is at 0.25 degrees spatial resolution and extends between 50 degrees latitudes. For this project you are going to use the monthly TRMM product that is created using a range of satellite data and supported by ground observed preciptation (product 3B43). The temporal availability of the TRMM 3B43 is from to November 2018.

In a previous post on TRMM the data is both cleaned, filled and masked to include only land. The xml file below tiles the monthly data to all MODIS tiles covered by the region “karttur-africasubsahara”. The spatial resolution of the destinations tiles equals the resolutions of the MODIS product MCD43A4 (463 m)

With the parameter asscript set to True you need to run the shell script fine reported when the process ends. And then rerun the same xml command to register the tiles in the database.

The spatial resolution of the TRMM tiles created from the xml above (463) is artificially high. It will only be used for cell by cell correlation with soil moisture derived from the MODIS product MCD43A4. The original TRMM data is in 0.25 degrees spatial resolution, which approximately translates to 28 km. The xml below creates the same tiles as the xml above, but at a spatial resolution of 27798.78 m. This odd number translates into exactly 40 columns and 40 rows for each MODIS tile. When you later create a mosaic and projects that mosaic back to Geographic coordinates you will see that the data exactly fits the original TRMM.

To compare TRMM with GRACE data wee need to create a third version of the TRMM tiles, in approximately 1 degree resolutions. That translates to a spatial resolution of 111195.12 (or 10 rows and 10 lines for each modis tile). As this tiling reduces the spatial resolution, you should set the parameter resample to average.

You can run all the three tiling processes together, but you must then set the parameter asscript to False and run all tiling on the fly.

One of the ideas with the Framework is that processing should not be redundant. For Ancillary time-series data all time-series processing should be done using the original (system:ancillary) rather than the tiled (system:modis or mgrs) data. The TRMM time series analysis should hence be done in system = ancillary and any statistical, trend or model results transferred using tiling (rather than processing at the tile level). The xml below tiles the TRMM time-series data to all tiles included in the default region “africasubsahara” (the region on which the tract “karttur-africasubsahara” is built).

SNULLE

GRACE

GRACE animation with TRMM overlay

Assume that the GRACE estimated mm water pillar is primarily related to local rainfall. We are going to test that assumption by cross-correlating the local TRMM rainfall with the local variation in GRACE mm water pillar. But before we do the statistical cross correlation we can create an animation that show both time series in the same movie frame. The basic concept for how to create such an animation is found in the blog post Animation with inset map.

Inthe Framework, the process movieoverlayframeModisRegion creates the map parts of the movie frames, with one base maps and one overlay map. The xml below creates a map with the GRACE mm water pillar as the base map and the TRMM rainfall as the overlay. Both the GRACE and TRMM data must be mosaicked and exported as shown above.

<?xml version='1.0' encoding='utf-8'?>
<africasubsaharaproject>
	<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
	<period startyear = "2002" startmonth='4' startday='23' endyear = "2016" endmonth='12' endday='31' timestep='M'></period>

	<!-- Create movie frame -->
	<process processid = 'movieoverlayframeModisRegion' version = '1.3'>
		<overwrite>False</overwrite>
		<parameters name = 'gracetrmm' basewidth = '800' basecrop='800,532,0,0'
		overlaywidth = '300' gravity='southwest' geomx='0' geomy='100'
		emboss='KARTTUR' embossdims='720,150' embossptsize='100' asscript='True'></parameters>
		<srcpath volume = "karttur3tb" hdrfiletype = 'tif' datfiletype = 'tif'></srcpath>
		<dstpath volume = "karttur3tb/movieclockoverlay" hdrfiletype = 'png' datfiletype = 'png'></dstpath>
		<srccomp>
			<grace-ave id = 'base' source = "NASA-GRACE" product = "cmwater" folder = "cmwater" band = "grace-ave" prefix = "grace-ave" suffix = "RL05-f-1deg">
			</grace-ave>

			<trmm-3b43v7-precip id ='over' source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m-30km">
			</trmm-3b43v7-precip>
		</srccomp>
	</process>
</africasubsaharaproject>

The movie clock is created in the same way as above.

<?xml version='1.0' encoding='utf-8'?>
<africasubsaharaproject>
	<userproj userid = 'karttur' projectid = 'karttur-africasubsahara' tractid= 'karttur-africasubsahara' siteid = '*' plotid = '*' system = 'modis'></userproj>
	<period startyear = "2002" startmonth='4' startday='23' endyear = "2016" endmonth='12' endday='31' timestep='M'></period>

	<!-- Create movie frame and movie -->
	<process processid = 'movieclockModisRegionToRegion' version = '1.3'>
		<overwrite>False</overwrite>
		<parameters name = 'default' width = '800' asscript='True'></parameters>
		<dstpath volume = "karttur3tb/movieclockoverlay" hdrfiletype = 'png' datfiletype = 'png'></dstpath>
		<dstcomp>
			<grace-ave id = 'base' source = "NASA-GRACE" product = "cmwater" folder = "cmwater" band = "grace-ave" prefix = "grace-ave" suffix = "RL05-f-1deg">
			</grace-ave>
		</dstcomp>
	</process>
</africasubsaharaproject>

BALL

The process movieclockModisRegionToRegion produces the shell scripts needed for assembling the final frames and for converting them to a movie.