You must have completed the data processing of TRMM data in the previous post.
TRMM
Project module
Other processing of TRMM data is done from within Karttur’s GeoImagine Framework. The project module file (projTRMM.py) is available in the project package projects.
from geoimagine.kartturmain.readXMLprocesses import ReadXMLProcesses, RunProcesses
if __name__ == "__main__":
verbose = True
projFN ='doc/TRMM/TRMM_YYYYMMDD.txt'
projFN ='doc/Layout/TRMM-layout_YYYYMMDD.txt'
procLL = ReadXMLProcesses(projFN,verbose)
RunProcesses(procLL,verbose)
Process chain
The project file links to an ASCII text file that contains a list of the xml files to execute. The layout and export processes are under the folder Layout.
projFN ='doc/Layout/TRMM-layout_YYYYMMDD.txt'
As the path to the project file does not start with a slash “\”, the path must be relative to the project module itself. The project package available on Karttur’s GitHub page contains the path and the files required for running the process chain. Both the text file and the xml files are available under the subfolder doc/TRMM.
###################################
###################################
### TRMM ###
###################################
###################################
###################################
### Layout ###
###################################
## Create scaling for TRMM data ##
#TRMM-0001_createscaling.xml
## Create the TRMM palettes ##
#TRMM-0002_createpalettes.xml
## Create legends for TRMM ##
#TRMM-0003_createlegends.xml
## Create TRMM movieclock ##
#TRMM-0004_addmovieclock.xml
## Export legend ##
#TRMM-0005_exportlegend.xml
###################################
### Export media ###
###################################
## Export shoreline vector to fit the TRMM data
#TRMM-0925_ExporttoSVG.xml
## Export png images ##
#TRMM-0900_ExporttoByte_M.xml
#TRMM-0910_ExporttoByte_timespanA_1998-2017.xml
#TRMM-0920_ExporttoByte_timespanM.xml
## Create TRMM movieframes (1998 to 2017) ##
#TRMM-0950_movieframes_M.xml
## Create movieclock, the process creates two shell scripts that must by run ##
#TRMM-0960_movieclock_M.xml
Layout
If you want to create graphical outputs you need to define layouts for the TRMM data, including for scaling the data to Byte (0-255) range and define color palettes. if you want to produce an animated movie you can also define a movie clock and timeline for each composition you want animate (you can also use the default movie clock and timeline).
Scaling
Annual precipitation across the globe varies from 0 to over 8000 mm. Linearly scaling this range into the byte range of 0-255 is not useful. Some non-linear function is required. Either you can transform the data as a pre-process, or apply a power function on the fly when exporting the data. The scaling must be preset and defined in the database prior to export. The createscaling defined in the (hidden) xml file below defines a power function for rescaling the TRMM precipitation data on the fly when exporting layout maps.
The default Karttur palette for precipitation comes in two versions, one for linear data, and one for logarithmic data. The former can, however, also be used after applying a power function as part of the layout export process as done in previous section.
Because of the relative complexity of the precipitation palettes, the default palettes for precipitation variation and error are defined separately, as is the palette for precipitation change. All the standard precipitation palettes are created from the xml below callingaddrasterpalette process:
<?xml version='1.0' encoding='utf-8'?>
<palette>
<userproj userid = 'karttur' projectid = 'karttur' tractid= 'karttur' siteid = '*' plotid = '*' system = 'system'></userproj>
<path></path>
<!-- addrasterpalette preciplinear-->
<process processid = 'addrasterpalette'>
<overwrite>Y</overwrite>
<delete>N</delete>
<parameters palette = 'preciplinear' compid='test'>
<setcolor id = '0' value = '0' red = '215' green ='76' blue='56' alpha ='0' label='0' hint='0' ></setcolor>
<setcolor id = '25' value = '25' red = '241' green ='115' blue='46' alpha ='0' label='25' hint='NA' ></setcolor>
<setcolor id = '50' value = '50' red = '255' green ='170' blue='51' alpha ='0' label='50' hint='50' ></setcolor>
<setcolor id = '75' value = '75' red = '255' green ='222' blue='66' alpha ='0' label='75' hint='NA' ></setcolor>
<setcolor id = '100' value = '100' red = '210' green ='251' blue='75' alpha ='0' label='100' hint='100' ></setcolor>
<setcolor id = '125' value = '125' red = '138' green ='253' blue='75' alpha ='0' label='125' hint='NA' ></setcolor>
<setcolor id = '150' value = '150' red = '90' green ='220' blue='90' alpha ='0' label='150' hint='150' ></setcolor>
<setcolor id = '175' value = '175' red = '30' green ='177' blue='125' alpha ='0' label='150' hint='NA' ></setcolor>
<setcolor id = '200' value = '200' red = '10' green ='120' blue='165' alpha ='0' label='175' hint='200' ></setcolor>
<setcolor id = '225' value = '225' red = '0' green ='79' blue='140' alpha ='0' label='200' hint='NA' ></setcolor>
<setcolor id = '250' value = '250' red = '0' green ='43' blue='117' alpha ='0' label='200' hint='250' ></setcolor>
<setcolor id = '253' value = '253' red = '240' green ='240' blue='132' alpha ='0' label='dry (0)' hint='NA' ></setcolor>
<setcolor id = '254' value = '254' red = '32' green ='32' blue='32' alpha ='255' label='frame' hint='frame' ></setcolor>
<setcolor id = '255' value = '255' red = '250' green ='250' blue='250' alpha ='255' label='255' hint='no data' ></setcolor>
</parameters>
</process>
<!-- addrasterpalette precipln-->
<process processid = 'addrasterpalette'>
<overwrite>Y</overwrite>
<delete>N</delete>
<parameters palette = 'precipln' compid='test'>
<setcolor id = '0' value = '0' red = '215' green ='76' blue='56' alpha ='0' label='0' hint='0' ></setcolor>
<setcolor id = '8' value = '8' red = '241' green ='115' blue='46' alpha ='0' label='8' hint='NA' ></setcolor>
<setcolor id = '13' value = '13' red = '255' green ='170' blue='51' alpha ='0' label='13' hint='NA' ></setcolor>
<setcolor id = '20' value = '20' red = '255' green ='222' blue='66' alpha ='0' label='20' hint='NA' ></setcolor>
<setcolor id = '31' value = '31' red = '210' green ='251' blue='75' alpha ='0' label='31' hint='31' ></setcolor>
<setcolor id = '49' value = '49' red = '138' green ='253' blue='75' alpha ='0' label='49' hint='NA' ></setcolor>
<setcolor id = '76' value = '76' red = '90' green ='220' blue='90' alpha ='0' label='76' hint='76' ></setcolor>
<setcolor id = '120' value = '120' red = '30' green ='177' blue='125' alpha ='0' label='120' hint='120' ></setcolor>
<setcolor id = '188' value = '188' red = '0' green ='90' blue='150' alpha ='0' label='188' hint='188' ></setcolor>
<setcolor id = '250' value = '250' red = '0' green ='43' blue='117' alpha ='0' label='250' hint='250' ></setcolor>
<setcolor id = '253' value = '253' red = '240' green ='240' blue='132' alpha ='0' label='dry (0)' hint='NA' ></setcolor>
<setcolor id = '254' value = '254' red = '32' green ='32' blue='32' alpha ='255' label='frame' hint='frame' ></setcolor>
<setcolor id = '255' value = '255' red = '250' green ='250' blue='250' alpha ='255' label='255' hint='no data' ></setcolor>
</parameters>
</process>
<!-- addrasterpalette precipdelta-->
<process processid = 'addrasterpalette'>
<overwrite>Y</overwrite>
<delete>N</delete>
<parameters palette = 'precipdelta' compid='test'>
<setcolor id = '0' value = '0' red = '215' green ='76' blue='56' alpha ='0' label='0' hint='%(d)s' ></setcolor>
<setcolor id = '63' value = '63' red = '255' green ='170' blue='51' alpha ='0' label='63' hint='%(d)s' ></setcolor>
<setcolor id = '125' value = '125' red = '224' green ='224' blue='224' alpha ='0' label='125' hint='%(d)s' ></setcolor>
<setcolor id = '187' value = '187' red = '30' green ='177' blue='125' alpha ='0' label='188' hint='%(d)s' ></setcolor>
<setcolor id = '250' value = '250' red = '0' green ='43' blue='117' alpha ='0' label='250' hint='%(d)s' ></setcolor>
<setcolor id = '253' value = '253' red = '245' green ='237' blue='182' alpha ='0' label='253' hint='NA' ></setcolor>
<setcolor id = '254' value = '254' red = '32' green ='32' blue='32' alpha ='255' label='frame' hint='NA' ></setcolor>
<setcolor id = '255' value = '255' red = '250' green ='250' blue='250' alpha ='255' label='255' hint='no data' ></setcolor>
</parameters>
</process>
<!-- addrasterpalette preciplinearstd-->
<process processid = 'addrasterpalette'>
<overwrite>Y</overwrite>
<delete>N</delete>
<parameters palette = 'preciplinearstd' compid='test'>
<setcolor id = '0' value = '0' red = '224' green ='224' blue='224' alpha ='0' label='0' hint='%(d)s' ></setcolor>
<setcolor id = '83' value = '83' red = '210' green ='251' blue='75' alpha ='0' label='83' hint='%(d)s' ></setcolor>
<setcolor id = '167' value = '167' red = '30' green ='177' blue='125' alpha ='0' label='167' hint='%(d)s' ></setcolor>
<setcolor id = '250' value = '250' red = '0' green ='43' blue='117' alpha ='0' label='250' hint='%(d)s' ></setcolor>
<setcolor id = '253' value = '253' red = '245' green ='237' blue='182' alpha ='0' label='253' hint='NA' ></setcolor>
<setcolor id = '254' value = '254' red = '32' green ='32' blue='32' alpha ='255' label='frame' hint='NA' ></setcolor>
<setcolor id = '255' value = '255' red = '250' green ='250' blue='250' alpha ='255' label='255' hint='no data' ></setcolor>
</parameters>
</process>
<!-- addrasterpalette preciplnstd-->
<process processid = 'addrasterpalette'>
<overwrite>Y</overwrite>
<delete>N</delete>
<parameters palette = 'preciplnstd' compid='test'>
<setcolor id = '0' value = '0' red = '224' green ='224' blue='224' alpha ='0' label='0' hint='%(d)s' ></setcolor>
<setcolor id = '8' value = '8' red = '241' green ='115' blue='46' alpha ='0' label='25' hint='NA' ></setcolor>
<setcolor id = '13' value = '13' red = '255' green ='170' blue='51' alpha ='0' label='50' hint='NA' ></setcolor>
<setcolor id = '20' value = '20' red = '255' green ='222' blue='66' alpha ='0' label='20' hint='NA' ></setcolor>
<setcolor id = '31' value = '31' red = '210' green ='251' blue='75' alpha ='0' label='31' hint='31' ></setcolor>
<setcolor id = '49' value = '49' red = '138' green ='253' blue='75' alpha ='0' label='49' hint='NA' ></setcolor>
<setcolor id = '76' value = '76' red = '90' green ='220' blue='90' alpha ='0' label='76' hint='76' ></setcolor>
<setcolor id = '120' value = '120' red = '30' green ='177' blue='125' alpha ='0' label='120' hint='120' ></setcolor>
<setcolor id = '188' value = '188' red = '0' green ='90' blue='150' alpha ='0' label='188' hint='188' ></setcolor>
<setcolor id = '250' value = '250' red = '0' green ='43' blue='117' alpha ='0' label='250' hint='250' ></setcolor>
<setcolor id = '253' value = '253' red = '245' green ='237' blue='182' alpha ='0' label='253' hint='NA' ></setcolor>
<setcolor id = '254' value = '254' red = '32' green ='32' blue='32' alpha ='255' label='frame' hint='NA' ></setcolor>
<setcolor id = '255' value = '255' red = '250' green ='250' blue='250' alpha ='255' label='255' hint='no data' ></setcolor>
</parameters>
</process>
</palette>
Create Legend
The process createlegend defines the legend layout for compositions. When producing the legends the colors and denominations of the actual legend items are taken from the definition of palettes and scalings respectively.
<?xml version='1.0' encoding='utf-8'?>
<palette>
<userproj userid = 'karttur' projectid = 'karttur' tractid= 'karttur' siteid = '*' plotid = '*' system = 'system'></userproj>
<!-- Create legend -->
<!-- NOTE THE ID IS ONLY USED IN THE SCRIPTING SO IT CAN HAVE (UNIQUE) DUMMY VALUES HERE -->
<!-- VWB monthly cmwater-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters columnhead='Precipitation (mm/month)' precision='0'></parameters>
<comp id = '1' source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f-m" timestep='M'></comp>
</process>
<!-- trmm-3b43v7-precip mm/month monthly data-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precip' columnhead='Rainfall (mm)' precision='0'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall" band='trmm-3b43v7-precip' suffix = "_v7"></comp>
</process>
<!-- TRMM annual precip-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipln' columnhead='Rainfall (mm)' precision='0'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" band = 'avg-trmm-3b43v7-precip' folder = "rainfall-annual-stats" suffix = "_A"></comp>
<comp id = '2' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ts-ic-trmm-3b43v7-precip' suffix = "_A"></comp>
<comp id = '3' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ols-ic-trmm-3b43v7-precip' suffix = "_A"></comp>
</process>
<!-- TRMM annual std-->
<process processid = '' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipln' columnhead='Rainfall variation (mm)' precision='0'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall-annual-stats" band = 'std-trmm-3b43v7-precip' suffix = "_A"></comp>
<comp id = '2' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ols-rmse-trmm-3b43v7-precip' suffix = "_A"></comp>
</process>
<!-- Mann Kendall-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='zhydro' columnhead='MK test (Z-value)'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'mk-z-trmm-3b43v7-precip' suffix = "_A"></comp>
</process>
<!-- slope of change-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipdelta' columnhead='Rainfall trend:(mm/yr)' precision='0'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ts-mdsl-trmm-3b43v7-precip' suffix = "_A"></comp>
<comp id = '2' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ts-losl-trmm-3b43v7-precip' suffix = "_A"></comp>
<comp id = '3' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ts-upsl-trmm-3b43v7-precip' suffix = "_A"></comp>
<comp id = '4' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ols-sl-trmm-3b43v7-precip' suffix = "_A"></comp>
</process>
<!-- r2-->
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='hydror2' columnhead='Correlation (r2)' palmax='100'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall-annual-trend" band = 'ols-r2-trmm-3b43v7-precip' suffix = "_A"></comp>
</process>
<process processid = 'createlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipdelta' columnhead='Rainfall change:(mm)' precision='0'></parameters>
<comp id = '1' source = "TRMM" product = "3B43" folder = "rainfall-change" band = 'TRMM-3B43v7-precip-delta' suffix = "_A"></comp>
</process>
</palette>
Export legend
With the process exportlegend you can create stand alone legends. To work you mtust have defined the scaling, the palette and the legend in the previous steps. The process produces two or three legends: a Scalable Vector Graphics (.svg file) a .png file, and if requested also a compressed .jpg file. The svg is saved as a www compliant xml, and you can use it either for editing in a graphics program or inclusion in a web-page (or both).
<?xml version='1.0' encoding='utf-8'?>
<exportlegend>
<userproj userid = 'karttur' projectid = 'karttur' tractid= 'karttur' siteid = '*' plotid = '*' system = 'system'></userproj>
<!-- Export legend -->
<!-- NOTE THE ID IS OLNLY USED IN THE SCRIPTING SO IT CAN HAVE (UNIQUE) DUMMY VALUES HERE -->
<!-- GRACE monthly cmwater-->
<process processid = 'exportlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipln'></parameters>
<dstpath volume='/Users/thomasgumbricht'></dstpath>
<comp id = '0' source = "trmm" product = "3b43" folder = "rainfall" band = "trmm-3b43v7-precip" prefix = "rainfall" suffix = "v7-f"></comp>
<comp id = '1' source = "trmm" product = "3b43" folder = "rainfall-A-stats" band = "avg-trmm-3b43v7-precip" prefix = "avg-trmm-3b43v7-precip" suffix = "v7-f-A"></comp>
<comp id = '2' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ols-ic-trmm-3b43v7-precip" prefix = "ols-ic-trmm-3b43v7-precip" suffix = "v7-f-A"></comp>
<comp id = '3' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ts-ic-trmm-3b43v7-precip" prefix = "ts-ic-trmm-3b43v7-precip" suffix = "v7-f-A"></comp>
</process>
<process processid = 'exportlegend' version = '1.3'>
<overwrite>True</overwrite>
<parameters palette='precipdelta'></parameters>
<dstpath volume='/Users/thomasgumbricht'></dstpath>
<comp id = 'layer4' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ols-sl-trmm-3b43v7-precip" prefix = "ols-sl-trmm-3b43v7-precip" suffix = "v7-f-A">
</comp>
<comp id = 'layer5' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ts-mdsl-trmm-3b43v7-precip" prefix = "ts-mdsl-trmm-3b43v7-precip" suffix = "v7-f-A">
</comp>
<comp id = 'layer6' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ts-losl-trmm-3b43v7-precip" prefix = "ts-losl-trmm-3b43v7-precip" suffix = "v7-f-A">
</comp>
<comp id = 'layer7' source = "trmm" product = "3b43" folder = "rainfall-A-trend" band = "ts-hisl-trmm-3b43v7-precip" prefix = "ts-hisl-trmm-3b43v7-precip" suffix = "v7-f-A">
</comp>
</process>
</exportlegend>
Export overlay
When you export the actual data as maps in the next step, you can add a vector overlay. In the Framework you can convert any vector to an svg and then include it as an overlay. Only a single svg file can be used as overlay when exporting raster data to image maps. But you can manually compose an svg from multiple svg exported vectors, e.g. using a graphics editor.
For the TRMM data I have added the global shoreline to distinguish the outline of the continents.
Having defined a palette and the scaling for the different layers, you can export the layers as color maps (colored GeoTiff images). The process for doing that is exporttobyte.
Details on defining and creating a movieclock within Karttur´s GeoImagine Framework is covered in the post on SMAP processing.
An animation requires a pre-defined movieclock (but usually the “default” movieclock is OK), and that the time series data is exported and then converted to movie frames.
A movieclock object defines the colors, types and sizes of the elements building the clock and timeline in the animation. Basically you do not need to define any of the elements, they are all defaulted. But if you want other widhts, margins, colors etc, all can be set using the process addmovieclock.
As default the script produces a shell script file that you must execute manually to produce the movie frames. The advantage with the script file is that you can edit the area of the map to appear in the movie as well as the embossed watermark text. The link to the script file is reported by the process. If you want to make the processing fully automated, you must set the parameter asscript to False.
Movie clock
The final step before producing the time series animation is to create the movie-clock that fits with your movie frames. You need to set the dimensions of the movie clock manually as parameters in the process movieclockancillary.
The process uses a combination of python image processing and ImageMagick commands.
The process also produces two shell script files. The first (in the subfolder frames) overlays the movieclock and the image frames, and the second (in the subfolder movie) produces the movie using ffmpeg. If you set the parameter asscript to True the module will execute the scripts internally.