Average rainfall 2001-2016, global tropics

Map: Average rainfall 2001-2016, global tropics

Process chain for Copernicus DEM

Introduction

Processing a global DEM all the way form downloading tiles to create coherent indices and metrics of landscapes, is a typical task for which Karttur’s GeoImagine Framework was built. Once the DEM is imported and organized within the Framework, it is easy to test different algorithms and visualizations for DEM. If you have access to some ground data you can also apply the Framework for comparing the results of different algorithms and parameters against ground data, and thus select the most appropriate landscape model.

Prerequisits

You must have installed Karttur’s GeoImagine Framework, and registered a user with a project that covers the geographic regions of the DEM you want to process.

Overview

This post summarises the processes chain for DEM data in Karttur’s GeoImagine Framework. Each step in the process chain is explained in more detail in linked posts.

A note on hydrological corrections

Before running the DEM processing you need to decide how you want to manage hydrological errors in your DEM. If there are no errors, then you do not need to worry, but usually there are at least both pits and flat area. Both of these problems primarily affects the extraction of river basins. Different algorithms handle these problems differently. The simplest solution is to fill up all pits completely, with or without tilting the flow across flat surfaces. This, however, causes the normally known relative DEM accuracy to be replaced by an unknown error that might, or might not, have spatial or contextual biases. If the DEM is going to be solely used for flow routing this is perhaps the best method. The alterantive is to to only fill up minor errors and errors that can be logically identied. The latter category would then include depressed channels and negative elevations along shorelines - and this is how it is done in this manual. But then you must also apply a flow routing algorithm that can pass depressions on the fly. The GRASS GIS module r.watershed does just that. And r.watershed is used for the DEM modelling in this post. To fill up all depressions, you can instead use the GRASS GIS module r.terraflow, or any of the DEM filling algorithms of SAGA GIS. If you apply hydrological DEM corrections, you must also decide which DEM aleternative (corrected or non-corrected) to use for ectracting other indexes, like slope, curvature etc.

Process chain

The process chain is available at Kartturs GitHub account and also found under the Hide/show button below. The process chain is an ordinary text file linking to a series of json files. Empty rows and rows starting with a “#” are ignored. When called from karttur’s GeoImagine Framework, the processes of each json file are sequentially run. The structure of the json files and the commands are summarised in the post [#]#().

Layout

To produce symbolised and labeled map using the Framework, you have to define some Layout features and parameters. All map layout is done using unsigned byte range maps with values ranging from 1 to 255, Where values 251 to 255 are reserved for specific purposes and not allowed for representing map values. To produce a symbolised map using the Framework you need to define the scaling and the palette.

Layout processes include:

  • CreateScaling,
  • AddRasterPalette,
  • CreateLegend, and
  • ExportLegend.

CreateScaling

json/0001_CreatesScaling_DEM_v090.json

The CreateScaling process converts any input raster into an unsigned byte (0-255) range file. The byte file is then used for assigning a color ramp (defined by the process AddRasterPalette, see next section).

Scaling is thus not defined on the fly, instead each map composition (see the post on ??? for an explanation on the Framework map composition concept) must be associated with a predefined scaling. Once defined, this scaling can not be changed.

AddRasterPalette

json/0002_AddRasterPalette_DEM_v090.json

In contrast to the scaling, the palette can be freely set for any map composition (but using the same scaling). The palette to use for any map layout is defined when calling the process that generates the layout.

CreateLegends

0003_createlegends_DEM_v090.json

Legend are defined using the process CreateLegends. While most entries can be set by their default values, a legend can also be complex to define.

ExportLegends

0005_exportlegend_DEM_v090.json

The process ExportLegends export predefined legends for particular compositions and raster palettes.

DownloadCopernicus

json/0110_DownloadCopernicus_CopDEM-90m.json

Also downloading the Copernicus DEM 90 m version is defined with a customized process, DownloadCopernicus.

UnZipRawData

json/0120_UnZipRawData_CopDEM-90m.json

Downloaded data that is retrieved as zip files can be unzipped with the command UnZipRawData.

MosaicAncillary

json/0125_MosaicAncillary_CopDEM-90m.json

Downloaded data that are delivered as tiles (except for MODIS and Sentinel data where the original tiling systems have been adopted by the Framework) need to be mosaicked together before importing to the framework.

OrganizeAncillary

json/0160_OrganizeAncillary_CopDEM-90m.json

With the Copernicus data downloaded, unzipped and mosaicked, the data can be organized for (or imported to) the Framework. The the Copernicus DEM data, the process OrganizeAncillary as defined in the json object below only imports the virtual mosaic linked to the orignal (downloaded and unzipped) tiles.

TileAncillaryRegion

json/0180_TileAncillaryRegion_CopDem-90m.json

All basic processing in the Framework is done using predefined tiling systems. You must chose the appropriate system (e.g. ease2n for the Copernicus DEM data defining the river basins feeding into the Arctic Ocean) and then TileAncillaryRegion.

MosaicAdjacentTiles

0190_MosaicAdjacentTiles_CopDEM-90m

Spatial processing tile by tile causes edge effects for kernel and regional based processes. To overcome that the Framework can use virtually expanded tiles as input in some processes. The process MosaicAdjacentTiles creates the expanded virtual tiles. Note that the vritual datasets created using MosaicAdjacentTiles depend on a rigid and unchanged file structure. Moving the datasets to another disk (volume) collapses the references to the orginal datasets.

Hydrological DEM corrections

Hydrological corrections of the Copernicus DEM is done in several steps:

  • Filling/flattening of single pits/peaks adjacent to streams
  • Adjacent tile mosaic the single cell corrected DEM
  • Filling of stream associated pits
  • Adjacent tile mosaic dual corrected DEM

Filling/flattening of single pits/peaks adjacent to streams

0230_GrassDemFillDirTiles_CopDEM_90m.json

The first hydrological correction is the filling up of all single cell pits and the flattening of all single cell peaks adjacent to streams. The Framework process is GrassDemFillDirTiles, which is a wrapper for the GRASS GIS module r.fill.dir.

Adjacent tile mosaic the single cell corrected DEM

0191_MosaicAdjacentTiles_CopDEM-90m.json

Create expanded virtual tiles of the single cell corrected DEM tiles for use in the next step.

Fill stream associated pits

0240_GrassDemHydroDemTiles_CopDEM-90m.json

The second hydrological correction is the filling up of larger pits in streams. The Framework process is GrassDemHydroDemTiles, which is a wrapper for the GRASS GIS addon module r.hydrodem.

Adjacent tile mosaic the dual corrected DEM

json/0192_MosaicAdjacentTiles_CopDEM-90m.json

Create expanded virtual tiles of the corrected DEM tiles for use in the next step.

River mouth and shoreline corrections

0210_GrassOnetoManyTiles-correct-shoreline_CopDEM-90m.json

The river mouth and shoreline correction processing fills up all negative pits adjacent to the sea. No special process is used, instead the generic Framework wrapper for GRASS GIS is used by defining a sequence of GRASS commands under the process GrassOnetoManyTiles.

GRASS script

The above process generates a script file that loops over all the tiles belonging to the defined region. An example is shown under the Hide/show button below, the commented lines (starting with “#”) have been added for explanation.

Kernel DEM derivates

With Karttur’s GeoImagine Framework, kernel (filter) indexes derived from DEM data can be calculated using GDAL, GRASS GIS, SAGA GIS or using internal processes.

GDAL DEM derivatives

0301_GdalDemTiles_CopDEM-90m.json

The Framework process GdalDemTiles is a wrapper for running the Geographic Data Abstraction Library (GDAL) DEM (gdaldem) program.

Framework DEM derivatives (NumpyDemTiles)

0303_NumpyDemTiles_CopDEM-90m.json

The Framework process for internal DEM derivates is focusing of calculating Terrain Position Index [TPI] landforms and can thus only retrieve the derivates for slope and TPI plus the landform categogization. The TPI extraction and thus the landform categorisation is, however, more advanced (thetorically more correct) compared to most other solutions.

GRASS DEM derivatives (GrassOnetoManyTiles)

0305_GrassOnetoManyTiles-DEM-derivates-3+9cell_CopDEM-90m.json

GRASS GIS has more options for retrieving DEM derivates than either GDAL or the internal Framework process. GRASS modules for retrieving DEM derivatives include: r.slope.aspect, r.params.scale, r.tpi, r.tri and r.geomorphon. Retrieving DEM derivates using GRASS GIS though the Framework is done through the standard GRASS wrapper GrassOnetoManyTiles.

Hillslope DEM derivates

Hillslope indexes dereived from DEM data are calcualted using either GRASS GIS or SAGA GIS.

GRASS GIS hillslope DEM derivates

0311_GrassOnetoManyTiles_hillslope-derivates_copDEM-90m.json

Hillslope DEM indexes can be created by the general GRASS GIS wrapper GrassOnetoManyTiles.

GRASS script

The above process generates a script file that loops over all the tiles belonging to the defined region. An example is shown under the Hide/show button below, the commented lines (starting with “#”) have been added for explanation.

DEM basin extraction

0321_GrassOnetoManyTilesCopDEM-basin-extract-stage2_copDEM.json

One (additional) problem with extrating river basins is the definition of outlets points, a problem unrelated to the problem of hydrological flow routing. This particualr problem is, however, of relevance only if you intend to use the basin dat for hydrolocail modeling. If you only intend to use the basin map for extracting spatial statistics, you do not need an un-ambiguosly identifed outlet point. This part is for creating a distinct, single cell, outlet point for basins. It maske use of GRASS GIS and the Framework wrapper GrassOnetoManyTiles.

GRASS script

The above process generates a script file that loops over all the tiles belonging to the defined region. An example is shown under the Hide/show button below, the commented lines (starting with “#”) have been added for explanation.