This post is a manual for how to use Karttur’s GeoImagine Framework for searching, downloading and checking the Copernicus DEM (CopDEM) 90 m version freely available at gisco-services. Note that at time of writing this manual (January 2022), the gisco-services version of CopDEM 90 m has a data gap between -180 and -179 degrees latitudes. Thus, if you need a truly global version of the CopDEM 90 m, you should use the processing chain in the post using the AWS version of the CopDEM 90 m.
Introduction
Searching, downloading, unzipping and checking tiled datasets available online, is a typical task for which Karttur’s GeoImagine Framework was built.
Prerequisits
You must have installed Karttur’s GeoImagine Framework.
Process chain
The process chain is available at Kartturs GitHub account and also found under the button below. The process chain is an ordinary text file linking to a series of json files. Empty rows and rows starting with a “#” are ignored. When called from karttur’s GeoImagine Framework, the processes of each json file are sequentially run. In the process chain below you can click on each json file name to see the complete content. The following sections also explain in more detail the processes of each jscon command file.
Search and download
Search and download differ for almost all datasets. Details for searching and different DEMs is in the post Digital Elevation Models (DEMs).
Search
0100_SearchCopernicusProducts_CopDEM-90m.json
The search for the Copernicus DEM 90 m product is done using the special process SearchCopernicusProducts. The search requires that you installed wget, see the post on Python virtual environment, that also covers installing wget.
The process SearchCopernicusProducts results in an html file with url links to all the resources to download. For the process defined in 0100_SearchCopernicusProducts_CopDEM-90m.json, the result is stored in the file copernicusDEM90.html
Download
0110_DownloadCopernicus_CopDEM-90m.json
Also downloading the Copernicus DEM 90 m version is defined with a customized process, DownloadCopernicus. This process reads the html file above (copernicusDEM90.html), extracts the url tag (<a>), looks for a local copy of the DEM tile, and if not found (or overwrite set to true) downloads the tile. The downloading is predefined to a path on the destination disk and can not be changed. If you want to store that data somewhere else you have to move them manually, but the system will hten lose track of them. The original tiles are not registered in the database. It is thus recommended that you leave the original data in the predefined location (for the data used here the default path is /DISKPATH/DOWNLOADS/CopernicusDem90).
Unzip
0120_UnZipRawData_CopDEM-90m.json
Downloaded data that is retrieved as zip files can be unzipped with the command UnZipRawData. This is a generic process, and you need to give the full path to both the source (zip) and target (expanded) datasets. The disk (volume) path is as usual given as the parameters srcpath and dstpath, but additioally you must also give the directories on the disks for both srcdir and dstdir under parameters. If srcdir is a file, the script assumes that it is a list of the files to unzip; if srcdir is a directory the script will explode all zip files in that directory.
For the example here I created a file listing all the zip files to explode (CopernicusDem90_rawtiles.csv). In MacOS or Linux you create the csv list by piping the ls command to a file:
$ ls *.zip > CopernicusDem90_rawtiles.csv
Bounding Boxes for raw tiles
The process BBoxTiledRawData creates bounding boxes for each raw tile. In the example used here, each tile to include is listed in a csv file, created in a similar way as the list of zip files in the provious section.
The process generates a single vector data source with all the included tiles.
As clearly seen in the figure above, the raw tiles are of arbitrary sizes with geographical coverage restricted to land. The next steps in the process chain converts these more arbitrary tiles to regular grids fitted to the projection system where you want to use the DEM.
A closer scrutiny of the north-western corner of the map reveals that the tiles do not cover the entire western hemisphere, but leave a gap west of -179 degrees longitude. The map to the left shows the part of Russia between -180 and -170 degrees longitude and how the most western parts are not covered by the tiles of the CopDEM 90 m version available from gisco-services.
Mosaic raw tiles
0125_MosaicAncillary_CopDEM-90m.json
Downloaded data that are delivered as arbitrary tiles need to be mosaicked together before importing to the framework. (SMAP, MODIS and Sentinel data do not need mosaicking of raw tiles as these datasets either are downloaded in predefined tiling systems or as complete global datasets). The mosaicking must be done as virtual (.vrt) dataset. To actually create a true global mosaic at e.g 90 m, as in this example, would take a long time and also take up unnecessary much disk space. The raw tile mosaic is only going to be used as an intermediate for creating the projection system predefined tiles that will later be used for all further analysis. The process MosaicAncillary does not import (organize in the terminology of the Framework) into the database, that is done in the next step.
Organzie
0125_MosaicAncillary_CopDEM-30m.json
With the CopDEM 90 m data downloaded, unzipped and mosaicked, the data can be organised for (or imported to) the Framework. For the Copernicus DEM data, the process OrganizeAncillary as defined in the json object below only imports the virtual mosaic linked to the original (downloaded and unzipped) tiles.
Once you have run the json command, you can check in the Framework database, and you should find that the ancillary schema contains a layer with the compid dem_copdem.
Next step
The next step is tile the imported CopDEM to fit the projection system that you want to use for working with the data. The processing chain starting with the tiling, is, however, the same regardless of whihc version of CopDEM you choose to download, and is covered in the next instruction post.