Average rainfall 2001-2016, global tropics

Map: Average rainfall 2001-2016, global tropics

Python virtual environment

Introduction

This post is an extended version of the post on Conda virtual environments in the blog on setting up the Spatial Data Integrated Development Environment (SPIDE).

With Conda, you can create, update, export and import virtual Python environments that have different versions of Python and/or packages installed in them. If you use Eclipe as your Integrated Development Environment (IDE) you can easily reset your Python source to a virtual version created in Conda. You can also share an environment by first exporting and then importing it - the exported environment files for the GeoImagine Framework are linked below.

Prerequisites

Anaconda/conda must be installed as described in this post. If you created a conda virtual environment when setting up the SPIDE you can skip to the section Install additional packages in your environments.

Conda virtual environments

Karttur’s GeoImagine Framework requires a large set of Python packages to work. You have to install these packages and then link them to the SPIDE. Many high level packages depend on other, more basic, packages. When installing many packages there is a large risk of forcing conflicting requirements regarding the versions of shared (i.e. more basic) packages. Sequentially installing (high level) packages can easily lead to a corrupt system due to conflicting requirements regarding shared packages.

To avoid having your complete system corrupted, it is recommended that you build the Python system and packages using a “virtual” environment. In essence this means that you build a system that is working as a stand-alone solution not affecting the core (or ‘base’) system. This is easily done in conda. This tutorial will take you through the steps of creating a virtual python environment in conda.

Check your conda installation and environment

Open a Terminal window, and confirm that Anaconda is installed by typing at the prompt:

$ conda -V

By default, the active environment—the one you are currently using—is shown in parentheses () or brackets [] at the beginning of your command prompt. If you have not installed any virtual environments, the default environment is base:

(base) $,

if you have defined, and activated, a virtual environment it will be shown instead:

(myenv) $.

If you do not see this, run:

conda info

and the first returned line should tell which environment that is active.

To update or manage your conda installation you need to deactivate any customized environment and return to the base environment. The best way to do that is to use the activate command with no environment specified:

$ conda activate

Alternatively you can deactivate the present environment, but if you do that while in base, it might crash your conda setup. Thus I do not write out the command for that.

When in the base environment the terminal prompts should look like this:

(base) $

To update your Anaconda distribution, type:

$ conda update conda

$ conda update anaconda

Import conda environment

The quickest way to build a virtual conda environment is to build (import) from an existing environment exported as a .yml file. Exporting an environment can be done in 2 modes - a full export or a replica of the installation comman. The former is more complete, but might be platform dependent, the latter is conequently less completet, but platform independent. I would recommend using the latter, history approach. Both types on exported .yml files are available in the repo geoimagine03-conda-yml. The two available environment files are:

  • env_geoimagine202110_full.yml (complete environment specification [for MacOS]), and
  • env_geoimagine202110_history.yml (core [history] installation with subpackags automatically chosen dependent on OS [OS independent]).

.condarc

To create a virtual environment from scratch you might need a .condarc configuration file in you personal folder.

.condarc is not included by default when you install conda. To find out if you have a .condarc file open a terminal window and type: $ conda info

Look for the line user config file: in the results.

If you do not have a .condarc file, you can create it by using a text editor (e.g. Atom), directly from the command line ( ~$ pico .condarc) or by running the command:

$ conda config

You can set a lot of parameters and functions in .condarc (as described in the conda document Using the .condarc conda configuration file), but for now you will only use it for defining a set of default packages that by default will be included when creating a new environment (but you can omit this default if required).

Default packages

The manual for setting default packages to install with every new environment is also described in the conda document Using the .condarc conda configuration file. Consult the conda document Anaconda package lists to see available packages for different conda distributions.

For creating virtual conda python environments for Karttur’s GeoImagine Framework, add the following lines to your .condarc file:

create_default_packages:
  - cartopy
  - fiona
  - gdal
  - geopandas
  - h5py
  - matplotlib
  - netcdf4
  - numba
  - numpy
  - pandas
  - pip
  - psycopg2
  - rasterio
  - scipy
  - statsmodels
  - xmltodict

The above list will also install several other packages that are required by the Framework, including for instance GDAL, fiona and shapely, that are packages for geographic data processing and topology.

The advantage with installing the core components in a single command is that conda will solve conflicts among dependencies. In other words, it is best to install all packages at once, so that all of the dependencies are installed at the same time.

Create a new environment

If you now create a new environment:

$ conda create —-name geoimagineXYZ,

the rather short list of default packages will create a rather long list of package to install.

Just press y when conda asks Proceed ([y]/n)? and let conda setup your environment. The terminal response should then be like this:

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate geoimagine001
#
# To deactivate an active environment, use
#
#     $ conda deactivate

If something goes wrong you just simply remove the virtual environment:

$ conda remove —-name geoimagine001 —-all

The conda base setup is not affected by either installing or deleting a virtual environment.

Activate your environment

To list available environments type:

$ conda info -e

If you want to activate a specific environment (other than ‘base’ that is now the default) type at the terminal:

$ conda activate geoimagine001

The prompt should change to say

(geoimagine001) … $

You can now make additional installations to your environment. But first have to locate your virtual environment to learn its path. You need to know that in order to link it as the Python interpreter in Eclipse.

Locate your virtual environments

You can tell conda to put your virtual environment under any path, but by default it is put under your Anaconda installation at:

../anacondaX/envs, which in my example (for Anaconda3) then becomes ../anaconda3/envs/geoimagine001.

If you explore that path you can find the packages installed under ../anaconda3/envs/geoimagine001/lib/python3.7/site-packages. (where python3.7 is the Python version I installed, but can differ if you installed another version).

Set your Python interpreter in Eclipse

With Eclipse workbench up and running, select from the top menu:

Eclipse : preferences

In the Preferences window that opens, click the PyDev expansion icon (>) in the menu to the left. In the expanded sub-list click the expansion icon for Interpreters and click Python interpreter. In the window that opens, click the Browse for python/pypy exe button in the upper right corner. The dialog window Select Interpreters opens.

Click the Browse button next to the textbox Interpreter Executable.

If you created a virtual environment, navigate to where you stored it and find the Python binary file (e.g. …/anaconda3/envs/geoimagine001/bin/python) and choose that file. Then edit the textbox Interpreter Name to something like ‘Python3.x geoimagine001’.

If you did not setup a virtual Python environment you can use the Anaconda default (or ‘base’) environment as your Python interpreter. Click the Browse button and navigate to where you installed Anaconda, and drill down to the Python binary:

…/anaconda3/bin/python

Regardless of which interpreter you selected, click Finish/OK, and the dialog window Selection Needed appears. Accept the default selection (all listed items), and click Finish/OK again. All the selected Libraries and their associated Packages will be linked to your project, and show up in the lower frame of the Preferences window. When finished, click Apply and Close.

At this stage you can continue with setting up Karttur’s GeoImagine Framework. Dependent on how you set it up and what functions you require, you will need to to install additional Python packages to your virtual environment. The topic of the reminder of this post.

Install additional packages in your environments

There are two main methods for installing additional Python packages to your environment, either using conda install or pip. The general recommendaton is to use one of these for individual environments, with conda install being the preferred method. But not all packages are available as default conda installations. All available packages are listed here. Some of the packages required by Karttur´s GeoImagaine Framework are only available via pip, and you thus have to mix conda install and pip as installations methods when setting up the complete library needed.

conda install

You can install new packages into your environment in the usual way that conda packages are installed. Just make sure that the terminal prompt points at your environment:

(geoimagine???) … $ conda install -c omnia svgwrite

or tell conda under which environment to install the packages:

$ conda install –name geoimagine002 -c omnia svgwrite

Once the installation is finished you should see the installed packages under the site-packages path and with the $ conda list

Install non-listed conda packages

If you want to install a package that is not listed as an available package for your conda distribution (see above) you should first search for it as it might have been a recent addition.

$ conda search plotnine

If the search identifies the package just go ahead and install it. If the package is not found you might still find it by looking for it using your web-browser, as also suggested by a non-successful search:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

For the example with plotnine, it is available for installation from a conda -forge channel, e.g.:

$ conda install -c conda-forge plotnine

If your package is available, it is likely that it is listed under a “-forge” command. This means that when conda install tries to solve the environment it will first report any conflicts with existing packages, and then forge conflicts. There are three alternative conflicts:

  • SUPERSEED
  • UPGRADE
  • DOWNGRADE

Superseed means that the new package comes with a dependency that is already installed, but that the dependency will be replaced by the alternative found with the new package. It should be exactly the same package with the same content as the existing dependency, but it will nevertheless be replaces. For all cases I have encountered it has worked out fine to accept the superseed. Otherwise this is the reason you are using a (virtual) environment. You just repeat the setup until the point where it went wrong, and then try another solution.

If the -forge installation reports that a package is to be upgrade(d) that might work. If it reports that a package is in for a downgrade then proceeding with the installation is likely to cause problems with other packages. One solution is then to export your environment and try other versions or alternatives. Change the conflicting packages to use the same version of the common dependency (you have to look in the documentation to find versions that can go together).

Install using pip

If the package you want to install is neither available with $ conda install, nor available as a $ conda -forge installation, you need to use an alternative package manager (as described here). This more or less always boils down to pip.

The package sentinelsat, used for searching and downloading satellte data from the European Space Agency (ESA) is only avalailable suing pip:

(geoimagine0) … $ pip install sentinelsat

Again you can check that the package was installed in the correct place by exploring the site-packages path.

Additional installations

The complete GeoImagine Framework requires a handful of additional installations:

  • landsatexplore
  • plotnine
  • pypng
  • seasonal
  • sentinelsat
  • svgis
  • svgwrite
landsatxplore

Landsatxplore is a package for searching and downloading Landsat satellite image scenes from EarthExplorer. There are alternative packages that can be used for the same task, but Karttur’s GeoImagine Framework is set up for using Landsatxplore. If you want to use Landsat data from EarthExplorer you need to register as an EarthExplorer user.

Landsatxplore is not available at any conda channel and you need to use the pip installation manager.

$ pip install landsatxplore

plotnine

plotnine is a powerful graphics editor that you can use for composing maps and layouts in Python. It is like a Python version of the popular “Grammar of graphics” concept used by ggplot. The grammar allows users to compose plots by explicitly mapping data to the visual objects that make up the plot. plotnine is available on several conda channels, and can for instance be installed using the command:

$ conda install -c conda-forge plotnine

seasonal

The seasonal package estimate and remove trend and periodicity in time-series. In the Framework it is used for time-series decomposition and trend estimations.

NOTE that in the Framework the seasonal package is edited to include more options and with altered default settings.

The seasonal package is installed with pip install

$ pip install seasonal

sentinelsat

In the Framework Sentinelsat is used for searching, downloading and retrieving the metadata and the actual data of Sentinel satellite images from the Copernicus Open Access Hub.

Sentinelsat is preferably installed via conda-forge

$ conda install -c conda-forge sentinelsat.

Sentinelsat can also be installed with pip install:

$ pip install sentinelsat.

svgis

SVGIS converts vector geodata to Scalable Vector Graphics (SVG). SVG can be styled using Cascaded Style Sheets (CSS) and also read and manipulated by drawing programs. In the Framework SVGIS is primarily used for exporting vector data to use as overlays in map layouts.

SVGIS is installed with pip install

$ pip install svgis

svgwrite

SVGwrite is a more general library for writing SVG formated vector graphics. It is used for creating legends and other layout items for maps. The preferred installation is using conda:

$ conda install -c conda-forge svgwrite

svgwrite is also available as pip install

$ pip install svgwrite

ggtools

GRACE & GLDAS Tools (ggtools) is a library for GRACE(Gravity Recovery and Climate Experiment) and GRACE-FO(Follow-on) GSM data(RL06 Level-2 monthly solutions) and GLDAS grid data. The package itself is installed using pip. It is dependent on cartopy, netcdf4 and h5py; all included as default packages (listed in .condarc).

$ pip install ggtools

tdqm

tdqm is preferably installed via conda-forge:

$ conda activate geoimagine202110 $ conda install -c conda-forge tqdm

Export and import environments

Once you have created a virtual environment that satisfies your needs, you can export it as .yml file. You can then use the .yml file to setup a new virtual environment, or share it to allow others to set up identical environments.

Export environment file (yml)

Activate the environment you want export $ source activate geoimagine0,

and then export

(geoimagine) … $ conda env export > geoimagine0.yml.

The exported .yml file below shows the complete installation of all packages required for Karttur’s GeoImagine Framework.

Import

Test if the following works (not installed with create_default_packages)

  • array
  • struct
  • os
  • shutil
  • subprocess
  • gc
  • collections
  • sys
  • math

Other installations

The full suit of opportunities that come with the Framework requires some additional resources, not directly linked ty python:

  • wget,
  • AWSCLIV2, and
  • aria2
  • inkscape

wget

Wget is a command-line tool for retrieving files using HTTP, HTTPS, FTP and FTPS (the most widely-used Internet protocols). In Karttur´s GeoImagine Framework, wget is used for accessing online available data from e.g.https://earthdata.nasa.gov, including MODIS and SMAP. To install Wget on Mac osx you can use Homebrew.

$ brew install wget

The installation of Homebrew itself is covered in the blog post on ImageMagick.

Amazon Web Services Command Line Interface 2 (AWSCLIV2)

Some DEMs are available as Amazon Web Services (AWS) Open Data. To access registries on AWS Open Data I use the free Amazon Web Services Command Line Interface 2 (AWSCLIV2). To install AWSCLIV2 on your machine you can check out Installing, updating, and uninstalling the AWS CLI version 2. Here is a summary of the steps needed for installation on MacOS:

  1. start a Terminal session.

  2. $ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"

  3. $ sudo installer -pkg AWSCLIV2.pkg -target /

Give your machine password and the AWSCLIV2.pkg package should install and report back:

installer: Package name is AWS Command Line Interface
installer: Installing at base path /
installer: The install was successful.

aria2

aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. You can use brew for installation:

$ brew install aria2

Inkscape

Inkscape is a drawing app that can also be accessed using the command line. You can install it as an ordinary binary app, and then access the command line features. it is used for transforming between vector and raster formats, for example for legends and maps.

Resources

Managing conda environments