Average rainfall 2001-2016, global tropics

Map: Average rainfall 2001-2016, global tropics

Conda virtual environments I

- Introduction - Prerequisites - Conda virtual environments - Check your conda installation - .condarc - Default packages - Create a new environment - Activate your environment - Locate your environment - More later - Resources

Introduction

With conda, you can create, update, export and import virtual Python environments that have different versions of Python and/or packages installed in them. If you use Eclipse as your Integrated Development Environment (IDE) you can easily reset your Python source to a virtual version created in conda. You can also share an environment by exporting and then importing it.

Prerequisites

Conda must be installed as described in this post.

Conda virtual environments

Karttur’s GeoImagine Framework requires a large set of Python packages to work. You have to install these packages and then link them to the Framework Spatial Data IDE (SPIDE). Many high level packages depend on other, more basic, packages. When installing many packages there is a large risk of forcing conflicting requirements regarding the versions of shared (i.e. more basic) packages. Sequentially installing (high level) packages can easily lead to a corrupt system due to conflicting requirements regarding shared packages.

To avoid having your complete system corrupted, it is recommended that you build the Python system and packages using a “virtual” environment. In essence this means that you build a system command for creating a role is is working as a stand-alone solution not affecting the core (or ‘base’) system. This is easily done in conda. This tutorial will take you through the steps of creating a virtual python environment in conda. Then you can proceed to install Eclipse and setup PyDev using the virtual environment created in this post.

Check your conda installation

Open a Terminal window, and confirm that Anaconda is installed by typing at the prompt:

$ conda -V

By default, the active environment—the one you are currently using—is shown in parentheses () or brackets [] at the beginning of your command prompt. If you have not installed any virtual environments, the default environment is base:

(base) $,

if you have defined and activated a virtual environment it will be shown instead:

(myenv) $.

If you do not see this, run:

conda info --envs

To update or manage your conda installation you need to deactivate any customized environment and return to the base environment. The best way to do that is to use the activate command with no environment specified:

$ conda activate

Alternatively you can deactivate the present environment, but if you do that while in base, it might crash your conda setup. Thus I do not write out the command for that.

When in the base environment the terminal prompts should look like this:

(base) $

To update your Anaconda distribution, type:

$ conda update conda

$ conda update anaconda

.condarc

To create a virtual environment from scratch you need to have a .condarc configuration file in you personal folder.

The .condarc was not included by default when you installed conda. To find out if you have a .condarc file open a terminal window and type: $ conda info

Look for the line user config file: in the results.

If you do not have a .condarc file, you can create it by using a text editor (e.g. Atom), directly from the command line ( ~$ pico .condarc) or by running the command:

$ conda config

You can set a lot of parameters and functions in .condarc (as described here), but for now you will only use it for defining a set of default packages that will always be included when creating a new environment.

Default packages

The manual for setting default packages to install with every new environment is described here. Lists of available default packages for different conda distributions are found here. The advantage with installing the core components in a single command is that conda will solve conflicts among dependencies. In other words, it is best to install all packages at once, so that all of the dependencies are installed at the same time. The above list will also install several other packages that are required by the Framework.

Default packages only

For creating virtual conda python environments only using conda default packages for Karttur’s GeoImagine Framework, add the following lines to your .condarc file:

create_default_packages:
  - cartopy
  - fiona
  - gdal
  - geopandas
  - h5py
  - matplotlib
  - netcdf4
  - numba
  - numpy
  - pandas
  - pip
  - psycopg2
  - rasterio
  - scipy
  - statsmodels
  - xmltodict
channels:
  - defaults

Default and conda-forge packages

conda-forge is a community-led collection of recipes, build infrastructure and distributions for the conda package manager. Some of the conda packages required to run the complete Framework are not available as default packages, but with conda-forge. You can extent the .condarc configuration to also include packages available at conda-forge. You can also install the conda-forge packages after you have set up the default packages - the commands for the latter are given further down in this post. If you want to ty out using extending the configuration, update your .condarc file:

create_default_packages:
  - cartopy
  - fiona
  - gdal
  - geopandas
  - h5py
  - matplotlib
  - netcdf4
  - numba
  - numpy
  - pandas
  - pip
  - psycopg2
  - rasterio
  - scipy
  - statsmodels
  - xmltodict
  - plotnine
  - pypng
  - sentinelsat
  - svgwrite
  - tqdm
channels:
  - conda-forge
  - defaults

Default packages only

The example below is the result from creating a new environment only using conda default packages.

If you now create a new environment:

$ conda create —-name geoimagineXYZ,

the rather short list of default packages will create a rather long list of package to install.

Just press y when conda asks Proceed ([y]/n)? and let conda setup your environment. The terminal response should then be like this:

Proceed ([y]/n)? y

Downloading and Extracting Packages
affine-2.3.0         | 17 KB     | ##################################################################################################################### | 100%
mkl_random-1.2.2     | 263 KB    | ##################################################################################################################### | 100%
geotiff-1.6.0        | 120 KB    | ##################################################################################################################### | 100%
aws-checksums-0.1.9  | 47 KB     | ##################################################################################################################### | 100%
rtree-0.9.7          | 47 KB     | ##################################################################################################################### | 100%
mapclassify-2.4.3    | 36 KB     | ##################################################################################################################### | 100%
numba-0.53.1         | 3.2 MB    | ##################################################################################################################### | 100%
olefile-0.46         | 49 KB     | ##################################################################################################################### | 100%
psycopg2-2.8.6       | 139 KB    | ##################################################################################################################### | 100%
fiona-1.8.13.post1   | 639 KB    | ##################################################################################################################### | 100%
libboost-1.73.0      | 12.7 MB   | ##################################################################################################################### | 100%
scikit-learn-0.24.2  | 4.8 MB    | ##################################################################################################################### | 100%
python-3.7.11        | 18.1 MB   | ##################################################################################################################### | 100%
xerces-c-3.2.3       | 1.9 MB    | ##################################################################################################################### | 100%
rasterio-1.1.0       | 6.5 MB    | ##################################################################################################################### | 100%
pixman-0.40.0        | 341 KB    | ##################################################################################################################### | 100%
aws-c-common-0.4.57  | 140 KB    | ##################################################################################################################### | 100%
munch-2.5.0          | 13 KB     | ##################################################################################################################### | 100%
libpq-12.2           | 2.0 MB    | ##################################################################################################################### | 100%
poppler-data-0.4.10  | 2.1 MB    | ##################################################################################################################### | 100%
certifi-2021.10.8    | 151 KB    | ##################################################################################################################### | 100%
fontconfig-2.13.1    | 211 KB    | ##################################################################################################################### | 100%
cycler-0.10.0        | 14 KB     | ##################################################################################################################### | 100%
geopandas-base-0.9.0 | 919 KB    | ##################################################################################################################### | 100%
h5py-3.2.1           | 894 KB    | ##################################################################################################################### | 100%
cligj-0.7.2          | 13 KB     | ##################################################################################################################### | 100%
matplotlib-3.4.2     | 26 KB     | ##################################################################################################################### | 100%
matplotlib-base-3.4. | 5.5 MB    | ##################################################################################################################### | 100%
mkl_fft-1.3.0        | 159 KB    | ##################################################################################################################### | 100%
bottleneck-1.3.2     | 107 KB    | ##################################################################################################################### | 100%
aws-c-event-stream-0 | 21 KB     | ##################################################################################################################### | 100%
proj-6.2.1           | 8.8 MB    | ##################################################################################################################### | 100%
kealib-1.4.14        | 139 KB    | ##################################################################################################################### | 100%
postgresql-12.2      | 3.3 MB    | ##################################################################################################################### | 100%
libgdal-3.0.2        | 10.4 MB   | ##################################################################################################################### | 100%
poppler-0.81.0       | 1.5 MB    | ##################################################################################################################### | 100%
tornado-6.1          | 588 KB    | ##################################################################################################################### | 100%
libkml-1.3.0         | 425 KB    | ##################################################################################################################### | 100%
libspatialite-4.3.0a | 2.1 MB    | ##################################################################################################################### | 100%
numexpr-2.7.3        | 124 KB    | ##################################################################################################################### | 100%
giflib-5.2.1         | 70 KB     | ##################################################################################################################### | 100%
pip-21.0.1           | 1.8 MB    | ##################################################################################################################### | 100%
numpy-base-1.21.2    | 4.6 MB    | ##################################################################################################################### | 100%
hdf4-4.2.13          | 682 KB    | ##################################################################################################################### | 100%
cairo-1.16.0         | 911 KB    | ##################################################################################################################### | 100%
pandas-1.3.3         | 8.8 MB    | ##################################################################################################################### | 100%
hdf5-1.10.6          | 3.0 MB    | ##################################################################################################################### | 100%
patsy-0.5.2          | 268 KB    | ##################################################################################################################### | 100%
cached-property-1.5. | 11 KB     | ##################################################################################################################### | 100%
aws-sdk-cpp-1.8.185  | 1.6 MB    | ##################################################################################################################### | 100%
pyshp-2.1.3          | 37 KB     | ##################################################################################################################### | 100%
scipy-1.7.1          | 15.3 MB   | ##################################################################################################################### | 100%
snuggs-1.4.7         | 9 KB      | ##################################################################################################################### | 100%
libdap4-3.19.1       | 1.1 MB    | ##################################################################################################################### | 100%
pyproj-2.6.1.post1   | 330 KB    | ##################################################################################################################### | 100%
numpy-1.21.2         | 23 KB     | ##################################################################################################################### | 100%
pillow-8.3.1         | 607 KB    | ##################################################################################################################### | 100%
cftime-1.5.0         | 168 KB    | ##################################################################################################################### | 100%
libnetcdf-4.6.1      | 782 KB    | ##################################################################################################################### | 100%
setuptools-58.0.4    | 787 KB    | ##################################################################################################################### | 100%
freexl-1.0.6         | 38 KB     | ##################################################################################################################### | 100%
gdal-3.0.2           | 1.1 MB    | ##################################################################################################################### | 100%
geopandas-0.9.0      | 10 KB     | ##################################################################################################################### | 100%
shapely-1.7.1        | 372 KB    | ##################################################################################################################### | 100%
llvmlite-0.36.0      | 296 KB    | ##################################################################################################################### | 100%
json-c-0.13.1        | 59 KB     | ##################################################################################################################### | 100%
statsmodels-0.12.2   | 8.2 MB    | ##################################################################################################################### | 100%
tiledb-2.2.9         | 1.5 MB    | ##################################################################################################################### | 100%
geos-3.8.0           | 815 KB    | ##################################################################################################################### | 100%
cfitsio-3.470        | 782 KB    | ##################################################################################################################### | 100%
netcdf4-1.5.7        | 387 KB    | ##################################################################################################################### | 100%
cartopy-0.18.0       | 1.7 MB    | ##################################################################################################################### | 100%
click-plugins-1.1.1  | 10 KB     | ##################################################################################################################### | 100%
mkl-service-2.4.0    | 44 KB     | ##################################################################################################################### | 100%
kiwisolver-1.3.1     | 52 KB     | ##################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: \

    Installed package of scikit-learn can be accelerated using scikit-learn-intelex.
    More details are available here: https://intel.github.io/scikit-learn-intelex

    For example:

        $ conda install scikit-learn-intelex
        $ python -m sklearnex my_application.py



done
#
# To activate this environment, use
#
#     $ conda activate geoimagine001
#
# To deactivate an active environment, use
#
#     $ conda deactivate

If something goes wrong you just simply delete the virtual environment:

$ conda remove —-name geoimagine001 –all

or

$ conda env remove —-name geoimagine001

The conda base setup is not affected by either installing or deleting a virtual environment.

Default and conda-forge packages

Create a new environment with specified python version

If you now create a new environment with a specified python environment:

$ conda create –n geoimagine_py37_01 python=3.7

or

$ conda create –no-default-packages –n ortho_py38 python=3.8 numpy pandas scikit-learn

Activate your environment

To list available environments type:

$ conda info –e

If you want to activate a specific environment (‘other than ‘base’ that is now the default) type at the terminal:

$ conda activate geoimagine001

The prompt should change to say

(geoimagine001) … $

You can now make additional installations to your environment. But for now you only have to locate your virtual environment to learn its path. You need to know that in order to link it as the Python interpreter in Eclipse later.

Locate your environment

You can tell conda to put your virtual environment under any path, but by default it is put under your Anaconda installation at:

../anacondaX/envs, which in my example (for Anaconda3) then becomes ../anaconda3/envs/geoimagine001.

If you explore that path you can find the packages installed under ../anaconda3/envs/geoimagine001/lib/python3.7/site-packages. (where python3.7 is the Python version I installed, but can differ if you installed another version).

Set your Python interpreter in Eclipse

With Eclipse workbench up and running, select from the top menu:

Eclipse : preferences

In the Preferences window that opens, click the PyDev expansion icon (>) in the menu to the left. In the expanded sub-list click the expansion icon for Interpreters and click Python interpreter. In the window that opens, click the Browse for python/pypy exe button in the upper right corner. The dialog window Select Interpreters opens.

Click the Browse button next to the textbox Interpreter Executable.

If you created a virtual environment, navigate to where you stored it and find the Python binary file (e.g. …/anaconda3/envs/geoimagine001/bin/python) and choose that file. Then edit the textbox Interpreter Name to something like ‘Python3.x geoimagine001’.

If you did not setup a virtual Python environment you can use the Anaconda default (or ‘base’) environment as your Python interpreter. Click the Browse button and navigate to where you installed Anaconda, and drill down to the Python binary:

…/anaconda3/bin/python

Regardless of which interpreter you selected, click Finish/OK, and the dialog window Selection Needed appears. Accept the default selection (all listed items), and click Finish/OK again. All the selected Libraries and their associated Packages will be linked to your project, and show up in the lower frame of the Preferences window. When finished, click Apply and Close.

Install additional packages in your environments

There are two main methods for installing additional Python packages to your conda environment, either using conda install or pip. The general recommendation is to use just one of these for individual environments, with conda install being the preferred method. But not all packages are available as default conda installations. All conda default available packages are listed here. Some of the packages required by Karttur´s GeoImagaine Framework are only available via pip, and you thus have to mix conda install and pip as installations methods when setting up the complete library needed.

conda install

You can install new packages into your environment in the usual way that conda packages are installed. Just make sure that the terminal prompt points at your environment:

(geoimagine???) … $ conda install -c conda-forge svgwrite

or tell conda under which environment to install the packages:

$ conda install –name geoimagine??? -c conda-forge svgwrite/span>

Once the installation is finished you should see the installed packages under the site-packages path and with the $ conda list

Install non-listed conda packages

If you want to install a package that is not listed as an available package for your conda distribution (see above) you should first search for it as it might have been a recent addition.

$ conda search plotnine

If the search identifies the package just go ahead and install it. If the package is not found you might still find it by looking for it using your web-browser, as also suggested by a non-successful search:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

For the example with plotnine, it is available for installation from the conda-forge channel, e.g.:

$ conda install -c conda-forge plotnine

If your package is available, it is likely that it is listed under a “-forge” command. This means that when conda install tries to solve the environment it will first report any conflicts with existing packages, and then forge conflicts. There are three alternative conflicts:

  • SUPERSEED
  • UPGRADE
  • DOWNGRADE

Superseed means that the new package comes with a dependency that is already installed, but that the dependency will be replaced by the alternative found with the new package. It should be exactly the same package with the same content as the existing dependency, but it will nevertheless be replaces. For all cases I have encountered it has worked out fine to accept the superseed. Otherwise this is the reason you are using a (virtual) environment. You just repeat the setup until the point where it went wrong, and then try another solution.

If the -forge installation reports that a package is to be upgrade(d) that might work. If it reports that a package is in for a downgrade then proceeding with the installation is likely to cause problems with other packages. One solution is then to export your environment and try other versions or alternatives. Change the conflicting packages to use the same version of the common dependency (you have to look in the documentation to find versions that can go together).

Install using pip

If the package you want to install is neither available with $ conda install, nor available as a $ conda -forge installation, you need to use an alternative package manager (as described here). This more or less always boils down to pip.

The package seasonal, used for estimating and removing trend and periodicity in time-series is only available using pip:

(geoimagine0) … $ pip install seasonal

Again you can check that the package was installed in the correct place by exploring the site-packages path.

Additional installations

The complete GeoImagine Framework requires a handful of additional installations:

  • landsatexplore
  • plotnine*
  • pypng*
  • seasonal
  • sentinelsat*
  • svgis
  • svgwrite*
  • tqdm*

* = installed with the extended channel conda-forge above.

landsatxplore

Landsatxplore is a package for searching and downloading Landsat satellite image scenes from EarthExplorer. There are alternative packages that can be used for the same task, but Karttur’s GeoImagine Framework is set up for using Landsatxplore. If you want to use Landsat data from EarthExplorer you need to register as an EarthExplorer user.

Landsatxplore is not available at any conda channel and you need to use the pip installation manager.

$ pip install landsatxplore

plotnine

plotnine is a powerful graphics editor that you can use for composing maps and layouts in Python. It is like a Python version of the popular “Grammar of graphics” concept used by ggplot. The grammar allows users to compose plots by explicitly mapping data to the visual objects that make up the plot. plotnine is available on several conda channels, and can for instance be installed using the command:

$ conda install -c conda-forge plotnine

seasonal

The seasonal package estimate and remove trend and periodicity in time-series. In the Framework it is used for time-series decomposition and trend estimations.

NOTE that in the Framework the seasonal package is edited to include more options and with altered default settings.

The seasonal package is installed with pip install

$ pip install seasonal

sentinelsat

In the Framework Sentinelsat is used for searching, downloading and retrieving the metadata and the actual data of Sentinel satellite images from the Copernicus Open Access Hub.

Sentinelsat is preferably installed via conda-forge

$ conda install -c conda-forge sentinelsat.

Sentinelsat can also be installed with pip install:

$ pip install sentinelsat.

svgis

SVGIS converts vector geodata to Scalable Vector Graphics (SVG). SVG can be styled using Cascaded Style Sheets (CSS) and also read and manipulated by drawing programs. In the Framework SVGIS is primarily used for exporting vector data to use as overlays in map layouts.

SVGIS is installed with pip install

$ pip install svgis

svgwrite

SVGwrite is a more general library for writing SVG formated vector graphics. It is used for creating legends and other layout items for maps. The preferred installation is using conda:

$ conda install -c conda-forge svgwrite

svgwrite is also available as pip install

$ pip install svgwrite

ggtools

GRACE & GLDAS Tools (ggtools) is a library for GRACE(Gravity Recovery and Climate Experiment) and GRACE-FO(Follow-on) GSM data(RL06 Level-2 monthly solutions) and GLDAS grid data. The package itself is installed using pip. It is dependent on cartopy, netcdf4 and h5py; all included as default packages (listed in .condarc).

$ pip install ggtools

tqdm

tqdm is used for adding a progress line for large processes, it is preferably installed via conda-forge:

$ conda install -c conda-forge tqdm

Export and import environments

Once you have created a virtual environment that satisfies your needs, you can export it as .yml file. You can then use the .yml file to setup a new virtual environment, or share it to allow others to set up identical environments.

Export environment file (yml)

Activate the environment you want export $ source activate geoimagine0,

and then export

(geoimagine) … $ conda env export > geoimagine0.yml.

The exported .yml file below shows the complete installation of all packages required for Karttur’s GeoImagine Framework.

Import

TO be completed

Resources

Using the .condarc conda configuration file

Anaconda package lists

Conda - Managing environments

Using the .condarc conda configuration file