Average rainfall 2001-2016, global tropics

Map: Average rainfall 2001-2016, global tropics

GitHub Submodules with git

Thomas Gumbricht bio photo By Thomas Gumbricht

These instructions are for using the git command line tool for creating a GitHub repository with a frame- (or super-) project (repository [repo]) linking together individual repos each containing one of the python packages that constitute Karttur’s GeoImagine Framework. You can also use Eclipse for creating the submodules and the superproject as outliend in the next post. If you are looking for how to clone the ready version of the complete framework, continue to the post Git clone with Eclipse - NOT YET AVAILABLE.

Introduction

Setting up a repository (repo) for code projects is offered by several companies. Karttur’s GeoImagine Framework is hosted on GitHub. This post is a manual on how to use the git command line tool to create a superproject on GitHub that links together a set of submodules to a complete system.

If you are primarily interested in Git submodules I recommend a more general instruction, like Using submodules in Git - Tutorial, Git Submodules basic explanation, Working with submodules or the youtube introduction Git Tutorial: All About Submodules.

Prerequisites

To follow the instructions in this post you need a GitHub account.

To create a fully functional GeoImagine Framework you must also have installed the Spatial Data Integrated Development Environment (SPIDE) as outlined in my blog Install and setup spatial data IDE.

Python packages repositories

The first step is to create all the repositories for the individual python packages. The approach I use for that is to give all the repos with individual packages a double name:

geoimagineXX-package

where the prefix is composed of two parts: the name geoimagine and then a version number (XX) then a hyphen (“-“) followed by the name of the actual python package. In October 2021, the Framework (version 03) is composed of the following packages/repos:

spackage repository
ktpandas geoimagine03-ktpandas
userproj geoimagine03-userproj
dem geoimagine03-dem
setup_processes geoimagine03-setup_processes-x
ktnumba geoimagine03-ktnumba
grass geoimagine03-grass
ktgdal geoimagine03-ktgdal
zipper geoimagine03-zipper
updatedb geoimagine03-updatedb
basins geoimagine03-basins
gis geoimagine03-gis
landsat geoimagine03-landsat
layout geoimagine03-layout
projects geoimagine03-projects
grace geoimagine03-grace
smap geoimagine03-smap
params geoimagine03-params
timeseries geoimagine03-timeseries
copernicus geoimagine03-copernicus
setup_processes geoimagine03-setup_processes
npproc geoimagine03-npproc
ktgrass geoimagine03-ktgrass
ancillary geoimagine03-ancillary
sentinel geoimagine03-sentinel
setup_db geoimagine03-setup_db
extract geoimagine03-extract
region geoimagine03-region
modis geoimagine03-modis
assets geoimagine03-assets
support geoimagine03-support
postgresdb geoimagine03-postgresdb
export geoimagine03-export

Create repos

To create the repos listed above, open your GtHub account in your web-browser. Then manually create all the repos as outlined in the post on git cheat sheet. You must also clone all the repos manually using the git clone command.

Each created repo will only contain two visible files: README.md and LICENCE. The next step is to add the python package code belonging to each repo. This can either be done by manual copy/paste from your locally developed IDE, or using the python module submodule_stage-commit-push_script. This module (or script) is part of the Framework itself, available in the support package. The script copies a list of python packages from one directory tree to another. If the target file already exist it is overwritten if the source file in more recent. The script also creates a predefined .gitignore. As the script is not dependent on any other part of the Framework, you can just copy and paste it from under the Hide/Show button.

Stage, commit and push

When you have filled the repos with the python packages, you need to stage (add), commit and push the locally edited repos to you online origin. If you used the script supplied above, it will generate a script file that will execute all the git commands.

Create the superproject repo

To create the superproject repo that will contain the python frame package, return to your online GtHub account and create another repo with only a README.md and a LICENCE. There is no need for a strict naming, in this example I named it geoimagine03frame, without any hyphen to separate it from the submodules. Clone the superproject repo to your local machine with the command git clone.

The git command for linking other repos as submodules is git submodule add. The command expects a url link to a repo; by default the name of the submodule will be identical to the original repo, but by adding a second variable the name can be customised. For the GeoImagine Framework the name should be set to the actual name of the python package. Thus the command for using the repo geoimagine03-ancillary as a submodule in the superproject becomes:

$ git submodule add https://github.com/karttur/geoimagine03-ancillary ancillary

Instead of writing the commands for adding all the individual package repos, the script submodule_stage-commit-push_script.py (above) generates the shell script code for adding all. To execute the script, open a Terminal window and change directory cd to the superproject repo, for example:

$ cd /Users/thomasgumbricht/GitHub/geoimagine03frame

and execute the git submodule add scripts from within the superproject.

After adding all package repos as submodules in October 2021, the complete GeoImagine Framework consists of 826 individual files. The complete directory tree in under the Hide/Show button.

Prepare superproject for Eclipse

Stage, commit and push

With the local clone of the superproject assembled, stage, commit and push it to the online origin repo.