These instructions are for creating a GithHub superproject and linking together all python packages that constitute Karttur’s GeoImagine Framework. If you are looking for how to clone the ready version of the complete framework, continue to the post Git clone with Eclipse.
Introduction
Karttur’s GeoImagine Framework available at GitHub is built from a “superproject” (or container) repository with around 40 individual repositories attached as submodules. Each submodule (or repo) contains a customised Python package. Together all the submodules constitute the complete Framework. This post explains the git concept of submodules, and details how Karttur’s GeoImagine Framework is organised online.
This post is a shorter summary of the post git submodules under my blog on git command line.
Superproject and Submodules
A superproject repo is like a container that links to other repos, or submodules. All that is required is to setup links in the hidden file .gitmodules in the root of the superproject. The submodules as such can be located at any url, including other repos. git supports adding, updating, synchronizing, and cloning submodules. This post is rather a barebone manual for how to create a superproject with non-complicated submodules, Managing submodules linked to repos that change (i.e. with commited updates) is substantially more complex.
Create superproject
Create an ordinary repo, as outlined in my post on git cheat sheet. In this example I created a superproject repo for Kartturs GeoImagine Framework called geoimagine02_framework. In the Terminal, change directory cd to the empty superproject repo:
$ cd geoimagine02_framework
To create the “superproject” repo as outlined in this post you do not need the more advanced submodule commands. In case you need to access more advanced function you need to initiate them with the command:
$ git submodule init
Define project hierarchy
In the framework repo create a directory that will eventually become the name of your Eclipse project (I called it karttur_v202003 because this version is for Eclipse version 4.15 or 2020 03):
$ mkdir karttur_v202003
cd to the created directory:
$ cd karttur_v202003
Inside the project folder (karttur_v202003), create another directory, that will later be converted to the top python package. I call this folder geoimagine.
$ mkdir geoimagine
To work properly the submodules with the individual python packages that build the Framework must be linked in under the repo path:
./karttur_v202003/geoimagine/
Add modules to the superproject
Adding submodules to the superproject can either be done by the git command git submodule add or by creating a file called .gitmodules and direct editing. For Karttur’s GeoImagine Framework to work out of the box when imported from the superproject repo, each module (i.e. python package) must be renamed when added as a submodule. Each linked submodule should be changed by removing the prefix (geoimagineXX-).
A link in .gitmodules for the package setup_db (repo submodule = geoimagineXX-setup_db) looks like this:
[submodule "setup_db"]
path = karttur_v202003/geoimagine/setup_db
url = https://github.com/karttur/geoimagineXX-setup_db
The same link generated by git command:
$ git submodule add –name setup_db https://github.com/karttur/geoimagine02-setup_db karttur_v202003/geoimagine/setup_db
Skipping the –name option, the submodule will be named from the path within the repo.
List of submodule repos
When writing this in February 2021, Kartturs Geoimagine Framework contains the following Python packages (repos of modules):
repo -> package
geoimagineXX-ancillary -> ancillary
geoimagineXX-dem -> dem
geoimagineXX-endmembers -> endmembers
geoimagineXX-export -> export
geoimagineXX-extract -> extract
geoimagineXX-gdalutilities -> gdalutilities
geoimagineXX-gis -> gis
geoimagineXX-grace -> grace
geoimagineXX-image -> image
geoimagineXX-jekyllise -> jekyllise
geoimagineXX-ktgraphics -> ktgraphics
geoimagineXX-ktnumba -> ktnumba
geoimagineXX-ktpandas -> ktpandas
geoimagineXX-landsat -> landsat
geoimagineXX-layout -> layout
geoimagineXX-mask -> mask
geoimagineXX-modis -> modis
geoimagineXX-overlay -> overlay
geoimagineXX-params -> params
geoimagineXX-postgresdb -> postgresdb
geoimagineXX-projects -> projects
geoimagineXX-region -> region
geoimagineXX-scalar -> scalar
geoimagineXX-sentinel -> sentinel
geoimagineXX-setup_db -> setup_db
geoimagineXX-setup_processes -> setup_processes
geoimagineXX-smap -> smap
geoimagineXX-soilmoisture -> soilmoisture
geoimagineXX-specials -> specials
geoimagineXX-sqldumps -> sqldumps
geoimagineXX-support -> support
geoimagineXX-timeseries -> timeseries
geoimagineXX-transform -> transform
geoimagineXX-updatedb -> updatedb
geoimagineXX-userproj -> userproj
geoimagineXX-zipper -> zipper
Shell script for setting up repos
The shell commands below will generate all the submodules of the Framework as of February 2021.
git submodule add --name ancillary https://github.com/karttur/geoimagine02-ancillary karttur_v202003/geoimagine/ancillary
git submodule add --name dem https://github.com/karttur/geoimagine02-dem karttur_v202003/geoimagine/dem
git submodule add --name endmembers https://github.com/karttur/geoimagine02-endmembers karttur_v202003/geoimagine/endmembers
git submodule add --name export https://github.com/karttur/geoimagine02-export karttur_v202003/geoimagine/export
git submodule add --name extract https://github.com/karttur/geoimagine02-extract karttur_v202003/geoimagine/extract
git submodule add --name gdalutilities https://github.com/karttur/geoimagine02-gdalutilities karttur_v202003/geoimagine/gdalutilities
git submodule add --name gis https://github.com/karttur/geoimagine02-gis karttur_v202003/geoimagine/gis
git submodule add --name grace https://github.com/karttur/geoimagine02-grace karttur_v202003/geoimagine/grace
git submodule add --name image https://github.com/karttur/geoimagine02-image karttur_v202003/geoimagine/image
git submodule add --name jekyllise https://github.com/karttur/geoimagine02-jekyllise karttur_v202003/geoimagine/jekyllise
git submodule add --name ktgraphics https://github.com/karttur/geoimagine02-ktgraphics karttur_v202003/geoimagine/ktgraphics
git submodule add --name ktnumba https://github.com/karttur/geoimagine02-ktnumba karttur_v202003/geoimagine/ktnumba
git submodule add --name ktpandas https://github.com/karttur/geoimagine02-ktpandas karttur_v202003/geoimagine/ktpandas
git submodule add --name landsat https://github.com/karttur/geoimagine02-landsat karttur_v202003/geoimagine/landsat
git submodule add --name layout https://github.com/karttur/geoimagine02-layout karttur_v202003/geoimagine/layout
git submodule add --name mask https://github.com/karttur/geoimagine02-mask karttur_v202003/geoimagine/mask
git submodule add --name modis https://github.com/karttur/geoimagine02-modis karttur_v202003/geoimagine/modis
git submodule add --name overlay https://github.com/karttur/geoimagine02-overlay karttur_v202003/geoimagine/overlay
git submodule add --name params https://github.com/karttur/geoimagine02-params karttur_v202003/geoimagine/params
git submodule add --name postgresdb https://github.com/karttur/geoimagine02-postgresdb karttur_v202003/geoimagine/postgresdb
git submodule add --name projects https://github.com/karttur/geoimagine02-projects karttur_v202003/geoimagine/projects
git submodule add --name region https://github.com/karttur/geoimagine02-region karttur_v202003/geoimagine/region
git submodule add --name scalar https://github.com/karttur/geoimagine02-scalar karttur_v202003/geoimagine/scalar
git submodule add --name sentinel https://github.com/karttur/geoimagine02-sentinel karttur_v202003/geoimagine/sentinel
git submodule add --name setup_db https://github.com/karttur/geoimagine02-setup_db karttur_v202003/geoimagine/setup_db
git submodule add --name setup_processes https://github.com/karttur/geoimagine02-setup_processes karttur_v202003/geoimagine/setup_processes
git submodule add --name smap https://github.com/karttur/geoimagine02-smap karttur_v202003/geoimagine/smap
git submodule add --name soilmoisture https://github.com/karttur/geoimagine02-soilmoisture karttur_v202003/geoimagine/soilmoisture
git submodule add --name specials https://github.com/karttur/geoimagine02-specials karttur_v202003/geoimagine/specials
git submodule add --name sqldumps https://github.com/karttur/geoimagine02-sqldumps karttur_v202003/geoimagine/sqldumps
git submodule add --name support https://github.com/karttur/geoimagine02-support karttur_v202003/geoimagine/support
git submodule add --name timeseries https://github.com/karttur/geoimagine02-timeseries karttur_v202003/geoimagine/timeseries
git submodule add --name transform https://github.com/karttur/geoimagine02-transform karttur_v202003/geoimagine/transform
git submodule add --name updatedb https://github.com/karttur/geoimagine02-updatedb karttur_v202003/geoimagine/updatedb
git submodule add --name userproj https://github.com/karttur/geoimagine02-userproj karttur_v202003/geoimagine/userproj
git submodule add --name zipper https://github.com/karttur/geoimagine02-zipper karttur_v202003/geoimagine/zipper
Prepare Eclipse project
if you want to prepare the Framework for the IDE Eclipse you need to add two hidden system files .project and .pydevproject. Both files should be located under the project directory - for this example that is the karttur_v202003 directory created above.
.project
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>kartturv202003</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.python.pydev.PyDevBuilder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.python.pydev.pythonNature</nature>
</natures>
</projectDescription>
.pydevproject
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?eclipse-pydev version="1.0"?><pydev_project>
<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
<path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
</pydev_project>
Git add, commit and push
Your project should now be ready for deployment. Use the standard git commands add, commit and push:
$ git add .
$ git commit -m “initial commit”
$ git push origin main
Review the GitHub superproject repo
If you return to your online git repo, e.g. at GitHub, you can inspect your superproject repo online. If all worked out, the project should be ready for cloning directly into Eclipse as described in the post Git clone with Eclipse.