NOTE, 1 October 2020 the label ‘master’ was replaced by ‘main’ for the default repo branch.
Introduction
Karttur’s GeoImagine Framework available at Github is built from an (almost) empty “superproject” repository with around 40 individual repositories attached as submodules. Each submodule (or repo) contains a customised Python package. Together all the submodules constitute the complete Framework. This post explains the git concept of submodules, and details how Karttur’s GeoImagine Framework is organised online.
The tutorial on Submodules at Atlassian Bitbucket contains all the essentials on git submodules. The reference page for git submodule is more barebone. Pro Git - Everything you need to know about git by Scott Chacon and Ben Straub is the most complete reference source.
If you are primarily interested in git submodules I recommend a more general instruction, like Using submodules in Git - Tutorial, Git Submodules basic explanation, Working with submodules or the youtube introduction Git Tutorial: All About Submodules.
Prerequisits
You need to install git for command line use as outlined in the parallel post on Local git control. You do not need a GitHub account to follow this post unless you want to try to manage changes between an online account linked as a submodule to a local superproject.
Working with submodules
Submodules are repositories that are linked to a superproject repository. All that is required is to setup links to other repos in the hidden file .gitmodules in the root of the superproject. The submodule(s) as such can be located at any url. git supports adding, updating, synchronizing, and cloning submodules. This post is rather a barebone manual for how to create a superproject with non-complicated submodules. How to update and manage superprojects is the topic of the post Managing superproject.
Create three repos
To test the function of submodules, create three directories and initiate git in each of them:
- container-repo
- ch01-introduction
- ch02-the_sun
Change directory (cd) to the parent folder where you want to create your test repos, then execute the following commands:
$ mkdir container-repo
$ cd container-repo
$ git init
$ cd ..
$ mkdir ch01-introduction
$ cd ch01-introduction
$ git init
$ cd ..
$ mkdir ch02-the_sun
$ cd ch02-the_sun
$ git init
$ cd ..
Create README and content
Continuing our example with chapters in a book, let us use the container-repo as a superproject with only a README file:
$ cd container-repo
$ pico README.md
## Parent repository for chapters in book on ...
This repo is an (almost) empty container linking to other repos (submodules) as defined in the hidden .gitmodules files.
Hit [ctrl]+[X] to exit pico and save the edits by pressing Y when asked.
stage and commit the container-repo and return to the parent folder.
$ git add .
$ git commit -am "Initial commit"
$ cd ..
The submodules in this example are the chapters in the book. To avoid keeping track of the chapters except as submodules, the directories are named as the chapters, but with the prefix “ch-nn” to also force a correct ordering.
Add a README and another md file with the chapter title and an outline of the content to each chapter repo, for example:
$ cd ch01-introduction
$ pico README.md
## Chapter 1 in book on ...
Hit [ctrl]+[X] to exit pico and save the edits by pressing Y when asked.
$ pico ch01-introduction.md
## Introduction
Narrative with a personal touch, skip long pieces on what you wanted to do. Just write what you did.
Hit [ctrl]+[X] to exit pico and save the edits by pressing Y when asked.
stage and commit the changes made to ch01-introduction and return to the parent folder.
$ git add .
$ git commit -am "Initial commit"
$ cd ..
Repeat the same steps for the 2nd chapter. The content of the markdown (md) files does not matter, as long as they contain some text.
Add submodule
The command for adding a submodule requires only the source path (or SSH or HTTPS path if you are adding an online repo). Optionally you can state the local path in the receiving repo (i.e. the superproject). If no receiving path is given the submodule will be saved in the root of the receiving repo under its original name, git submodule add source_path [receiving_path].
Add local repo as submodule
As our receiving repo (container-repo) and the submodules (ch01-xx and ch02-yy) are saved in parallel directories, you can give the path using the relative syntax: ../ch01-xx etc. Thus, to add the local repo ch01-introduction as a submodule at the root of the repo container-repo without changing its name:
$ cd container-repo
$ git submodule add ../ch01-introduction
or if you want to save the submodule under a predefined path in the receiving repo:
$ git submodule add ../ch02-the_sun submodules/ch02-sun
Check the status of your superproject:
$ git status
On branch master
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: .gitmodules
new file: ch01-introduction
new file: submodules/ch02-sun
The first submodule that you added (ch01-introduction) is located directly under the root; the second submodule (ch02-sun) under the path /submodules/ (and with the submodule having a different name comparaed to the source repo). Then also the file .gitmodules has been added. This file contains the link between the submodules and their origin:
[submodule "ch01-introduction"]
path = ch01-introduction
url = ../ch01-introduction
[submodule "submodules/ch02-sun"]
path = submodules/ch02-sun
url = ../ch02-the_sun
stage and commit the local submodules to save them in your history tree:
$ git add .
$ git commit -am "initial commit for local submodules"
Add remote repo as submodule
To add a remote repo from GitHub.com, open your web browser and navigate to the repo you want to add as a submodule. For the repo (“my-first-repo”) you created in the post on Remote repositories with GitHub. If you did not create this repo, you can use the GitHub example repo the Spoon-Knife project.
With the GitHub repo you want to add as a submodule in your browser, click the button. Copy the link to the repo as shown to the right. Then paste the link to the command:
$ git submodule add git@github.com:”YourGitHubAccount”/my-first-repo.git
remote/my-first-repo
Cloning into '/Users/"YourGitHubAccount"/temp/git-dummy/container-repo/remote/my-first-repo'...
git sremote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 9 (delta 1), reused 4 (delta 1), pack-reused 0
Receiving objects: 100% (9/9), 1.64 KiB | 840.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.
$ git add .
$ git commit -am "initial commit for remote submodules"
Put your terminal window in the root of your container-repo and list (ls) the content:
$ ls
README.md ch01-introduction remote submodules
This corresponds to the commands executed above.
Edit submodule repo
Return to one of the repos that you added as a submodule. Edit one of the text files, or add a new file. stage and commit the changes. Make sure that you have no outstanding content by running git status. Here is an example:
$ cd ..
$ cd ch01-introductio
$ pico ch01-introduction.md
...
Add a text string ...
Hit [ctrl]+[X] to exit pico and save the edits by pressing Y when asked.
$ git commit -am "edited ch01-introduction.md"
[master 60b6923] edited ch01-introduction.md
1 file changed, 2 insertions(+)
Pull changes to submodule
The commited changes in the repo ch01-introduction are not yet reflected in the submodule built from this repo. In your terminal window, return to the superproject repo that contains the submodules, then to location (directory) where you saved ch01-introduction as a submodule(in my example above I did not give any path and the sublodule is saved under its original name), and then pull.
$ cd ..
$ cd container-repo
$ cd ch01-introduction
$ git pull
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /Users/thomasgumbricht/temp/git-dummy/ch01-introduction
ac86f33..60b6923 master -> origin/master
Updating ac86f33..60b6923
Fast-forward
ch01-introduction.md | 2 ++
1 file changed, 2 insertions(+)
Return to the root of the repo containing the submodules and check status:
$ cd ..
$ git status
On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: ch01-introduction (new commits)
no changes added to commit (use "git add" and/or "git commit -a")
Even if you have pulled the changes in the original repo linked to the submodule (ch01-introduction), these changes are not saved to the superproject history tree. You have to run a commit:
$ git commit -am “pulled ch01-introduction”
Resources
Submodules at Atlassian Bitbucket
Pro Git - Everything you need to know about git by Scott Chacon and Ben Straub (20200219).