Introduction
Having spent 6 months trying to develop the business side of my projects while leaving my python programming untouched, I realise that the documentation of my python projects, packages and modules have much to desire. This post lists some options for documentation of python code and explains how to install the python package sphinx and then use it for html documentation of packages and modules.
Python documentation generators
Looking for a python documentation generator I searched for reviews and comparisons. I only knew about Doxygen, but then also found Sphinx, pdoc and pydoctor.
For a comparing review see Peter Kong (2018) Comparison of Python documentation generators. For a more complete (but still not up-to-date) list see python wiki page on Documentation tools.
Having looked around, I chose Sphinx as the documentation generator for my python coding.
Install Sphinx
Sphinx is installed using the Terminal. The Sphinx package is not part of Karttur´s Geoimagine Framework, it is thus better to setup a separate virtual python environment that is only used for Sphinx. Note, however that this environment also must contain all the python packages used by the packages and modules you intend to document. Thus if you have a default environment for your virtual python environment, keep it. Otherwise you can add the missing packages as they are requested by Sphinx.
(base) user ~ % conda create -n sphinx python
Activate the new environment
% conda activate sphinx
Install Sphinx to your new environment using either conda or pip.
Conda
Install Sphinx using conda:
(sphinx) user ~ % conda install -c anaconda sphinx
Then also install the add-on packages sphinx_rtd_theme, autodocsumm and nbsphinx:
(sphinx) user ~ % conda install -c conda-forge sphinx_rtd_theme
(sphinx) user ~ % conda install -c conda-forge autodocsumm
(sphinx) user ~ % conda install -c conda-forge nbsphinx
Pip
With pip you can install both sphinx and sphinx_rtd_theme with a single command:
(sphinx) user ~ % pip install sphinx sphinx_rtd_theme
And then separately install autodocsumm and nbsphinx:
(sphinx) user ~ % pip install autodocsumm
(sphinx) user ~ % pip install nbsphinx
Generate python documentation with Sphinx
This part of the post was inspired by the article Documenting Python code with Sphinx by Yash Salvi. I also looked at the youtube instruction Auto-Generated Python Documentation with Sphinx by avcourt. If you want to get a better grip on Sphinx I suggest that you follow the hands-on manual by Yash Salvi. The 3-part tutorial by Kay Jan Wong is more up to date.
Prepare your python package and modules
My python projects are built using clusters of packages linked together using a git hyperproject. I save the packages as separate GitHub repos. Consequently, the documentation is also created and saved on a per package (repo) basis. Thus the title of this section package in singular and modules in plural.
To make use of Sphinx you need to prepare your python modules by writing Docstrings.
Docstrings
Sphinx can be parameterised to produce different types of documentation (html, latex or linkcheck) with different formats and contents. But regardless of how you parameterise and which extensions or themes (extensions and themes are introduced further down) you use, you must have created strictly defined comments, or Docstrings for Sphinx to read.
For Sphinx to work, a typical python class (“class”) or function (“def”) must start with an initial, triple quote (""") enclosed Docstring. This comment is read and interpreted by Sphinx. As we are going to use the theme “read_the_docs” or rtd to generate the documentation in this tutorial, we need to use a syntax format that fits the rtd theme. We can choose between the default format, a google format or a numpy format. Kay Jan Wong summarises the differences in syntax.
Sphinx rtd theme default format
The Sphinx rtd theme standard format syntax (called sphinx) looks like the example below (see Sphinx-RTD-Tutorial on Writing Docstrings for details):
def ReadImportParamsJson(jsonFPN):
""" Read the parameters for importing OSSL data
:param jsonFPN: path to json file
:type jsonFPN: str
:return paramD: nested parameters
:rtype: dict
"""
with open(jsonFPN) as jsonF:
paramD = json.load(jsonF)
return (paramD)
Sphinx rtd theme numpy format
The numpy format for the Sphinx rtd theme is more comprehensive. But also more demanding. Here is the numpy format syntax using the same example as above:
def ReadImportParamsJson(jsonFPN):
""" Read the parameters for importing OSSL data
Parameters
----------
jsonFPN : str
path to json file.
Returns
-------
paramD
nested paramters
"""
with open(jsonFPN) as jsonF:
paramD = json.load(jsonF)
return (paramD)
The advantage with the numpy and google formats are that more complex information can be conveyed by extending the underscored heading system. For complex packages and packages assembled into projects, this is an advantage. The rest of this post will thus use the numpy format syntax.
Quickstart sphinx
Open a Terminal window. To do the following operations you need to activate the conda environment with sphinx, e.g.:
% conda activate sphinx
cd to the parent folder of the python package you want to document. Create a sibling folder to the actual package folder, and call the sibling docs.
(sphinx) pythonPackageParentFolder % mkdir docs
cd to the docs directory:
(sphinx) pythonPackageParentFolder % cd docs
Create the back-bone for a sphinx documentation by running sphinx-quickstart:
(sphinx) docs % sphinx-quickstart
A set of interactive questions follows, accept any default suggested and add the project name and yourself as the author.
docs % sphinx-quickstart
Welcome to the Sphinx 5.0.0 quickstart utility.
Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).
Selected root path: .
You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]:
The project name will occur in several places in the built documentation.
> Project name: sphinx-test
> Author name(s): Thomas G
> Project release []:
If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.
For a list of supported codes, see
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language.
> Project language [en]:
Creating file /Users/.../docs/conf.py.
Creating file /Users.../docs/index.rst.
Creating file /Users/.../docs/Makefile.
Creating file /Users/.../docs/make.bat.
Finished: An initial directory structure has been created.
You should now populate your master file /Users/.../docs/index.rst and create other documentation
source files. Use the Makefile to build the docs, like so:
make builder
where "builder" is one of the supported builders, e.g. html, latex or linkcheck.
Check out the directory and file structure created with the tree command:
% tree
.
├── Makefile
├── _build
├── _static
├── _templates
├── conf.py
├── index.rst
└── make.bat
To put on flesh (that is create an output document) on this back-bone, the following steps are required:
- edit the configuration file (conf.py),
- generate .rst files,
- edit the .rst files, and
- build (“make”) the output files.
Edit conf.py
The configuration file contains information and instructions for the generation of the rst files. Thus you need to make some edits in conf.py.
- Uncomment the three lines (while changing the last to have double dots):
- import os
- import sys
- sys.path.insert(0, os.path.abspath(‘..’))
- Edit project information (in case you did a typo):
- project = ‘sphinx-test’
- copyright = ‘2022, Thomas G’
- author = ‘Thomas G’
- Add extensions (in our case that should include):
- ‘sphinx.ext.autodoc’,
- ‘sphinx.ext.viewcode’,
- ‘sphinx.ext.napoleon’,
- ‘autodocsumm’
- Change the html_theme:
- html_theme = ‘sphinx_rtd_theme’
You should recognise the extension autodocsumm and the html_theme sphinx_rtd_theme as the two add-on packages we installed before. conf.py should now look like this:
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
# -- Project information -----------------------------------------------------
project = 'OSSL-import'
copyright = '2022, Thomas Gumbricht'
author = 'Thomas Gumbricht'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'autodocsumm'
]
autodoc_default_options = {"autosummary": True}
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
Generare restructred text (rst) files
To use the configuration that you set above and let sphinx create the content (but not the skin so to say), run the command sphinx-apidoc. We just want to create a set of output (option: -o) files; we want them to be saved in the directory we are in, and the source is the parent folder. The default extension for the output files is .rst. The command then becomes:
(sphinx) docs % sphinx-apidoc -o . ..
or if your python package/module to document is at another path:
(sphinx) docs % sphinx-apidoc -o . /path/to/python/package
If your terminal cursor is not in the docs sibling folder to the python package that you are documenting you need to adjust the command.
If you rerun the tree command you should see how sphinx-apidoc generated a new set of rst and (empty) directories :
% tree
.
├── Makefile
├── _build
├── _static
├── _templates
├── conf.py
├── index.rst
├── make.bat
├── maths.rst
└── modules.rst
Edit rst files
Designing the final documentation content, style and layout is done by editing the rst files. For our first trial we are just going to add the package modules (modules.rst) to the front (main) page (index.rst). Open index.rst and add the line
modules
.. sphinx-test documentation master file, created by
sphinx-quickstart on Wed Sep 28 14:29:36 2022.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to sphinx-test's documentation!
=======================================
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Generate output
You are now ready to generate the output, for instance html pages. To do that you run the make.bat file created by sphinx in the docs folder:
(sphinx) docs % make html
Regenerate output
If you make changes and want to regenerate the output, you should first clean the existing output:
(sphinx) docs % make clean html
followed by
(sphinx) docs % make html
Publish on GitHub
If you want to publish your documentation on GitHub, you should have a separate gh-pages repo just for the documentation where you need to add an empty file called .nojekyll in the root.
% touch .nojekyll
With the file .nojekyll in the root you can upload (git push) the local repo to your online gh-pages GitHub repo.
Resources
Documenting Python code with Sphinx by Yash Salvi.
Auto-Generated Python Documentation with Sphinx by avcourt.