sphinxcontrib-jupyter Documentation

This sphinx extension can be used to build a collection of Jupyter notebooks for Sphinx Projects.

Note

It has mainly been written to support the use case of scientific publishing and hasn’t been well tested outside of this domain. Please provide feedback as an issue to this repository.

Requires: Sphinx >= 1.7.2 (for running tests).

One of the main benefits of writing Jupyter notebooks as RST files is to simplify the task of version control for large projects.

Installation

To install the extension:

pip install sphinxcontrib-jupyter

to upgrade your current installation to the latest version:

pip install --upgrade sphinxcontrib-jupyter

Todo

Add installation to distribute via conda-forge. See Issue #160.

You can refer to the release notes for information on each release.

Alternative

Another way to get the latest version it is to install directly by getting a copy of the repository:

git clone https://github.com/QuantEcon/sphinxcontrib-jupyter

and then use

python setup.py install

Developers

For developers it can be useful to install using the develop option:

python setup.py develop

this will install the package into the site-wide package directory which is linked to the code in your local copy of the repository. It is not recommended to install this way for common use.

Sphinx Setup

To initially setup a Sphinx project, please refer here.

Note

QuantEcon has developed a jupinx-quickstart that can assist with setting up a repository to get you up and running quickly. Please refer to the Jupinx Project for more details.

You can use the sphinx quickstart and update the project conf.py file to include the jupyter extension and add the desired configuration settings (see Extension Configuration section for details):

extensions = ["sphinxcontrib.jupyter"]

once the extension is installed you can then run:

make jupyter

The Extension Configuration section includes details on how to configure the extension in your conf.py.

Extension Configuration and Options

The options are split into the different parts of the compilation pipeline that are available in this extension:

Constructing Jupyter Notebooks

jupyter_conversion_mode

Specifies which writer to use when constructing notebooks.

Option Description
“all” (default) compile complete notebooks which include markdown cells and code blocks
“code” compile notebooks that only contain the code blocks.

conf.py usage:

jupyter_conversion_mode = "all"

jupyter_static_file_path

Specify path to _static folder.

conf.py usage:

jupyter_static_file_path = ["source/_static"]

jupyter_header_block

Add a header block to every generated notebook by specifying an RST file

conf.py usage:

jupyter_header_block = ["source/welcome.rst"]

jupyter_default_lang

Specify default language for collection of RST files

conf.py usage:

jupyter_default_lang = "python3"

jupyter_lang_synonyms

Specify any language synonyms.

This will be used when parsing code blocks. For example, python and ipython have slightly different highlighting directives but contain code that can both be executed on the same kernel

conf.py usage:

jupyter_lang_synonyms = ["pycon", "ipython"]

jupyter_kernels

Specify kernel information for the jupyter notebook metadata.

This is used by jupyter to connect the correct language kernel and is required in conf.py.

conf.py usage:

jupyter_kernels = {
    "python3": {
        "kernelspec": {
            "display_name": "Python",
            "language": "python3",
            "name": "python3"
            },
        "file_extension": ".py",
    },
}

Todo

See Issue 196

jupyter_write_metadata

write time and date information at the top of each notebook as notebook metadata

Note

This option is slated to be deprecated

jupyter_options

An dict-type object that is used by dask to control execution

Todo

This option needs to be reviewed

jupyter_drop_solutions

Drop code-blocks that include :class: solution

Values
False (default)
True

Todo

This option needs to be reviewed

jupyter_drop_tests

Drop code-blocks` that include ``:class: test

Values
False (default)
True

Todo

This option needs to be reviewed

jupyter_ignore_no_execute:

Values
False (default)
True

When constructing notebooks this option can be enabled to ignore :class: no-execute for code-blocks. This is useful for html writer for pages that are meant to fail but shouldn’t be included in coverage tests.

conf.py usage:

jupyter_ignore_no_execute = True

jupyter_ignore_skip_test

When constructing notebooks this option can be enabled to ignore :class: skip-test for code-blocks.

Values
False (default)
True

conf.py usage:

jupyter_ignore_skip_test = True

jupyter_allow_html_only

Enable this option to allow .. only:: html pass through to the notebooks.

Values
False (default)
True

conf.py usage:

jupyter_allow_html_only = True

jupyter_target_html

Enable this option to generate notebooks that favour the inclusion of html in notebooks to support more advanced features.

Values
False (default)
True

Supported Features:

  1. html based table support
  2. image inclusion as html figures

conf.py usage:

jupyter_target_html = True

jupyter_images_markdown

Force the inclusion of images as native markdown

Values
False (default)
True

Note

when this option is enabled the :scale: option is not supported in RST.

conf.py usage:

jupyter_images_markdown = True

jupyter_dependencies

Specify file or directory level dependencies

conf.py usage:

jupyter_dependencies = {
    <dir> : ['file1', 'file2'],
    {<dir>}/<file.rst> : ['file1']
}

this allows you to specify a companion data file for a given RST document and it will get copied through sphinx to the _build folder.

Executing Notebooks

jupyter_execute_nb

Enables the execution of generated notebooks

Values
False (default)
True

Todo

deprecate this option in favour of jupyter_execute_notebooks

jupyter_execute_notebooks

Enables the execution of generated notebooks

Values
False (default)
True

conf.py usage:

jupyter_execute_notebooks = True

jupyter_dependency_lists

Dependency of notebooks on other notebooks for execution can also be added to the configuration file above in the form of a dictionary. The key/value pairs will contain the names of the notebook files.

conf.py usage:

# add your dependency lists here
jupyter_dependency_lists = {
   'python_advanced_features' : ['python_essentials','python_oop'],
   'discrete_dp' : ['dp_essentials'],
}

jupyter_dependencies

Specify support (dependencies) for notebook collection at the file or the directory level.

conf.py usage:

jupyter_dependencies = {
    <dir> : ['file1', 'file2'],
    {<dir>}/<file.rst> : ['file1']
}

Note

to specify a support file at the root level of the source directory the key should be “”

jupyter_number_workers

Specify the number cores to use with dask

Values
Integer (default = 1)

conf.py usage:

jupyter_number_workers = 4

jupyter_threads_per_worker

Specify the number of threads per worker for dask

Values
Integer (default = 1)

conf.py usage:

jupyter_threads_per_worker = 1

Converting Notebooks to HTML

jupyter_generate_html

Enable sphinx to generate HTML versions of notebooks

Values
False (default)
True

conf.py usage:

jupyter_generate_html = True

jupyter_html_template

Specify path to nbconvert html template file

Note

Documentation on nbconvert templates can be found here

conf.py usage:

jupyter_html_template = "theme/template/<file>.tpl"

jupyter_make_site

Enable sphinx to construct a complete website

Todo

Document all the extra elements this option does over jupyter_generate_html

This option:

  1. fetches coverage statistics if coverage is enabled.

conf.py usage:

jupyter_make_site = True

jupyter_download_nb

Request Sphinx to generate a collection of download notebooks to support a website

conf.py usage:

jupyter_download_nb = True

jupyter_download_nb_images_urlpath

Apply a url prefix when writing images in Jupyter notebooks for download notebook set.

conf.py usage:

jupyter_images_urlpath = "s3://<path>/_static/img/"

jupyter_theme

Specify theme name

conf.py usage:

jupyter_theme = <theme-name>

The theme should be located in the path of jupyter_theme_path. The default path would be: theme/<theme-name>/

jupyter_theme_path

Specify location for theme files

Value
“theme” (default)

conf.py usage:

jupyter_theme_path = "theme"

jupyter_template_path

Specify path for templates

Value
“templates” (default)

conf.py usage:

jupyter_template_path = "templates"

jupyter_template_html

Specify html template to be used by nbconvert

conf.py usage:

jupyter_template_html = <path to tpl file>

The template file should be located in the path of jupyter_template_path. The default path would be: templates/<tpl file>

Computing Coverage Statistics

Warning

make coverage will currently produce a json report that can be used to support an execution status page. But adding badges and a status page needs to be made into an option and specified via the default theme. See #237

jupyter_make_coverage

Enable coverage statistics to be computed

Values
False (default)
True

jupyter_template_coverage_file_path

Provide path to template coverage file

Todo

Document format for template

conf.py usage:

jupyter_template_coverage_file_path = "theme/templates/<file>.json"

Configuration for Exercise Directives

exercise_include_exercise

Enable inclusion / exclusion of exercises directives contained in the rst

Values
True (default)
False

conf.py usage:

exercise_include_exercise = False

exercise_inline_exercises

Add inline exercises as a formatting choice

Values
False (default)
True

If the config variable exercise_inline_exercises is set to false (the default), then where ever the exercise initially appeared in the body of the document, you will instead see a link that says See exercise # which refers to an ExerciseList.

Converting Notebooks to PDF

jupyter_latex_template

Provide path to latex nbconvert template file

conf.py usage:

jupyter_latex_template = "theme/templates/latex.tpl"

jupyter_bib_file

Provide path to bibtex file for reference support

conf.py usage:

jupyter_bib_file = "_static/references.bib"

jupyter_pdf_author

Specify Author Field for PDF document

conf.py usage:

jupyter_pdf_author = "QuantEcon Developers"

jupyter_pdf_showcontentdepth

Specify which depth of the local contents directives to add to generated pdf files

Values
2 (Default)

Note

this shows second level contents by default as the pdf document adds the page or document title to the top of the article format.

jupyter_pdf_urlpath

Enable local links within the project to link a hosted located via a urlprefix and link modification.

conf.py usage:

jupyter_pdf_urlpath  = "https://lectures.quantecon.org/"

jupyter_pdf_excludepatterns

Exclude certainly documents from getting compiled as pdf files.

conf.py usage:

jupyter_pdf_excludepatterns = ["index", "404", "search"]

This can be useful for make site when pdf construction is part of a broader project that supports html targets.

This extension also offers additional directives that can be used while writing your documents

Directives

The following directives are provided by this extension:

Directive: exercise

Exercise directives can be added to your text such as:

.. exercise::

    Contains the Exercise

If you would like to exclude them from your documents you can set:

exercise_include_exercises = False``

in your conf.py file.

Directive: exerciselist

Collect exercises from the document and compile an exercise list where the directive is placed in the RST

.. exerciselist::

Beneath each exercise in the list there is another link “(back to text)” that points back to the location where you originally authored/placed the exercise.

Provided Options
.. exerciselist::
    :scope: SCOPE
    :from: FILE
    :force:
Option Description Values
:from: import exercise defined in a file  
:scope: provide scope of exercises file, section, or all
:force: force exercises to render where they are defined True/False
:title: Specify title used for exercise block  
:scope:

file then only exercises defined in the same file as the exerciselist directive will be included section then all exercises defined in the same section will be included all then all exercises anywhere in the project will be included

:force:

By default, when the conf.py config setting exercise_inline_exercises is true, all exercises will render where they are defined and the exerciselist node will be removed from the doctree.

Warning

However, if the :force: option is given the exerciselist will always be rendered and when exercise_inline_exercises is True, each problem will be rendered twice.

Additional Info

If an exercise is included in an exercise list, it is only removed from its original location if the exercise list is in the same file.

For example, if I define an exercise in A.rst and in B.rst I both define an exercise and an exerciselist with :scope: section then the following will happen:

  • both exercises will render at the point of the exerciselist in B.rst
  • The exercise in B.rst will not be rendered where it was defined, but instead a link to the exercise list and number will be given
  • The exercise in A.rst will still be rendered in file A. This means its contents are rendered two times.

It can also be useful to have multiple configurations when working on a large project, such as generating notebooks for working on locally while compiling the project for HTML in a deployment setting.

Further details on how to manage large projects can be found here.

An example conf.py is available here

Example conf.py file

After running a sphinx-quickstart you can add the jupyter options needed for your project in a similar fashion to what is shown belows.

The below configuration settings are the default ones provided by the jupinx quickstart tool

# Configuration file for the Jupinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))


# -- Project information -----------------------------------------------------

project = 'DEMO'
copyright = '2019, AUTHOR'
author = 'AUTHOR'

# The short X.Y version
version = '0.1'

# The full version, including alpha/beta/rc tags
release = '0.1'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    'sphinxcontrib.jupyter',
    'sphinxcontrib.bibtex',
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'

# The master toctree document.
master_doc = 'index'

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['static']


# -- Extension configuration -------------------------------------------------

# -- jupyter build configuration ---------------------------------------------------
jupyter_kernels = {
    'python3': {
        'kernelspec': {
            'display_name': 'Python',
            'language': 'python3',
            'name': 'python3'
        },
        'file_extension': '.py'
    },
    'python2': {
        'kernelspec': {
            'display_name': 'Python',
            'language': 'python2',
            'name': 'python2'
        },
        'file_extension': '.py'
    },
    'julia-1.1': {
        'kernelspec': {
            'display_name': 'Julia 1.1',
            'language': 'julia',
            'name': 'julia-1.1'
        },
        'file_extension': '.jl'
    }
}

# --------------------------------------------
# jupyter Sphinx Extension conversion settings
# --------------------------------------------

# Conversion Mode Settings
# If "all", convert codes and texts into notebook
# If "code", convert codes only
jupyter_conversion_mode = "all"

jupyter_write_metadata = False

# Location for _static folder
jupyter_static_file_path = ["source/_static"]

# Configure jupyter headers
jupyter_headers = {
    "python3": [
        # nbformat.v4.new_code_cell("%autosave 0")      #@mmcky please make this an option
        ],
    "julia": [
        ],
}

# Filename for the file containing the welcome block
jupyter_welcome_block = ""

#Adjust links to target html (rather than ipynb)
jupyter_target_html = False

#path to download notebooks from
jupyter_download_nb_urlpath = None

#allow downloading of notebooks
jupyter_download_nb = False

#Use urlprefix images
jupyter_download_nb_image_urlpath = None

#Allow ipython as a language synonym for blocks to be ipython highlighted
jupyter_lang_synonyms = ["ipython"]

#Execute skip-test code blocks for rendering of website (this will need to be ignored in coverage testing)
jupyter_ignore_skip_test = True

#allow execution of notebooks
jupyter_execute_notebooks = False

# Location of template folder for coverage reports
jupyter_template_coverage_file_path = False

# generate html from IPYNB files
jupyter_generate_html = False

# html template specific to your website needs
jupyter_html_template = ""

# latex template specific to your website needs
jupyter_latex_template = ""

#make website
jupyter_make_site = False

#force markdown image inclusion
jupyter_images_markdown = True

#This is set true by default to pass html to the notebooks
jupyter_allow_html_only=True

Managing Large Projects

Large projects may require different build pathways due to the time required for execution of embedded code. This can be done by modifying the Makefile to accommodate multiple build pathways.

You may, for example, wish to leave make jupyter simply building notebooks while setting up an alternative make command to target a full website build.

In the Makefile you can add an alternative build target such as:

BUILDWEBSITE  = _build/website

and then you can modify options (set in the conf.py file) using the -D flag.

website:
    @$(SPHINXBUILD) -M jupyter "$(SOURCEDIR)" "$(BUILDWEBSITE)" $(SPHINXOPTS) $(O) -D jupyter_make_site=1 -D jupyter_generate_html=1 -D jupyter_download_nb=1 -D jupyter_execute_notebooks=1 -D jupyter_target_html=1 -D jupyter_images_markdown=0 -D jupyter_html_template="theme/templates/lectures-nbconvert.tpl" -D jupyter_download_nb_urlpath="https://lectures.quantecon.org/"

this will setup a new folder _build/website for the new build pathway to store resultant files from the options selected. See Builders for further details.

Note

This method also preserves the sphinx cache mechanism for each build pathway.

Warning

Issue #199 will alter this approach to include all configuration settings in the conf.py file and then the different pipelines can be switched off in the Makefile which will be less error prone.

Builders

This extension has the following Builders

jupyter

This builder currently handles jupyter, html, and coverage output

@$(SPHINXBUILD) -M jupyter "$(SOURCEDIR)" "$(BUILDCOVERAGE)" $(FILES) $(SPHINXOPTS) $(O)

Warning

If your project needs to build jupyter and html then configuration for html and coverage is currently handled through Makefile overrides. The project is working on separate builders for html and coverage

Example Configuration for HTML production

@$(SPHINXBUILD) -M jupyter "$(SOURCEDIR)" "$(BUILDWEBSITE)" $(FILES) $(SPHINXOPTS) $(O) -D jupyter_make_site=1 -D jupyter_generate_html=1 -D jupyter_download_nb=1 -D jupyter_execute_notebooks=1 -D jupyter_target_html=1 -D jupyter_download_nb_image_urlpath="https://s3-ap-southeast-2.amazonaws.com/lectures.quantecon.org/py/_static/" -D jupyter_images_markdown=0 -D jupyter_html_template="python-html.tpl" -D jupyter_download_nb_urlpath="https://lectures.quantecon.org/" -D jupyter_coverage_dir=$(BUILDCOVERAGE)

jupyterpdf

This builder handles production of pdf

@$(SPHINXBUILD) -M jupyterpdf "$(SOURCEDIR)" "$(BUILDCOVERAGE)" $(FILES) $(SPHINXOPTS) $(O)

Credits

This project is supported by QuantEcon

Many thanks to the lead developers of this project.

Contributors

Projects using Extension

  1. QuantEcon Lectures

If you find this extension useful please let us know at contact@quantecon.org

LICENSE

Copyright © 2019 QuantEcon Development Team: BSD-3 All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Indices and tables