JupyOps
Install jupyter
with pipx
First, issues with other methods I have tried:
- Using system's package manager: some extensions may not be actively maintained (example (opens in a new tab))
- Using project's venv or conda env: have to duplicate the setup in each environment
What I settled on for now is using pipx
to manage a separate virtual environment for everything Jupyter. Here is how I do it:
- Install the metapackage that pulls in all Jupyter components
pipx install jupyter --include-deps pipx ensurepath
- Inject other useful things into the
jupyter
venvpipx inject jupyter ipywidgets pipx inject jupyter jupyterlab_widgets pipx inject jupyter jupyterlab-vim pipx inject jupyter jupyterlab-git pipx inject jupyter jupyterlab-lsp 'python-lsp-server[all]' pipx inject jupyter jupyterlab-code-formatter black
To keep the packages updated, run
pipx upgrade-all
If the system's Python is updated, you may also need to re-install packages under pipx
pipx reinstall-all
Manage kernel registry
By default, JupyterLab only has access to the ipython
kernel from the same virtual environment. To add another kernel, run the following command using the ipython
you would like to register
<your kernel path>/ipython kernel install --user --name=<your kernel name>
For example, this adds the systems's ipython
kernel to the kernel selection list in the Notebook/Lab UI.
/bin/ipython kernel install --user --name=system
To make ipywidgets
work, it also needs to be installed in each kernel's environment, e.g., for conda
,
conda install -c conda-forge ipykernel ipywidgets
To see a list of registered kernels
jupyter kernelspec list
To remove a registered kernel from the list, run
jupyter kernelspec uninstall <unwanted kernel>
(uninstall
and remove
are equivalent here).
Avoid Git repository bloating
When added to a git
repository, the notebooks' output can be filtered out to avoid bloating.
- Add a filter to repo-wise git config (use
--global
flag to make it global). We only need to define theclean
command, which is used to convert the contents of worktree file upon checkin.git config filter.strip-notebook-output.clean \ 'jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR'
- Tell git to apply this filter to
.ipynb
filesecho '*.ipynb filter=strip-notebook-output' >> .git/info/attributes
- If the repo has existing unscrubbed notebooks, add them to be renormalized
and then commit.
git add --renormalize .
Note that the filter only cleans the checked-in notebooks, the local worktree remains untouched.