Configure the kernel to have packages automatically installed

[martinRenou]

Problem

It is currently not possible to automatically install Python packages inside the pyolite kernel.

Proposed Solution

It would be nice to be able to configure Pyolite to include some packages that would automatically be installed with micropip.

cc. @psychemedia who suggested it here https://github.com/jtpio/jupyterlite-demo/pull/7#issuecomment-870144941

Additional context

It might be possible to have it once we use IPython for the execution (see related work https://github.com/jtpio/jupyterlite/pull/171), as it's possible to configure IPython itself to run code at the shell start.

[bollwyvl]

I feel we need to figure out some way to get these hosted with the build, get the service worker going, use dat/ipfs, etc. before installing more stuff magically. Spamming pypi with requests without any kind of caching isn't very nice... They probably weren't signing up to be a static cdn for web page assets.

[jtpio]

Maybe one way could be to make that part that @psychemedia linked to in https://github.com/jtpio/jupyterlite-demo/pull/7#issuecomment-870144941 configurable at build time:

https://github.com/jtpio/jupyterlite/blob/5225266445d65c05cc2f545fceb1fa9f44380709/packages/pyolite-kernel/src/worker.ts#L20-L25

To download the wheels only once and place them at the right location.

But that would be specific to the pyolite kernel, and the jupyterlite toolchain should ideally be kernel agnostic.

[bollwyvl]

I haven't dug enough into micropip, but presumably it can accept additional (preferred) index pages... given a list of wheels, that should be easy enough to add. What we don't want to do is end up having the maintenance burden of little PRs for every little patch/build someone needs.

At any rate, wheels will get special treatment in the toolchain anyway as they are the most reliable source of labextensions, as we can predict precisely where the assets will be inside them... until we have a wasm-32 target from conda-forge, which would probably be the most desirable way for all non-basically-pure JS kernels. Further, there are a number of wrapper kernels that reuse ipykernel, etc. which would be able to use wheel-based mechanisms.

[psychemedia]

One of the things I hacked to get the folium demo working was a prebuilt universal wheel for one of the packages served from a Github page: https://github.com/OpenComputingLab/vce-wheelhouse

import micropip
await micropip.install("https://opencomputinglab.github.io/vce-wheelhouse/wheelhouse/MarkupSafe-2.0.1-py2.py3-none-any.whl")

It's probably a good idea to separate out the ideas of installing from pypi, from a JupyterLite wheelhouse, or from a local wheel.

In building an http servable distribution, one approach might be:

• for the CLI builder to grab an appropriate wheel and bundle it with the distribution;
• micropip to then install it from the local wheel.

If micropip can only install from a URL, then the distribution could publish its own wheelhouse from which the the wheel could then be installed. Of course, other folk then might start using your website as a wheelhouse...

[rth]

Spamming pypi with requests without any kind of caching
isn't very nice... They probably weren't signing up to be a static cdn for
web page assets.

Installs with micropip should be cached by the browser same as when downloading any other Pyodide package.

Also,

$ dig files.pythonhosted.org
...
files.pythonhosted.org. 67618   IN      CNAME   dualstack.r.ssl.global.fastly.net.

so the requests are being received by Fastly, which is really not that different from mirroring those files via some other CDN (including JsDelivr which will end up also using Fastly). Compared to the overall PyPi traffic I think this is still negligble, but we can certainly ask them this question.

Generally if you have question or feature requests about micropip, please open an issue in the pyodide repo.

[RandomFractals]

well, I ran a test with pyolite in vscode, and also question the need to load all of those libraries for simple notebooks that don't use most of them: joyceerhl/vscode-pyolite#2 (comment)

this is what's being loaded for a simple notebook with 2 imports to load json data: https://github.com/RandomFractals/vscode-data-table/blob/main/notebooks/chicago-red-light-cameras.ipynb

import json
from js import fetch

response = await fetch('https://data.cityofchicago.org/resource/thvf-6diy.json')
text = await response.text()
data = json.loads(text)
data

Why do we need to load matplotlib and other libraries pyodide has configured by default?

I think they should be loaded dynamically based on imports in the notebook cell code. It's exactly what ObservableHQ and other intelligent JS notebook platforms do since dynamic modules loading is readily available in most web browsers now.

Have you considered optimizing your kernel initialization with just the basics without adding all the other dataviz packages most devs and data scientists might not be using in their pyodide notebooks.

[jtpio]

@RandomFractals for some libraries it's a bit trickier as they require patches to be rendered properly in output cells in JupyterLite.

Have you considered optimizing your kernel initialization with just the basics without adding all the other dataviz packages most devs and data scientists might not be using in their pyodide notebooks.

Yes, see jupyterlite/jupyterlite#239 as an example. Unfortunately there were some issues with the recursion limit in upstream Pyodide, so the change had to be reverted. But hopefully will come back at some point: jupyterlite/jupyterlite#254

[bollwyvl]

ain't nothing free... there isn't a 1-1 mapping of imported names to installable python packages, and the need for patching is very real.

but we certainly need some better approaches for handling these things: there's a sketch of a path forward over on jupyterlite/jupyterlite#151 (comment) and i'll get around to it when i can, but there's also a lot of other stuff on our plates.... and host environments other than real browsers are going to be best-effort for the foreseeable.

[RandomFractals]

@bollwyvl understood. I was just sharing my first impression after trying it in vscode. I will try to find some time soon to provide good data viz examples and notebooks for a deeper dive. I think porting some of these real data notebooks could be fun to validate and troubleshoot some of the hooks you are working on: https://github.com/RandomFractals/Chicago-transportation-notebooks

Other than those extraneous js libs loading nitpicks, I like what you've created so far, and def. plan on using your Py lite stack in browser and vscode.

[RandomFractals]

I added vscode Pyolite example docs here: https://github.com/RandomFractals/vscode-data-table#pyolite-notebook-example