TL;DR I have been thinking about alternative frontends to the Jupyter notebook. The first step is to abstract communications with Jupyter kernels. I have started writing one of these.
I am a huge fan of the Jupyter notebook environment. It is a fantastic way to explore a new data analysis approach, and keep results together with the implementation and explanation.
I wanted to understand how the Jupyter notebook/lab frontend interacts with the Kernel backends, and as such have started writing a client library.
The Jupyter suite consists of two parts:
Within the frontend is typically a kernel communication abstraction component, what I call the “client”:
The frontend handles nicely presenting the execution results and collecting input from the user (i.e. the notebook). The kernel is the execution engine that can be written in many languages. The client handles communicating with a kernel. It communicates over 41 ZeroMQ sockets:
shell: for client -> kernel requests (and responses)
iopub: for kernel -> client messages
heartbeat: for health checks
stdin: for kernel -> client requests (and responses from the frontend)
This abstraction is what allows the frontend to communicate with different kernels.
Writing a client library
The documentation on the Jupyter wire protocol is pretty good at describing how messages are sent to and from the kernel over the “wire” (the various ZeroMQ sockets). This documentation was used to build a
WireMessage abstraction, which:
- is created from a
Commandenum variant (conversion method),
- gets passed to a
Socket; an abstraction around a ZeroMQ socket,
- gets sent over the wire to the kernel,
- a response is sent back and converted to a
WireMessage(conversion performed here),
- which is then turned back into a
Responseobject and generally passed back to the user.
I hope that this library will form the beginning of a new Rust-based Jupyter frontend. I have some ideas but that’s for another time.
- Project repository
- Description of the Jupyter wire protocol
- Creating language kernels for IPython
Technically 5: a “control” socket is a duplicate of the “shell” socket but for high-priority messages. ↩︎