#9.8. Jupyter kernel integration

This section defines the rok kernel of the Jupyter surface.

#9.8.1. Kernel

Jupyter runs the rok kernel as a separate process and communicates over the ZeroMQ-based Jupyter wire protocol. The kernel announces protocol_version as "5.3" (the value jupyter_client expects; works with clients 5.3 and later). In this form all OS resources, i.e. files, network and external processes, are available.

The kernel runs on the same cell session as the terminal REPL and follows the rules of 9.7.0: each cell runs exactly once, bindings and items carry over to later cells, and a failed cell leaves no trace in the bindings. The kernel's engine is the same JIT session as the REPL.

#Notebook cells and REPL cells

A notebook cell is split into several REPL cells by the same rules the terminal REPL uses to split input, and they run in order. As in IPython, only the last value is displayed as the result. If a later part of a notebook cell fails, the effects of the earlier parts (the preceding REPL cells) remain.

#9.8.2. Local native kernel: operation

When the user selects "rok" as the notebook's kernel, Jupyter runs the command of the kernelspec (9.8.3) as a separate process.

rokc jupyter --connection-file <path>       # also accepts rokc options such as --extern
rok jupyter --connection-file <path>        # resolves dependencies, then runs rokc jupyter

The kernel reads the ZMQ ports and key from the connection file, then binds and listens on its five sockets. Once all five accept connections, it writes the line rokc jupyter: ready to its stderr, before output capture starts. Only the tcp transport is supported (no ipc), and CURVE encryption is not used. Messages are HMAC-signed with the connection file's key.

#Socket channels

#Message format

Each message is serialized as six or more ZMQ frames.

[b"u-u-i-d",         # zmq identity(ies)
 b"<IDS|MSG>",       # delimiter
 b"baddad42",         # HMAC signature (with key)
 b"{header}",          # header JSON
 b"{parent_header}",   # parent header JSON
 b"{metadata}",        # metadata JSON
 b"{content}",         # content JSON
]

The detailed format follows the Jupyter wire protocol standard. Every request is wrapped in iopub status: busy and status: idle.

#Supported messages

#Execution

On receiving an execute_request, the kernel sends:

1. iopub execute_input (code and execution count).
2. The cell's output after it finishes: stdout / stderr as iopub stream. Output is not sent incrementally during execution.
3. If there is a value, iopub execute_result: text/plain holding the same type: value as the REPL (9.7.7).
4. On failure, iopub error: evalue is the first line of stderr, traceback is the lines of stderr.
5. shell execute_reply (including execution_count, with success / failure status).

The only output MIME type is text/plain.

#9.8.3. kernelspec installation

Jupyter only recognizes kernels installed in a kernelspec directory.

rok jupyter install [--env <name>]

The kernelspec is kernels/rok/kernel.json under the directory below. Installed with --env <name>, it is kernels/rok-<name>/, with display name rok (<name>) and command rok --env <name> jupyter ....

1. $JUPYTER_DATA_DIR if set
2. ~/Library/Jupyter on macOS
3. Otherwise $XDG_DATA_HOME/jupyter, or ~/.local/share/jupyter if unset

kernel.json runs the rok proxy (9.10.6) and records the ROK_HOME and PATH at installation time in env.

{
  "argv": ["/home/u/.rok/bin/rok", "jupyter", "--connection-file", "{connection_file}"],
  "display_name": "rok",
  "language": "rok",
  "interrupt_mode": "message",
  "env": { "ROK_HOME": "/home/u/.rok", "PATH": "<PATH at install time>" }
}

Once installed, the rok kernel is usable from Jupyter, VS Code, nteract and others.

#9.8.4. Interrupt / shutdown

#Interrupt

Since interrupt_mode is "message", an interrupt arrives as an interrupt_request on the control socket. The running cell is terminated and the session continues from the state before that cell (as for a failed cell, 9.7.0). A cell still being compiled cannot be interrupted.

#Shutdown (kernel restart)

After replying to a shutdown_request on the shell or control socket, the kernel cleans up the session and exits the process. On restart, Jupyter launches it again automatically.

#9.8.5. Code completion

complete_request matches the fragment before the cursor against the session's bindings, items, module paths, members and keywords, and replies with { matches, cursor_start, cursor_end }.

#9.8.6. inspect_request: help

inspect_request (Shift+Tab) replies with found: true and, as text/plain, the name at the cursor as name: type, its signature and doc comments (/// / //! of Lexical structure). For an unknown name it is found: false.

#9.8.7. Code sharing between REPL and Jupyter

Code written in the REPL can be moved to Jupyter cells unchanged, and vice versa, because both follow the same cell session and the same semantics (9.7.0).

#Differences

The two surfaces differ in a few respects, summarized below.

#9.8.8. Security / isolation

The kernel runs arbitrary code with the user's privileges. No isolation is performed.

#9.8.9. Next

The following Debug / diagnostic tools defines gdb-friendly features, profiling, --emit and more.

Edit this page on GitHub