Hot tips for `tractor` hackers#

This is a (perpetually WIP) guide for newcomers to the project, mostly to do with dev, testing, CI and release gotchas, reminders and best practises.

tractor is a fairly novel project compared to most since it is effectively a new way of doing distributed computing in Python and is much closer to working with an “application level runtime” (like erlang OTP or scala’s akka project) than it is a traditional Python library. As such, having an arsenal of tools and recipes for figuring out the right way to debug problems when they do arise is somewhat of a necessity.

Making a release#

Nothing fancy: the traditional PyPA flow on the hatchling build backend, with uv doing the driving and towncrier generating the changelog.

collect news fragments: user-facing changes should land with a small .rst snippet under nooz/ (see nooz/HOWTO.rst; fragment types are feature, bugfix, doc and trivial),

render them into NEWS.rst:

uvx towncrier build --version <version>

build and upload (testpypi first if you’re being careful):

uv build
uvx twine upload -r testpypi dist/*
uvx twine upload dist/*

How you organize built artifacts under dist/ locally (per release sub-dirs and such) is entirely up to you.

Keep in mind that PyPi releases tend to lag the main branch since we develop in the open — main is usually the thing to run when you want the latest.

Debugging and monitoring actor trees#

Your “what is my tree doing right now?” toolbox, in escalation order:

Live process-tree view — keep a watch-ed pstree running in a side terminal; actor procs are recognizable by their _subactor[<name>@<uuid-prefix>] process titles. The exact one-liners (plus the pgrep marker recipes) live in Testing tips.

SIGUSR1 task-tree dumps — boot any tree with enable_stack_on_sig=True (or export TRACTOR_ENABLE_STACKSCOPE=1) and every actor installs a stackscope signal handler. Then from any shell:

# dump every actor's live trio task tree:
pkill --signal SIGUSR1 -f <part-of-your-cmd>

# or for a single process:
kill -SIGUSR1 $(pgrep -f <part-of-your-cmd>)

Each dump is also tee’d (append-mode) to /tmp/tractor-stackscope-<pid>.log so you still get output under pytest capture or in CI. This works without debug-mode being enabled — it’s the lightest-weight hang-investigation tool in the box.

The built-in multi-process debugger — debug_mode=True plus tractor.pause() and friends: the heavyweight champ for interactive, REPL-driven inspection of a whole tree (including crash handling). Remember pytest capture interplay — see Testing tips.

Post-mortem zombie sweeps — scripts/tractor-reap for the (should-be-rare!) cases where hacking on the runtime itself wedges a tree: a SIGINT-first, structured concurrency (SC) polite escalation, plus --shm and --uds leaked-resource sweeps.

Using the log system to trace `trio` task flow#

The logging system is oriented around the stack “layers” of the runtime, letting you trace logical abstraction layers in the code — errors, cancellation, IPC and streaming, the low level transport and wire protocols — independently of one another.

Concretely, tractor.log.get_logger() returns a StackLevelAdapter sporting extra level-methods beyond the stdlib set, including:

.cancel() — cancellation-machinery flow,
.runtime() — actor-runtime lifecycle chatter,
.devx() — debugger/devx tooling internals,
.transport() — wire-level msging events.

To get console output at any level from your own code:

from tractor.log import get_console_log
get_console_log('cancel')

or, runtime-wide without touching code, just export TRACTOR_LOGLEVEL=cancel (the env-var wins over caller-passed levels; great for test runs).

When you want only one subsystem cranked, the suite’s --tl flag (and tractor.log.apply_logspec()) accept a per-sublogger spec:

uv run pytest tests/... --tl 'devx:runtime,trionics:cancel'

Every record’s header includes the emitting actor and task names, so cross-process flows can be stitched back together by eyeball (or grep).

Building these docs#

The site you’re reading is sphinx + the pydata-sphinx-theme, with diagrams authored in d2 (via our local .. d2:: directive) and every code block literalinclude-d straight from examples/ — so what you read is what CI runs, and it can’t rot.

The one-liner, from any dev shell:

uv run --group docs make -C docs html

then open docs/_build/html/index.html.

Nix users: the d2 diagram renderer is deliberately kept out of the default dev-shell so casual envs stay lean; it lives in an opt-in docs shell:

# enter the docs shell (puts `d2`, `uv` + python on PATH)
nix develop .#docs

# ...then build (diagrams re-render from docs/diagrams/*.d2)
uv run --group docs make -C docs html

or as a one-shot without staying in the shell:

nix develop .#docs -c uv run --group docs make -C docs html

Live-reload while editing — rebuild + browser refresh on every save:

nix develop .#docs -c uv run --with sphinx-autobuild \
    --group docs sphinx-autobuild docs docs/_build/html
# then open http://127.0.0.1:8000

Share on your LAN: bind the server to all interfaces (--host 0.0.0.0) instead of localhost, then hand a peer on your subnet http://<your-lan-ip>:8000:

nix develop .#docs -c uv run --with sphinx-autobuild \
    --group docs sphinx-autobuild docs docs/_build/html \
    --host 0.0.0.0 --port 8000

or statically serve an already-built tree (no rebuild-on-save):

python -m http.server -d docs/_build/html --bind 0.0.0.0 8000

hostname -I prints the IP to share; note it’s an unauthenticated server bound to every interface, so keep it to a trusted LAN.

How the diagrams resolve,

.d2 sources live in docs/diagrams/; their rendered SVGs are git-committed under docs/_diagrams/ as a fallback,
with a d2 binary on PATH (the docs shell, or set D2_BIN='nix run nixpkgs#d2 --') any stale SVG re-renders at build time,
with NO binary the committed SVGs are served as-is, so CI and casual builds need no d2 at all,
a .d2 that fails to compile is a hard build error under sphinx-build -W (the last-good committed SVG is left untouched).

The build is currently warning-free; keep it that way — the -W flag turns any sphinx warning into a hard failure:

uv run --group docs sphinx-build -b html -W docs docs/_build/html

A terser command cheat-sheet also lives at notes_to_self/howtodocs.md.

Hot tips for tractor hackers#