Debugging and devx: tractor.devx#

Multi-process debugging that actually works: boot the tree with open_root_actor(debug_mode=True) (or pass it to open_nursery()) and any crash or explicit pause in any actor acquires a tree-global TTY lock and drops you into a pdbp-powered REPL — one actor at a time, SIGINT shielded, no garbled terminals. The top-level helpers below are the daily drivers; the rest of the toolbox lives under tractor.devx.

Pausing and post-mortems#

async tractor.pause(*, hide_tb=True, api_frame=None, shield=False, **_pause_kwargs)[source]#

A pause point (more commonly known as a “breakpoint”) interrupt instruction for engaging a blocking debugger instance to conduct manual console-based-REPL-interaction from within tractor’s async runtime, normally from some single-threaded and currently executing actor-hosted-trio-task in some (remote) process.

NOTE: we use the semantics “pause” since it better encompasses the entirety of the necessary global-runtime-state-mutation any actor-task must access and lock in order to get full isolated control over the process tree’s root TTY: https://en.wikipedia.org/wiki/Breakpoint

Parameters:

  • hide_tb (bool)

  • api_frame (FrameType | None)

  • shield (bool)

Return type:

None

tractor.pause_from_sync(hide_tb=True, called_from_builtin=False, api_frame=None, allow_no_runtime=False, **_pause_kwargs)[source]#

Pause a tractor scheduled task or thread from sync (non-async function) code.

When greenback is installed we remap python’s builtin breakpoint() hook to this runtime-aware version which takes care of all bg-thread detection and appropriate synchronization with the root actor’s Lock to avoid mult-thread/process REPL clobbering Bo

Parameters:

  • hide_tb (bool)

  • called_from_builtin (bool)

  • api_frame (FrameType | None)

  • allow_no_runtime (bool)

Return type:

None

Note

pause_from_sync() needs the greenback portal: boot with open_root_actor(maybe_enable_greenback=True) (mind the performance implications). With debug_mode on, the built-in breakpoint() is also remapped to a tractor-safe equivalent.

async tractor.post_mortem(*, tb=None, api_frame=None, hide_tb=False, **_pause_kwargs)[source]#

Our builtin async equivalient of pdb.post_mortem() which can be used inside exception handlers.

It’s also used for the crash handler when debug_mode == True ;)

Parameters:
Return type:

None

Deprecated since version 0.1.0a6: tractor.breakpoint() warns and simply calls pause() — use tractor.pause() (async) or tractor.pause_from_sync() in new code.

Crash handling for CLIs and sync entrypoints#

tractor.devx.open_crash_handler(catch={<class 'BaseException'>}, ignore={<class 'KeyboardInterrupt'>, <class 'trio.Cancelled'>}, hide_tb=True, repl_fixture=None, raise_on_exit=True)[source]#

Generic “post mortem” crash handler using pdbp REPL debugger.

We expose this as a CLI framework addon to both click and typer users so they can quickly wrap cmd endpoints which get automatically wrapped to use the runtime’s debug_mode: bool AND pdbp.pm() around any code that is PRE-runtime entry

  • any sync code which runs BEFORE the main call to trio.run().

Parameters:
tractor.devx.maybe_open_crash_handler(pdb=None, hide_tb=True, **kwargs)[source]#

Same as open_crash_handler() but with bool input flag to allow conditional handling.

Normally this is used with CLI endpoints such that if the –pdb flag is passed the pdb REPL is engaed on any crashes B)

Parameters:

Note

Both are sync context managers usable before (or without) trio.run() — wrap your CLI main() to get a post-mortem REPL on any uncaught exception instead of a bare traceback.

Runtime hang-hunting#

tractor.devx.enable_stack_on_sig(sig=Signals.SIGUSR1)[source]#

Enable stackscope tracing on reception of a signal; by default this is SIGUSR1.

HOT TIP: a task/ctx-tree dump can be triggered from a shell with fancy cmds.

For ex. from bash using pgrep and cmd-sustitution (https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution) you could use:

>> kill -SIGUSR1 $(pgrep -f <part-of-cmd: str>)

OR without a sub-shell,

>> pkill –signal SIGUSR1 -f <part-of-cmd: str>

Parameters:

sig (int)

Return type:

ModuleType

With stackscope integration enabled (also via open_root_actor(enable_stack_on_sig=True) or the TRACTOR_ENABLE_STACKSCOPE env var) a SIGUSR1 triggers a full trio task-tree dump from every actor — works on live, non-debug-mode trees too:

pkill --signal SIGUSR1 -f <part-of-your-cmd>

Dumps also tee to /tmp/tractor-stackscope-<pid>.log so you still get output under captured/CI stdio.

Lower-level debug plumbing#

tractor.devx.mk_pdb()[source]#

Deliver a new PdbREPL: a multi-process safe pdbp.Pdb-variant using the magic of tractor’s SC-safe IPC.

Our pdb.Pdb subtype accomplishes multi-process safe debugging by:

  • mutexing access to the root process’ std-streams (& thus parent process TTY) via an IPC managed Lock singleton per actor-process tree.

  • temporarily overriding any subactor’s SIGINT handler to shield during live REPL sessions in sub-actors such that cancellation is never (mistakenly) triggered by a ctrl-c and instead only by explicit runtime API requests or after the pdb.Pdb.interaction() call has returned.

FURTHER, the pdbp.Pdb instance is configured to be trio “compatible” from a SIGINT handling perspective; we mask out the default pdb handler and instead apply trios default which mostly addresses all issues described in:

The instance returned from this factory should always be preferred over the default pdb[p].set_trace() whenever using a pdb REPL inside a trio based runtime.

Return type:

PdbREPL

async tractor.devx.maybe_wait_for_debugger(poll_steps=2, poll_delay=0.1, child_in_debug=False, header_msg='', _ll='devx')[source]#
Parameters:
Return type:

bool

maybe_wait_for_debugger() is mainly useful in runtime/test code that must avoid tearing down a tree while a child still holds the global debug lock.

See also

Errors and cancellation types for the boxed error types you’ll inspect from the REPL, Runtime and spawning for the debug_mode / maybe_enable_greenback / enable_stack_on_sig boot flags on tractor.open_root_actor(), and “Native” multi-process debugging for the guided multi-actor REPL tour.