Debug Running Asyncio Services with pstree and ps in Python 3.14

An asynchronous service can be alive while making no useful progress.

The process still responds to signals. CPU usage may be low. The event loop is still running. Yet a request, worker, or shutdown path appears stuck somewhere inside a chain of coroutines.

Traditional stack traces are only part of the answer. An asyncio application is organized around tasks and await relationships, so the useful question is often not merely “where is this thread?” but “which task is waiting for which other task?”

Python 3.14 adds command-line introspection for exactly that situation. A separate Python process can inspect a running process with:

python -m asyncio pstree PID
python -m asyncio ps PID

These commands expose the pending task graph without requiring the target application to provide a debug HTTP endpoint or to be restarted with custom instrumentation.

This article focuses on using that capability safely and interpreting what it tells you.

Build a service that is easy to inspect

Consider a small program with several long-running jobs:

import asyncio
import os


async def fetch_partition(partition: int) -> None:
    while True:
        await asyncio.sleep(30)
        print(f"polled partition {partition}")


async def consumer_group() -> None:
    async with asyncio.TaskGroup() as group:
        for partition in range(3):
            group.create_task(
                fetch_partition(partition),
                name=f"partition-{partition}",
            )


async def main() -> None:
    print(f"PID={os.getpid()}")
    await consumer_group()


asyncio.run(main())

Run it normally:

python worker.py

Suppose it prints:

PID=42137

Leave that process running. From another terminal, inspect it with Python 3.14:

python -m asyncio pstree 42137

The exact addresses, line numbers, and internal frames vary, but the important shape is the task hierarchy. You should see the main task leading into consumer_group(), followed by the named partition tasks and their suspended coroutine stacks.

Naming tasks is not required for introspection, but it makes operational output much easier to read.

Use pstree when the relationship matters

pstree renders tasks and coroutine stacks as a tree based on await relationships.

That is particularly useful with structured concurrency. A TaskGroup creates a relationship between its parent coroutine and its child tasks, and the tree makes that structure visible.

For example, imagine a service shaped conceptually like this:

main
└── serve
    ├── http-server
    ├── queue-consumer
    │   ├── partition-0
    │   ├── partition-1
    │   └── partition-2
    └── metrics-publisher

If shutdown hangs, a tree can quickly reveal that main is waiting for serve, which is waiting for a task group, which still contains partition-1.

The stack beneath that task then tells you where the coroutine is suspended.

This is more actionable than a flat list when the problem is an ownership or dependency chain.

Use ps when you need an inventory

The second command is:

python -m asyncio ps 42137

Instead of drawing a tree, ps prints a flat table of pending tasks.

Its output includes information such as:

• the event-loop thread ID;
• the task ID;
• the task name;
• the coroutine stack;
• the awaiting task’s stack;
• the awaiter’s name and task ID.

The flat representation is useful when you are answering questions such as:

• How many tasks are pending?
• Are tasks running in multiple event-loop threads?
• Do many workers have the same suspended stack?
• Which task names are represented?
• Is one family of tasks unexpectedly absent?

A tree emphasizes relationships. A table emphasizes inventory and repeated patterns.

In an incident, it is often useful to capture both.

Treat task names as operational metadata

The default task names are technically sufficient, but application-specific names turn introspection into a much better diagnostic tool.

Compare:

task = asyncio.create_task(handle_connection(reader, writer))

with:

task = asyncio.create_task(
    handle_connection(reader, writer),
    name="connection-42",
)

For a task group:

async with asyncio.TaskGroup() as group:
    for shard in shards:
        group.create_task(
            consume(shard),
            name=f"consumer:{shard.name}",
        )

Good task names identify a role or bounded unit of work. Avoid putting secrets, credentials, authorization headers, full request bodies, or other sensitive data into them.

Remember that diagnostic output may be copied into incident tickets, terminals, logs, or chat systems.

A useful naming scheme might include a non-sensitive worker role and an internal identifier:

consumer:orders:shard-03

rather than customer-supplied content.

Diagnose a hanging shutdown

A common async failure mode is a shutdown that never completes.

Consider:

import asyncio


async def worker() -> None:
    try:
        while True:
            await asyncio.sleep(60)
    except asyncio.CancelledError:
        await flush_before_exit()
        raise

If flush_before_exit() waits forever, cancellation reached the worker but the worker still does not finish.

A normal process monitor only tells you that the process remains alive. pstree can show the parent task still awaiting the child and the child suspended inside the cleanup path.

That distinction matters. The fix is not necessarily “send cancellation again.” The actual bug may be an unbounded cleanup operation.

Make cleanup waits explicit and bounded when appropriate:

async def worker() -> None:
    try:
        while True:
            await asyncio.sleep(60)
    except asyncio.CancelledError:
        try:
            async with asyncio.timeout(5):
                await flush_before_exit()
        finally:
            raise

Introspection does not solve the lifecycle bug, but it can make the lifecycle state visible.

Diagnose a fan-out bottleneck

Suppose a coordinator starts many tasks:

async def refresh_all(accounts) -> None:
    async with asyncio.TaskGroup() as group:
        for account in accounts:
            group.create_task(
                refresh(account),
                name=f"refresh:{account.internal_id}",
            )

During an incident, asyncio ps might show hundreds of refresh:* tasks with the same final coroutine frame.

That repeated shape is evidence worth investigating. Perhaps all tasks are waiting for a shared semaphore, database pool, DNS operation, or downstream response.

Do not infer the root cause from the frame name alone. A suspended coroutine tells you where execution is waiting at the captured moment, not why the awaited operation has failed to make progress.

Correlate the snapshot with metrics, downstream health, pool utilization, traces, and logs.

Distinguish a snapshot from a trace

The command-line tools inspect current state. They do not reconstruct the history that led to that state.

If a task is shown in:

wait_for_message -> consume_partition

you know where it is suspended now.

You do not automatically know:

• how long it has been there;
• how many messages it processed previously;
• whether it repeatedly leaves and re-enters the same await;
• whether latency is caused by the event loop or an external service;
• what arguments were passed unless that information is represented elsewhere.

For duration and causality, use normal observability techniques such as metrics, tracing, structured logs, and application-level timing.

External task introspection complements those tools rather than replacing them.

Capture multiple snapshots before declaring a deadlock

One snapshot can make healthy waiting look suspicious.

A server task is supposed to spend much of its life awaiting I/O. A queue consumer may legitimately wait for new work. A timer task may remain in asyncio.sleep() for minutes.

When investigating a possible stall, capture snapshots separated by an appropriate interval:

python -m asyncio ps 42137
# wait long enough for expected work to progress
python -m asyncio ps 42137

If the same tasks remain at the same logical boundary while request latency and queue depth continue to increase, the evidence becomes stronger.

Task IDs and exact output should not be treated as a stable serialization format. Use the command as an operational diagnostic interface, not as a durable application protocol.

Understand cycles in the await graph

pstree needs a tree-shaped representation. Python’s documentation notes that if the await graph contains a cycle, pstree reports an error instead of rendering the tree.

A cycle is unusual and generally points to a programming problem.

For example, two tasks should not be designed so that each can only finish after the other finishes:

Task A waits for Task B
Task B waits for Task A

When pstree reports an await-graph cycle, switch to:

python -m asyncio ps PID

The flat ps view prints pending tasks even when the graph contains cycles. That gives you a way to inspect the participants without requiring the graph to be representable as a tree.

The diagnostic lesson is important: failure to print a tree is itself useful information when the reason is a cycle.

Multi-threaded event loops change the picture

Python applications are often described as having “the event loop,” but a process can have event loops in multiple threads.

Python 3.14’s asyncio introspection infrastructure can inspect tasks running across threads, and asyncio ps includes the event-loop thread ID in each row.

This becomes especially relevant for free-threaded Python or applications that deliberately isolate async workloads in different threads.

Do not assume two tasks shown in the same process necessarily execute on the same event-loop thread.

When diagnosing shared-state problems, record both task identity and thread identity.

External inspection is not application instrumentation

One of the most useful properties of the Python 3.14 CLI is that the target service does not need to expose a custom endpoint for this task graph.

The inspection command runs separately and reads the target process state without executing application code inside that target.

That reduces the temptation to add an emergency endpoint such as:

/debug/all-tasks

and accidentally expose internal state over the network.

It does not mean external inspection is permission-free or universally available. Python documents these commands as supported only on appropriate platforms, and inspecting another process may require operating-system permission.

Treat process-inspection access as privileged operational capability.

Do not build a production API around CLI output

It can be tempting to run python -m asyncio ps from a monitoring agent every few seconds and parse the table.

That is usually the wrong abstraction.

The command is designed for introspection by operators. Output contains implementation-level details such as task IDs, thread IDs, stacks, file names, and line numbers. Those details can change with code layout and Python versions.

For continuous monitoring, export stable application metrics instead:

pending_jobs.set(queue.qsize())
active_workers.set(len(active_workers_by_id))

Use tracing for request relationships and timing. Use logs for lifecycle events. Reach for external asyncio inspection when you need a high-resolution snapshot of a live process.

Pair the CLI with programmatic introspection when appropriate

Python 3.14 also provides programmatic APIs for async call-graph introspection in the current process.

That is a different boundary.

Programmatic inspection is useful when a trusted diagnostic subsystem inside your application deliberately captures task relationships. The CLI is useful when you need to inspect a process from outside it, especially when changing or restarting the target would destroy the state you want to observe.

Choose based on the operational boundary:

Inside the process  -> programmatic asyncio introspection
Outside the process -> python -m asyncio pstree / ps

Neither should become an excuse to execute untrusted diagnostic requests against production internals.

Keep version compatibility explicit

The pstree and ps command-line inspection functionality discussed here was added in Python 3.14.

If your fleet contains Python 3.13 and Python 3.14 services, do not assume the same incident command works everywhere.

Record the runtime version first:

python --version

For containers, also make sure the Python executable used for inspection is the one you intend. Host Python, sidecar Python, and the target container’s runtime can differ.

A runbook should state the minimum supported Python version instead of leaving responders to discover it during an outage.

Build an incident runbook

A compact workflow for a suspected async stall can look like this:

1. Identify the exact target process and runtime version.
2. Confirm that you are authorized to inspect that process.
3. Capture python -m asyncio pstree PID.
4. Capture python -m asyncio ps PID when a flat inventory is useful or the graph has a cycle.
5. Record relevant service metrics at the same time.
6. Repeat the snapshot after an interval appropriate to the workload.
7. Compare task roles and suspended stacks, not only unstable numeric IDs.
8. Correlate the result with traces, logs, pool statistics, and downstream health.
9. Preserve only the diagnostic data allowed by your organization’s security and retention policy.

The important step is correlation. A task snapshot becomes much more useful when paired with evidence about what the service was expected to be doing.

Design async code for diagnosability

The new tools are most effective when the application already has understandable concurrency structure.

Prefer named tasks:

asyncio.create_task(run_scheduler(), name="scheduler")

Prefer structured task ownership where it fits:

async with asyncio.TaskGroup() as group:
    group.create_task(run_api(), name="api")
    group.create_task(run_consumer(), name="consumer")

Bound operations that must not wait forever:

async with asyncio.timeout(10):
    await dependency.close()

And keep task roles semantically meaningful. A graph containing Task-137, Task-138, and Task-139 is less useful than one containing api, consumer:orders, and checkpoint-writer.

Diagnosability is partly a property of application structure, not only of debugging tools.

Test the behavior, not the exact rendering

If you add tests or operational checks around task introspection, avoid snapshotting an entire CLI rendering unless you specifically maintain tooling that requires it.

Exact line numbers, task identifiers, internal asyncio frames, and thread IDs are runtime details.

For application tests, validate your own invariants instead:

async def test_workers_have_names():
    async with asyncio.TaskGroup() as group:
        task = group.create_task(
            asyncio.sleep(0),
            name="checkpoint-writer",
        )
        assert task.get_name() == "checkpoint-writer"

For an integration runbook test, it can be enough to verify that the supported Python 3.14 environment permits inspection and that a known task name appears.

That gives confidence in the operational path without treating human-oriented output as a permanent schema.

Know what the tools cannot prove

A task graph is strong evidence about async control flow, but it has limits.

It cannot by itself prove that a database is healthy, that a remote peer will respond, or that a thread outside asyncio is making progress. It does not replace native debugging for crashes, profiling for CPU hotspots, or distributed tracing for cross-service causality.

It also does not turn every long-lived await into a bug. Waiting is the normal state of many asynchronous tasks.

Use the output to narrow a question:

Which async branch is not progressing, and what is it currently awaiting?

Then use the appropriate subsystem-specific evidence to answer why.

Conclusion

Python 3.14 makes a live asyncio process substantially easier to inspect from the outside.

Use:

python -m asyncio pstree PID

when you need to understand task ownership and await relationships, and:

python -m asyncio ps PID

when you need a flat inventory of pending tasks or need to inspect a graph that cannot be rendered as a tree.

The most effective operational pattern is to name tasks, use structured concurrency, capture more than one snapshot, and correlate task state with normal observability data.

Most importantly, treat these commands as diagnostic snapshots rather than a replacement for metrics, traces, logs, or stable application interfaces. They are valuable precisely because they expose the async control-flow state that those higher-level systems often cannot show directly.