Launching a command from Python is easy. Capturing its output is also easy. The trouble starts when a parent process waits for a child while the child is waiting for the parent to read from a pipe.

That circular wait is a deadlock: neither process can make progress even though neither has crashed.

This problem is especially confusing because the same code may work during testing and hang only when a command produces more output. Small output fits in an operating-system pipe buffer. Larger output can fill that buffer and expose the incorrect coordination.

The safest mental model is: a pipe has finite capacity, so creating a pipe also creates a responsibility to keep draining it while the child runs.

Python’s subprocess module already provides high-level operations that handle this coordination correctly. The important part is choosing the right one for the amount and shape of output you expect.

Start with subprocess.run for bounded commands

If a command should run to completion and its output is reasonably bounded, subprocess.run() is usually the simplest choice.

import subprocess
import sys

result = subprocess.run(
    [
        sys.executable,
        "-c",
        "print('ready')",
    ],
    capture_output=True,
    text=True,
    check=True,
)

print(result.stdout.strip())

capture_output=True asks Python to capture both standard output and standard error. text=True decodes them as text rather than returning bytes. check=True raises subprocess.CalledProcessError when the child exits with a non-zero status.

The important behavior is less visible: run() does not simply start the child, wait for it, and read the pipes afterward. It uses the lower-level process communication machinery so output can be collected without the classic pipe deadlock.

For one-shot commands with bounded output, that is usually preferable to manually coordinating Popen.

Understand why wait() plus PIPE can hang

A pipe connects a writer to a reader through a kernel-managed buffer. That buffer is finite.

Suppose a parent creates a pipe for a child’s standard output:

import subprocess
import sys

process = subprocess.Popen(
    [sys.executable, "-c", "print('x' * 1_000_000)"],
    stdout=subprocess.PIPE,
)

process.wait()
output = process.stdout.read()

This order is risky.

The parent calls wait() before reading. Meanwhile, the child may keep writing output. Once the pipe buffer fills, the child’s next write blocks until the parent reads some data.

Now both processes are waiting:

  1. The parent waits for the child to exit.
  2. The child waits for the parent to drain the full pipe.

The child cannot exit because it is blocked in a write, and the parent will not read until the child exits.

The exact amount of output required to trigger this depends on the operating system and runtime environment. That is why code with this bug can appear reliable with small test cases.

The fix is not to guess a pipe-buffer size. The fix is to coordinate reading and waiting correctly.

Use communicate() when working directly with Popen

When you need Popen rather than run(), use communicate() to exchange data and collect piped output.

import subprocess
import sys

process = subprocess.Popen(
    [
        sys.executable,
        "-c",
        (
            "import sys; "
            "print('normal output'); "
            "print('diagnostic output', file=sys.stderr)"
        ),
    ],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)

stdout, stderr = process.communicate()

print("exit:", process.returncode)
print("stdout:", stdout.strip())
print("stderr:", stderr.strip())

communicate() reads standard output and standard error until end-of-file and waits for the child to terminate. If standard input is also a pipe, it can send input before finishing the exchange.

That matters when both output streams are captured. Reading one entire stream before touching the other can deadlock if the child fills the stream you are not currently reading.

For example, this pattern is unsafe:

stdout = process.stdout.read()
stderr = process.stderr.read()
process.wait()

Even though the parent starts reading before waiting, it reads the streams sequentially. If the child writes enough to stderr while the parent is blocked waiting for EOF on stdout, the child can block on stderr and never close stdout.

communicate() exists to handle this multi-stream coordination.

Send input through communicate() instead of manual writes

The same principle applies when the parent writes to the child’s standard input.

import subprocess
import sys

process = subprocess.Popen(
    [
        sys.executable,
        "-c",
        "import sys; print(sys.stdin.read().upper(), end='')",
    ],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)

stdout, stderr = process.communicate("hello\n")

print(stdout, end="")

Passing input to communicate() lets Python coordinate closing the child’s input and reading its outputs.

A risky alternative is to write a large amount to process.stdin while ignoring the child’s output. The parent can block because the child is not reading input yet, while the child can simultaneously block because the parent is not draining output.

When all communication is bounded and can be buffered in memory, communicate() is the straightforward solution.

Treat standard output and standard error as independent pressure points

Capturing only stdout does not make stderr irrelevant. What matters is where each child stream goes.

If stderr is inherited from the parent, the child writes directly to the parent’s standard error destination and there is no Python-created pipe for that stream.

If you set both streams to PIPE, both must be drained.

If you do not need separate streams, you can merge standard error into standard output:

import subprocess
import sys

process = subprocess.Popen(
    [
        sys.executable,
        "-c",
        "import sys; print('out'); print('err', file=sys.stderr)",
    ],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
)

combined_output, _ = process.communicate()

This reduces two captured output streams to one. The trade-off is that you lose the ability to distinguish which bytes came from standard output and which came from standard error.

Choose based on how the output will be consumed, not merely on convenience.

Add timeouts without abandoning pipe cleanup

Timeouts introduce another lifecycle question: what happens to the child after the timeout?

With subprocess.run(), Python handles that cleanup for you. If its timeout expires, the child is killed and waited for before TimeoutExpired is re-raised.

import subprocess
import sys

try:
    subprocess.run(
        [sys.executable, "-c", "import time; time.sleep(30)"],
        timeout=2,
        check=True,
    )
except subprocess.TimeoutExpired:
    print("command timed out")

Popen.communicate() behaves differently. A communication timeout raises TimeoutExpired, but it does not kill the child automatically.

A robust cleanup pattern is:

import subprocess
import sys

process = subprocess.Popen(
    [sys.executable, "-c", "import time; time.sleep(30)"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

try:
    stdout, stderr = process.communicate(timeout=10)
except subprocess.TimeoutExpired:
    process.kill()
    stdout, stderr = process.communicate()

The second communicate() is important. It finishes draining the pipes and waits for the terminated child.

Calling only kill() and then abandoning the Popen object can leave output unread and process cleanup incomplete.

communicate() solves deadlocks by buffering output, so output size matters

communicate() is designed for finite communication. The data it reads is buffered in memory.

That creates a different failure mode for commands with huge or unbounded output. A process that emits gigabytes of logs may not deadlock, but collecting all of those logs into Python strings or byte arrays can consume unacceptable memory.

This is not a reason to return to wait() plus delayed reads. Instead, change the output destination or adopt a streaming design.

If you only need a durable log, let the child write directly to a file:

import subprocess
import sys

with open("worker.log", "w", encoding="utf-8") as log_file:
    result = subprocess.run(
        [
            sys.executable,
            "-c",
            "import sys; print('done'); print('details', file=sys.stderr)",
        ],
        stdout=log_file,
        stderr=subprocess.STDOUT,
        check=False,
    )

print("exit:", result.returncode)

Here, Python does not create an output pipe. The child writes to the file handle, so there is no bounded pipe buffer that the parent must drain.

This approach also avoids storing the complete output in Python memory.

Stream output only when you actually need incremental processing

Sometimes you need to react to output while the child is still running: update a progress display, parse events, or forward logs to another system.

Streaming is legitimate, but it changes the problem. You are now responsible for continuously servicing every pipe you create.

A simple case is one captured stream while the other is inherited or redirected:

import subprocess
import sys

process = subprocess.Popen(
    [
        sys.executable,
        "-c",
        "for i in range(3): print(f'event {i}', flush=True)",
    ],
    stdout=subprocess.PIPE,
    stderr=None,
    text=True,
)

for line in process.stdout:
    print("child:", line, end="")

returncode = process.wait()

This can be appropriate when:

  • only one output stream is piped,
  • the child does not depend on parent input that might create another blocking cycle,
  • line buffering behavior is acceptable for the child and protocol,
  • and you intentionally want incremental processing.

The situation becomes more complex when both stdout and stderr must be processed live. A sequential loop over one pipe is not sufficient because the other pipe can fill.

Portable solutions include reading the two streams concurrently, for example with separate threads, or using asyncio subprocess support when the surrounding application is asynchronous. Platform-specific readiness APIs are another option, but their behavior differs enough across operating systems that they should not be treated as a universal drop-in pattern.

If you do not need live processing, communicate() is easier to reason about.

Common mistakes come from mismatched lifecycle assumptions

Most subprocess pipe bugs fit a few recurring patterns.

Waiting before draining a pipe

If the child can produce enough output to fill a pipe, wait() before reading can deadlock. Use run() or communicate() for bounded capture.

Reading stdout fully before reading stderr

Two pipes are two independent bounded buffers. Sequential full reads can block each other. Use communicate() unless you have designed concurrent readers.

Writing all input before reading any output

Bidirectional communication can deadlock in either direction. Use communicate(input=...) for bounded request-response interactions.

Capturing unlimited output in memory

communicate() avoids the pipe coordination deadlock, but it buffers captured data. Redirect huge output to a file or implement deliberate streaming.

Assuming a timeout kills a Popen child

A timeout from Popen.communicate() does not terminate the process for you. Kill or terminate according to your application’s policy, then call communicate() again to finish cleanup.

Choose the simplest communication model that fits

Subprocess coordination becomes easier when you decide first what kind of interaction you actually need.

Use subprocess.run() when the child is a one-shot command and captured output is bounded.

Use Popen.communicate() when you need lower-level process control but can still buffer the complete input and output.

Redirect output to a file, DEVNULL, or another non-pipe destination when you do not need to hold it in Python.

Use explicit streaming only when incremental processing is a real requirement. If multiple streams must be live at once, drain them concurrently rather than reading them one after another.

The central rule is simple: if you create a pipe, the other process may eventually block until you service it. High-level subprocess APIs are valuable because they encode that coordination correctly. Reach for manual reads, writes, and waits only when the application genuinely needs a more specialized communication pattern.

Python Subprocess Concurrency I/O