Python: Multiprocessing

Why Multiprocessing Instead of Threading for CPU-bound Tasks

Python's threading module is genuinely useful for I/O-bound work, where threads spend most of their time waiting on network responses, disk reads, or database queries. The GIL releases during those waits, so other threads make progress. For CPU-bound work, however, the GIL means that only one thread executes Python bytecode at any moment, regardless of how many CPU cores the machine has. A program that uses four threads to compress files may actually run slower than a single-threaded version because of the contention and context-switching overhead around the GIL.

The multiprocessing module solves this by launching separate OS processes instead of threads. Each process has its own Python interpreter, its own GIL, and its own memory space. Four processes on a quad-core machine can genuinely run four CPU-bound tasks simultaneously. The trade-off is that sharing data between processes requires explicit mechanisms, because they do not share memory by default.

The multiprocessing Module

Python's multiprocessing module is designed to feel similar to threading. Many of the concepts carry over: you create a Process object, give it a target function and arguments, call start(), and then join() to wait for it. The module also provides process pools, shared memory primitives, inter-process queues and pipes, manager objects, and synchronisation primitives that work across process boundaries.

One important implementation note: on Windows and macOS (using the default "spawn" start method), Python serialises (pickles) the target function and its arguments to send them to the child process. This means the target function and its arguments must be picklable. Lambda functions and locally defined functions cannot be pickled, so they must be defined at the module level. On Linux, the default start method is "fork", which copies the parent process's memory directly, so pickling is not needed for the initial arguments, but it brings its own complications around resources held by the parent.

Creating and Starting Processes

Creating a process follows the same pattern as creating a thread: instantiatemultiprocessing.Process with a target callable and optional args orkwargs, then call start(). The join() method blocks the calling process until the child finishes. You can also check whether a process is still alive withp.is_alive(), access its exit code via p.exitcode (available after it finishes), and give it a name with the name parameter.

Because child processes do not share memory with the parent, return values from the target function are not automatically available in the parent. You must use a Queue,Pipe, shared memory object, or a process pool to get results back. This is a common source of confusion for developers coming from threading.

Process vs Thread: Key Differences

Understanding the practical differences between Process and Thread helps you choose the right tool and avoid wasting time debugging communication issues that arise from choosing the wrong one.

Process Pools: Pool.map() and Pool.starmap()

Creating one process per task is rarely what you want. OS processes are heavyweight, and spawning hundreds of them would overwhelm the machine. A multiprocessing.Pool solves this by maintaining a fixed number of worker processes. Tasks are assigned to idle workers as they become available. The pool handles all the bookkeeping of distributing work, collecting results, and reusing processes for multiple tasks.

Pool.map(func, iterable) applies a function to every item in an iterable and returns the results in the same order as the input. It blocks until all results are ready. Pool.starmap(func, iterable_of_tuples) works the same way, but it unpacks each tuple as separate positional arguments to the function, which is useful when the function takes more than one argument.

Shared Memory: Value and Array

Most of the time you communicate between processes by sending messages through a queue or pipe. But for simple numeric data that processes need to update at high frequency, message-passing has overhead. multiprocessing.Value and multiprocessing.Arraycreate objects backed by shared memory that multiple processes can read and write directly. No serialisation is involved, so they are fast.

Value(typecode, initial_value) creates a single shared scalar. Array(typecode, size_or_initializer) creates a fixed-length shared array. The typecode uses the same single-character codes as Python's array module: 'i' for signed integer,'d' for double, 'f' for float, and so on. Both objects have an internal lock accessible as .get_lock(), but you should acquire it explicitly when you need atomic read-modify-write operations to avoid race conditions.

Queue and Pipe for Inter-process Communication

When you need to pass Python objects between processes, rather than raw numeric data, the two main tools are multiprocessing.Queue and multiprocessing.Pipe. Both serialise objects with pickle to move them across the process boundary.

A Queue is multi-producer/multi-consumer: many processes can put items in and many can get items out, making it the natural choice for task queues and result collection. A Pipe creates two connected Connection objects. Each end can send and receive, but a pipe is optimised for one-to-one communication because there are only two endpoints. For a simple producer-consumer pair a pipe is faster and more direct; for fan-out/fan-in patterns with many workers, a queue is cleaner.

Manager Objects

multiprocessing.Value and Array only support simple C-style numeric types. When you need processes to share a Python list, dictionary, namespace, or other complex object, you use a Manager. A manager launches a separate server process that owns the data and exposes proxy objects. All other processes interact with the data by sending method calls through the proxy, which the manager server handles and applies to the real object.

The proxy approach means manager objects work across machines (with the right connection settings), not just across local processes. The downside is that every operation involves inter-process communication through the manager server, which adds latency compared to direct shared memory. For frequent, fine-grained updates, Value or Arraywith a lock is faster. For high-level objects that are updated less frequently, managers are the right choice.

concurrent.futures.ProcessPoolExecutor

multiprocessing.Pool is powerful, but its API predates modern Python idioms.concurrent.futures.ProcessPoolExecutor provides the same underlying capability through a higher-level interface that mirrors ThreadPoolExecutor. If you have used the threading tutorial's ThreadPoolExecutor examples, the pattern here will look identical: you swap the class name and get parallelism instead of concurrency.

The key concept is the Future object returned by executor.submit(). A future represents a computation that may not have completed yet. You can call future.result()to block until it finishes and get the return value, or pass the future toconcurrent.futures.as_completed() to process results in the order they finish rather than the order they were submitted. Exceptions raised inside the worker function are re-raised when you call future.result(), which makes error handling straightforward.

Synchronisation: Lock and Semaphore

Even though processes have separate memory, you still need synchronisation when they share a resource: a Value, an Array, a file, or a database connection pool. The multiprocessing module provides Lock, RLock, Semaphore,Event, Condition, and Barrier, which mirror their threadingequivalents but work across process boundaries. You must pass them to child processes as constructor arguments; they cannot be created inside the child and shared with the parent because the child has a separate memory space.

A critical point: you cannot use threading.Lock to protect shared state across processes. Each process would get its own independent lock object, providing no mutual exclusion. Always use multiprocessing.Lock when coordinating multiple processes.

Quiz - Test Your Knowledge

Ten questions covering the GIL and why it matters, the distinction between processes and threads, process pools, shared memory primitives, IPC options, Manager objects, the ProcessPoolExecutor API, and synchronisation across process boundaries. Several questions hinge on subtle but important distinctions, so read each option carefully.