# Concurrency vs Parallelism

Szymon Kopyciński · 21 October 2026

Will more workers actually help? 

## Introduction

A researcher runs a feature pipeline over a day's market data. It takes two hours. The server has sixteen CPU cores, so someone suggests sixteen workers.

If each worker handles a sixteenth of the data, shouldn't the job finish in about eight minutes?

Perhaps. But what if they all read the same files? What if each starts its own multithreaded numerical library? What if the calculation needs yesterday's state before it can process today?

We need to discover which work can happen independently, which resource limits it, and what running more of it at once would cost.

## What needs to finish sooner?

Use the selected dataset from [Choosing the Right Database](/blog/engineering-tradeoffs-2627/choosing-the-right-database): 100 million normalised events per trading day. The pipeline reads published Parquet, constructs per-instrument features and combines them into a cross-sectional dataset.

Assume a shared research server with sixteen physical CPU cores and 64 GiB of RAM. Its two-hour baseline and a proposed thirty-minute completion target are illustrative, not measured benchmarks. Other researchers need the machine too. We are improving historical research turnaround, rather than an order gateway's response time.

Functionally, the output must contain the same instruments, observations and feature definitions as before. Non-functional requirements include completion time, peak memory, shared storage load and reproducibility. A faster calculation which silently changes a rolling window has failed the functional requirement.

Also ask whether the thirty-minute target is a deadline or a preference. A deadline may justify hardware or algorithm changes. A preference might justify stopping after a modest improvement which leaves the server usable.

## Several tasks in progress, or several tasks executing?

*Concurrency* means managing tasks whose execution can overlap. *Parallelism* means executing work simultaneously, for example on different cores.

One thread can start a network read, let another task make progress while the request is outstanding, then resume when the data arrives. Several tasks are in progress even though that thread executes only one at a time. Two cores calculating two independent features at the same moment provide parallelism.

This distinction matters because waiting and computation need different remedies.

A pipeline can be limited by storage bandwidth, network response time, memory bandwidth or CPU execution. It can also wait for a lock: permission to use some shared state. A worker count is not a diagnosis.

Measure elapsed time alongside CPU time, bytes read, memory and queueing. Aggregate CPU time across subprocesses; the coordinator's CPU usage alone misses their work. Low CPU use might indicate waiting, but it does not establish whether the problem is a disk, a remote service or a dependency between tasks.

## Find the part that can shrink

Suppose investigation produces this **hypothetical** decomposition:

| Stage | Baseline elapsed time | Assumption for the first redesign |
| --- | ---: | --- |
| Read and prepare inputs | 20 minutes | Remains unchanged |
| Calculate instrument features | 80 minutes | Can divide across independent workers |
| Combine, validate and publish | 20 minutes | Remains unchanged |

For this model the stages do not overlap. More feature workers accelerate only the middle stage.

```mermaid
flowchart TD
    accTitle: The serial stages remain on the completion path
    accDescr: Input preparation must finish before independent feature tasks begin. Publication waits for all feature tasks, so their parallelism cannot remove the two serial stages.
    R[Read and prepare: 20 min] --> A[Feature partition A]
    R --> B[Feature partition B]
    R --> C[Other feature partitions]
    A --> J[Combine and publish: 20 min]
    B --> J
    C --> J
```

The fixed-work argument associated with [Amdahl's Law](https://doi.org/10.1145/1465482.1465560) separates an unchanged fraction from a parallelisable fraction. Let the original time be $T_1$, the unchanged fraction be $s$, and the worker count be $n$. With ideal division and no new overhead:

$$
T_n = T_1\left(s+\frac{1-s}{n}\right)
$$

Here $T_1=120$ minutes and $s=1/3$. The model becomes $T_n=40+80/n$ minutes.

| Workers | Ideal completion time | Speed-up over the baseline |
| --- | ---: | ---: |
| 1 | 120 minutes | 1.0 times |
| 2 | 80 minutes | 1.5 times |
| 4 | 60 minutes | 2.0 times |
| 8 | 50 minutes | 2.4 times |
| 16 | 45 minutes | About 2.67 times |

Even infinitely many feature workers leave forty minutes. The thirty-minute target requires changing something else: less input work, a cheaper combination stage, overlap between stages, or a different calculation.

This is a conditional model. Workers may be unevenly loaded; coordination adds time. Changing the algorithm or making data fit in cache can change the work itself. “One third is serial” describes this decomposition, not an immutable property of the problem.

## Can the engine already do the work?

Before adding Python workers, inspect the expensive operation. Are we repeatedly scanning the same day for separate features? Materialising columns which no calculation uses? Converting columnar data into Python objects and then back again?

A shared scan, earlier filtering or an array operation can remove work. Those improvements can benefit every researcher without introducing another scheduler.

An analytical engine may already distribute a query across cores. DuckDB's [performance guidance](https://duckdb.org/docs/stable/guides/performance/how_to_tune_workloads) discusses parallel execution and cases where reducing thread counts helps. Sixteen Python processes, each running an engine configured to use sixteen threads, do not create another 240 cores.

An engine's parallelism and a Python worker pool are alternative places to allocate resources. Combining them requires a deliberate budget. Prefer one layer to own most of the parallel work initially; introduce another when a measured limitation justifies it.

## Which Python execution model fits?

Threads share a process's memory, making it convenient to work with read-only data without serialising each task's inputs. They can overlap blocking I/O. Shared mutable state also becomes a correctness responsibility: two workers updating the same output array need an ownership rule or synchronisation.

In a GIL-enabled CPython interpreter, the global interpreter lock restricts execution of Python code to one thread at a time. That matters for a CPU-heavy Python loop. It does not mean every operation called from Python runs under the lock.

Many NumPy operations release the GIL, allowing suitable numerical work in different threads to run in parallel. Object-array operations have different behaviour, and concurrently mutating an array can be unsafe. The [NumPy thread-safety documentation](https://numpy.org/doc/stable/reference/thread_safety.html) explains these distinctions. “Written in Python” is insufficient information for choosing processes.

Separate processes can execute Python work on different cores under a GIL-enabled build. In return, ordinary task arguments and results cross process boundaries. Python's [executor documentation](https://docs.python.org/3.15/library/concurrent.futures.html#processpoolexecutor) specifies picklable inputs and results for `ProcessPoolExecutor`: serialisation turns objects into a representation another process can reconstruct.

That transfer may be a significant part of the job. Worker start-up and separate memory allocations also cost resources. Do not assume a platform's process start method gives every worker a free, safely shared copy of the coordinator's dataset.

The version and build matter. CPython has offered optional free-threaded builds since 3.13; extension compatibility can affect whether the GIL remains disabled. The [free-threading guide](https://docs.python.org/3.15/howto/free-threading-python.html) documents how to check the running interpreter. Removing the GIL does not remove shared-state races or guarantee that memory bandwidth scales with cores.

There are also interpreter pools, introduced in Python 3.14. They provide separate interpreters within worker threads, with isolated runtime state and explicit data-transfer constraints. The [executor reference](https://docs.python.org/3.15/library/concurrent.futures.html#interpreterpoolexecutor) describes them. They broaden the options; their compatibility and transfer costs still need checking against the actual workload. These details were checked against Python 3.15 documentation in October 2026.

Finally, asynchronous tasks let an event loop manage work which can suspend while waiting. This suits many outstanding network operations when the client library supports it. A long CPU calculation or blocking call inside the event-loop thread prevents other tasks on that loop from progressing. Python documents this [cooperative scheduling model](https://docs.python.org/3.15/library/asyncio-task.html#task-object). Adding `async` to a feature function does not distribute its arithmetic across cores.

| Approach | Evidence which favours it | Cost to account for |
| --- | --- | --- |
| One engine with internal parallelism | Most work already fits analytical operations | Engine memory, threads and shared storage |
| Threads | I/O waits or suitable GIL-releasing computation | Shared-state safety and native thread counts |
| Processes | Substantial independent Python computation | Start-up, serialisation and private memory |
| Interpreter pools or free-threaded execution | Compatible runtime and dependencies; a demonstrated benefit | Isolation or synchronisation requirements; build-specific behaviour |
| Asynchronous I/O | Many independent waits through compatible APIs | Event-loop blocking, cancellation and bounded outstanding work |

## Moving the data can cost more than moving the task

Suppose each process reaches a hypothetical peak of 4 GiB of private memory, with another 8 GiB for the coordinator and its buffers. Sixteen workers would require 72 GiB under that accounting, before the operating system and other researchers. Our machine has 64 GiB.

Eight workers would require 40 GiB. That fits the stated memory capacity more comfortably, but does not establish that eight is the right count. Simultaneous jobs, temporary arrays and storage pressure still matter. Count bytes in flight as well as active workers; a queue of completed results can consume memory too.

A better task boundary often consists of an input reference, a dataset version and a partition description. The worker reads the required columns, writes its own bounded output and returns a small result reference.

```mermaid
flowchart TD
    accTitle: Dispatch references and bound data in flight
    accDescr: The coordinator admits only a limited number of tasks. Workers read immutable partitions directly and return output references rather than sending full datasets through the coordinator.
    M[Versioned input manifest] --> C[Coordinator: bounded admission]
    C -->|Partition reference| A[Worker A]
    C -->|Partition reference| B[Worker B]
    D[Immutable input files] --> A
    D --> B
    A --> O[Separate task outputs]
    B --> O
    A -->|Small result reference| V[Validate complete task set]
    B -->|Small result reference| V
    O --> V
    V --> P[Publish complete dataset version]
```

This reduces coordinator transfers; it does not make storage free. Multiple readers may saturate the same disk or network link. Shared memory can avoid some copies but adds ownership and lifetime management, as [How Should Your Services Communicate?](/blog/engineering-tradeoffs-2627/how-should-your-services-communicate) discusses. Use it when transfer measurements justify the added responsibility.

## Independence is an algorithm property

Partitioning by instrument looks natural. It works when instrument A's feature state does not depend on unfinished work for instrument B. A market-wide ranking needs a later combination step, or a separately supplied market-wide input.

Splitting a rolling feature at arbitrary row boundaries is different. A five-minute window needs earlier observations. A worker starting at 10:00 cannot pretend the history begins there. Supply sufficient preceding data or validated initial state, and exclude overlap from the published output.

For order-book reconstruction, retain the required snapshot and apply deltas in their source order. Do not let workers independently initialise consecutive chunks to empty books. The [order-book article](/blog/quant-foundations-2627/how-market-data-becomes-an-orderbook) explains why sequence and state matter.

Tasks also vary in size. One heavily traded instrument can outlast hundreds of quiet ones. Several bounded tasks per worker allow scheduling around that imbalance, while excessively tiny tasks spend too much time on dispatch. Split according to measured work and valid state boundaries, not merely equal instrument counts.

Parallel results can arrive in different orders. Define final ordering explicitly. Floating-point addition can change with reduction order, so agree whether reproducibility means identical bits or a justified numerical tolerance. A small difference near a trading threshold can change an order; the tolerance must respect downstream use. Assign random streams using stable task identities, rather than whichever worker happens to execute next.

A failed task must not leave a partially published dataset. Give retries identifiable outputs and publish only after validating the complete task set. Correctness includes missing and duplicate partitions, boundary observations and late failures, not just comparing one successful aggregate.

## The whole server has a worker count

Even four processes can oversubscribe the machine if each invokes a library that creates eight compute threads. Up to 32 such threads may compete for sixteen cores, alongside the coordinator and other jobs. This is a possible configuration, not a statement that every NumPy call creates threads.

Numerical linear algebra may use a multithreaded BLAS backend: a library providing optimised matrix and vector routines. Its controls depend on the backend; NumPy's [configuration documentation](https://numpy.org/doc/stable/reference/global_state.html#number-of-threads-used-for-linear-algebra) identifies that distinction. Budget outer workers and inner library threads together.

More runnable threads can increase scheduling overhead. More simultaneous scans can exhaust storage bandwidth. More private arrays can trigger memory pressure and spilling. Keeping every core busy is useful only if it advances the required work at an acceptable shared cost.

This is also why a cluster is not the automatic next step. It adds data movement, deployment and failure handling. Additional machines help when the task graph and data placement can use their resources; they do not remove a global dependency or a single saturated storage service.

## What I would change first

Keep the versioned Parquet inputs and publication contract. Profile representative quiet and busy days, then remove repeated scans and unnecessary materialisation. Test engine-native parallel execution before introducing an outer pool.

If substantial independent Python computation remains, start a pilot with two and four processes, initially limiting each task's native compute threads. Send partition references, bound active tasks and result buffering, and validate rolling-state boundaries. Four is a test configuration with headroom, not a benchmark-derived optimum.

Compare elapsed time, total CPU consumption, peak memory, transferred bytes, storage throughput and the slowest tasks. Record runtime builds, library thread settings, dataset versions and warm versus cold cache conditions. Verify the complete output against the reference, including numerical tolerances. Repeat with another research job active.

Under the illustrative stage breakdown, even ideal sixteen-worker execution misses the thirty-minute target. Report that clearly. Improve the unchanged stages or revisit the target before promising success from additional workers. Keep an improvement only when it helps the intended workload and its neighbours.

## Your turn

Your four-worker pipeline finishes sooner when run alone. Three researchers then launch it together. All three jobs take longer than the original single-job baseline, although CPU utilisation is high.

Which measurements would distinguish too many native threads, storage contention, excessive data copying and memory pressure? How would you test those explanations without changing everything at once?

Propose a server-wide admission policy and a worker/thread budget. Decide whether to optimise one job's completion time, the last researcher's completion time, or fairness between them. Include a rolling-feature correctness check and explain what evidence would justify adding another machine.


Part of [Engineering Trade-offs](/blog/engineering-tradeoffs-2627) · Previous: [Latency vs Throughput](/blog/engineering-tradeoffs-2627/latency-vs-throughput)