# Latency vs Throughput

Szymon Kopyciński · 13 October 2026

What Are You Actually Optimising For?

## Introduction

Four strategies send order requests through one execution gateway. The venue allows an illustrative 40 order-related API requests per second, but one request may contain several orders.

The team proposes collecting orders in 10 ms windows and submitting each group together. Fewer API calls, less overhead. That sounds like an improvement.

Then a strategy asks to cancel an order just after a window begins. Its quote is now exposed at a price it no longer wants. Should that cancellation wait for the same window?

We need to decide which work benefits from batching, what delay it introduces, and which constraint actually limits dispatch.

## Define what needs to improve

Functionally, the gateway must submit compatible requests, match responses to individual orders and report outcomes. Non-functional requirements include quota compliance, bounded queue growth, recovery and latency targets by request class.

The 40-request limit is hypothetical, not a quoted venue rule. Before implementation, establish what the real venue counts: HTTP calls, individual orders, weighted endpoint operations, or messages across several sessions. Does a batch cost one unit? How many orders can it contain? Can some entries succeed while others fail? Are cancellations and modifications batchable?

For the examples, assume a batch of up to ten compatible new orders costs one request, all order-related dispatch shares one quota, and sufficient transport capacity exists. We will initially ignore network and processing time to isolate batch-formation delay. These are deliberately simplifying assumptions.

Also assume the strategies have different deadlines. A slower rebalance can tolerate 10 ms of extra waiting; a rapidly expiring signal may not. A latency budget needs a strategy's measured opportunity lifetime, not a generic label such as “systematic”. This API-limited example is distinct from an ultra-low-latency exchange gateway.

## Latency has boundaries

Latency is elapsed time between two defined events. “Order latency” could mean strategy decision to gateway receipt, receipt to dispatch, dispatch to venue acknowledgement, or decision to execution.

Those quantities answer different questions. A resting limit order may wait minutes for a fill even if submission takes milliseconds. Venue acknowledgement does not prove execution; local cancellation acceptance does not prove the venue removed an order.

For submission, one useful breakdown is:

$$
T_{\text{decision-to-ack}} =
T_{\text{internal}} + T_{\text{batch}} + T_{\text{quota queue}}
+ T_{\text{encode/send}} + T_{\text{venue response}}
$$

Measure each term rather than assigning every unexplained delay to “the network”. A 10 ms formation timeout bounds only one term under stated scheduling assumptions.

Use a monotonic clock for elapsed durations within a process. Comparing timestamps from different hosts requires clock synchronisation and an estimate of its error. Preserve the request ID across stages so observations describe the same operation.

## Throughput needs a unit too

Throughput is completed work per unit time. Here we have strategy messages, logical order operations, API requests and actual executions. They are not interchangeable.

If each permitted request contains ten new orders, the illustrative quota permits up to $40 \times 10 = 400$ new-order submissions per second before other constraints. It does not promise 400 fills. If cancellations share the quota, some capacity must serve them. If batches are usually half full, the attainable order rate changes again.

Count what completed means: queued locally, sent, accepted individually, or filled. A dashboard showing 400 orders “processed” because they entered a buffer can conceal a gateway sending only 200.

## Why batching can save work

Requests have fixed costs: creating envelopes, validating shared fields, invoking the operating system, and processing protocol headers. Some work is repeated per order too.

In a simple illustrative model, let fixed cost be $a$ and per-order cost be $b$. The cost of a batch containing $n$ orders is:

$$
C(n)=a+nb, \qquad \frac{C(n)}{n}=\frac{a}{n}+b
$$

Suppose $a=200\,\mu\text{s}$ and $b=20\,\mu\text{s}$. Ten individual requests cost $10\times220=2{,}200\,\mu\text{s}$ of this resource's work. One ten-order request costs $400\,\mu\text{s}$, or $40\,\mu\text{s}$ per order.

These invented costs demonstrate amortisation, not a benchmark or a prediction of venue acknowledgement time. The first order might wait 9 ms to save a much smaller amount of processing work. Resource efficiency and individual elapsed time are different measurements.

Databases batch inserts and networking can combine writes for similar reasons. The database article in this series discusses how small inserts create storage work. The acceptable wait depends on the application: historical publication and cancellation dispatch need different budgets.

## What does a 10 ms window cost?

Consider fixed boundaries at 10, 20, 30 ms and so on. Non-empty windows flush at their boundary. An order arriving at time $t$ within a window ending at $e$ waits:

$$
W_{\text{batch}}=e-t
$$

With ideal scheduling and a defined boundary rule, the additional wait is between zero and approximately 10 ms. An order just after the previous boundary waits almost the full window. An order just before the next boundary waits almost nothing. For arrivals whose phase is uniformly distributed within windows, the mean formation wait is 5 ms.

That mean is an assumption about arrivals. Strategies often react to the same event, so their arrivals can cluster. The actual waiting distribution may look quite different. Timers can also fire late when a process is descheduled; measure realised formation delay rather than treating the configured timeout as a hard real-time guarantee.

```mermaid
sequenceDiagram
    accTitle: Alternative dispatch timelines for three arrivals
    accDescr: With processing and network time ignored, immediate submission makes three calls at 1, 4 and 9 milliseconds. Fixed-window batching makes one call at 10 milliseconds and adds waits of 9, 6 and 1 milliseconds.
    participant S as Order arrivals
    participant G as Gateway
    participant V as Venue
    alt Immediate submission
        S->>G: A at 1 ms
        G->>V: A at 1 ms
        S->>G: B at 4 ms
        G->>V: B at 4 ms
        S->>G: C at 9 ms
        G->>V: C at 9 ms
    else Fixed 10 ms boundary
        S->>G: A at 1 ms
        S->>G: B at 4 ms
        S->>G: C at 9 ms
        Note over G: Wait until boundary
        G->>V: A, B and C at 10 ms
    end
```

This schematic compares alternative policies, not duplicate submissions in one system. It saves two calls while delaying the three orders by 9, 6 and 1 ms. Each order keeps its own identity and outcome.

Now compare a full second. Assume all orders are compatible, batch capacity is ten, and windows end at multiples of 10 ms. Ignore quota waiting in this table so we can see whether batching alone is sufficient.

| Illustrative arrivals in one second | Orders | Immediate calls | Calls with 10 ms windows | Mean formation wait | Within a 40-call ceiling? |
| --- | ---: | ---: | ---: | ---: | --- |
| One order at 5, 105, …, 905 ms | 10 | 10 | 10 | 5 ms | Yes |
| Ten simultaneous orders at 1, 101, …, 901 ms | 100 | 100 | 10 | 9 ms | Yes |
| One every 2.5 ms, starting at 1.25 ms | 400 | 400 | 100 | 5 ms | No |

In the last row, every window contains four orders. The waits repeat as 8.75, 6.25, 3.75 and 1.25 ms. Batching cuts calls by 75%, but still emits 100 calls per second. This is why reduced API utilisation does not establish quota compliance.

## Batching and rate limiting are separate

Batching changes how many logical operations share a request. Rate limiting decides when a request is permitted. A 10 ms timer is neither a 40-per-second limiter nor a substitute for checking the venue's rules.

A rolling-window quota could prohibit more than 40 dispatches in any preceding second. A token bucket instead replenishes capacity at a rate and permits bursts up to its capacity. Coinbase Exchange's [REST documentation](https://docs.cdp.coinbase.com/exchange/rest-api/rate-limits) gives a real token-bucket example; it is not the source of our hypothetical quota or batching assumptions.

A bucket refilling at 40 tokens per second with capacity 40 can permit an initial burst and further requests as tokens replenish. It therefore does not reproduce a strict 40-in-every-rolling-second ceiling. Even pacing at nominal 25 ms intervals needs boundary handling, scheduling allowance and accounting for other callers to match a real quota.

Enforce the documented accounting scope across all strategies, retries, sessions and other clients sharing it. A limiter inside each of four strategies would let each spend the same allowance. Use venue feedback to detect divergence, and account for weighted endpoints if applicable. Urgent work cannot bypass a venue restriction merely because we assign it high priority.

## Which batching policy fits?

| Policy | Benefit | Constraint to watch |
| --- | --- | --- |
| Immediate submission | No intentional formation wait | Can consume quota with many small calls |
| Fixed-time windows | Predictable nominal boundaries | Sparse traffic waits without saving calls |
| Maximum size only | Full batches amortise overhead | An incomplete batch can wait indefinitely |
| Size-or-time | Flush on size or oldest-order timeout | Timeout still needs a strategy-specific budget |
| Adaptive batching | Adjusts to measured load and queue state | Harder to explain and validate under changing traffic |

A size-or-time policy starts its timeout when the first order enters an empty batch. Later arrivals must not reset that oldest-order deadline. Fixed periodic boundaries are a different policy, even if both use 10 ms.

When quota is unavailable, compatible orders may collect during unavoidable waiting. Avoid adding another full formation window once a dispatch slot becomes available. Bound size using the venue's limit and preserve per-order deadlines. An “adaptive” policy which silently lengthens those deadlines changes the service promised to strategies.

## Queues turn overload into delay

Let $\lambda$ be incoming order operations per second and $\mu$ the sustainable completion capacity in the same units under the current request mix. Utilisation is $\rho=\lambda/\mu$.

If $\lambda>\mu$ persists, backlog grows unless we reject or remove work. At $\lambda<\mu$, bursts and variable processing times can still produce queues. Running near capacity leaves little room to absorb them; there is no universal utilisation threshold which predicts our tail latency.

Suppose effective capacity is 400 new orders per second, but arrivals remain at 500. After ten seconds, there are approximately 1,000 additional queued orders. If arrivals then stop, clearing that backlog takes about 2.5 seconds at the assumed capacity. With continuing arrivals, catch-up takes longer or never happens.

[Little's Law](https://pubsonline.informs.org/doi/abs/10.1287/opre.9.3.383) relates long-run averages:

$$
L=\lambda W
$$

Here $L$ is the average number of items within a defined system boundary, $\lambda$ its effective arrival rate, and $W$ their average residence time. Use a stable observation regime with finite averages, consistent units and balanced flow over the long run. No Poisson-arrival assumption is necessary. Exclude rejected requests from the admitted population, or account for their exits consistently.

If an admitted order queue averages 20 orders and processes 200 orders per second, its mean residence time is $20/200=0.1$ seconds. This does not reveal p99 waiting time. Nor can we apply the same steady-state interpretation to an indefinitely growing backlog, or combine a count of orders with a rate of API calls.

## The slowest requests matter

p50 is the median latency. p95, p99 and p99.9 describe increasingly slow parts of the distribution. At 100,000 observations, roughly 100 lie in the slowest 0.1%; that percentile needs enough samples to be meaningful.

A mean can hide cancellations which occasionally sit behind a burst of new orders. Scheduling, shared-resource contention, garbage collection, retransmission and venue variation can contribute. [The Tail at Scale](https://research.google/pubs/the-tail-at-scale/) discusses sources of latency variability in large services; our gateway must measure which sources actually apply.

Report latency per strategy and request class. Include queue age, formation wait, limiter wait, dispatch-to-acknowledgement latency, batch occupancy, rejects, expiries and timeouts. Count terminal outcomes too: a chart containing only successful requests hides overload failures.

Record raw correlated timings or compatible histograms. You cannot add stage p99s to obtain end-to-end p99, or average four strategy p99s into a meaningful system percentile. Test bursts on a schedule independent of prior completions; a load generator which waits for each response can stop injecting work precisely when the system stalls.

## What I would put in the gateway

Start with a single dispatch authority for each quota scope, bounded queues and explicit per-order state. Keep urgent cancellations and authorised risk-reducing actions out of ordinary formation windows, but run them through the same quota enforcement. Reserve some dispatch headroom for urgent work if measurements and venue rules support it.

```mermaid
flowchart TD
    accTitle: A gateway separates order policy from quota enforcement
    accDescr: Four strategies share admission and dispatch authority. Compatible ordinary requests are grouped, urgent requests avoid formation delay, and both pass the same venue-aware limiter.
    S[Four strategies via OMS] --> A[Validate and admit]
    A --> B[Compatible new-order batching]
    A --> U[Urgent cancel and risk-reducing queue]
    B --> Q[Deadline and priority scheduler]
    U --> Q
    Q --> R[Venue-aware quota enforcement]
    R --> D[Dispatch and track individual outcomes]
    D <-->|Requests and reports| V[Venue]
    D --> J[Durable state and reconciliation]
    A -.-> M[Per-strategy timings and outcomes]
    R -.-> M
    D -.-> M
```

“Compatible” must include endpoint, account, supported order types and dependency rules. Do not aggregate orders by netting strategies' quantities unless the trading policy explicitly authorises it. A batch container should preserve each strategy's intent and attribution.

Priority also has limits. A cancel cannot overtake creation of the order it references. If creation is still queued, resolve the cancel locally and report that outcome. If creation may already have been sent, treat it as an uncertainty to resolve. Modifications need ordering and venue-specific replace semantics. A priority queue cannot remove causal dependencies.

```mermaid
flowchart TD
    accTitle: Queue age and venue quota control dispatch
    accDescr: Ordinary batches and urgent requests share a bounded scheduler. Unsent expired requests leave explicitly, while valid work waits for quota and is reconsidered when capacity returns.
    B[Ordinary batches] --> S[Bounded priority scheduler]
    U[Urgent requests] --> S
    S --> E{Deadline and risk valid?}
    E -->|No| F[Report unsent expiry or rejection]
    E -->|Yes| R{Quota available?}
    R -->|No| W[Wait and observe queue age]
    W --> S
    R -->|Yes| D[Account quota and dispatch]
```

Select work again as capacity becomes available so a previously selected low-priority batch does not monopolise the next slot. Apply fairness where deadlines permit; otherwise urgent traffic can starve ordinary strategies. Monitor that outcome instead of assuming priorities create capacity.

Initially, use size-or-time batching only for strategies whose budgets permit it. Treat 10 ms as a candidate maximum formation interval to test, not a universal setting. More urgent work submits without deliberate formation delay when quota is available. If every strategy needs immediate dispatch beyond available capacity, software scheduling cannot satisfy the requirements: reduce traffic or obtain different venue access.

Bound queues by count, memory and age. Reject new unsent work explicitly when it cannot meet its deadline; do not silently forget accepted intent. Preserve uncertain sent work for reconciliation. Admission must leave capacity for critical cancellation requests and report when even those cannot be accepted.

During venue unavailability, stop accumulating unbounded new exposure. Retain required state, expire unsent stale signals visibly, and retry with bounded backoff according to the protocol. After a timeout, do not resubmit an entire partially successful batch blindly. Recover outcomes by individual identifiers and documented venue facilities. The communications article in this series explains why an unanswered request may already have had an effect.

Compare policies using the same captured arrival trace and a test venue with controlled acknowledgement delays, quotas and failures. Measure the whole latency distribution by strategy, not only calls saved. That evidence should determine whether batching earns its delay.

## Your turn

Your gateway batches every 10 ms. API utilisation falls, but one latency-sensitive strategy's execution performance deteriorates. Other strategies appear unaffected.

Which per-strategy and per-request timings would you inspect? Test whether the cause is formation wait, quota contention, changed arrival clustering, or venue response time. How would you distinguish a slower acknowledgement from worse fills or delayed cancellations?

Propose a revised policy, explain what it costs in quota capacity and fairness, and design a comparison which could disprove your hypothesis. Include overload and venue-outage cases; a quieter average should not conceal worse behaviour when urgent requests arrive together.


Part of [Engineering Trade-offs](/blog/engineering-tradeoffs-2627) · Previous: [How Should Your Services Communicate? Sockets, Queues and Shared Memory](/blog/engineering-tradeoffs-2627/how-should-your-services-communicate) · Next: [Concurrency vs Parallelism](/blog/engineering-tradeoffs-2627/concurrency-vs-parallelism)