# How Should Your Services Communicate? Sockets, Queues and Shared Memory

Szymon Kopyciński · 12 October 2026

A feed handler, four strategies, an OMS and a gateway. Which IPC method do they use?

## Introduction

A feed handler receives market updates. Four strategies consume them and send order requests to an order management system, or OMS. A risk component evaluates exposure, an exchange gateway talks to the venue, and an event logger records what happened.

Some components are separate processes on one machine. Others may move to different servers. The team proposes choosing one communication technology for all of them.

Consider three messages: a new best bid, a request to buy 100 shares, and a report that 40 shares traded. Losing each message has a different consequence. Recovering each also requires different information.

That difference should shape the architecture before we compare sockets, Redis, RabbitMQ and shared memory.

## Which guarantees do we need?

Use illustrative requirements: market data peaks at 100,000 updates per second across subscriptions. Target a publication-to-strategy-receipt latency of at most 1 ms at p99: 99% of measured updates should arrive within that budget under the agreed load. Orders arrive much less frequently, and these strategies can accept several milliseconds of internal order processing. These are design targets, not measured results or requirements for every systematic platform.

Functionally, strategies must subscribe, submit requests, receive status and recover after restarting. Risk and positions must account for executions. Non-functional requirements specify delay, bounded memory, recoverability and independent restarts.

Require accepted order intent and execution records to survive process crashes and host restarts with intact storage. Permanent disk loss is a stronger requirement needing separate durable copies or replication. A slow logger must not block market-data delivery indefinitely. That does not mean we may discard the only record of a fill.

| Path | What matters most | When the consumer falls behind |
| --- | --- | --- |
| Book deltas to strategies | Ordered, trustworthy, fresh state | Detect a gap and recover before trading |
| Order requests to OMS | Stable identity and explicit outcome | Reject overload visibly; resolve uncertain requests |
| Execution events to positions and risk | Recoverable effects and consistent state | Retain history; gate trading if required state is stale |
| Dashboard notifications | Convenient updates | Reconnect and fetch authoritative state |

Some market information can be replaced. If a consumer wants only the latest complete quote, an older quote may be dispensable. Incremental book updates are different: skipping one can corrupt state. The recovery mechanics are covered in [From Market Data to Order Book](/blog/quant-foundations-2627/how-market-data-becomes-an-orderbook). Here the question is where our internal channel enforces them.

## A process boundary changes the problem

Processes normally have separate address spaces. A Python object in the feed handler is not automatically visible to a strategy. *Inter-process communication*, or IPC, transfers information across that boundary.

With a socket, we usually serialise an object into bytes, pass those bytes through operating-system buffers and reconstruct a message at the other end. That involves encoding, copying, scheduling and possibly networking. The proportions depend on implementation and workload; a slow serialiser can dominate a local transport comparison.

Shared memory instead maps a region into both processes. They can agree on a binary layout and communicate through that region, but must coordinate access.

These candidates are not at the same abstraction level. Sockets are transports. Shared memory is a memory-sharing mechanism. Redis and RabbitMQ are services offering messaging or data structures, generally reached through network connections. A broker does not replace networking; it adds behaviour above it.

## When a socket is enough

Unix domain sockets communicate locally. TCP sockets also work across hosts. On Linux, Unix sockets support stream, datagram and sequenced-packet modes. The latter two preserve message boundaries; stream mode does not. Local datagram semantics must not be confused with network UDP. See [unix(7)](https://man7.org/linux/man-pages/man7/unix.7.html).

[TCP](https://man7.org/linux/man-pages/man7/tcp.7.html) provides an ordered byte stream on a connection. It does not preserve the boundaries between application writes. If the sender writes two requests, a receiver may read part of one or both together. A protocol must frame messages, perhaps with a length prefix, and validate lengths, message types and schema versions. It must handle partial reads and writes.

Neither successful `send` nor a TCP acknowledgement proves that the OMS accepted an order. Bytes might be in a buffer, or the OMS might crash before recording them. Transport reliability describes the stream; application reliability describes the work performed with it.

Direct sockets suit a small, known topology. We accept responsibility for subscriptions, reconnects, heartbeats, stale-state detection and bounded send buffers. A strategy restart should obtain a valid snapshot and sequence boundary before consuming deltas. Ordering across reconnects or several connections requires application identifiers; one connection's ordered bytes do not establish a global order.

Changing Unix sockets to TCP later preserves much of a framed protocol, but introduces authentication, encryption, network partitions and remote failure detection. It is a deployment change with consequences, not merely a new address.

## What a broker adds

A broker can route messages, retain them, track consumer acknowledgements and redeliver work after a consumer disconnects. That can save considerable application infrastructure when consumers restart independently.

RabbitMQ routes publications through exchanges to queues. Four consumers on one queue generally share the work; four components which each need every execution event need separate queues or another suitable fan-out arrangement.

A *publisher confirm* reports broker acceptance according to the destination's semantics. A *consumer acknowledgement* tells the broker it may consider a delivery handled. These are separate, as the [RabbitMQ acknowledgement documentation](https://www.rabbitmq.com/docs/confirms) explains. Neither proves a trading venue accepted an order.

For crash recovery, specify durable queues, persistent messages where applicable, publisher confirms and manual consumer acknowledgements. If broker-node failure must be tolerated, [quorum queues](https://www.rabbitmq.com/docs/quorum-queues) replicate and confirm through a majority. Loss of that majority prevents progress; replication trades additional work and availability constraints for data safety. A single broker with a disk is a different failure model.

RabbitMQ [queue ordering](https://www.rabbitmq.com/docs/queues) is also scoped. A single publishing channel establishes enqueue order, but interleaved publishers, priorities, redelivery and concurrent consumer processing complicate the observed sequence. Preserve per-order sequence numbers and serialise dependent state transitions. An acknowledged queue message is normally removed; a queue is not automatically a permanent audit history. RabbitMQ also has streams, a distinct retained-log abstraction.

The broker adds monitoring, capacity planning, credentials, upgrades and recovery procedures. An unavailable broker must lead to defined behaviour: retain unpublished critical events durably, stop accepting work when storage fills, and prevent components with stale risk state from authorising new exposure. Buffering forever is not a recovery policy.

## Redis is two different proposals

[Redis Pub/Sub](https://redis.io/docs/latest/develop/pubsub/) delivers to currently subscribed clients with at-most-once semantics. If a client disconnects, it does not replay missed publications when that client returns. It can be useful for dashboard invalidations when the dashboard can fetch current state again. Using it as the sole execution-event record would not meet our recovery requirement.

[Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/) stores entries with IDs, supports reading retained history and tracks pending deliveries in consumer groups. Consumers in one group divide work; separate groups can each consume the stream. Reclaiming abandoned pending entries must be part of recovery, not an assumption that every failed delivery automatically retries itself.

`XACK` removes an entry from a group's [pending list](https://redis.io/docs/latest/commands/xack/); it does not ordinarily delete the stream entry. Retention and trimming are separate decisions. Trim too early and a disconnected consumer may lose the history it needs. Processing order can differ from entry order when consumers work concurrently.

Retained in Redis memory is not the same as durable through a power failure. The [persistence policy](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) determines what reaches disk; AOF with an every-second synchronisation policy can lose recent acknowledged writes. [Asynchronous replication](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/) also leaves failure cases where acknowledged writes are lost. Specify the tolerated loss and failover configuration before calling the path durable.

Streams can be reasonable where Redis is already operated and its recovery guarantees fit. They still need consumer recovery, retention and memory planning. Check the selected version's [licence](https://redis.io/legal/licenses/): Redis 8 offers AGPLv3, RSALv2 and SSPLv1 choices; do not assume every release has the older BSD terms. Deployment terms are part of the decision, not a latency metric.

## Delivery is not correct processing

Suppose a positions consumer receives fill `F17`, increases a position by 40, then crashes before acknowledging. A broker may deliver `F17` again. That is *at-least-once delivery*: retries aim to prevent loss, but duplicates are possible within the stated failure and retention assumptions.

The effect must be *idempotent*: processing the same fill again leaves the position unchanged. Record the fill's identity and the position update in one transaction. Otherwise a crash between those writes can cause a duplicate effect or a missed effect.

For order requests, the strategy creates a stable request ID before sending and retains it across retries and restarts. The OMS stores the request ID, payload and result atomically with the resulting intent. Concurrent duplicates must resolve to the same record; reuse with different parameters is rejected. Retention must cover the permitted retry horizon. This follows the reasoning behind [idempotent API design](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/), rather than inferring intent from identical payloads.

An OMS acceptance means “intent recorded”, not “order working at the venue”. A venue timeout creates a second uncertainty. Use venue-supported client-order identifiers and status queries, execution replay or reconciliation. A client-order ID is only a deduplication guarantee if the venue documents that behaviour and its scope. Internal deduplication alone cannot make an external side effect atomic with our database transaction.

## Why shared memory earns a separate discussion

[POSIX shared memory](https://man7.org/linux/man-pages/man7/shm_overview.7.html) lets processes share a mapped region. A ring buffer can hold fixed-layout messages in reusable slots: a producer publishes completed slots and a consumer advances its read position.

This can reduce transport copying and per-message system calls. It does not eliminate encoding, cache traffic or scheduling, and it does not automatically beat sockets for every message size.

The implementation must establish ownership, memory visibility and what happens when producer and consumer positions wrap. Writing payload bytes before publishing a position needs appropriate synchronisation; otherwise a consumer may observe an incomplete message. Multiple consumers need independent progress tracking or separate rings. A normal process-local mutex or casual Python assignment is not a portable cross-process protocol.

We must also handle a producer crash mid-write, consumer restarts, schema compatibility and full buffers. Shared pages are not a durable event log. Processes on another host cannot map the same RAM through this mechanism.

Measured socket overhead could justify a proven shared-memory channel in a tightly controlled local deployment. It is a poor default merely because the system trades financial instruments.

## Slow consumers still exist

At 100,000 updates per second, a stalled consumer accumulates 50,000 messages in half a second. Buffers hide the stall briefly. They do not increase the consumer's processing capacity.

```mermaid
flowchart TD
    accTitle: Independent buffers prevent one subscriber blocking all others
    accDescr: Each subscriber has a bounded buffer. A lagging strategy enters recovery, while an exhausted durable capture path stops trading rather than silently losing required history.
    F[Feed handler] --> A[Strategy A buffer]
    F --> B[Strategy B buffer]
    F --> C[Capture buffer]
    B --> H{Lag or capacity limit?}
    H -->|Yes| R[Invalidate subscription and resynchronise]
    H -->|No| S[Process deltas]
    C --> D[Durable capture writer]
    C -->|Cannot retain required history| T[Controlled trading halt]
```

*Backpressure* means making upstream work slow, stop or be rejected when downstream capacity is exhausted. An exchange market feed may not be pausable, so stopping our read is not a lossless solution. Isolate subscribers, measure message age, and invalidate a strategy which cannot stay current.

Keep diagnostic logging off the critical path, with an explicit drop policy if appropriate. Critical event capture needs a bounded durable spool and an exhaustion policy. Isolation prevents indefinite blocking; it cannot promise indefinite lossless operation during a storage outage.

## What I would connect first

For our small topology, start with framed Unix domain stream sockets between colocated processes, and TCP where a component is already remote. Use per-strategy market-data buffers, sequence checks and snapshot recovery. Apply the same identity-based order protocol over either transport.

Keep the OMS as the authoritative sequencer of order intent, live positions and risk reservations. The risk component supplies checks and versioned limits; the OMS atomically reserves against current exposure when recording intent. Four strategies must not each receive approval against the same unreserved capacity. If required risk information is stale, block new exposure.

Use an existing transactional store, such as PostgreSQL with [durable commits enabled](https://www.postgresql.org/docs/18/wal-reliability.html), for intent and an execution-event journal. An append-only journal records state changes so they can be replayed. Commit an event with its state update, and let downstream consumers advance durable cursors after their own committed effects. Use a committed publication order: an increasing ID allocated before commit can otherwise cause readers to skip a late transaction. The logger builds its archive from that journal. This initially avoids operating a broker solely to move low-volume execution records.

```mermaid
flowchart TD
    accTitle: Separate fresh market data from recoverable trading state
    accDescr: Strategies use sockets for fresh updates and identified commands. OMS and gateway retain critical records in a transactional journal, which downstream consumers can replay independently.
    F[Feed handler] -->|Sequenced sockets| S[Four independent strategies]
    F --> C[Durable market-data capture]
    S -->|Request ID and status protocol| O[OMS]
    O <-->|Checks and limit versions| R[Risk component]
    O <-->|Intent and event transactions| J[State store and event journal]
    O -->|Identified commands| G[Exchange gateway]
    G <-->|Venue protocol| V[Venue]
    G -->|Persist reports| J
    J -->|Recoverable status feed| O
    J -->|Replay by cursor| P[Positions and reconciliation]
    J -->|Replay by cursor| R
    J -->|Replay by cursor| L[Persistent event logger]
    O -->|Status and fills| S
```

The gateway deduplicates commands by intent ID and records dispatch state and venue reports; the OMS consumes them idempotently. Gate new exposure when required execution state is behind or uncertain. If a report cannot be persisted, stop new exposure and recover through the venue's protocol. Do not claim survival of all venue events without verifying its replay or reconciliation facilities.

```mermaid
sequenceDiagram
    accTitle: Acceptance and execution are separate milestones
    accDescr: OMS acceptance follows durable intent and reservation. Venue reports are recorded and applied before being delivered to downstream consumers.
    participant S as Strategy
    participant O as OMS
    participant D as Journal
    participant G as Gateway
    participant V as Venue
    S->>O: Submit request R42
    O->>D: Commit R42 and reservation
    D-->>O: Committed
    O-->>S: Intent accepted
    O->>G: Dispatch R42
    G->>D: Record dispatch intent
    G->>V: Submit with client-order ID
    V-->>G: Ack, then fill F17
    G->>D: Commit venue reports
    O->>D: Apply F17 once
    O-->>S: Status and F17
    Note over D: Independent replay for consumers
```

At the ambiguous dispatch boundary, a recorded attempt does not prove receipt by the venue. Recovery must resolve that state before resending an order.

```mermaid
sequenceDiagram
    accTitle: A lost acknowledgement does not create a new request
    accDescr: The strategy retries the same persisted request ID after reconnecting. The OMS returns recorded status without creating another intent.
    participant S as Strategy
    participant O as OMS
    participant D as State store
    S->>O: Submit R42
    O->>D: Commit R42 and intent
    D-->>O: Committed
    O--xS: Acceptance lost on disconnect
    Note over S: Outcome unknown, preserve R42
    S->>O: Reconnect and query or retry R42
    O->>D: Look up R42 and compare payload
    D-->>O: Existing intent and current status
    O-->>S: Same intent, current status
```

## What would change the design?

**Everything on one machine:** keep sockets and the journal; processes remain independently restartable. The shared host remains a common failure boundary.

**Strategies across servers:** use authenticated, encrypted TCP, and revisit bandwidth, reconnect recovery and partition behaviour. If many downstream consumers need independent delivery, add RabbitMQ through a transactional *outbox*: commit an event and pending-publication record together, then publish with confirms and retry. Consumers remain idempotent. The journal retains audit history even after queues acknowledge messages.

**Recoverability dominates latency:** require replicated storage and, if using RabbitMQ, quorum queues; acknowledge only at the chosen durability boundary. Test failover and reconciliation. Replication cost belongs in the latency budget.

**Much lower and more predictable latency:** profile encoding, scheduling and queue age first. A proven shared-memory channel, careful CPU placement and allocation control may then be justified. Keep durable execution recovery even if market-data delivery changes.

## Your turn

A strategy submits an order. The OMS processes it, but the connection fails before confirmation arrives.

Design the protocol for the strategy's retry, including request identity, persisted state, acknowledgement meaning and a status query. What happens if the OMS also restarts? If the retry changes quantity? If the OMS dispatched to the venue but never received its response? Explain where deduplication works, where the outcome remains uncertain, and how reconciliation prevents a second unintended order.