Engineering Trade-offs
Exploring the decisions and trade-offs behind software engineering, from databases to distributed systems, through practical scenarios drawn from quantitative trading.
Introduction
You're building a market data platform. You need to store millions of events, move data between processes, and make sure your trading strategies receive information quickly and reliably.
Do you use DuckDB, ClickHouse, or kdb+? Should your processes communicate over Unix sockets, shared memory, or a message broker? Is it worth sacrificing latency to improve throughput?
Ask three engineers and you might get four different answers.
The frustrating (and interesting) thing about engineering is that there often isn't a single correct answer. A solution that works brilliantly in one environment might be completely inappropriate in another.
What matters is understanding why.
What this series is about
Engineering Trade-offs explores the decisions, compromises, and constraints behind building software systems, with a particular focus on quantitative trading infrastructure.
Rather than teaching you how to use a particular programming language, framework, or database, we want to explore how engineers decide which technologies to use, when to use them, and why.
We'll cover topics such as:
- Databases: When would you choose DuckDB over ClickHouse or kdb+? What makes a database suitable for a particular workload?
- Inter-process communication: Unix sockets, shared memory, Redis, RabbitMQ. How should different components of a trading system communicate?
- Performance: What's the difference between latency and throughput? When does batching help, and when does it hurt?
- System design: How do you balance correctness, reliability, performance, and complexity?
These aren't questions with universally correct answers; they're questions that require you to understand the problem before choosing a solution.
How we'll approach it
Each article starts with a realistic engineering scenario, usually inspired by challenges encountered in high-frequency trading, proprietary trading, or systematic investment management.
We'll establish the requirements, explore possible approaches, and examine their trade-offs. Then we'll return to the original problem and make a decision, explaining what assumptions justify it and what might change our minds.
Where useful, we'll include diagrams, quantitative examples, and small exercises to get you thinking.
The aim isn't to make you an expert in every technology we discuss. It's to help you develop the intuition to evaluate technologies you haven't encountered before.
Who should read this series?
Anyone interested in software engineering, quantitative development, or the systems behind financial markets.
You don't need experience building trading infrastructure, or even particularly advanced programming knowledge. Some familiarity with programming will help, but we'll introduce more specialised concepts as we encounter them.
Good engineering isn't about knowing every tool or memorising language syntax off by heart. It's being able to define problems well, and making well-informed trade-offs.
NEFS Quant · About · Blog · News & Events · People · Privacy policy · Terms of use