name: ddia-systems

description: 'Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", or "LSM tree vs B-tree". Also trigger when choosing between SQL and NoSQL, designing data pipelines, or debugging distributed system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.'

license: MIT

metadata:

author: wondelai

version: "1.2.0"

Designing Data-Intensive Applications Framework

A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.

Core Principle

Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.

Scoring

Goal: 10/10. Rate any data architecture 0-10 against the principles below: deliberate trade-off choices for data models, storage, replication, partitioning, transactions, and pipelines score high; accidental complexity and ignored failure modes score low. Report the current score and the improvements needed to reach 10/10.

The DDIA Framework

Seven domains for reasoning about data-intensive systems:

1. Data Models and Query Languages

Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.

Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.

Key insights:

• Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
• Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
• Polyglot persistence -- different stores for different access patterns -- is often the right answer
• Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates

Code applications:

See: references/data-models.md for relational/document/graph trade-offs and query language comparisons.

2. Storage Engines

Core concept: Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.

Why it works: Understanding your database's storage engine lets you predict performance characteristics, choose appropriate indexes, and avoid pathological workloads.

Key insights:

• LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
• B-trees: in-place updates, predictable read latency, write amplification from page splits
• Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles
• Column-oriented storage dramatically improves analytical queries through compression and vectorized processing
• In-memory databases are fast because they avoid encoding overhead, not because they avoid disk

Code applications:

See: references/storage-engines.md for LSM vs B-tree internals, compaction, and column storage.

3. Replication

Core concept: Replicat