Ddia systems

--- name: ddia-systems description: 'Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use this when the user mentions "database choice," "replication lag," "partitioning strategy," "consistency vs. availability," "stream processing," "ACID transactions," "eventual consistency," or "LSM tree vs. B-tree." Also trigger when choosing between SQL and NoSQL, designing data pipelines, or debugging distributed system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.' license: MIT metadata: author: wondelai version: "1.2.0" --- # Designing Data-Intensive Applications Framework A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance. ## Core Principle Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades:prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs distinguish robust systems from fragile ones. ## Scoring Goal: 10/10. Rate any data architecture on a scale of 0:10 based on the principles below: deliberate trade-off choices for data models, storage, replication, partitioning, transactions, and pipelines score high; accidental complexity and ignored failure modes score low. Report the current score and the improvements needed to reach 10/10. ## The DDIA Framework Seven domains for reasoning about data-intensive systems: ### 1. Data Models and Query Languages Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns. Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time. Key insights: - Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data - Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility - Polyglot persistence:different stores for different access patterns:is often the right answer - Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates Code examples: | Context | Pattern | Example | |---------|---------|---------| | User profiles with nested data | Document model for self-contained aggregates | Profile, addresses, and preferences in a single MongoDB document | | Social network connections | Graph model for relationship traversal | Neo4j Cypher: `MATCH (a)-[:FOLLOWS*2]->(b)` for friend-of-friend | | Financial ledger with joins | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, and entries | See: references/ data-models.md for relational/document/graph trade-offs and query language comparisons. ### 2. Storage Engines Core concept: Storage engines balance read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes. Why it works: Understanding your database’s storage engine allows you to predict performance characteristics, choose appropriate indexes, and avoid pathological workloads. Key insights: - LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification - B-trees: in-place updates, predictable read latency, write amplification from page splits - Write amplification (one logical write causing multiple physical writes) is important for SSDs with limited write cycles - Column-oriented s