Chaos engineer

--- name: chaos-engineer description: Designs chaos experiments, creates failure injection frameworks, and facilitates game day exercises for distributed systems:producing runbooks, experiment manifests, rollback procedures, and post-mortem templates. Use when designing chaos experiments, implementing failure injection frameworks, or conducting game day exercises. Invoke for chaos experiments, resilience testing, blast radius control, game days, antifragile systems, fault injection, Chaos Monkey, and Litmus Chaos. license: MIT metadata: author:https://github.com/Jeffallan version: "1.1.0" domain: devops triggers: chaos engineering, resilience testing, failure injection, game day, blast radius, chaos experiment, fault injection, Chaos Monkey, Litmus Chaos, antifragile role: specialist scope: implementation output-format: code related-skills s: sre-engineer, devops-engineer, kubernetes-specialist --- # Chaos Engineer ## When to Use This Skill - Designing and executing chaos experiments - Implementing failure injection frameworks (Chaos Monkey, Litmus, etc.) - Planning and conducting Game Day exercises - Building blast radius controls and safety mechanisms - Setting up continuous chaos testing in CI/CD - Improving system resilience based on experiment findings ## Core Workflow 1. System Analysis - Map architecture, dependencies, critical paths, and failure modes 2. Experiment Design - Define hypotheses, steady state, blast radius, and safety controls 3. Execute Chaos - Run controlled experiments with monitoring and quick rollback 4. Learn & Improve - Document findings, implement fixes, and enhance monitoring 5. Automate - Integrate chaos testing into CI/CD for continuous resilience ## Reference Guide Load detailed guidance based on context: | Topic | Reference | Load When | |-------|-----------|-----------| | Experiments |references/experiment-design.md | Designing hypotheses, blast radius, rollback | | Infrastructure |references/infrastructure-chaos.md | Server, network, zone, and region failures | | Kubernetes |references/kubernetes-chaos.md | Pod, node, Litmus, and chaos mesh experiments | | Tools & Automation |references/chaos-tools.md | Chaos Monkey, Gremlin, Pumba, CI/CD integration | | Game Days |references/game-days.md | Planning, executing, learning from game days | ## Safety Checklist Non-obvious constraints that must be enforced for every experiment: - Steady state first : define and verify baseline metrics before injecting any failure - Blast radius cap : start with the smallest possible impact scope; expand only after validation - Automated rollback ≤ 30 seconds : the abort path must be scripted and tested before the experiment begins - Single variable : change only one failure condition at a time until behavior is well understood - No production without safety nets : customer-facing environments require circuit breakers, feature flags, or canary isolation - Close the loop : every experiment must produce a written summary of lessons learned and at least one tracked improvement ## Output Templates When implementing chaos engineering, provide: 1. Experiment design document (hypothesis, metrics, blast radius) 2. Implementation code (failure injection scripts/manifests) 3. Monitoring setup and alert configuration 4. Rollback procedures and safety controls 5. Learning summary and improvement recommendations ## Concrete Example: Pod Failure Experiment (Litmus Chaos) The following shows a complete experiment:from hypothesis to rollback:using Litmus Chaos on Kubernetes. ### Step 1 : Define the steady state and apply the experiment ``bash # Verify baseline: p99 latency < 200ms, error rate < 0.1% kubectl get deploy my-service -n production kubectl top pods -n production -l app=my-service ` ### Step 2 : Create and apply a Litmus ChaosEngine manifest ``yaml #chaos-pod-delete.yaml apiVersion : litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: my-service-pod-delete namespace: production spec: appinfo: appns: production applabel: "app=my-service" appkind: deployment # Limit blast radius: only 1 replica at a time engineState : active