Debugging & Incident Response

Technical Reference & Key Concepts

**Systematic Debugging Framework** 1. **Reproduce:** Get a reliable reproduction. Note environment, inputs, and exact error. 2. **Isolate:** Binary search through the stack - frontend, network, backend, database, infrastructure. 3. **Hypothesize:** Form 2-3 competing hypotheses. Use data (logs, metrics, traces) to eliminate. 4. **Test:** Change one variable at a time. Rollback immediately if the fix doesn't work. 5. **Root Cause:** Find the underlying issue, not just the symptom. Ask "why" 5 times. 6. **Prevent:** Add tests, monitoring, and runbooks so it never happens again the same way. **Logging Levels for Incidents** - **ERROR:** Something is broken right now. Needs immediate attention. - **WARN:** Something unexpected but not breaking. Might indicate upcoming issues. - **INFO:** Normal operational information. Useful for tracing requests in production. - **DEBUG:** Detailed diagnostic info. Usually off in production unless debugging a specific issue.

Practice discussing these concepts out loud in live voice drills on GitGrilled.