Debugging: Production Incident

Interviewer

Your team's API latency just spiked from 50ms to 5 seconds at the 99th percentile. How do you debug this?

Candidate

First, I'd check if this is a full outage or partial degradation. I'd look at our dashboards: is CPU high? Memory? Network? Is it all endpoints or just specific ones? If it's all endpoints, the issue is likely infrastructure (database, cache, or network). If it's specific endpoints, it's application-level. Let me check the database first since that's the most common bottleneck. I'd look at: connection pool exhaustion, slow queries, or a missing index. If the DB is fine, I'd check the cache - maybe Redis evicted all keys and now every request hits the DB directly. If it's a specific endpoint, I'd check recent deploys. Roll back the latest change and see if latency recovers. If it does, the new code has a bug or regression.

Interviewer

You find that the database CPU is at 100%. What next?

Candidate

I'd enable slow query logging immediately if it's not already on. Check pg_stat_activity (Postgres) or equivalent for running queries. Look for queries with high execution times or those stuck waiting on locks. Common causes: a missing index on a new query pattern, an unoptimized JOIN added in a recent deploy, or a full table scan on a large table. If I can't fix it immediately, I'd consider: (1) Kill the offending query, (2) Add a read replica to offload read traffic, (3) Rate-limit the endpoint, (4) If critical, fail open and serve stale cached data.