Debug Complex Issue

# Debug Complex Issue

Systematically analyze and debug a complex issue using structured problem-solving techniques and comprehensive investigation.

## Investigation Framework

### 1. Problem Definition

**Issue Description**
- What is the problem? (be specific)
- When did it start occurring?
- How often does it occur? (always, intermittently, specific conditions)
- What is the impact? (severity, affected users, business impact)

**Expected vs Actual Behavior**
- What should happen?
- What is actually happening?
- What error messages or symptoms are present?

**Environment Details**
- Application version
- Operating system and version
- Browser/client version (if applicable)
- Database version
- External dependencies versions
- Configuration differences (dev vs production)

### 2. Reproduction Steps

**Minimal Reproduction Case**
1. Step-by-step instructions to reproduce
2. Required test data
3. Pre-conditions that must be met
4. Expected result at each step
5. Actual result observed

**Reproduction Rate**
- Consistently reproducible? (Yes/No)
- Percentage of attempts that reproduce issue
- Specific conditions required
- Time-dependent factors

### 3. Evidence Collection

**Logs and Traces**
- Application logs (with timestamps)
- Error logs and stack traces
- System logs
- Network traffic logs
- Database query logs
- Third-party service logs

**Metrics and Monitoring**
- CPU usage patterns
- Memory consumption
- Network latency
- Database performance
- API response times
- Error rates

**State Information**
- Application state before/during/after issue
- Database state
- Cache state
- Session information
- Environment variables
- Configuration values

### 4. Root Cause Analysis

**Hypothesis Generation**
For each potential cause:
- What could cause these symptoms?
- Is it consistent with all evidence?
- What would disprove this hypothesis?
- How can we test it?

**Common Categories to Investigate**

**Code-Level Issues**
- Logic errors
- Race conditions
- Memory leaks
- Null/undefined handling
- Type mismatches
- Incorrect algorithms

**Data Issues**
- Data corruption
- Invalid data states
- Missing data
- Data type mismatches
- Encoding issues

**Integration Issues**
- API contract mismatches
- Network timeouts
- Authentication failures
- Rate limiting
- Service unavailability

**Infrastructure Issues**
- Resource exhaustion (CPU, memory, disk)
- Network problems
- Database connection pool exhaustion
- Cache invalidation issues
- Load balancer configuration

**Configuration Issues**
- Incorrect environment variables
- Missing configuration
- Feature flags
- Permission settings
- Timeout values

### 5. Debugging Techniques

**Code-Level Debugging**

1. Add detailed logging at critical points
2. Use debugger with breakpoints
3. Add assertions to verify assumptions
4. Isolate the problematic code section
5. Binary search (comment out sections)
6. Rubber duck debugging

**System-Level Debugging**

1. Monitor resource usage
2. Check process states
3. Analyze thread dumps
4. Review network traffic
5. Examine database query plans
6. Check file system state

**Experimental Debugging**

1. Change one variable at a time
2. Compare working vs broken states
3. Test with different inputs
4. Test in different environments
5. Rollback recent changes
6. Bisect commit history

### 6. Resolution Strategy

**Quick Fixes (Immediate Mitigation)**
- Restart services
- Clear caches
- Rollback recent changes
- Adjust resource limits
- Enable circuit breakers
- Implement rate limiting

**Proper Solutions**
- Code fixes with tests
- Configuration updates
- Infrastructure improvements
- Process improvements
- Documentation updates

**Prevention Measures**
- Add monitoring and alerts
- Implement health checks
- Add input validation
- Improve error handling
- Add integration tests
- Update documentation

### 7. Solution Validation

**Testing Checklist**
- [ ] Issue no longer reproduces in test environment
- [ ] Solution works for all known reproduction cases
- [ ] No new issues introduced (regression testing)
- [ ] Performance impact acceptable
- [ ] Solution works in all environments
- [ ] Edge cases handled
- [ ] Error handling tested

**Monitoring Plan**
- Key metrics to watch
- Alert thresholds
- Dashboard for tracking
- Log analysis queries

## Debugging Output Template

### Issue Summary
- **Problem**: [One-line description]
- **Severity**: Critical / High / Medium / Low
- **Status**: Investigating / Root Cause Found / Fixed / Verified
- **First Observed**: [Date/Time]
- **Affected**: [Users/Systems affected]

### Root Cause
[Detailed explanation of what caused the issue]

### Evidence
- [Key log entries]
- [Relevant metrics]
- [Code snippets]
- [Screenshots]

### Solution
[Step-by-step fix with code changes]

### Testing
[How solution was verified]

### Prevention
[Steps to prevent recurrence]

### Timeline
- **Detected**: [Time]
- **Investigation Started**: [Time]
- **Root Cause Found**: [Time]
- **Fix Deployed**: [Time]
- **Verified**: [Time]

## Best Practices

- Stay objective and methodical
- Document everything as you investigate
- Don't assume - verify with data
- Test hypotheses systematically
- Communicate status regularly
- Keep stakeholders informed
- Learn from each issue
- Update documentation and runbooks
- Share knowledge with team
- Implement preventive measures