SOC2 PI1.4 - Error Detection and Correction

Control: PI1.4 - Error Detection and Correction Status: Procedure template Last Updated: 2025-12-16 Owner: Infrastructure Administrator

---

Overview

This document defines infrastructure-level error detection capabilities and application-level error handling requirements for CloudForge CI. These procedures are a template for addressing SOC2 PI1.4 (Error Detection and Correction) requirements.

---

Infrastructure Error Detection (Automated)

CloudForge CI configures the following infrastructure monitoring:

CloudWatch Alarms

Thresholds depend on the security profile (production / staging / dev):

Alarm Type	Metric	Threshold	Created by	Action
High CPU	CPUUtilization	70% / 80% / 90%, two 5-minute periods	SecurityMonitoringFactory	SNS notification
High Memory	MemoryUtilization	80% / 85% / 90%, two 5-minute periods	SecurityMonitoringFactory	SNS notification
5xx Errors	HTTPCode_ELB_5XX_Count	5 / 7 / 10 per minute	AlarmFactory	SNS notification
4xx Errors	HTTPCode_ELB_4XX_Count	20 / 30 / 50 per 5 minutes, two periods	AlarmFactory	SNS notification
Target Response Time	TargetResponseTime	2s / 3s / 5s, two 5-minute periods	AlarmFactory	SNS notification

Unhealthy target detection relies on the load balancer health checks below.

Health Checks

// AlbFactory configures health checks:
.healthCheck(HealthCheck.builder()
    .path(healthCheckPath)
    .healthyHttpCodes("200-299")
    .interval(Duration.seconds(30))
    .timeout(Duration.seconds(5))
    .healthyThresholdCount(2)
    .unhealthyThresholdCount(3)
    .build())

Database Monitoring

Metric	Threshold	Indicates
DatabaseConnections	> 80% max	Connection pool exhaustion
FreeableMemory	< 256MB	Memory pressure
FreeStorageSpace	< 10GB	Disk space critical
ReadLatency/WriteLatency	> 100ms	I/O bottleneck
ReplicaLag	> 60s	Replication issues

Container/ECS Monitoring

Metric	Threshold	Action
Service Desired vs Running	Mismatch	Deployment issue
Task Stopped	Unexpected stop	Container crash
CPU/Memory Reservation	> 80%	Scaling needed

---

Application Error Handling Requirements

Applications deployed on CloudForge CI must implement:

1. Structured Error Logging

Requirement: All errors must be logged in structured JSON format.

{
  "timestamp": "2025-12-16T10:30:00Z",
  "level": "ERROR",
  "service": "payment-service",
  "traceId": "abc123",
  "error": {
    "type": "ValidationError",
    "message": "Invalid card number format",
    "code": "CARD_001",
    "stackTrace": "..."
  },
  "context": {
    "userId": "user-123",
    "requestId": "req-456"
  }
}

Compliance: Enables CloudWatch Logs Insights queries for error analysis.

2. Error Classification

Applications must classify errors:

Category	HTTP Status	Retry	Alert
Client Error	4xx	No	Aggregate
Validation Error	400	No	No
Authentication Error	401/403	No	Threshold
Server Error	5xx	Yes	Immediate
Dependency Error	502/503/504	Yes	Immediate
Timeout	504	Yes	Immediate

3. Error Response Format

Standard error response structure:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Human-readable error message",
    "details": [
      {
        "field": "email",
        "message": "Invalid email format"
      }
    ],
    "requestId": "req-456"
  }
}

4. Retry Logic with Exponential Backoff

For transient errors, implement retry:

// Example retry configuration
RetryConfig.builder()
    .maxAttempts(3)
    .waitDuration(Duration.ofMillis(500))
    .retryOnResult(response -> response.getStatusCode() >= 500)
    .retryOnException(e -> e instanceof TimeoutException)
    .exponentialBackoffMultiplier(2)
    .build();

5. Circuit Breaker Pattern

Prevent cascade failures:

// Example circuit breaker
CircuitBreakerConfig.builder()
    .failureRateThreshold(50)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowSize(10)
    .build();

---

Error Correction Procedures

Automated Correction

Error Type	Automated Response
Unhealthy container	ECS replaces task
Failed health check	ALB removes target
High CPU	Auto Scaling adds capacity
Deployment failure	CloudFormation rollback

Manual Correction

Error Type	Procedure
Application bug	Hotfix → PR → Deploy
Data corruption	Restore from backup
Configuration error	Update deployment context, redeploy
Dependency failure	Implement fallback, notify vendor

Incident Response

Error Detected
     ↓
Automated alert (CloudWatch → SNS → Slack/Email)
     ↓
Acknowledge alert
     ↓
Triage: Auto-recoverable or manual intervention?
     ↓
[Auto-recoverable]              [Manual intervention]
     ↓                                  ↓
Monitor recovery              Investigate root cause
     ↓                                  ↓
Document                      Implement fix
                                        ↓
                              Test & deploy
                                        ↓
                              Document & post-mortem

---

Monitoring Dashboards

Application Team Dashboard Requirements

Each application should have a CloudWatch dashboard with:

1. Error Rate Panel

   • 5xx errors per minute
   • 4xx errors per minute
   • Error rate percentage

2. Latency Panel

   • P50, P90, P99 response times
   • Target response time trend

3. Availability Panel

   • Healthy host count
   • Request count
   • Success rate

4. Resource Panel

   • CPU utilization
   • Memory utilization
   • Connection count

Example Dashboard Definition

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "title": "Error Rate",
        "metrics": [
          ["AWS/ApplicationELB", "HTTPCode_Target_5XX_Count", "TargetGroup", "${tg}"],
          [".", "HTTPCode_Target_4XX_Count", ".", "."]
        ],
        "period": 60,
        "stat": "Sum"
      }
    }
  ]
}

---

Error Reporting

Daily Error Summary (Automated)

CloudWatch Logs Insights query for daily error report:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(1h)
| sort @timestamp desc

Weekly Error Review

Agenda:

1. Review top errors by frequency
2. Identify recurring issues
3. Prioritize fixes
4. Update error handling if needed

Evidence: Weekly review notes stored in [INTERNAL]/engineering/error-reviews/

---

Validation and Data Integrity

Input Validation Requirements

// All user input must be validated
public class ValidationRules {
    // Required fields
    @NotNull
    @NotEmpty
    private String requiredField;

    // Format validation
    @Email
    private String email;

    // Range validation
    @Min(0) @Max(100)
    private Integer percentage;

    // Pattern validation
    @Pattern(regexp = "^[A-Z]{2}[0-9]{6}$")
    private String referenceCode;
}

Output Validation

Applications must validate:

• Response data matches expected schema
• Sensitive data is masked/excluded
• Error responses don't leak internal details

---

Evidence Collection

Automated Evidence

• CloudWatch alarm history
• CloudWatch Logs (error logs)
• X-Ray traces (if enabled)
• ECS task stopped events
• ALB access logs

Manual Evidence

Evidence Type	Location	Retention
Error review meeting notes	Confluence	3 years
Incident post-mortems	Confluence	7 years
Bug fix tickets	Jira	7 years
Dashboard screenshots	Confluence	1 year

---

Compliance Mapping

Requirement	Implementation	Evidence
SOC2 PI1.4	Error logging, monitoring, correction	CloudWatch, incident tickets
SOC2 CC7.2	System monitoring	CloudWatch alarms
PCI-DSS Req 10.2	Audit logging	CloudTrail, application logs
HIPAA §164.312(b)	Audit controls	CloudWatch Logs

---

Responsibilities

Role	Responsibility
Infrastructure Administrator	Infrastructure monitoring, alerting setup, error triage, incident response
Application Developer	Application error handling, logging, dashboards

---

Document Control

Version	Date	Author	Changes
1.0	2025-12-16	CloudForge CI	Initial release