The Code AI Can't Write (Yet)

​

The Honest Assessment

​

The Seven Failure Categories

​

1. Novel Problems (No Pattern to Match)

// AI's approach: Runtime validation
fn get_client(&self, scope: Scope) -> Client {
    if scope.is_valid() {  // Runtime check
        Client::new(scope)
    } else {
        panic!("Invalid scope")
    }
}

// Compile-time enforcement
impl CapsuleClient {
    fn table_name(&self, base: &str) -> String {
        format!("{}_{}", self.capsule.code, base)
    }
}

impl PlatformClient {
    fn table_name(&self, base: &str) -> String {
        base.to_string()  // No prefix
    }
}

​

2. Framework Quirks (Middleware Ordering, Lifecycle)

// ❌ AI's initial code (wrong order)
App::new()
    .wrap(ConfigMiddleware::new(config_service))  // Runs first, needs capsule
    .wrap(CapsuleExtractor::new())                // Runs second, provides capsule

// ✅ Human fix
App::new()
    .wrap(CapsuleExtractor::new())                // Runs first
    .wrap(ConfigMiddleware::new(config_service))  // Runs second, has capsule

​

3. Performance Optimization (Needs Profiling Data)

// AI's code: Assume role on every request
pub async fn operator_client(&self) -> Client {
    let sts = self.sts_client();
    let creds = sts.assume_role()  // 200-500ms latency
        .role_arn(&self.role_arn)
        .send()
        .await?;

    Client::new_with_credentials(creds)
}

// Human optimization: Cache credentials
pub async fn operator_client(&self) -> Client {
    if let Some(creds) = self.cache.get(&self.role_arn) {
        return Ok(Client::new_with_credentials(creds));
    }

    let creds = self.sts_client()
        .assume_role()
        .send()
        .await?;

    self.cache.insert(self.role_arn.clone(), creds.clone());
    Ok(Client::new_with_credentials(creds))
}

​

4. Security Hardening (Requires Threat Modeling)

• Cross-tenant queries via timestamp-based GSI (leaked data)
• Missing tenant validation in batch operations
• Race conditions in tenant-scoped locks
• Token substitution attacks (swap tenant_id in JWT)

​

5. Operational Concerns (Caching, Monitoring, Scaling)

​

6. Business Logic (Domain-Specific Rules)

• Charge per API call
• Monthly aggregation
• Pro-rated refunds

• Don’t charge for failed requests (500 errors)
• Don’t charge during maintenance windows
• Don’t charge for health checks
• Cap monthly charge at contract limit
• Handle timezone boundaries for “monthly”

​

7. Edge Cases (Requires Deep Context)

​

Real Examples: What AI Did vs. What AI Missed

​

Week 6: AWS Client Factory

​

Why These Boundaries Exist

​

1. Training Data Limitations

• Production deployment configs: Not in repos (secrets, scaling params)
• Incident post-mortems: Private docs, not public code
• Performance profiling results: Runtime data, not source code
• Security threat models: Confidential, not open-source
• Business domain knowledge: In people’s heads, not documentation

​

2. Context Window Limits

• Single crate implementation
• Related test files
• Architecture docs

• Entire workspace (9 crates, 180 files)
• Cross-crate dependency chain
• Historical evolution (why code changed)

​

3. Operational Knowledge Gap

• AWS STS assume-role latency
• DynamoDB 400KB item size limit
• EventBridge PutEvents throttling
• CORS preflight optimization
• HTTP/2 connection pooling

​

What This Means for Workflows

​

Where Humans Add Value

​

The Decision Tree: When to Use AI vs. Intervene

New task arrives:

Is it a novel problem AI hasn't seen?
├─ YES → Human designs, AI implements
└─ NO → Continue

Does it involve framework-specific quirks?
├─ YES → AI implements, human reviews framework behavior
└─ NO → Continue

Does it need performance optimization?
├─ YES → Human profiles first, then AI optimizes hotspots
└─ NO → Continue

Is it security-critical?
├─ YES → Human threat models, AI implements controls
└─ NO → Continue

Does it involve operational concerns?
├─ YES → AI builds infrastructure, human adds monitoring/caching
└─ NO → AI can handle end-to-end

Is it domain-heavy business logic?
├─ YES → Human validates with domain expert, AI implements
└─ NO → Continue

Are there cascading errors (>10)?
├─ YES → Manual intervention (batch fix)
└─ NO → Let AI fix incrementally

​

Principles for Working Within Boundaries

​

The Meta-Insight

1. Novel problems: Training data limitation
2. Framework quirks: Hidden execution order
3. Performance: No profiling data
4. Security: Requires adversarial thinking
5. Operational: Production experience needed
6. Business logic: Domain knowledge gap
7. Edge cases: Global context required

• ❌ “Let AI do everything, fix what breaks”
• ✅ “Use AI where it excels, human where it doesn’t”

​

Actionable Takeaways

1. Document constraints before asking AI to design - ADRs, requirements docs, isolation rules. Let AI design within boundaries.
2. Review for the seven gaps - Performance (add caching), security (threat model), operational (monitoring), framework (quirks), novel patterns (human designs first).
3. Watch for cascade signals - Diminishing returns (errors per commit dropping), “partial fix” in commit messages, AI making same fix repeatedly.
4. Add operational layer before shipping - Metrics, tracing, circuit breakers, caching. AI builds infrastructure, humans harden for production.
5. Design prevents debugging - Week 6’s proactive design: 3 days, 0 bugs. Week 5’s reactive fixing: 24 hours, still broken. Design wins.

​

Discussion