Week 5: When AI Fails Spectacularly

​

The Ironic Timing

• End-to-end test coverage (21 scenarios)
• Organization modeling
• API architecture
• Usage metering pipelines

​

The Setup: A “Simple” Macro Enhancement

// BEFORE: Macro generated struct, but methods were manual
#[derive(DynamoDbRepository)]
#[aggregate = "Account"]
pub struct DynamoDbAccountRepository;

impl DynamoDbAccountRepository {
    // Manual implementation of save, get, query, delete
    pub async fn save(&self, account: Account) -> Result<()> { /* ... */ }
    pub async fn get(&self, id: AccountId) -> Result<Option<Account>> { /* ... */ }
}

// AFTER: Macro generates methods too
#[derive(DynamoDbRepository)]
#[aggregate = "Account"]
pub struct DynamoDbAccountRepository;

// Macro now generates db_save, db_get, db_query, db_delete automatically

​

What We Built (or Tried To)

​

The CRUD Macro Enhancement

• Method renames: save() → db_save(), get() → db_get()
• Factory pattern change: self.client field → self.client() method
• Error type change: RepositoryError → EventStoreError
• New dependencies: aws-runtime, serde_dynamo

​

The Cascade: Hour by Hour

​

Hour 1: “This is fine”

feat(dynamodb-derive): add CRUD method generation to DynamoDbEntity macro

• cd7c3e1: fix(crm): remove duplicate DynamoDbLegalEntityRepository export
• 1ef9150: fix(crm): fix syntax errors and add missing imports

​

Hour 2-3: The Pattern Emerges

fix(crm): partial fix for organization_dynamodb.rs factory pattern

// Manual code expected:
let client = self.client.clone();  // self.client is a field

// Macro generated:
let client = self.client();  // self.client() is a method

• 9e0815f: replace self.client with client
• 11fe195: add client creation to save_calendar method
• b1a1ff0: batch fix organization_dynamodb.rs methods
• eb5e215: fix pipeline_dynamodb client calls
• a7fbb9b: fix remaining client() method calls

​

Hour 4-6: Type System Whack-a-Mole

fix(crm): update pk_for_id calls to new single-parameter signature

• Old: pk_for_id(tenant_id, capsule_id, id)
• New: pk_for_id(id) (macro infers tenant/capsule from entity)

• 9249626: fix pk_for_id and gsi method signatures
• 7503f9f: add GSI methods and fix syntax errors
• 47fd4ae: fix CrmError variants, field access errors
• 3afc5a0: move contact methods to ContactRepository impl block
• 3862c3e: fix E0425 errors (missing values) and self.client() calls

​

The Breaking Point: When I Stopped Trusting AI

fix(crm): fix CrmError::Repository tuple variant usage

// AI's fix:
return Err(CrmError::Repository("save failed".to_string()));

// But CrmError::Repository is not a tuple variant!
// It's a struct variant:
CrmError::Repository {
    source: RepositoryError,
    entity: String,
}

​

The Fix: Human Intervention

1. Stopped the AI session
2. Read the original macro change (commit 508f43e)
3. Made a list of ALL changes the macro introduced:
   • New method names (save → db_save)
   • New factory pattern (client field → client() method)
   • New error types (EventStoreError instead of RepositoryError)
   • New dependency requirements (serde_dynamo)
4. Fixed them systematically in 3 commits instead of 30:

# Used batch find/replace for all save() → db_save() calls
rg "\.save\(" -t rust | cut -d: -f1 | sort -u | xargs sd '\.save\(' '.db_save('

// Updated trait definition once, all implementations inherited it
trait Repository {
    fn client(&self) -> &CapsuleClient;  // Method, not field
}

// Map old error type to new error type
impl From<RepositoryError> for CrmError {
    fn from(e: RepositoryError) -> Self {
        CrmError::Repository {
            source: e,
            entity: "unknown".to_string(),
        }
    }
}

• AI: 30 commits, 24 hours, 63 errors remaining
• Human: 3 commits, 90 minutes, 0 errors

​

What Went Wrong: AI’s Fix-in-Session Anti-Pattern

​

The Problem

1. Read the error message
2. Understand the immediate cause
3. Fix that specific error
4. Compile
5. If more errors, repeat from step 1

• Each fix addresses symptoms, not root cause
• No holistic understanding of “what changed upstream”
• No batching of related fixes
• Creates more errors by partial fixes

​

The Example

error[E0599]: no method named `save` found for type `DynamoDbAccountRepository`
  --> crm/src/services/account_service.rs:45:28
   |
45 |         self.repo.save(account).await?;
   |                   ^^^^ method not found

AI Fix: Rename save → db_save in account_service.rs

New Error:
error[E0599]: no method named `save` found for type `DynamoDbContactRepository`
  --> crm/src/services/contact_service.rs:67:28
   |
67 |         self.repo.save(contact).await?;
   |                   ^^^^ method not found

AI Fix: Rename save → db_save in contact_service.rs

New Error:
error[E0308]: mismatched types - expected `CrmError`, found `EventStoreError`
  --> crm/src/services/account_service.rs:45:9
   |
45 |         self.repo.db_save(account).await?;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

AI Fix: Add error conversion in account_service.rs

New Error: ...

1. Recognize pattern: “All save() calls need to be db_save()”
2. Fix ALL save() calls in one commit
3. Recognize pattern: “All error types changed”
4. Add unified error conversion
5. Compile once → done

​

What I Learned: When to Stop Using AI

​

Red Flags That AI Is Stuck

• Commits 1-5: Average 12 errors fixed each
• Commits 6-15: Average 4 errors fixed each
• Commits 16-30: Average 1 error fixed each (sometimes net increase!)

• “fix client() calls” (7 commits)
• “fix pk_for_id signature” (5 commits)
• “fix error variants” (6 commits)

• “partial fix for organization_dynamodb.rs factory pattern”
• “fix remaining client() method calls”
• “batch fix organization_dynamodb.rs methods”

​

Principles Established This Week

# Find all usage sites
rg "\.save\(" --type rust

# Batch rename
sd '\.save\(' '.db_save(' crm/**/*.rs

# Verify
cargo check

Fix compilation errors in crm crate.

Constraints:
- Maximum 5 commits
- Group related fixes in single commit
- Do NOT add new methods/types
- Only update call sites to match new signatures
- If stuck after 3 commits, report and stop

​

The Surprising Discovery: AI Can Prevent This

Review the macro change that caused 30 fix commits.

Could this have been prevented?
What checks should run before making breaking macro changes?

1. Impact Analysis:
   • Run cargo tree to find all crates using the macro
   • Grep for all usage sites across workspace
   • Estimate affected lines of code
2. Breaking Change Detection:
   • Compare old vs new generated code (cargo expand)
   • List method signature changes
   • List type changes
   • List new dependencies
3. Migration Plan:
   • Create checklist of required updates per crate
   • Batch fixes by category (method renames, type updates, etc.)
   • Prepare workspace-wide test command
4. Atomic Commit Strategy:
   • Update macro
   • Update ALL downstream crates
   • Verify workspace compilation
   • Commit atomically

​

Week 5 By The Numbers

If error count > 10 and related to single upstream change:
  → Manual batch fix (2-3 commits, tools: rg + sd)
Else if errors decreasing steadily (5+ per commit):
  → Let AI continue
Else if errors stalled (< 3 per commit for 3 commits):
  → Stop AI, analyze root cause manually

​

The Contrast: Week 4 vs Week 5

• AI built features from scratch
• Clear requirements, no existing code to conflict with
• 107 commits, all green
• Productivity: 6-8 weeks of work in 5.5 days

• AI fixed breaking changes it created
• Complex interdependencies, cascading effects
• 31 commits, increasing errors
• Productivity: 2 hours of work took 24 hours

AI for NEW work (creation) → Excellent
AI for FIXING work (reactive debugging) → Poor
AI for PREVENTING work (planning) → Excellent

​

Takeaways from Week 5

1. Always create migration plan first (Week 2 principle applies)
2. Fix all affected sites atomically (never commit broken states)
3. Use batch tools (rg, sd) for systematic renames
4. Verify with cargo check --workspace before committing

1. Monitor error-fixing progress (errors per commit metric)
2. Intervene when progress stalls (< 3 errors per commit)
3. Use AI for planning/prevention, human for reactive fixes
4. Fresh AI sessions for post-mortem analysis (worked brilliantly)

​

What’s Next

• AI for feature planning and implementation
• Human for migration plans and systematic refactoring
• Clear decision trees for when to use which approach