​

Week 4: When AI Truly Excels

​

The Surprising Realization

• Complete end-to-end test coverage for event flows (21 test scenarios)
• Organization model documentation (agent responsibilities, workflows, decision frameworks)
• API visibility architecture for SDK filtering
• Bundle/unbundle workflow state machine with examples
• Usage metering infrastructure with aggregation pipelines

​

What We Built This Week

​

1. Event Flow E2E Testing (The Work Nobody Wants to Do)

• Miss an event → audit trail broken
• Wrong event order → state corruption
• Cross-tenant event leak → compliance violation

1. Set up test infrastructure (EventBridge, SQS, DynamoDB Streams)
2. Create test data for each entity type
3. Trigger actions and wait for async event delivery
4. Verify event payload, ordering, and side effects
5. Clean up test resources
6. Repeat for every entity and workflow combination

    Planning session for comprehensive E2E event flow testing.

    Context:
    - Event-sourced CRM with 7 entities (Account, Contact, Lead,
      Opportunity, Activity, Product, Address)
    - Events flow: DynamoDB Streams → EventBridge → SQS → Consumers
    - Need to verify event delivery, ordering, and payload correctness
    - Must test cross-entity workflows (e.g., Lead conversion)

    Requirements:
    1. Test each entity's event flow independently
    2. Test multi-entity workflows (create Account → add Contact → convert Lead)
    3. Test failure scenarios (event delivery failure, consumer errors)
    4. All tests must run against LocalStack (no AWS resources)

    Design comprehensive test suite with:
    - Test data fixtures
    - Event verification utilities
    - Async event waiting helpers
    - Clear assertion patterns

eva-crm/tests/integration/event_flow/
├── mod.rs                      (shared utilities)
├── contact_events_test.rs      (5 scenarios)
├── lead_events_test.rs         (6 scenarios)
├── opportunity_events_test.rs  (4 scenarios)
└── partner_events_test.rs      (3 scenarios)

eva-crm/tests/e2e/
└── lead_conversion_workflow_test.rs (multi-entity)

​

2. Organization Model Documentation (The Work Nobody Does)

• Evaluator sometimes made decisions outside its scope
• Builder occasionally asked questions Evaluator should answer
• Verification reports varied in quality
• No clear escalation path for conflicts

Planning session for agent organization model.

Context:

- Using 3 agents: Evaluator (Opus), Builder (Sonnet), Verifier (Sonnet)
- Working well for features, but roles blur on complex issues
- Need clear boundaries, responsibilities, and escalation paths

Design an organization model with:

1. Agent roles and responsibilities (what each agent owns)
2. Decision rights (who decides what)
3. Communication protocols (how agents interact)
4. Conflict resolution (what happens when agents disagree)
5. Quality gates (when work can progress to next agent)

Model after real organizations:

```text
- Evaluator = Principal Architect
- Builder = Senior Engineer
- Verifier = Tech Lead / Reviewer

Include failure modes and handling strategies.

• Verifier would sometimes say “looks good” without checking edge cases
• Builder occasionally added features not in the plan (“seemed useful”)
• Unclear when to involve human vs. let agents decide

• Verifier now follows checklist: requirement coverage, test adequacy, edge cases, consistency
• Builder stays strictly within plan (escalates if deviation needed)
• Clear decision rights eliminate back-and-forth

​

3. Documentation Strategy (The Work That Never Gets Done)

“The organization model references several documents that don’t exist yet (PRD templates, ADR process, verification report format). Should we create a documentation strategy to ensure consistency?”

    Design a documentation strategy for the multi-agent workflow.

    Context:
    - Solo developer using AI agents
    - Building production SaaS platform
    - Need to maintain context across weeks/months
    - AI agents need consistent documentation format to work effectively

    Requirements:
    - Documentation types (what to document)
    - Documentation templates (consistent format)
    - Documentation lifecycle (when to create/update)
    - Documentation storage (where to keep)
    - Documentation quality (how to verify)

    Optimize for:
    - AI agents can find and use docs effectively
    - Human can understand project after 3-month break
    - Minimal maintenance overhead

​

The Journey: When Tedious Work Becomes Systematic

​

API Visibility Architecture (Creative + Systematic)

• Customer SDK (only public-facing routes)
• Platform SDK (all routes including internal)
• Partner SDK (partner portal routes only)

• Design visibility tagging system
• Choose implementation approach (compile-time vs runtime)
• Define SDK filtering rules

• Tag 127 existing API routes with visibility
• Update OpenAPI generation to filter by visibility
• Create SDK generation scripts for each audience
• Write migration guide for future routes

​

Product Features (Bundle/Unbundle Workflow)

• Can’t unbundle a simple product (must be a bundle)
• Can’t bundle already-bundled products (no nested bundles)
• Pricing must be recalculated on bundle/unbundle
• Events must be emitted for audit trail

1. Design state machine (1 day)
2. Implement domain logic (1 day)
3. Write tests (1 day)
4. Write example showing how to use it (2 hours)
5. Total: 3+ days

1. Evaluator designs state machine (2 hours)
   • Identifies 7 states and 12 transitions
   • Defines validation rules for each transition
   • Plans event emission strategy
2. Builder implements (4 hours)
   • State machine with typed states (compile-time enforcement)
   • Validation logic per state
   • Event emission + repository integration
   • 15 unit tests covering all transitions
3. Builder creates example (1 hour)
   • bundle_unbundle_workflow.rs showing real usage
   • Includes error handling and edge cases
   • Documented with comments explaining each step
4. Verifier catches issues (30 minutes)
   • Missing negative test: “What if unbundle fails mid-operation?”
   • Missing validation: “Can bundle quantity be zero?”
   • Unclear example: “Should show rollback scenario”
5. Builder fixes (1 hour)
   • Added rollback test
   • Added quantity validation
   • Enhanced example with rollback scenario

​

Usage Metering Pipeline (Pure Systematic Work)

• Emit usage events on each API call
• Consume events from queue
• Aggregate usage by partner, date, endpoint
• Store in DynamoDB for billing queries

• Define 5 event types (API call, storage, bandwidth, feature usage, error)
• Create DynamoDB schema for events + aggregates
• Implement event consumer with batch processing
• Create aggregation pipeline (sum, count, percentiles)
• Write 12 integration tests (event → consumer → aggregate → query)

• Evaluator designed comprehensive pipeline (2 hours)
• Builder implemented everything including edge cases (5 hours)
• Verifier validated correctness (1 hour)

• Event deduplication (idempotency)
• Out-of-order event handling
• Aggregate correction on late-arriving events
• Efficient querying with GSI design

​

What We Learned: The Taxonomy of AI-Suitable Work

Is the work...
  ├─ Systematic? (repetitive pattern)
  │  └─ YES → AI excels (Builder)
  │
  ├─ Well-specified? (clear requirements)
  │  └─ YES → AI good (Evaluator → Builder)
  │
  ├─ Needs thoroughness? (edge cases, testing)
  │  └─ YES → AI better than human (Verifier)
  │
  ├─ Creative? (novel solution needed)
  │  └─ NO → Human designs, AI implements
  │
  └─ Ambiguous? (unclear requirements)
     └─ NO → Clarify first, then use AI

​

Principles Established This Week

​

Metrics: Week 4 by the Numbers

​

The Mistake I Made (And What It Taught Me)

Error: PartnerCostMatrix query failed: Access denied

​

What’s Next: Week 5 Preview

• Refactor DynamoDB entities to single-table design (breaking change)
• Migrate event schema to versioned events (backward compatibility required)
• Consolidate API routes (eliminate duplication)
• Performance optimization (query patterns, indexing strategy)

• Understanding existing code deeply enough to refactor safely?
• Maintaining backward compatibility?
• Verifying refactoring didn’t change behavior?

​

Code Examples (Sanitized)

/// Async event collector with timeout and filtering
pub struct EventCollector {
    queue_url: String,
    sqs_client: aws_sdk_sqs::Client,
    timeout: Duration,
}

impl EventCollector {
    /// Collect events matching predicate within timeout
    pub async fn collect_events<F>(
        &self,
        predicate: F,
        expected_count: usize,
    ) -> Result<Vec<Event>>
    where
        F: Fn(&Event) -> bool,
    {
        let start = Instant::now();
        let mut collected = Vec::new();

        // Exponential backoff with jitter
        let mut delay = Duration::from_millis(100);

        while start.elapsed() < self.timeout {
            // Poll SQS queue
            let messages = self.sqs_client
                .receive_message()
                .queue_url(&self.queue_url)
                .max_number_of_messages(10)
                .wait_time_seconds(1)
                .send()
                .await?
                .messages
                .unwrap_or_default();

            for msg in messages {
                let event: Event = serde_json::from_str(&msg.body)?;

                if predicate(&event) {
                    collected.push(event);

                    if collected.len() >= expected_count {
                        return Ok(collected);
                    }
                }
            }

            // Exponential backoff with jitter
            tokio::time::sleep(delay).await;
            delay = (delay * 2).min(Duration::from_secs(5));
        }

        Err(Error::EventCollectionTimeout {
            expected: expected_count,
            received: collected.len(),
            elapsed: start.elapsed(),
        })
    }
}

#[tokio::test]
async fn test_lead_conversion_emits_events() {
    let collector = EventCollector::new("test-queue-url", Duration::from_secs(10));

    // Trigger lead conversion
    convert_lead_to_opportunity(lead_id).await?;

    // Collect events
    let events = collector
        .collect_events(
            |e| e.entity_type == "Lead" || e.entity_type == "Opportunity",
            2, // Expect: LeadConverted + OpportunityCreated
        )
        .await?;

    // Verify event ordering and payload
    assert_eq!(events[0].event_type, "LeadConverted");
    assert_eq!(events[1].event_type, "OpportunityCreated");
    assert_eq!(events[1].payload["lead_id"], lead_id);
}

• Exponential backoff handles EventBridge → SQS delays
• Predicate filtering allows flexible event matching
• Helpful error messages on timeout (shows expected vs received)
• Reusable across all 21 test scenarios

​

Discussion: Where Does AI Excel for You?