The Challenge: Build a CRM Domain Layer
Build the entire CRM foundation for a multi-tenant platform. The scope:- 7 core domain models (Account, Contact, Lead, Opportunity, Activity, Product, Address)
- Event sourcing integration
- DynamoDB repository pattern
- Financial configuration system
- ISO reference data (countries, currencies)
- Full API layer with permissions
- Integration tests with LocalStack
The Journey in Detail
Phase 1: Planning
I started with a single GitHub issue: “Initialize CRM crate with domain models” Instead of diving into code, I opened an Evaluator (Opus) session:- Read existing domain models from the auth crate
- Grep for event sourcing patterns in the kernel crate
- Analyze DynamoDB entity implementations
- Review repository trait patterns
Key Architecture Decisions from the Plan
Key Architecture Decisions from the Plan
Decision 1: Single-Table Design
- Use one DynamoDB table for all CRM entities
- PK pattern:
TENANT#{tenant_id}#ACCOUNT#{account_id} - GSI patterns for cross-entity queries
Decision 2: Event Sourcing Integration
- Each domain model emits events via
EventStoretrait - Events follow naming convention:
{Entity}{Action}(e.g.,AccountCreated) - Repository pattern wraps both state storage and event publishing
Decision 3: Macro-Driven DynamoDB Entities
- Create
#[derive(DynamoDbEntity)]macro - Auto-generate partition key, sort key, GSI attributes
- Eliminate boilerplate across 7 domain models
Decision 4: Four-Level Testing
- L1: Domain model validation (unit tests)
- L2: Repository CRUD (LocalStack integration tests)
- L3: Event publishing flow (EventBridge → SQS verification)
- L4: End-to-end CRM workflows
Phase 2: Implementation
Instead of one massive implementation session, I broke it into 7 sub-tasks:- Create
DynamoDbEntityderive macro - Implement Account domain model
- Implement Contact domain model
- Implement Lead domain model
- Implement Opportunity domain model
- Implement Activity domain model
- Add API layer with routes
Sub-Task Example: DynamoDB Macro
Builder session (Sonnet):- Created
eva-dynamodb-derivecrate - Implemented proc macro with
synandquote - Generated 15 unit tests covering various attribute patterns
- All tests passing locally
Builder fixed both issues
Re-verification: PASSED ✅ Merge to main. This pattern - finding issues in verification that Builder missed - repeated across all 7 sub-tasks.The Pattern That Emerged
After several iterations, I noticed something fascinating: Verifier caught different types of bugs than Builder’s own tests.- Builder's Tests
- Verifier's Catches
What Builder tested:
- Happy path functionality
- Basic edge cases (empty strings, null values)
- Compilation and type safety
Phase 3: Integration
With all 7 domain models implemented and verified individually, I assumed integration would be smooth. Wrong. Integration task: Wire all CRM entities to the API layer with permission-based routes. Builder session (Sonnet):- Created API handlers for 7 entities (35 routes total)
- Applied
#[eva_api]macro to each route - Wired to DynamoDB repositories
- Tests passing
- Created
eva-api-common/src/permissions.rswith 45 typed constants - Refactored all 35 routes to use constants
- Added compile-time enforcement (routes won’t compile without valid permission constant)
“Routes now use type-safe constants, but the authorization middleware still accepts arbitrary strings. Recommend: update middleware to only accept permission constants.”This is where AI really impressed me. Verifier didn’t just check requirements - it suggested architectural improvements.
The Verification Report That Mattered Most
The most valuable verification happened during event sourcing integration, when Verifier caught an issue that would’ve been catastrophic in production. Context: Implementing the event sourcing integration between CRM domain events and the kernel’s EventStore. Builder’s implementation: Straightforward - emit events after each repository save. Verifier’s question:“What happens if DynamoDB save succeeds but EventStore append fails? You now have state divergence between the database and the event log.”Uh oh. Builder had implemented this pattern:
“DynamoDB and EventStore are separate systems. If event append fails, account exists in DB but has no audit trail. Violates event sourcing guarantees.”The fix required rethinking the architecture. After discussing with Evaluator (in a new planning session), we introduced a two-phase commit pattern:
Key Learning: Verifier’s independent perspective caught an architectural flaw that Builder’s tests would
never find. Tests can’t verify what you didn’t think to test.
What We Built
By the end of the iteration, the CRM crate was complete:Domain Models
7 core entities:
- Account (companies)
- Contact (people)
- Lead (prospects)
- Opportunity (deals)
- Activity (tasks/events)
- Product (catalog items)
- Address (multi-address support)
- Tenant-scoped settings
- Currency preferences
- Fiscal year configuration
Infrastructure
Custom DynamoDB macro:
#[derive(DynamoDbEntity)]- Auto-generates PK/SK/GSI
- PII field encryption support
- Trait-based abstraction
- In-memory + DynamoDB implementations
- Event publishing integration
Event Sourcing
21 domain events:
- AccountCreated, AccountUpdated
- ContactAdded, ContactEmailChanged
- LeadConverted, OpportunityWon
- etc.
- EventStore trait implementation
- DynamoDB Streams → EventBridge
- Audit trail for compliance
Testing
4-level test suite:
- L1: 85 unit tests (domain validation)
- L2: 42 integration tests (LocalStack)
- L3: 12 event flow tests
- L4: 3 E2E workflows
- 6,800 lines of production code
- 2,400 lines of test code
- 23 files created
- 216 commits
- 0 bugs in production
What We Learned: Real Lessons from Real Bugs
Lesson 1: Fresh Verifier Sessions Are Non-Negotiable
Experiment: Early in the process, I got lazy and reused Builder’s session for verification (to “save time”). Result: Verifier found 0 bugs. Wait, what? The code was perfect? No. I opened a fresh Verifier session later - found 4 bugs immediately. Analysis: When you reuse Builder’s session:- Verifier inherits Builder’s mental model
- Verifier “remembers” the shortcuts Builder took
- Verifier validates assumptions instead of challenging them
Lesson 2: AI Hallucinates Requirements, Not Just Code
Builder’s hallucination: During Contact domain model implementation, Builder added apreferred_name field:
“Contact struct includes preferred_name field.
This field is not mentioned in the PRD.
Is this intentional or hallucinated?”
Root cause: Builder saw first_name and last_name and “reasoned” that users would want a preferred
name too. Reasonable inference - but not in the requirements.
Fix: Removed the field. Created a GitHub issue: “Consider adding preferred_name field to Contact” for
future discussion.
Lesson: AI doesn’t just hallucinate code - it hallucinates features. Verifier prevents scope creep.
Lesson 3: Test Coverage ≠ Requirement Coverage
Builder proudly reported: “92% test coverage across all domain models!” Verifier asked: “Which requirement does test #14 verify?” Builder: ”…” Turns out: Builder wrote tests that covered code paths but didn’t map to requirements. Example:Principles We Established
Based on the bugs we caught and the patterns that worked, we codified these principles:Principle 1: Events Are the Source of Truth
Principle 1: Events Are the Source of Truth
What we learned: When integrating event sourcing with a database, always append events BEFORE updating state.Rule: Event append failure = operation fails. Database save failure = rollback event.Never: Save to database first, then emit event (leads to orphaned state).
Principle 2: Macro-Generated Code Needs Manual Review
Principle 2: Macro-Generated Code Needs Manual Review
What we learned: The
DynamoDbEntity macro generated code that compiled but violated
DynamoDB best practices (wrong projection types).Rule: All macro-generated code must be reviewed by Verifier, including expansion inspection.Tool added: cargo expand to verify macro output.Principle 3: Type Safety Beats Runtime Checks
Principle 3: Type Safety Beats Runtime Checks
What we learned: String-based permissions led to typos and inconsistencies across 35 routes.Rule: Use typed constants for all cross-cutting concerns (permissions, event names, table names).Example:
Principle 4: Verification Finds Different Bugs Than Tests
Principle 4: Verification Finds Different Bugs Than Tests
What we learned: Builder’s tests found code-level bugs. Verifier found requirement-level gaps.Rule: Both are necessary. Tests verify “does it work?” Verifier verifies “is it right?”Metrics tracked:
- Builder bugs: Compile errors, test failures
- Verifier bugs: Requirement gaps, edge cases, consistency issues
The Mistake I Made (And What It Taught Me)
After initial verification: After Verifier approved the Opportunity domain model, I merged to main and moved on. Later during integration testing: Integration tests failed. Cross-entity query between Opportunity and Account was broken. What happened? Verifier checked Opportunity in isolation. Tests passed. Requirements met. But Opportunity has a foreign key relationship to Account:opportunity.account_id → account.id
The bug: Opportunity’s account_id field used a different ID format than Account’s id field.
New Verification Prompt for Multi-Entity Domains
New Verification Prompt for Multi-Entity Domains
After verifying individual entities, perform cross-entity validation:
-
Foreign key consistency:
- Do foreign key types match primary key types?
- Are ID formats consistent across relationships?
-
Event naming consistency:
- Do all Created events have the same structure?
- Are event versioning patterns consistent?
-
Repository pattern consistency:
- Do all repositories implement the same trait?
- Are CRUD method signatures consistent?
-
API route consistency:
- Are path patterns consistent? (/accounts/:id vs /account/:account_id)
- Are HTTP methods consistent across entities?
- Are permission naming patterns consistent?
Metrics
- Velocity
- Quality
- Cost
- Rework
Planned work: Initialize CRM crate with 7 domain modelsActual completion:
- 7 domain models ✅
- DynamoDB macro ✅
- Event sourcing integration ✅
- API layer with 35 routes ✅
- Financial configuration ✅
- Reference data (ISO 3166, ISO 4217) ✅
- LocalStack integration tests ✅
Code Examples (Sanitized)
Here’s the final DynamoDB entity macro we built:- Zero boilerplate across 7 entities
- Compile-time validation of key patterns
- Automatic PII field encryption
- Type-safe event metadata