Integration & enterprise systems
Get your software working inside the customer's CRM, warehouse and legacy APIs.
Roadmap: Production GenAI & integration
Why it matters
Deployments succeed or fail at the integration layer. The model or the product is rarely the problem; the problem is a CRM with 400 custom fields, a nightly export that sometimes doesn't arrive, or an API with a 100-calls-per-minute quota. Interviewers reward the simplest reliable path and penalise "we'll just call the API."
What it's made of
- REST basics done right: status codes, errors, pagination, versioning
- Choosing batch, webhooks, events or embedded UI by freshness need
- Mapping and normalising messy customer data
- Incremental sync with cursors and checkpoints
- Handling schema drift, duplicates and quotas
- Configure vs fork: keeping one product, not one per customer
Pick the integration pattern by freshness
Start by asking how fresh the data really needs to be. The answer usually picks the pattern for you:
- Hours or a day is fine: batch sync. Simple, easy to replay, easy to debug. Start here.
- Minutes matter: webhooks, with idempotent handling and a periodic reconciliation job as backup.
- Seconds matter and volume is high: an event stream. Only when you need it.
- Users act inside their existing tool: an embedded UI or extension that calls your API.
Build for messy data from day one
Put a normalisation layer between the customer's systems and yours. Map their fields to your model in one place, log anything that doesn't fit, and never let raw customer data leak through your codebase. When their schema changes (it will), you fix one mapping instead of fifty call sites.
Make every write safe to repeat
Networks fail halfway. Give every write an idempotency key, use upserts instead of inserts, and keep a sync cursor so a crashed job resumes instead of starting over. Then test it: kill the job mid-run and check you get the same result.
Configure, don't fork
When a customer needs something different, look for the configuration option first: a setting, a mapping table, a feature flag. Custom code for one customer becomes a branch you maintain forever. If three customers ask for the same thing, it's a product feature. Write it up for the product team.
Mid-level looks like
Designs a working integration path and handles the obvious errors.
Senior looks like
Defends why it's batch-first, names what's deferred, and plans for schema changes, duplicate events and quota exhaustion before they happen.
Common pitfalls
- Defaulting to real-time without anyone needing fresh data.
- No idempotency, so a retried webhook creates duplicate records.
- Offset-based pagination over data that changes mid-sync.
- Forking the product for one big customer and maintaining it forever.
Practice drills
Drill 1
Design a two-week pilot connecting an AI assistant to a customer's CRM for 20 users. Include a trade-off table covering freshness, scope, auth, AI output and rollout.
A strong answer: Batch-first sync, a least-privilege service account, read-only scope for the pilot, and explicit deferrals with reasons.
Drill 2
Design webhook ingestion that tolerates duplicate and out-of-order deliveries.
A strong answer: Idempotency keys, event timestamps or versions for ordering, a dead-letter path, and a reconciliation job.
Drill 3
Write a cursor-based incremental sync and test it against records deleted halfway through a run.
A strong answer: Stable cursors, checkpointing, explicit handling of deletes (tombstones or a periodic full compare), and a test that proves it.
Watch path
Play all · 2 videos · 49 minRead next
Designing Data-Intensive Applications
Martin Kleppmann
The best single explanation of how databases, replication, partitioning, streams and batch processing really work. You will meet every one of these in customer environments, and this lets you reason about them quickly.
Fundamentals of Data Engineering
Joe Reis, Matt Housley
Covers the full data lifecycle (ingestion, storage, transformation, serving) in vendor-neutral terms. Most FDE deployments start with getting at messy customer data, and this gives you the vocabulary and patterns for it.
Release It!
Michael T. Nygard
Stability patterns and anti-patterns from real production failures: timeouts, circuit breakers, bulkheads, back-pressure. Read it before you ship an integration into infrastructure you do not control.
Fundamentals of Software Architecture
Mark Richards, Neal Ford
Catalogues architecture styles and, more importantly, how to analyse tradeoffs between them. Helps you justify design choices to customer architects and review boards with more than personal preference.