Integration Patterns Overview

Choose an interaction by timing and coupling

System integration connects independently owned applications or services. The main pattern choices differ in when a response is needed, how much coupling the caller accepts, and how each side handles failure. A synchronous API gives the caller an immediate result and a clear request boundary, but the caller depends on the receiver's availability and latency. Messaging lets a producer hand off work and continue, but introduces asynchronous delivery, eventual results, and queue or broker operations. Batch file exchange is useful for large transfers where a scheduled delay is acceptable and both sides can agree on a schema and handoff protocol.

Do not choose a pattern by fashion. Begin with the business interaction: must the caller know the outcome now, can work be delayed, what volume or burst shape exists, and who owns each system? A payment authorization may require an immediate response, while an overnight reconciliation file may be a natural batch. A notification can be accepted durably and delivered asynchronously if the product contract allows a pending state.

API integration

An API is appropriate when a caller needs a response to continue a user or system workflow. Define the contract: request and response shape, authentication, authorization, timeouts, error semantics, versioning, and rate limits. A successful HTTP response should mean something specific. If the work is merely accepted, return an accepted or pending state rather than implying the downstream task is complete.

Synchronous coupling creates failure propagation. A slow dependency can consume caller threads and exhaust connection pools. Use timeouts aligned to the user experience, bounded retries only for safe operations, and circuit breaking or bulkheads where they protect shared capacity. Retries can duplicate side effects, so use idempotency or a deduplication strategy. Observe latency percentiles and error classes at the integration boundary.

Message-oriented integration

Messaging decouples producer and consumer in time and often availability. A queue can buffer a burst or allow a consumer to recover after an outage. The producer's acknowledgment should specify whether the message is durably accepted. Consumers should expect retries and duplicate delivery, use idempotent processing, and move unprocessable messages to a dead-letter path with an owner and replay procedure.

Ordering is not free. A broker may preserve order only within a partition or key, while global order limits throughput. Define the ordering requirement and select a key that preserves it where needed. Schema evolution must allow producers and consumers to deploy independently. Version messages compatibly and validate consumers against representative payloads. Monitor oldest-message age, delivery rate, retry count, and dead-letter volume; queue depth alone can hide a growing delivery delay.

Events describe facts that occurred, while commands request an action. Confusing them creates unclear ownership: a consumer may treat an event as an instruction or a producer may assume a command was completed because it was published. Name messages accordingly, define the source of truth, and state what eventual consistency means to users and operators.

Batch and file exchange

Batch transfer is a fit when data is naturally collected and processed as a unit, real-time latency is not required, or a partner can only support scheduled files. Define file format, encoding, schema version, partitioning, encryption, delivery acknowledgment, naming, duplicate handling, and retention. Use checksums or manifests when completeness matters. Make the handoff idempotent so a retry does not process the same file twice.

Batch systems need operational visibility too. Track expected arrival, record counts, validation failures, processing duration, and reconciliation totals. A missing file should be distinguishable from an empty file. Quarantine malformed input with a safe replay path rather than silently skipping records. Secure the transport and storage, and delete sensitive files according to policy. Batch reduces coupling in time but can create a large recovery backlog if a scheduled run fails.

Hybrid integration

Real systems often combine patterns. An API can accept a request and publish a command; a worker can call an external API and emit a completion event; a nightly batch can reconcile asynchronous results. Each boundary needs a clear owner and contract. Avoid building a chain of patterns without explaining which latency, resilience, or compatibility problem each one solves. The simplest pattern that satisfies the interaction and failure requirements is usually the most operable choice.

Select a pattern for three workflows

Consider a retail platform that integrates with a payment provider, a warehouse system, and a finance partner. For payment authorization, the checkout flow needs an immediate approval or rejection to tell the customer whether the order can proceed. A synchronous API is appropriate, with a strict timeout, an idempotency key, and a clear pending or unknown outcome when the provider times out after possibly processing the request.

For warehouse fulfillment, the order can be accepted before a worker allocates stock. Publish a command or event to a durable queue, include a stable order id, and make the consumer idempotent. Define what the customer sees while fulfillment is pending, how retries are bounded, and who investigates a dead-letter message. Ordering may be needed per order but not across every customer, so choose a partition key that preserves only the necessary scope.

For a finance partner that reconciles settled payments once each night, a file exchange may be sufficient. Specify a versioned format, encryption, checksums, record totals, duplicate-file detection, delivery acknowledgment, and retention. If a file fails validation, quarantine it and notify the owner rather than partially importing it without a reconciliation trail.

Compare constraints explicitly

Write down latency, volume, failure behavior, consistency, ownership, and operational skill for each interaction. A synchronous API offers immediate response but couples availability. Messaging absorbs temporary failure but changes the completion contract. Batch handles bulk data efficiently but delays visibility and creates scheduled recovery concerns. Compare total cost, including on-call operation, schema management, replay, and reconciliation, not only development effort.

For the payment path, ask what a timeout means. If the provider might have committed authorization, a retry needs the same idempotency key or a status lookup. For the warehouse message, ask whether the producer should wait for stock confirmation. For the finance file, ask how the partner confirms complete receipt and how records are reconciled. These questions expose semantics that a technology choice alone cannot answer.

Define contracts and rollout

Version APIs and message schemas deliberately. Use backward-compatible additions where possible, and deploy consumers that accept the new shape before producers emit it. For files, include a schema version and a manifest so validation is deterministic. Document ownership and a deprecation path for each integration. A contract should include security and data classification, not only field names.

Instrument the boundary. API metrics should include latency percentiles, status classes, and timeout counts. Queue metrics should include age, delivery, retries, and dead letters. Batch metrics should include arrival time, record counts, validation errors, and reconciliation status. Use correlation identifiers across synchronous and asynchronous steps while avoiding sensitive values in logs.

Exercise the failure path

Disable the payment provider in a test and verify checkout returns within its timeout without making a duplicate authorization. Stop the warehouse consumer and observe queue growth, then restore it and measure drain time. Deliver the same finance file twice and verify it is not double-counted. Submit a malformed file and confirm an operator can identify and replay it after correction. These exercises reveal whether the pattern and contract match the real failure model.

Close by choosing the simplest pattern that meets each workflow's timing and reliability need. A single platform-wide integration style may be easier to govern, but forcing every interaction into it can create unnecessary coupling or delay. Standards should define security, naming, observability, and versioning while allowing a fit-for-purpose interaction model.

Make completion semantics explicit

An API response, broker acknowledgment, and file receipt prove different things. A 202 response may mean the request was accepted but not completed. A broker acknowledgment may mean the message is durably stored, not processed. A file-transfer confirmation may show that bytes arrived, not that records passed business validation. State the acknowledgment boundary so callers do not infer a stronger guarantee than the integration provides.

Retries and replay require idempotency. A caller may time out after the receiver performs the work. A consumer may crash after committing state but before acknowledging a message. A batch job may restart after importing part of a file. Use stable operation identifiers, deduplication records, transaction boundaries, or reconciliation appropriate to the pattern. Avoid claiming exactly-once end-to-end behavior without identifying where duplicates are prevented.

Account for organizational coupling

Integration creates a contract between owners. Synchronous APIs couple release timing and availability more tightly. Messages decouple deployment but require schema governance and consumer discovery. Batch files can preserve compatibility for external partners but need clear schedules and operational handoffs. Consider team ownership, support hours, incident coordination, and the ability to test against a representative environment.

Security requirements cross every pattern. Authenticate callers, authorize operations, encrypt sensitive data in transit and at rest, rotate credentials, and minimize payloads. Messages and files can persist longer than an API request, so retention and deletion need explicit treatment. Correlation IDs should support tracing without embedding personal or secret data.

Observe the actual contract

Monitor the user-visible outcome, not just transport health. An API can return success while downstream processing is incomplete. A queue can accept messages while age grows beyond the service objective. A file can arrive on time while reconciliation finds missing records. Define service indicators around the promise made to the user or partner, and connect alerts to owners and runbooks.

In an interview, choose one concrete workflow, clarify whether it needs an immediate result, and compare API, messaging, and batch against latency, availability, consistency, and operational cost. Explain acknowledgment semantics, duplicate handling, versioning, and failure recovery. Then describe a hybrid only when each boundary has a reason. This shows that integration patterns are contracts and operating models, not merely transport products.

A pattern should also match the partner's operating model. An external organization may only support a daily file, while an internal service can offer a versioned API. Agree on support hours, escalation, credential rotation, and change notice periods. The transport is only one part of the contract; ownership determines how quickly a failure is diagnosed and repaired.

Compatibility has a lifecycle. Define how long old API versions or message schemas remain supported, how consumers discover changes, and who approves a breaking change. For files, publish schema examples and validation results to both parties. For messages, plan producer and consumer deployment order. A technically robust pattern can still fail if teams cannot coordinate a contract change or replay work safely.

Compare the full operating cost. Synchronous APIs need capacity for dependency timeouts and incident coordination. Queues need consumer scaling, schema governance, replay tools, and dead-letter ownership. Batch exchange needs scheduling, secure handoff, reconciliation, and recovery windows. Include these responsibilities in the design review so the selected pattern is operable by the teams who inherit it.