Building a High-Throughput Financial Transaction Pipeline with Kafka, S3, PostgreSQL, Redis and DynamoDB
How we designed a FASTag transaction pipeline around duplicate webhooks, slow banks, and partial failures — using Kafka, S3, PostgreSQL, Redis, and a DynamoDB dead-letter queue.
Processing financial transactions is not just about moving messages from one service to another.
The system needs to answer some difficult questions:
- What happens if the external system sends the same transaction twice?
- What if our database is temporarily unavailable?
- How do we acknowledge a webhook quickly without losing data?
- How do we process thousands of transactions concurrently while maintaining ordering?
- What happens when an external API is slow?
- How do we recover transactions that could not be processed?
This article explores an architecture designed around these problems using Kafka, object storage, PostgreSQL, Redis, and a Dead Letter Queue.
1. The Problem#
Imagine an external financial institution sends us transaction callbacks. A single callback may contain hundreds of transactions.
For every transaction, our system may need to:
- Validate the transaction.
- Identify the associated account or financial instrument.
- Verify information with an external API.
- Store the transaction.
- Update the balance.
- Update downstream systems.
- Perform settlement-related operations.
Doing all of this synchronously inside the webhook is a bad idea.
Consider:
External System
|
v
Webhook
|
+--> Database
|
+--> External API
|
+--> Wallet
|
+--> Other services
|
v
Response
If any downstream dependency becomes slow, the webhook becomes slow. If something fails halfway through the processing, we now have a difficult question:
Did we process the transaction or not?
Instead, we want the webhook to do as little work as possible.
2. Separate Ingestion From Processing#
The first principle is simple:
Accept the event quickly. Process it asynchronously.
The architecture therefore looks like:
External System
|
v
Webhook
|
v
Object Storage
|
v
Kafka
|
v
Transaction Processing
|
+--> PostgreSQL
+--> Redis
+--> External APIs
The webhook's responsibility is primarily:
Receive
↓
Validate
↓
Persist
↓
Publish event
↓
Acknowledge
Webhook is simply a HTTP endpoint. It should do the basic validation and authentication of sender. This can be done with the help of IP-whitelisting at the API gateway level to allow calls from specific IPs of the sender. Second, we can also use a long-term token identifier that authenticates the sender. It should not perform the expensive business processing. This gives us an important separation:
Webhook latency ≠ transaction processing latency
The external system gets a fast acknowledgement, while our internal pipeline can continue processing independently.
3. Why Put the Original Event in Object Storage?#
Instead of putting the entire callback into Kafka, we first store the raw event in object storage (S3). Kafka then carries a lightweight reference to that object — just the S3 key and the source bank, not the full payload.
Conceptually:
Object Storage
|
| raw callback
|
v
Kafka
|
| object reference
v
Consumer
This gives us something extremely valuable in financial systems:
Durability + Replayability
Suppose Kafka is temporarily unavailable. The original callback is still safely stored. Or suppose we discover a processing bug tomorrow. We can retrieve the original event and replay it. Object storage therefore becomes our durable event archive, while Kafka becomes our processing pipeline. This separation also keeps Kafka messages relatively small.
4. What Happens Inside the Kafka Broker?#
Once the producer sends a batch, the Kafka broker receives the records for the relevant partition. A Kafka topic is divided into partitions:
Topic
├── Partition 0
├── Partition 1
├── Partition 2
└── Partition 3
Each partition is an ordered append-only log. New records are appended to the end:
Partition 1
Offset
100 → Transaction A
101 → Transaction B
102 → Transaction C
103 → Transaction D
The offset is effectively the position of the record within that partition. This is why Kafka can provide ordering within a partition. It does not provide global ordering across the entire topic.
Replication Factor
Kafka also provides durability through replication. Suppose we have:
Replication Factor = 3
A partition may have:
Partition 1
Broker 1 → Leader
Broker 2 → Replica
Broker 3 → Replica
The producer normally communicates with the leader for that partition. The leader writes the record and the followers replicate it. If one broker fails, another replica can become the leader. This is particularly important for financial transaction processing because we don't want a single broker failure to mean loss of transaction data.
Replication factor vs partitions
These solve two different problems: Partitions → scalability and parallelism Replication → availability and durability
For example:
8 partitions
Replication factor = 3
means the topic has 8 logical partitions, with each partition replicated across 3 brokers.
It does not mean there are 24 independent partitions.
5. Why One Kafka Topic Isn't Always Enough#
At first, a single Kafka topic may look sufficient:
External System
|
v
Webhook
|
v
Kafka
|
v
Consumers
But as transaction volume increases, another problem appears: where should each transaction be partitioned?
Suppose an incoming callback contains multiple transactions:
Callback
├── Transaction A → FASTag-101
├── Transaction B → FASTag-205
├── Transaction C → FASTag-101
└── Transaction D → FASTag-999
If we need transactions for the same FASTag to remain ordered, the partition key should be the fastagId.
One approach would be to determine the partition inside the HTTP webhook itself and publish each transaction to the appropriate partition.
Conceptually:
Webhook
|
+-------------+-------------+
| | |
Txn A Txn B Txn C
| | |
calculate key calculate key calculate key
| | |
+-------------+-------------+
|
Kafka
This looks simple, but it pushes additional work into the synchronous ingestion layer.
At low volumes, this may not matter. At high volumes, however, the webhook becomes responsible for:
- parsing large callbacks,
- extracting individual transactions,
- identifying the partition key,
- potentially performing lookups,
- producing multiple Kafka records,
- handling Kafka producer backpressure,
- and waiting for Kafka acknowledgement.
The more processing we put into the webhook, the more the ingestion layer becomes a bottleneck.
Now comes an interesting problem: ordering. Suppose an external callback belongs to a particular entity (bank). We want callbacks from the same entity to maintain their order.
So the first Kafka topic can use:
partition key = entity (bank)
Kafka guarantees ordering within a partition. But downstream, our actual requirement may be different. We don't necessarily care about ordering all transactions for the entity. We care about ordering transactions for the same account/tag/financial instrument.
So we introduce a second stage.
Kafka Topic 1
Key: Entity (bank)
|
v
Consumer
|
Parse XML, split batch
|
v
Kafka Topic 2
Key: FASTag ID
|
v
Transaction Consumer
This is one of the most important design decisions in the architecture. In practice, the first-stage consumer isn't just a passthrough — it's where the raw bank-specific payload (XML, in our case) actually gets parsed into a normalized transaction shape before it's re-published, keyed by fastag_id, to the second topic.
6. How Kafka Producer Batching Actually Works#
There is another important optimization in this architecture: producer batching.
When an application produces Kafka records, it doesn't necessarily send every record to the broker as an individual network request.
For example:
Transaction 1 ─┐
Transaction 2 ─┤
Transaction 3 ─┤
Transaction 4 ─┤
Transaction 5 ─┘
|
v
Kafka Producer
|
v
Accumulate
|
v
Send batch
The producer maintains an internal buffer for records that are going to Kafka.
Two important configuration concepts are typically involved:
- batch size — how much data (or how many messages) the producer tries to accumulate for a partition before sending.
- linger time — how long the producer can wait for additional records to arrive before sending the current batch.
In our producer this maps directly to batch.num.messages and queue.buffering.max.ms. We also set request.required.acks = -1, which means the broker only confirms the write once it's been replicated to all in-sync replicas — the extra latency buys us the durability guarantee that matters for financial data. Combined with compression.codec = snappy, this keeps throughput high without sacrificing the write guarantee.
The exact configuration depends on the workload.
For a high-throughput system, a small amount of intentional batching can significantly reduce network overhead because one request can carry many Kafka records.
But batching does not mean one Kafka message
This distinction is important.
Suppose the application produces:
Txn A → key=FASTag-101
Txn B → key=FASTag-205
Txn C → key=FASTag-101
The producer can accumulate these records and send them efficiently.
Kafka still treats them as individual records.
The producer determines the partition for each record based on its key, and records destined for the same partition can then be grouped into a batch.
Conceptually:
Producer
|
+---- Partition 1 batch
| ├── Txn A
| └── Txn C
|
+---- Partition 3 batch
└── Txn B
This gives us both:
Correct partitioning at the record level + efficient network transmission through batching.
7. Processing Transactions in Batches#
The second consumer is where most of the business processing happens.
Instead of processing:
Transaction → DB
Transaction → DB
Transaction → DB
we process a batch:
100 Transactions
|
v
Validation
|
v
Lookups
|
v
Bulk DB operations
Batching reduces:
- network round trips,
- database connection overhead,
- query execution overhead,
- application overhead.
For example, instead of checking 100 accounts individually:
SELECT account WHERE id = A
SELECT account WHERE id = B
SELECT account WHERE id = C
...
we can perform a bulk lookup:
SELECT ...
FROM accounts
WHERE account_id IN (...)
The same principle applies to inserts and updates — we bulk-insert the whole prepared batch of transactions in one call, rather than one INSERT per transaction.
But batch size matters.
A huge batch isn't automatically better.
Large batches increase:
- memory consumption,
- processing latency,
- failure scope,
- transaction duration.
The right batch size should be determined through load testing.
The bulk-first, individual-fallback pattern
One pattern that's paid off in practice: always try the bulk operation first, but never let a single bad record take the whole batch down.
Bulk insert (100 rows)
|
v
Integrity error?
|
Yes
|
v
Fall back: insert one row at a time
|
v
Duplicate rows → counted as success, skipped
Genuinely bad rows → pushed to DLQ
A UNIQUE constraint violation on a bulk insert doesn't tell you which row caused it — it just fails the whole batch. Falling back to row-by-row only when the bulk path raises an IntegrityError means 99 valid transactions still get written, and only the one problematic row needs individual attention. The same pattern applies when creating or updating FASTag records concurrently from multiple partitions — a duplicate-key error on create usually just means another thread already won the race, which is a successful outcome, not a failure.
8. External APIs Are Often the Real Bottleneck#
Database operations aren't necessarily the slowest part.
Suppose every transaction requires an external API call — for us, that's calling the bank to verify a FASTag's current status or balance before we trust the Kafka-reported value.
If one API call takes 300 ms and we process 100 transactions sequentially:
100 × 300ms = 30 seconds
That's too slow.
Instead, we can use bounded concurrency:
Batch
|
+---------+---------+
| | |
API API API
| | |
... ... ...
For example, a batch of 100 transactions might process 10 external calls concurrently — in our pipeline this is a ThreadPoolExecutor with a fixed worker pool, rather than firing off every call at once.
But we shouldn't create unlimited concurrency.
Too much parallelism can cause:
- API throttling,
- connection exhaustion,
- CPU/memory pressure,
- cascading failures.
So concurrency should be bounded and tuned according to the external provider's limits.
9. Retry Only What Is Retryable#
Not every failure should be retried.
For example:
HTTP 400
usually means the request itself is invalid. Retrying it won't fix the problem.
But:
Timeout
HTTP 500
Connection failure
may be temporary.
A good retry strategy is:
Attempt 1
↓
wait
↓
Attempt 2
↓
wait longer
↓
Attempt 3
↓
DLQ
This is exponential backoff. In our consumers, every network-facing step — reading the raw event from S3, publishing to the next Kafka topic, calling the bank's API, writing to the database — is wrapped with its own retry policy (fixed tries, increasing delay), and only gives up and routes to the DLQ once that specific step has exhausted its attempts. This matters: a transient S3 hiccup shouldn't need to retry the whole batch from the beginning, just the read that actually failed.
In production, adding jitter is also useful to prevent many consumers from retrying at exactly the same time.
10. Idempotency Is Non-Negotiable#
Distributed systems will produce duplicate messages.
For example:
Transaction X
|
v
Consumer processes X
|
X
Consumer crashes before offset commit
|
v
Kafka delivers X again
We cannot simply assume:
"Kafka delivered it once, so we'll process it once."
We should assume at-least-once delivery.
Therefore, the business operation must be idempotent.
Database-Level Idempotency
A transaction identifier can have a unique constraint:
UNIQUE(transaction_id)
First processing:
INSERT X
→ Success
Duplicate processing:
INSERT X
→ Duplicate
The database becomes the final protection against duplicate financial transactions.
This is much safer than doing:
SELECT → not found → INSERT
because two consumers could perform the SELECT simultaneously.
11. Redis for Short-Lived Idempotency#
Database uniqueness is ideal for persisted transactions.
For short-lived operations, Redis can provide distributed idempotency.
For example:
SET idempotency_key value NX EX ttl
NX means:
Only create the key if it doesn't already exist.
So:
Request A
↓
Redis SET NX
↓
Success
A duplicate request:
Request B
↓
Redis SET NX
↓
Already exists
This prevents duplicate concurrent operations — we use exactly this for recharge requests, where a short-lived "processing" lock blocks a rapid double-tap from the client while the first request is still in flight, separate from the longer-lived success cache used for the actual idempotency key.
Redis can also be used for caching frequently accessed, relatively stable information — like a bank's OAuth bearer token — reducing both pressure on PostgreSQL and the number of times we have to re-authenticate with an external API.
12. What Happens When Processing Fails?#
Retries won't solve every problem.
Eventually, some transactions will still fail.
We don't want:
Failure
↓
Drop message
Instead:
Process
↓
Retry
↓
Retry
↓
Retry
↓
Dead Letter Queue
We use DynamoDB as the DLQ store, with a TTL on each item so failed events don't accumulate forever. The DLQ entry contains enough information to understand:
- which event failed (the original payload),
- where it failed (which processing stage — S3 read, Kafka publish, bank API call, or DB write),
- why it failed (the exception message),
- the partition key, so a failed item can be traced back to the specific account/tag it belongs to,
- and how it can be replayed.
This gives operations teams a recovery path instead of silently losing transactions. It also means a single malformed record in a batch of 100 doesn't block the other 99 — the batch keeps moving, and only the bad record lands in the DLQ.
13. Why Offset Commit Order Matters#
Kafka consumers should not commit offsets before processing completes.
Bad:
Consume
↓
Commit offset
↓
Process
↓
Crash
If the consumer crashes after committing but before processing, Kafka won't redeliver the message.
Instead:
Consume
↓
Process
↓
Database commit
↓
Commit Kafka offset
Now if the consumer crashes before committing the offset, Kafka can redeliver the event.
And that's okay — because our database idempotency protects us from duplicate processing. In fact, if the handler raises any unhandled exception, we deliberately skip the commit entirely and move on to the next poll — the unprocessed batch gets redelivered on the next consume rather than silently disappearing.
This gives us the combination:
At-least-once delivery + idempotent processing = effectively-once business outcome.
14. Scaling the Pipeline#
Kafka makes horizontal scaling relatively straightforward.
Suppose we have:
8 partitions
We can have multiple consumers in the same consumer group:
Consumer 1 → Partition 1
Consumer 2 → Partition 2
Consumer 3 → Partition 3
...
More consumers allow more parallel processing — up to the number of partitions.
But simply adding consumers doesn't solve every problem.
If the database can process only:
2,000 transactions/sec
and Kafka can deliver:
10,000 transactions/sec
then the database is the bottleneck.
The effective throughput of the system is determined by its slowest critical dependency.
15. Monitor Consumer Lag#
One of the most important metrics in a Kafka-based architecture is consumer lag.
If producers are generating:
2,000 events/sec
while consumers process:
1,500 events/sec
the backlog grows by:
500 events/sec
This appears as Kafka consumer lag.
Monitoring only CPU isn't enough. A consumer can have low CPU utilization while spending most of its time waiting for:
- PostgreSQL,
- external APIs,
- network,
- connection pools.
Therefore, monitor:
- consumer lag,
- partition lag,
- processing latency,
- database latency,
- external API latency,
- error rate,
- retry rate,
- DLQ volume.
16. Putting Everything Together#
The complete flow becomes:
External System
|
v
+-----------+
| Webhook |
+-----+-----+
|
v
+-----------+
| Object |
| Storage |
+-----+-----+
|
v
+-----------+
| Kafka 1 |
| Entity Key|
+-----+-----+
|
v
+-----------+
| Consumer |
| Fan-out |
+-----+-----+
|
v
+-----------+
| Kafka 2 |
| Account Key|
+-----+-----+
|
v
+------------------+
| Transaction |
| Processor |
+--------+---------+
|
+-----------+-----------+
| | |
v v v
PostgreSQL Redis External APIs
|
v
Wallet/Settlement
|
failures after retry
|
v
DLQ
Each component has a focused responsibility.
Webhook → fast ingestion
Object storage → durable raw events and replay
Kafka → buffering, ordering and parallelism
Consumer 1 → transform and fan-out
Consumer 2 → business processing
PostgreSQL → transactional source of truth
Redis → caching and distributed idempotency
DLQ → failed-event recovery
17. The Bigger Lesson#
The interesting part of this architecture isn't Kafka. It isn't PostgreSQL. It isn't Redis. It's the way the components work together.
A production-grade financial pipeline should assume:
- messages will be duplicated,
- consumers will crash,
- databases will become unavailable,
- external APIs will timeout,
- events will arrive late,
- traffic will spike,
- and some transactions will inevitably fail.
The ultimate goal isn't simply to build a system that can process 10,000 transactions per second. It's to build one that can process those transactions correctly — even when things go wrong.
And in financial systems, correctness is ultimately more important than raw throughput.