Distributed Image Processing Pipeline
A Kafka-based pipeline that splits a large image into tiles, farms them to a pool of workers for OpenCV processing, and deterministically reassembles the result — with worker liveness tracking, durable job state, and a Flask monitoring API.
30-second version
I built the control plane for a distributed image-processing system. A Master splits an image into tiles, publishes them to Apache Kafka, and a pool of workers processes the tiles with OpenCV in parallel. The Master reassembles results by tile coordinates, so the output is deterministic no matter what order workers finish in. I added heartbeat-based liveness tracking to re-dispatch work from dead workers, SQLite for durable job state, and a Flask API for monitoring.
2-minute version
The point of the project is the distributed-systems mechanics, not the image math. Images are memory-heavy and CPU-bound, so I split one into a coordinate grid of tiles and push them onto a Kafka topic that is partitioned so a consumer group of workers can process tiles in parallel. Because delivery is at-least-once, I made every write idempotent — each tile is keyed by a deterministic tile_id, so a redelivered tile overwrites the same slot instead of being applied twice. Workers publish heartbeats to a second topic; when one goes quiet the Master marks it inactive and re-dispatches its unfinished tiles. Results come back on a third topic, and the Master stitches them by (row, col), using a small overlap/halo so edge filters don't leave seams. All job and tile state is persisted in SQLite, so the Master can recover after a crash, and a read-only Flask API exposes status without touching the hot path.
01 Overview
Processing a large image on one machine is bounded twice: the whole image must be decoded into RAM, and every pixel is transformed on a single thread. This project removes both bounds by splitting the image into independent tiles, sending them through Apache Kafka to a pool of workers, and reassembling the processed tiles by their grid coordinates. A single Master process — the control plane — owns tiling, dispatch, reassembly, and liveness, while SQLite keeps durable job state and Flask exposes a read-only monitoring API.
It matters because it is a small, working instance of the patterns behind MapReduce, Celery, and every data pipeline: partitioning for parallelism, message-based decoupling, at-least-once delivery, idempotent side effects, and crash recovery. Those are exactly the ideas a backend/systems interview probes, and this project let me internalize them by building them rather than reading about them.
02 Problem & Context
The constraints that shaped the design, in order of how hard they pushed:
- Workers can die at any time. A process that crashes mid-tile must not corrupt the job or lose the tile — the system has to detect the death and re-dispatch.
- Output must be deterministic. The final image must be byte-for-byte identical regardless of which worker handled which tile or in what order tiles finished.
- No shared memory between workers. Everything is message-passing over Kafka; a worker on another machine knows nothing but what a message tells it.
- Job state must survive a crash. If the Master dies, restarting it must recover every in-flight job, not lose them.
- Memory is the enemy. The full image should never need to be decoded into RAM at once — only tile-sized buffers.
- Monitoring must not slow the pipeline. Observability is a read-only sidecar, never a step in the critical path.
I chose this project because Kafka's core ideas — topics, partitions, consumer groups, and offsets — are easy to describe and hard to truly understand until you have watched a worker crash mid-tile and watched the system recover. Building it forced those concepts from vocabulary into instinct.
03 Foundations
The concepts in plain language, before the architecture. Each card is the "explain it to a smart non-expert" layer.
Apache Kafka
A durable, append-only log. Producers write messages to a topic; a topic is split into partitions; consumers in a consumer group each claim partitions and read by offset. Because it is durable and replayable, a message can be re-read after a crash — the property that makes re-dispatch possible.
Tiling & reconstruction
Cut an image into a fixed grid; each tile remembers its (row, col). Process each tile independently, then write the results back into the same grid. Placement by coordinates — not arrival order — is what makes reconstruction deterministic.
Heartbeats & liveness
Workers publish a tiny message to a heartbeats topic every couple of seconds. The Master keeps a last-seen table and marks any worker stale when it goes quiet too long — turning "dead" and "hung" into the same detectable signal.
At-least-once & idempotency
Delivery can repeat: a worker may crash after finishing work but before acknowledging it. The fix is not to prevent repeats, it is to make effects idempotent — a tile result is keyed by tile_id, so a re-sent tile overwrites the same slot instead of being applied twice.
OpenCV
The imaging library that decodes and filters. Workers apply blur, edge detection, or resize on a tile's pixel window. The overlap/halo trick exists precisely because these filters read a pixel's neighborhood, not just the pixel.
SQLite (WAL) + Flask
SQLite stores job and tile rows durably in one file; WAL mode lets a reader run while the Master writes. Flask is a read-only sidecar serving GET /jobs/{id} — observability with no write path into the pipeline.
04 Architecture
Three Kafka topics sit at the center — tasks, results, and heartbeats — with the Master on one side, the worker pool on the other, and SQLite plus Flask underneath as the durable and observable layer.
| Component | Responsibility |
|---|---|
| Client | Submits an image, then queries job status from Flask. |
| Master (control plane) | Tiles the image, dispatches tasks, consumes results and heartbeats, reassembles by coordinates, tracks liveness, persists state. |
| Kafka broker | Durable transport across three topics; the replayable log that makes re-dispatch possible. |
| Worker pool | Consumes tile tasks, runs OpenCV, publishes results and heartbeats. |
| SQLite | Durable job + tile state; the source of truth for crash recovery. |
| Flask API | Read-only monitoring endpoints, decoupled from the pipeline hot path. |
05 Execution / Data Flow
The life of one image, from submission to a finished, monitored job. Reconstruction is keyed on coordinates, so steps 7–9 tolerate workers finishing in any order.
- The client
POSTs an image to the Master. - The Master reads the image dimensions, computes the tile grid —
(row, col)plus an overlap/halo — and inserts a job row withstatus = queuedinto SQLite. - The Master produces one message per tile to the
taskstopic, keyed by a deterministictile_id. - Workers in the consumer group poll
tasks, decode just their tile's pixel window, and run the OpenCV operation. - Workers publish results —
tile_id,(row, col), and the processed bytes — to theresultstopic. - The Master consumes results, writes each tile into its
(row, col)slot, and marks the tiledonein SQLite. - When every tile of a job is done, the Master flushes the stitched image and sets the job to
status = done. - Flask reads the job store to serve
GET /jobs/{id}— percentage complete and each worker's last heartbeat.
06 Key Design Decisions & Trade-offs
| Decision | Chosen approach | Alternative rejected | Why / trade-off |
|---|---|---|---|
| Message bus | Apache Kafka (partitioned log) | Redis pub/sub, RabbitMQ, in-memory queue | Durable, replayable log with offsets → re-dispatch after a crash is free. Costs broker ops; accepted for a systems project. |
| Delivery semantics | At-least-once + idempotent processing | Exactly-once via Kafka transactions | Correctness from idempotent workers, not the broker; avoids transaction latency. Repeats are tolerated, duplicates suppressed. |
| Reconstruction order | Coordinate-keyed tiles (row, col) | Arrival/timestamp order | Deterministic regardless of completion order or worker identity; a straggler can't corrupt the image. |
| Job-state store | SQLite (single file, WAL mode) | Postgres, Redis | ACID durability + transactions with zero ops. Redis loses state on crash; Postgres is overkill at one-node scale. |
| Parallelism model | Kafka consumer groups (one partition per worker) | Thread pool in one process | Scales horizontally across machines; Kafka manages membership and rebalancing. |
| Worker liveness | Heartbeat topic + stale timeout | Direct TCP health checks | Broker-mediated and network-agnostic; "hung" and "dead" collapse into one signal. |
| Monitoring | Flask reads SQLite (read-only) | In-process admin endpoint | Observability never blocks the hot path; API failure can't stop processing. |
The durable log gives crash recovery and replay, but it isn't free — every tile pays a broker round-trip and a disk write. I accepted that latency because the alternative (an in-memory queue) is faster until the first crash, at which point it loses in-flight tiles. For a pipeline whose point is correctness under failure, I trade a few milliseconds of latency for never losing work — never the reverse.
07 Challenges & How I Solved Them
Duplicate tiles after a worker crash
Symptom: a tile got applied twice. Root cause: the worker processed and published, then died before committing its offset — at-least-once redelivered the tile. Fix: key every result by tile_id and dedupe on write, so a redelivered tile overwrites the same slot instead of being applied twice.
Wrong reassembly order
Symptom: images stitched incorrectly when workers finished out of order. Root cause: reconstruction keyed on arrival order, not position. Fix: tag each tile with (row, col) and place results into a fixed grid — order-agnostic and deterministic.
Seams at tile boundaries
Symptom: visible lines where tiles met. Root cause: filters like blur/edge read a pixel neighborhood, and border pixels had no real neighbors. Fix: process each tile with a halo of extra pixels on every side, then crop the halo before stitching so borders see true neighbors.
Silent worker stall
Symptom: a job hung with no error. Root cause: a worker was alive but stuck, so there was no disconnect signal. Fix: heartbeat timeout on the Master — go quiet too long and the worker is marked inactive and its unfinished tiles re-dispatched.
Memory blow-up on large images
Symptom: decoding a big image exhausted RAM. Root cause: the whole image was decoded into one buffer. Fix: tile streaming — only a tile-sized window is ever decoded at once, so peak memory is a tile, not the image.
08 Interview Q&A
Why Apache Kafka instead of a plain in-memory queue, Redis pub/sub, or RabbitMQ?
Kafka is a durable, replayable, partitioned log, and three properties map directly onto my needs. First, durability: messages are written to disk and replicated, so a crash doesn't lose in-flight tiles — Redis pub/sub drops messages the moment no subscriber is listening, and a plain in-memory queue vanishes with the process. Second, replay: consumers read by offset, so when a worker dies mid-tile the Master can re-read the same tile from the log instead of it being gone forever. Third, horizontal scaling: partitions let N workers consume in parallel, and consumer groups handle membership and rebalancing for me. RabbitMQ is a fine broker with strong routing and ack semantics, but Kafka's log-plus-offset model maps one-to-one onto "re-dispatch after failure", which is the core need here. The trade-off is real: Kafka needs a broker and has more operational surface than Redis — acceptable for a systems project where the point is the distributed mechanics.
How do partitions and consumer groups give you parallelism?
A topic is split into partitions, each an ordered, append-only log. A consumer group is a set of workers that share the workload, and Kafka's key invariant is that each partition is assigned to exactly one consumer in the group. So parallelism equals the number of partitions: if the tasks topic has P partitions, up to P workers consume concurrently. Add more workers than partitions and the extras sit idle — so you scale partitions and worker count together. Ordering is guaranteed only within a partition, never across them, which is exactly why my reconstruction doesn't rely on arrival order (it keys on tile coordinates). The "one partition → one consumer" invariant is also what stops two workers from grabbing the same tile.
How do you avoid processing the same tile twice?
Two layers. Offsets: each worker periodically commits the offset of the last tile it fully handled, so on restart or rebalance it resumes after that point and doesn't reprocess committed tiles. But offsets alone can't give exactly-once — a worker can process a tile, publish its result, and crash before committing, and on recovery Kafka redelivers that tile (at-least-once). So the second layer is idempotency: every tile carries a deterministic tile_id (job + row + col), workers write results keyed by it, and the Master dedupes on write. A redelivered tile either overwrites the same key or is recognized as already-done and skipped. Net effect: a tile may be attempted twice, but never applied twice.
How do you handle a worker dying mid-tile?
Workers publish a heartbeat to a heartbeats topic on a fixed interval (roughly every 2s — estimate). The Master consumes that topic and keeps a last-seen map. If a worker's heartbeat goes stale past a timeout (roughly 5s — estimate), the Master marks it inactive and re-dispatches its unfinished tiles to another worker. Two subtleties worth naming: a worker that is alive-but-hung also stops heartbeating, so the timeout catches slow workers too, not just dead ones; and re-dispatch means the original worker might still finish and publish a result later — which is why every write is idempotent (see above), so a late duplicate just overwrites the same tile. Delivery is at-least-once by design; correctness comes from idempotency, not from perfect failure detection.
Why is coordinate-based reconstruction deterministic, and why does that matter?
Each tile is tagged with its (row, col) grid position, so the Master places each result at a fixed slot regardless of when it arrives — the final image is the same no matter the completion order or which worker did which tile. If I had relied on arrival order, a single straggler would corrupt the image. The reason it also matters for seams: a filter like blur or edge detection reads a pixel neighborhood, so a tile's border pixels have no correct neighbors on the other side. I process each tile with a small overlap/halo — a few extra pixels on every side (estimate) — compute the filter over the halo, then crop back to the tile's true bounds before stitching, so borders see real neighboring pixels. Determinism plus halo equals byte-for-byte reproducible output with clean edges.
How does tiling avoid loading the whole image into memory?
Only the tile being processed is decoded into RAM at any moment; the full image is never held as a single decoded buffer. The Master reads just the header to get dimensions, computes the tile grid (row, col, overlap), and assigns each tile a coordinate window. Workers decode only their window — for compressed formats like PNG/JPEG you can read a region without decoding the whole raster — process it, and free it. So peak memory is roughly one tile per worker plus the final reassembled image, instead of a full-resolution decode. That is what lets very large images — bigger than a worker's RAM — get processed at all. The trade-off: tiling adds boundary handling, and the halo costs a little redundant compute.
How is job state persisted and recovered after a Master restart?
Every job and every tile lives as a row in SQLite. The Master writes the job as queued, advances each tile through pending → processing → done, and marks the job done when all tiles complete. Because SQLite commits are durable (I run WAL mode so a reader can read while the Master writes), a Master crash loses nothing. On restart the Master reads the job table, finds any job that isn't done, and re-dispatches the still-pending/processing tiles. Tiles a worker was mid-way through are simply re-issued — at-least-once plus idempotency covers the overlap. The key point I'd stress: SQLite is the source of truth for "what's left to do", while Kafka is only the transport. I could replay from the log instead, but the job table is a cheaper and more explicit recovery path.
How does Flask expose monitoring without blocking the pipeline?
Flask runs as a separate process from the Master's event loop and only ever reads. It opens a read-only connection to the same SQLite file the Master writes and serves endpoints like GET /jobs/{id} (status, percentage of tiles done, per-worker last heartbeat) and GET /jobs (list). Because it reads the same file rather than sharing in-memory state, there's nothing to synchronize and no lock contention on the hot path — workers never talk to Flask at all. If the API dies, processing continues unaffected. The one shared resource is SQLite, and WAL mode lets the reader run concurrently with the writer. Monitoring is a read-only sidecar, not a step in the pipeline.
How would you scale this to 100 workers or 100k images?
Horizontally, and by raising partition count in step with workers — more workers only help if partitions grow too, because of the one-partition-one-consumer rule. The plan: (1) partition the tasks topic into, say, 100+ partitions so all 100 workers stay busy; (2) the Master becomes the risk — one process consuming every result and heartbeat and doing all reassembly won't keep up, so I'd shard it by job range (multiple Master instances, each owning a slice, coordinated via the job table or Kafka itself). For 100k images, batching and backpressure matter more than raw worker count: cap in-flight jobs and persist results incrementally instead of holding everything in memory. The honest part: I'd need to measure where it saturates first — partition throughput, Master reassembly CPU, or SQLite write rate — and that's the first thing I'd profile. SQLite is likely the first casualty at that scale and moves to Postgres or a per-shard store.
How do you handle backpressure?
At two points. Producer side: the Master doesn't dump 100k tile tasks into the topic unboundedly — it throttles by job and only dispatches the next batch as workers report progress, keeping a bounded in-flight window so memory and broker backlog stay flat. Consumer side: workers pull (poll) at their own pace, so a slow worker simply consumes slower; Kafka buffers the rest in the log, and you watch consumer lag as the backpressure signal. If lag grows, you add workers and partitions rather than let the backlog balloon. The clean mental model I use: the durable log is the buffer, consumer lag is the gauge, and the Master's dispatch window is the valve.
What ordering guarantees do you have — and not have?
Within a single partition Kafka preserves produce order, so tiles sent to the same partition arrive in order. Across partitions there is none — and I deliberately don't rely on any. The system is order-agnostic: reconstruction keys on (row, col), not arrival, and every tile is independent, so out-of-order completion can't corrupt output. If I needed strict per-image order (say, a pipeline where step B must follow step A per image), I'd key messages by job_id so all of one job's messages land in one partition — gaining per-job order at the cost of parallelism. It's the classic order-vs-parallelism trade, and I chose parallelism because tiles are embarrassingly independent.
How would you test the distributed behaviour?
In layers. Unit: pure functions — tiling math, coordinate-to-pixel mapping, idempotency dedup — with plain asserts. Integration: a real local Kafka (testcontainers or a docker-compose broker) plus a few workers, driving real produce/consume for a small synthetic image and asserting the reconstructed output is byte-identical to a single-process reference. Failure injection is where it earns its keep: kill a worker mid-job and assert the job still completes with no tile applied twice; crash the Master and restart it, asserting recovery from SQLite; run workers out-of-order and assert deterministic output. Plus a small chaos loop that randomly kills and restarts workers over many runs. The things that are hard to unit-test — timing, rebalancing, at-least-once redelivery — are covered by those fault-injection runs, not by mocking the broker.
What breaks at 10x?
Honestly, three things, in order of likelihood. First, SQLite: a single file with a single writer becomes the contention point as job/tile write volume climbs 10x — the Master's writes serialize. Second, the Master itself: it consumes every result and heartbeat and does reassembly in one process, so at 10x it becomes a CPU/event-loop bottleneck before the workers do. Third, partition sizing: if I don't grow partitions with workers, the added workers idle. The fix for all three is the same shape — shard the Master by job range and move durable state to a real multi-writer store — which is the v2 plan, not something I'd bolt on now. The discipline is: at 10x you stop guessing and start profiling consumer lag, Master CPU, and SQLite write latency.
Exactly-once vs at-least-once — explain the difference and what you actually have.
At-least-once guarantees no message is lost, but a message may be delivered more than once. Exactly-once guarantees each message is processed exactly once end-to-end. The catch: "exactly-once" is two different problems. Kafka's exactly-once (idempotent producer + transactions + read_committed consumers) solves exactly-once delivery within Kafka — no duplicate writes to a topic. But exactly-once processing — making an external side effect like a file write or a DB row happen once — is impossible in a distributed system without coordination; the best you get is effectively-once by making the side effect idempotent. So my system is at-least-once delivery plus idempotent processing, which gives the same observable result (no duplicate tile applied) without the transaction machinery and its latency cost. If I genuinely needed Kafka transactions I'd add an idempotent producer and read_committed consumers — but for tiles that are idempotent by construction, that's complexity I don't pay for.
Why SQLite for the job store instead of Postgres or Redis?
For a single-node systems project, SQLite buys real ACID durability and a transaction API with zero operational overhead — no server, no auth, one file. Redis is faster but in-memory, so you accept losing job state on crash, which is unacceptable for "recover after restart". Postgres is the right long-term answer at scale, but running a server for this buys nothing when one Master writes and one Flask reads. The decision is honest: SQLite is the lazy-correct choice at this scale, and the migration path to Postgres is a clean data-layer swap. I'd say exactly that to an interviewer rather than pretend I needed Postgres for what is, at this scale, a single-writer workload.
How do you pick tile size and the overlap/halo width?
Tile size is a trade between parallelism and overhead: smaller tiles mean more parallel work and less memory per tile, but more messages, more per-tile overhead, and more boundary pixels to handle. I make it configurable and default to something like 256×256 or 512×512 px (estimate), chosen so a worker's peak memory sits comfortably under its limit. The halo width is driven by the filter's kernel radius: a blur with a k×k kernel needs roughly k/2 pixels of halo per side so border pixels see full neighborhoods. The rule I encode is simple — halo ≥ kernel radius, and crop the halo before stitching. Both are tuned per workload, not hard-coded religion.
09 Limitations & If I Rebuilt It
What is genuinely limited, and what v2 would change:
- SQLite is a single-writer bottleneck. One Master writing tiles serializes on the file. v2: Postgres (or a per-shard store) once write volume climbs.
- At-least-once, not exactly-once. Idempotency makes duplicates harmless, but the machinery to avoid them entirely is absent. v2: idempotent producer + read_committed consumers if I need true broker-level once.
- The Master is a single point of failure. If it dies, no dispatch or reassembly happens until restart. v2: active/standby Master with leader election via Kafka, or sharded Masters.
- Fixed, naive tile grid. Uniform tile size ignores image content. v2: adaptive tiling and a configurable overlap width per filter.
- Reassembly holds the final image in memory. For very large outputs the stitched buffer can get big. v2: stream tiles to a disk-backed canvas as they complete.
- Fixed heartbeat timeout. A one-size timeout risks false timeouts under load. v2: adaptive timeout with backoff.
10 Talking Points & Metrics
All numbers are design targets or labelled estimates — not measured benchmarks. I never quote throughput or latency I haven't measured.
Talking points
- Parallelism comes from partitions; one partition → one consumer in a group.
- Correctness under failure comes from idempotency, not from perfect detection.
- Coordinate-keyed reconstruction makes output deterministic regardless of order.
- Heartbeat timeout turns "dead" and "hung" into one detectable signal.
- SQLite is the recovery source of truth; Kafka is only the transport.
11 Project Glossary
- Master
- The control-plane process: tiles images, dispatches tasks, consumes results, reassembles, tracks liveness, and persists state.
- Worker
- A process in the consumer group that consumes tile tasks and runs OpenCV.
- Topic
- A named, append-only message log in Kafka.
- Partition
- A shard of a topic; the unit of parallelism and of per-partition ordering.
- Consumer group
- A set of consumers that share a topic's partitions; each partition goes to exactly one member.
- Offset
- A consumer's position in a partition log; committing it marks messages as processed.
- At-least-once delivery
- A guarantee that no message is lost, at the cost of possible duplicates.
- Heartbeat
- A periodic liveness message a worker publishes so the Master can detect death or stalls.
- Tile
- A rectangular sub-region of the image, tagged with its
(row, col). - Tiling
- Splitting the image into tiles so they can be processed independently and in parallel.
- Reconstruction
- Stitching processed tiles back into the image by coordinate.
- Halo / overlap
- Extra border pixels around a tile so edge filters see real neighbors; cropped before stitching.
- Idempotency
- Making an operation safe to repeat — a redelivered tile overwrites the same slot instead of applying twice.
- Job store
- The durable SQLite table holding job and tile state for recovery.
References & Further Reading
- Apache Kafka — official documentation
- Kafka docs — consumers, consumer groups, and offsets
- Kafka docs — delivery semantics (at-least-once, exactly-once)
- Confluent — What is Apache Kafka?
- OpenCV — official documentation
- Flask — official documentation
- SQLite — official documentation
- SQLite — Write-Ahead Logging (WAL mode)