Asynchronous by default
SQS decouples the request from the full catalog operation. Lambda can process smaller units instead of holding one synchronous execution open for the entire job.
← Back to selected work
Product Ingestion Pipeline
An asynchronous product-ingestion pipeline for a multi-store catalog, redesigned to remove a slow synchronous bottleneck and reduce manual rework.
RoleLead Frontend Developer · led three developers
FocusTypeScript · React · Next.js
01Context
The problem
A synchronous ingestion process took approximately two hours to complete across a large catalog. Long-running work made failures expensive, slowed feedback, and created avoidable manual intervention.
My contribution
I led the redesign and contributed across the application, backend, and AWS architecture. The solution moved the long-running work into an asynchronous serverless pipeline using Lambda and SQS.
02System view
How the pieces connect
A conceptual view of the system, recreated from the public-safe project description.
01CatalogProducts and store configuration
02QueueSQS distributes work
03WorkersLambda processes units
04UpdatesStore-specific results
05FeedbackFaster completion
03Outcome
Processing went from approximately two hours to ten minutes, a reduction of roughly 92%. The team also observed fewer errors and less manual rework, although no additional percentages are being claimed.
2happroximate processing time before
10mapproximate processing time after
~92%approximate reduction in duration
04Engineering notes
Design around the business shape
The pipeline reflects the actual dimensions of the catalog: products, stores, variants, and modifiers. That makes the architecture explainable to both the delivery team and the people waiting for a catalog update.
Measure the before and after
The clearest success signal was operational time: a process that took about two hours completed in about ten minutes. Error and rework improvements are included as observations, not overstated measurements.
05Scope & limits
Stack
What I would keep visible
- The client is described generically because the client name is not confirmed as publishable.
- The duration and scale figures are approximate historical project figures, not a current benchmark.
- No claim is made about a specific retry, delivery guarantee, or observability implementation beyond the confirmed architecture.
The key improvement came from changing the shape of the work, not simply optimizing a synchronous function. Queue-backed units of work gave the process a path to scale and made the bottleneck visible.
Explore another case