High-Volume Ecommerce Catalogue Automation

The problem

The business ran ecommerce on top of a supplier catalogue nobody fully controlled. More than 200,000 SKUs came in through:

• Nightly FTP drops in half a dozen different CSV/Excel dialects
• A handful of supplier REST and SOAP APIs, each with its own auth and pagination quirks
• Scraped product pages for suppliers who published no feed at all

Every source had its own idea of a SKU, a category tree, and a unit of measure. Pricing and inventory changed daily. Before automation, updating the catalogue was a recurring manual chore — slow, error-prone, and a hard ceiling on how many suppliers the business could realistically carry. The business problem wasn't "we need a script," it was "we can't grow the catalogue without this becoming someone's full-time job."

Architecture

What I built

Ingestion layer. A set of Python jobs per supplier source — FTP pull, API client, or scraper — each normalizing into a common intermediate schema before anything touches the database. New suppliers plug in as a new ingestion adapter, not a rewrite of the pipeline.

Normalization and matching. Supplier part numbers, UPCs, and free-text titles get reconciled against the existing catalogue using a mix of exact-key matching and fuzzy matching for the messy cases (title similarity, brand + attribute matching) with a manual review queue for anything under a confidence threshold — so ambiguous matches get a human decision instead of a silent guess.

Pricing and inventory sync. Cost and quantity updates flow through margin rules before they hit the storefront, with change thresholds that flag anything that moves further than expected (a price drop of 90% is far more likely to be a bad feed row than a real discount).

Publishing. Validated records write to Odoo and Shopify through their APIs, with diffs logged so every price or inventory change is traceable back to the source file or API call that caused it.

Reliability and validation

• Every ingestion run produces a diff report before anything is published — new items, removed items, and changed fields are all visible before they go live.
• Sanity thresholds catch outlier price and quantity changes for manual review instead of auto-publishing them.
• Failed or partial supplier feeds fail loudly (alerting) rather than silently publishing stale or incomplete data.
• A confidence-scored matching queue means uncertain SKU matches never get merged automatically.

Technologies used

Python for ingestion and transformation, PostgreSQL as the system of record, pandas for bulk transforms, and the Shopify Admin API and Odoo's API for publishing. Scheduled jobs run the pipeline on a nightly and hourly cadence depending on the source.

Business impact

The catalogue automation removed manual data entry as the ceiling on supplier growth — the business went from a catalogue that required constant manual upkeep to one that scales by adding ingestion adapters. Combined with the marketplace and advertising work described in the Shopify and Amazon marketplace synchronization case study, this system was part of growing this employer's ecommerce revenue from roughly $60,000 to $400,000 per month.

What I'd change next

The matching layer is the piece I'd keep investing in — better fuzzy-matching and a tighter feedback loop from the manual review queue back into the automated confidence scoring, so the system gets better at flagging its own uncertain calls over time.