[{"data":1,"prerenderedAt":171},["ShallowReactive",2],{"\u002Fwork\u002Fcase-studies\u002Fcatalogue-automation":3},{"id":4,"title":5,"body":6,"businessProblem":133,"confidentiality":134,"description":135,"employer":136,"extension":137,"featured":138,"heroStat":139,"meta":142,"navigation":138,"ogImage":143,"order":144,"path":145,"publishedAt":146,"results":147,"seo":159,"stem":160,"summary":161,"technologies":162,"updatedAt":143,"__hash__":170},"caseStudies\u002Fwork\u002Fcase-studies\u002Fcatalogue-automation.md","High-Volume Ecommerce Catalogue Automation",{"type":7,"value":8,"toc":121},"minimark",[9,14,18,31,34,38,43,47,54,60,66,72,76,90,94,97,101,114,118],[10,11,13],"h2",{"id":12},"the-problem","The problem",[15,16,17],"p",{},"The business ran ecommerce on top of a supplier catalogue nobody fully controlled. More than 200,000\nSKUs came in through:",[19,20,21,25,28],"ul",{},[22,23,24],"li",{},"Nightly FTP drops in half a dozen different CSV\u002FExcel dialects",[22,26,27],{},"A handful of supplier REST and SOAP APIs, each with its own auth and pagination quirks",[22,29,30],{},"Scraped product pages for suppliers who published no feed at all",[15,32,33],{},"Every source had its own idea of a SKU, a category tree, and a unit of measure. Pricing and\ninventory changed daily. Before automation, updating the catalogue was a recurring manual chore —\nslow, error-prone, and a hard ceiling on how many suppliers the business could realistically carry.\nThe business problem wasn't \"we need a script,\" it was \"we can't grow the catalogue without this\nbecoming someone's full-time job.\"",[10,35,37],{"id":36},"architecture","Architecture",[39,40],"architecture-diagram",{":steps":41,"title":42},"[{\"label\":\"Ingest\",\"detail\":\"FTP, REST\u002FSOAP APIs, scrapers\"},{\"label\":\"Normalize\",\"detail\":\"Units, categories, attributes\"},{\"label\":\"Match & validate\",\"detail\":\"Dedup, price\u002Fstock sanity checks\"},{\"label\":\"Publish\",\"detail\":\"Shopify, Odoo, internal reporting\"}]","Catalogue pipeline — simplified",[10,44,46],{"id":45},"what-i-built","What I built",[15,48,49,53],{},[50,51,52],"strong",{},"Ingestion layer."," A set of Python jobs per supplier source — FTP pull, API client, or scraper —\neach normalizing into a common intermediate schema before anything touches the database. New\nsuppliers plug in as a new ingestion adapter, not a rewrite of the pipeline.",[15,55,56,59],{},[50,57,58],{},"Normalization and matching."," Supplier part numbers, UPCs, and free-text titles get reconciled\nagainst the existing catalogue using a mix of exact-key matching and fuzzy matching for the messy\ncases (title similarity, brand + attribute matching) with a manual review queue for anything under\na confidence threshold — so ambiguous matches get a human decision instead of a silent guess.",[15,61,62,65],{},[50,63,64],{},"Pricing and inventory sync."," Cost and quantity updates flow through margin rules before they hit\nthe storefront, with change thresholds that flag anything that moves further than expected (a price\ndrop of 90% is far more likely to be a bad feed row than a real discount).",[15,67,68,71],{},[50,69,70],{},"Publishing."," Validated records write to Odoo and Shopify through their APIs, with diffs logged so\nevery price or inventory change is traceable back to the source file or API call that caused it.",[10,73,75],{"id":74},"reliability-and-validation","Reliability and validation",[19,77,78,81,84,87],{},[22,79,80],{},"Every ingestion run produces a diff report before anything is published — new items, removed\nitems, and changed fields are all visible before they go live.",[22,82,83],{},"Sanity thresholds catch outlier price and quantity changes for manual review instead of\nauto-publishing them.",[22,85,86],{},"Failed or partial supplier feeds fail loudly (alerting) rather than silently publishing stale or\nincomplete data.",[22,88,89],{},"A confidence-scored matching queue means uncertain SKU matches never get merged automatically.",[10,91,93],{"id":92},"technologies-used","Technologies used",[15,95,96],{},"Python for ingestion and transformation, PostgreSQL as the system of record, pandas for bulk\ntransforms, and the Shopify Admin API and Odoo's API for publishing. Scheduled jobs run the pipeline\non a nightly and hourly cadence depending on the source.",[10,98,100],{"id":99},"business-impact","Business impact",[15,102,103,104,109,110,113],{},"The catalogue automation removed manual data entry as the ceiling on supplier growth — the business\nwent from a catalogue that required constant manual upkeep to one that scales by adding ingestion\nadapters. Combined with the marketplace and advertising work described in the\n",[105,106,108],"a",{"href":107},"\u002Fwork\u002Fcase-studies\u002Fshopify-amazon-sync","Shopify and Amazon marketplace synchronization"," case study,\nthis system was part of growing this employer's ecommerce revenue from roughly ",[50,111,112],{},"$60,000 to\n$400,000 per month",".",[10,115,117],{"id":116},"what-id-change-next","What I'd change next",[15,119,120],{},"The matching layer is the piece I'd keep investing in — better fuzzy-matching and a tighter feedback\nloop from the manual review queue back into the automated confidence scoring, so the system gets\nbetter at flagging its own uncertain calls over time.",{"title":122,"searchDepth":123,"depth":123,"links":124},"",3,[125,127,128,129,130,131,132],{"id":12,"depth":126,"text":13},2,{"id":36,"depth":126,"text":37},{"id":45,"depth":126,"text":46},{"id":74,"depth":126,"text":75},{"id":92,"depth":126,"text":93},{"id":99,"depth":126,"text":100},{"id":116,"depth":126,"text":117},"Product data for 200,000+ SKUs arrived from dozens of suppliers in inconsistent formats — FTP flat files, REST and SOAP APIs, and, for suppliers with no feed at all, scraped catalogue pages. None of it agreed on units, categories, or even what counted as the same product.\n","anonymized","How I built a Python-based pipeline that ingests, cleans, matches, and publishes 200,000+ supplier products across FTP feeds, APIs, and scraped sources — with validation built in.","a Canadian industrial distributor","md",true,{"label":140,"value":141},"SKUs under management","200,000+",{},null,1,"\u002Fwork\u002Fcase-studies\u002Fcatalogue-automation","2026-01-15",[148,151,153,156],{"label":149,"value":150},"Manual catalogue hours per week","-80%",{"label":152,"value":141},"SKUs under active automation",{"label":154,"value":155},"Pricing\u002Finventory refresh cycle","Daily → hourly",{"label":157,"value":158},"Duplicate\u002Fmismatched listings","Caught pre-publish",{"title":5,"description":135},"work\u002Fcase-studies\u002Fcatalogue-automation","Automating ingestion, cleanup, matching, and pricing for 200,000+ supplier SKUs.",[163,164,165,166,167,168,169],"Python","PostgreSQL","Pandas","Odoo","Shopify Admin API","Cron \u002F scheduled jobs","FTP\u002FSFTP","54mWLZEHR2EO6_FtvYCFbJ_-UOYmblyE3ijgq8hWufM",1785688078907]