What actually costs the time
European procurement data is published for reuse. The licences permit it, the APIs are free, the documentation exists. And then you start building.
Tender counts arrive as two parallel arrays that mean nothing read separately. Amounts turn up in two decimal conventions. Contract periods occasionally end before they begin. A request asking for too many fields fails with an error that looks like a field problem and is not — and the obvious recovery fires one probe per field, trips the rate limiter, and hands you 429s that look like a third distinct problem.
Every code sample here comes from a pipeline that runs daily against the live APIs, including the parts that failed first.
What's inside
Nine chapters, in the order the problems actually appear.
- Two sources, two different jobs — TED and ANAC: coverage, latency, join keys, and when each is the wrong choice
- The TED Search API — ITERATION pagination, and the notices-per-page × field-count limit you should compute rather than discover
- Finding fields that exist — the 400 response enumerates every valid field name, and a probing method that separates rejected fields from ones that are simply always empty
- The shapes that break parsers — multilingual dictionaries, per-lot repetition, parallel arrays, NUTS codes interleaved with bare country codes
- Ingesting ANAC OCDS — streaming hundreds of megabytes of JSONL, and the three categories of data defect the publisher documents itself
- The normalisation layer — amounts, dates, company names, and where name collapsing goes wrong
- Schema and idempotent writes — migrations derived from the schema, and why tables → migrate → indexes is not optional
- Metrics worth deriving — discount, competition, contract expiry and recurrence, with the reasoning behind each
- Limits worth stating — what the data does not contain, and why publishing that makes your numbers defensible
Written from building it
Not assembled from documentation.
Verified field references
Probed against the live API, with the exact response shape for each — including the ones whose obvious name is rejected.
Working code
Python, dependency-light, copy-paste ready. No vendor SDK required.
Failure modes documented
The errors that mislead, and what they actually mean. That is most of the value.
A method that outlasts it
APIs evolve. The probing method lets you re-verify rather than trust a snapshot.
What this is not
Not an introduction to public procurement law. Not a vendor tutorial. Not a general data-engineering primer.
One honest caveat: field references were verified against the live APIs at the time of writing, and both services evolve. The handbook includes the probing method precisely so you can re-verify — that method will outlast any specific field name in it.