Your rate limit is a concurrency setting
Wayli is an AI-generated audio walking-tour app. As of early November the catalogue holds 5,824 places in 139 cities across 14 countries, 99.7% of them with a cover image. Almost everything a listener sees on a place page comes from someone else’s free infrastructure: the map data, the geocoding, the photos, the opening hours. We pay for the model calls and the speech synthesis, and we get the rest on trust.
That arrangement only works if we behave. Public services don’t send invoices; they send 429s, and eventually blocks. So we stopped treating etiquette as a footnote in the client code and started treating it as a design constraint, on the same level as cost. This post is what that looked like in practice, including the day we got it wrong.
One door for every write
Every place enters the system through a single queue, and the queue starts a Step Functions workflow. The steps are boring on purpose: create the place as pending, generate the text with an LLM, have a judge on a different provider validate it (one redo allowed, a second rejection marks the place flagged), publish, fetch a cover image, fetch visitor info such as opening hours and a website. The last two are non-fatal. A place without a photo is still a place.
Executions are named after the request id, so a duplicate request is a success, not a second run. Place ids are derived deterministically, a uuid5 of the request id and the OSM id, so a retry lands on the same row instead of creating a twin. Narration is lazy: speech is generated on first playback through an idempotent, claim-gated endpoint and the MP3 goes to S3. That is why only about 12% of stories have audio so far, and why that number is a feature rather than a backlog.
Cost gates go before the expensive step
The “what’s near me” feature is where tokens could leak fastest. Locations are indexed by geohash, and a lookup is a prefix query at length 4, 5 or 6, cells of roughly 40 km, 5 km and 1 km. If the area is thin, a discovery run goes looking for new places in the open data and asks a model to rank them. Every gate that can say no sits ahead of that call:
- a global kill switch and a daily cap on runs
- a cooldown per geohash-6 cell, so the same street corner can’t trigger a run twice in a row
- a growing-radius sweep (1, 2, 4, 8, then 10 km) that stops at the first radius with enough candidates
- deduplication against existing places within about 100 m, using a geohash-7 bucket, before the LLM ranking, so duplicates never cost a token
- a snapshot of the run configuration, so a change mid-flight doesn’t alter a run already in progress
A refused run ends successfully, not as a failure, and refunds the user’s meters in a single idempotent transaction. Discovery runs are also serialised: reserved concurrency of 1 plus a lock item in the table. It is slower than parallel, and that is the point.
Querying OpenStreetMap politely
Candidate places come from the Overpass API, which serves OpenStreetMap data. Our first version used one statement per tag. Over Tel Aviv it timed out with a 504. Overpass query cost scales with the number of statements, so we grouped the tags into six regex statements, and the same area answered in about two seconds. Same data, a fraction of the load on someone else’s server.
The client rules are short. Retry only on 429 and 504, since other errors won’t improve with repetition. Send a descriptive User-Agent that says who we are. Never exceed one request per second.
Geocoding followed the same logic. The public Nominatim usage policy forbids autocomplete, and a search box is autocomplete whether we call it that or not. So we use Photon, which is OSM-based and built for search-as-you-type. Results are cached in DynamoDB for 24 hours, the whole system is limited to 2 requests per second globally, and each call has a 2-second timeout. The pin sanity checks that follow have per-service circuit breakers, an 8-second HTTP timeout and a 150-second wall budget, so one slow upstream can’t hold a workflow open.
Images: only what we can legally show
The cover chain stops as soon as it has three images. It tries the images we were given, then Wikimedia Commons, then the image property on Wikidata, then Openverse. Only on a full miss do we generate an illustration. The whole chain has a 50-second budget.
The licence gate reads machine-readable fields only. Public domain, CC0, CC BY and CC BY-SA pass. NonCommercial and NoDerivatives are rejected: the app is commercial, and we crop every image. Beyond licences there is a name-relevance threshold, a ban on maps and logos, a 640 px minimum width and a dedupe across sources. Each stored image keeps its artist, licence, licence URL and source, and the app shows “Photo: artist · licence”. Generated ones are labelled “AI illustration”, always.
We considered Google image search and dropped it. Its Custom Search API shuts down on 2027-01-01, and licences scraped from result pages can’t be verified. An image whose licence we can’t prove is a liability, not a cover.
The day we hit 429 on Commons
A bulk seeding run started 215 workflow executions in 44 seconds. The cover-fetcher Lambda did what Lambda does and scaled out to 106 containers. Commons answered 119 of their requests with a 429. Worse, the failed rows had no status at all, so nothing downstream knew a cover was missing.
The cause was one sentence long: the container count is your global request rate. Our per-container politeness meant nothing once a hundred containers each made their “polite” request in the same second. The fix was equally short. We set reserved concurrency on the fetcher to 4. We made retries honour Retry-After, with jitter and a 30-second budget. We added workflow-level retries, and a CoverFailed metric so a failure becomes a number on a dashboard instead of a blank cell in a table.
Notice what that is: a rate limit expressed as a concurrency setting. Lambda has no native “at most N requests per second to this host”. It has a ceiling on parallel executions. For a function that makes one upstream call per invocation, that ceiling divided by how long each call takes is your request rate, and it is the only rate knob you actually hold.
Budgets for the paid parts
Speech synthesis is billed against a flat monthly character allowance, which is why narration is lazy. Pre-narration, when we do it, has its own character budget with a 10% reserve. If the primary voice is unavailable, a cheaper fallback speaks instead and marks the audio for regeneration later, so a degraded path repairs itself. Free accounts get per-user daily quotas. And the read path, the part users hit constantly, was load-tested at 50 requests per second with zero errors and a p50 around 241 ms. It is cheap because it reads precomputed rows and never calls anything external.
What we took from it
- Put every gate that can refuse work ahead of the expensive step, and make a refusal a clean, refunded success rather than an error.
- On serverless, your outbound rate is your concurrency. Cap it deliberately, per upstream, instead of hoping each container behaves.
- Read the usage policy before you pick the API. Choosing a service that is meant for your access pattern beats engineering around one that isn’t.
- Keep licence and attribution data next to every asset from the first fetch, and decide from machine-readable fields only.
Free public data is the reason a small team can build something like this at all. The least we can do is make our traffic look like a good neighbour’s: low, predictable, identified, and quick to back off. As a side effect, the same habits kept our own bill and our failure modes small, which is a trade we’ll take every time.