Most iCal polling is wasted work

Every iCal integration starts the same way: a background job that fetches a .ics file every ten or fifteen minutes and compares it with the last copy. It works, it is a hundred lines, and it quietly becomes the most expensive thing in the product.

The arithmetic nobody runs

A calendar for one holiday rental changes a handful of times a week. Polled every ten minutes, that is 10,080 requests per feed per week to discover maybe seven changes. The other 10,073 requests carry no news at all, and each one still costs:

• a DNS lookup and a TLS handshake,
• the provider's response, which for a busy calendar is tens of kilobytes,
• a parse of the whole file,
• a comparison against the stored copy,
• and a database write, if the job records that nothing happened.

Multiply that by a few thousand feeds and the queue is doing real work to learn nothing. This is what people mean when they say the integration "got slow one day": nothing changed except the number of calendars.

Conditional requests are the cheap half of the fix

Almost every provider supports ETag or Last-Modified, and every serious HTTP client will send them back if you keep them. The exchange then looks like this:

GET /villa-42.ics
If-None-Match: "8f3c1a9e"

HTTP/1.1 304 Not Modified

The body never crosses the wire, so there is nothing to parse and nothing to compare. In practice this is where the traffic goes: a healthy feed answers 304 on the overwhelming majority of polls, and the ones that matter are the few that answer 200.

That is not a micro-optimisation. It is the difference between a queue whose cost grows with feeds and one whose cost grows with changes.

The expensive half is the diff

A 304 tells you nothing moved. It does not tell you what moved, and that is the part integrations get wrong. Comparing two calendar files means deciding:

• an event was modified rather than deleted and re-added — otherwise every edit looks like a cancellation plus a new booking to whatever consumes it,
• recurring events are expanded the same way twice,
• a provider that rewrites its X-GENERATED-AT stamp on every request is not a change at all.

Get this wrong and the integration does something destructive on a schedule, which is worse than being slow.

What to keep, and what to hand over

The rule that has served us well: keep the decision, hand over the traffic.

Your application should own the webhook endpoint, the signature check and what a change means for your data. Everything before that — the schedule, the conditional requests, the backoff when a provider is having a bad day, the parse and the diff — is the same code in every product that does this, and it is the code that breaks quietly at 3am.

That is the whole shape of the service we run: you register the feed URLs your users connected, and an HTTP POST arrives when something actually changed. If you would rather keep polling, the two things above — conditional requests and a diff that distinguishes modification from replacement — are the ones worth doing first.