Scraping Istanbul's LRT arrival signs for a year: what broke and why

Last April I started scraping the public arrival boards for the T1 tram line in Istanbul. Not for a dashboard, just to answer a personal question: how often is the displayed “2 min” actually true? I pointed a small Python script at the same JSON endpoint the mobile app uses, every 30 seconds, and let it run on a €35 Orange Pi in my apartment. The dataset is now 11 months old, roughly 1.2 million rows. The scraping part took an afternoon. Keeping it alive took the rest of the year.

The first version was naive: requests.get on the endpoint, parse the ETA field, write to SQLite. Here is the core loop I still use, with the fixes folded in:

import requests, sqlite3, time
from datetime import datetime, timezone

DB = '/home/pi/tram.db'
URL = 'https://api.ibb.gov.tr/.../arrivals?line=T1'
HEADERS = {'User-Agent': 'personal-eta-check/1.0'}

def fetch():
    r = requests.get(URL, headers=HEADERS, timeout=8)
    r.raise_for_status()
    return r.json()

def store(stop_id, direction, eta_min):
    con = sqlite3.connect(DB)
    con.execute(
        'INSERT INTO arrivals (ts, stop_id, direction, eta_min) VALUES (?,?,?,?)',
        (datetime.now(timezone.utc).isoformat(), stop_id, direction, eta_min)
    )
    con.commit()
    con.close()

while True:
    try:
        data = fetch()
        for item in data.get('stops', []):
            store(item['stopId'], item['direction'], item['eta'])
    except Exception as e:
        print(datetime.now(), e)
    time.sleep(30)

That still looks reasonable, but it hides most of the real work. The endpoint has no documented rate limit, yet after about two weeks of 30-second polling I started getting 403s for six-hour stretches. I never found a published policy. I switched to 90 seconds and added jitter: time.sleep(90 + random.uniform(-10, 10)). The 403s mostly stopped. I also cache the last good response and only insert when the payload hash changes, which cut writes by about 70% and made the SD card last longer.

The bigger lesson was about time. The API returns eta in whole minutes, and the timestamp is when the server answered, not when the sign was rendered. If you compare that to actual arrival logs from the same stops (I later added a second script that records when the tram actually passes), the displayed minute is optimistic by roughly 40 seconds on average, and much worse during the evening rush. That is not a bug in the sign; it is the cost of rounding and network latency. I stopped treating the numbers as ground truth and started treating them as a noisy signal.

Where this breaks: the endpoint changes shape without notice. In August the field eta became a string for some stops and an integer for others. My parser crashed silently for three days because I caught the exception and only printed it. If you build something like this, log failures to a file, not just stdout. Also, the Orange Pi’s clock drifted by 11 minutes over five months because I never enabled NTP. Every row before I fixed it is off by that amount, and I only noticed when the evening peak looked like it started at 17:49 instead of 18:00. That is a data quality problem no amount of scraping discipline fixes after the fact.

I still do not know if the 403s were rate limiting or a CDN rule. If anyone has run a similar poll against IBB endpoints for more than a few weeks, I would like to compare notes on what interval you settled on and whether you saw the same silent string/integer switch.