SDE bulk-load into a local store (follow-up from #16) #51

Closed
opened 2026-06-27 18:31:32 +00:00 by Latte · 1 comment
Owner

Deferred from Phase 5 (#16) — it's a large, self-contained effort.

Download the Static Data Export, load it into a local store (e.g. SQLite), and back the universe/recipe lookups with it for offline, fast, name- and recipe-based queries (currently served via public ESI).

  • Pick a format/source (SDE SQLite conversion or YAML/CSV) and a download/refresh step
  • Local store + loader (cached on disk, lazy/one-time)
  • Back get_item / systems / stations / blueprints with the SDE where it beats ESI
  • Offline name <-> id resolution without a round-trip
  • Tests against a tiny fixture subset

The rest of Phase 5 (Expires caching, error-limit back-off, setup docs) shipped under #16.

Deferred from Phase 5 (#16) — it's a large, self-contained effort. Download the [Static Data Export](https://developers.eveonline.com/resource/resources), load it into a local store (e.g. SQLite), and back the universe/recipe lookups with it for offline, fast, name- and recipe-based queries (currently served via public ESI). - [ ] Pick a format/source (SDE SQLite conversion or YAML/CSV) and a download/refresh step - [ ] Local store + loader (cached on disk, lazy/one-time) - [ ] Back `get_item` / systems / stations / blueprints with the SDE where it beats ESI - [ ] Offline name <-> id resolution without a round-trip - [ ] Tests against a tiny fixture subset The rest of Phase 5 (Expires caching, error-limit back-off, setup docs) shipped under #16.
Latte added the epic label 2026-06-27 18:31:32 +00:00
Latte closed this issue 2026-06-27 19:27:41 +00:00
Author
Owner

Done on dev via PR #58: familiar sde download streams + bz2-decompresses the Fuzzwork SQLite SDE into the local store; offline system_by_* / station_by_id lookups added; universe.get_system/get_station prefer the SDE when present and fall back to ESI. 159 tests, 90.8% coverage. Closing.

Done on `dev` via PR #58: `familiar sde download` streams + bz2-decompresses the Fuzzwork SQLite SDE into the local store; offline `system_by_*` / `station_by_id` lookups added; `universe.get_system`/`get_station` prefer the SDE when present and fall back to ESI. 159 tests, 90.8% coverage. Closing.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Hiddenden/familiar#51