UniverseDB Data Pipeline

Data pipeline that extracts country, subdivision, and city data from public CSV sources, normalizes names, assigns deterministic UUIDv5 identifiers, and outputs parquet files.

Requirements

• Python 3.14
• uv

Setup

uv sync

Usage

# Interactive (marimo notebook)
uv run marimo edit main.py

# Headless / batch
uv run python main.py

Output is written to dist/:

• countries.parquet
• subdivisions.parquet
• cities.parquet

Data sources

See SOURCES.md.

ID scheme

All entities use deterministic UUIDv5 IDs derived from udb:// URI schemes with uuid.NAMESPACE_URL:

Entity	URI pattern
Country	udb://country/{alpha2}
Subdivision	udb://country/{alpha2}/subdivision/{code}
City	udb://country/{alpha2}/subdivision/{state_code}/locality/{slug}

Dependencies

• marimo — reactive notebook runtime
• pandas — CSV extraction and transformation
• polars / pyarrow — parquet I/O
• python-slugify — city name slugification
• duckdb, sqlglot — available for advanced queries