GIS-Scripts

A single home for all the GIS / geospatial data scripts used to build the Watrways map layers. These scripts create, convert, import, and inspect the data that powers the app — NHD hydrography, bathymetry (depth) tiles, hillshade, elevation / terrain DEMs, contours, and the live-camera dataset.

Note: This folder contains the scripts only. None of the large output artifacts (.pmtiles, .mbtiles, .pbf, .har, generated .json data dumps, decoded tile directories) are included — only the code used to produce them.

---

Tooling / prerequisites

Different scripts need different tools. Install what you need for the pipeline you're running:

Tool	Used by	Install
Node.js 16+	all .js / .mjs scripts	https://nodejs.org
Python 3.8+	all .py scripts	https://python.org
GDAL (ogr2ogr, gdalwarp, gdalbuildvrt, gdal_contour, gdal2tiles.py, gdal_translate)	NHD / terrain / contour builds	OSGeo4W / conda / brew
tippecanoe	vector tile (PMTiles) generation	brew / WSL build
pmtiles (pmtiles.exe)	.mbtiles → .pmtiles conversion	https://github.com/protomaps/go-pmtiles
rio (rio-rgbify)	terrain-RGB raster tiles	pip install rio-rgbify
psql	bulk-loading NHD into Supabase	PostgreSQL CLI tools
Playwright	HAR route recording	npm i playwright && npx playwright install chromium
Python libs	DEM / hillshade	pip install numpy scipy Pillow rasterio shapely geopandas

Common env vars (set in the repo's .env.local): SUPABASE_SERVICE_ROLE_KEY, R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET.

---

Folder map

GIS-Scripts/
├── nhdpipeline/          NHD GeoPackage → JSON (cross-platform, no GDAL)
├── elevationpipeline/    USGS elevation TIFFs → terrain/DEM PMTiles
└── scripts/              Everything else (NHD tiles+import, terrain, contours, cameras, R2)
    ├── nhdtiles/         Batch NHD GPKG → PMTiles
    └── TileScr/          Bathymetry/depth tile scraping → hillshade/depth PMTiles
        └── foldertotiles/  Snapshot of the depth-tile build scripts

---

1. nhdpipeline/ — NHD attributes → JSON

Cross-platform (Mac/Win/Linux, no GDAL) extraction of the National Hydrography Dataset. Reads NHD_H_*_State_GPKG.gpkg files and emits per-state JSON for waterbodies, flowlines, and pre-computed inflow/outflow/upstream/downstream connections. Geometry is intentionally skipped (rendering is handled by PMTiles).

cd nhdpipeline && npm install
node index.js          # auto-detects all GPKGs in the working dir

• index.js — the pipeline. Auto-detects state GPKGs, walks the network graph, writes {state}_flowline.json, {state}_waterbody.json, {state}_waterbody_connections.json, {state}_flowline_connections.json.
• See nhdpipeline/README.md for output format and R2 upload structure.

2. elevationpipeline/ — elevation TIFFs → terrain/DEM PMTiles

Converts USGS 1-meter 3DEP elevation TIFFs into PMTiles for 3D terrain. Two approaches (raster-direct, recommended; and legacy point-extraction).

• terrain-dem-pipeline.py — recommended. Raster-space pipeline: samples elevation only within a shoreline buffer and builds a terrain DEM MBTiles (zoom 13–14). Fastest, smallest output.
• terrain-dem-pipeline-single-dem.py — single-DEM variant of the above.
• elevationpipe.py — legacy point-extraction: TIFFs → elevation_points.json (supports --workers N for parallelism).
• merge_elevation_points.py — merges the per-split *-elevation_points.json files produced by parallel elevationpipe.py runs.
• build_dem.py — interpolates elevation_points.json into a DEM MBTiles.
• Docs: README.md, QUICKSTART.md, COMMANDS.md, TERRAIN-DEM-PIPELINE.md, SINGLE_DEM_PIPELINE.md, and the various *_SUMMARY.md / approach notes.

# Recommended path
python terrain-dem-pipeline.py PA.txt NHD_H_Pennsylvania_State_GPKG.gpkg --state pa
pmtiles convert pa-terrain-dem.mbtiles pa-terrain-dem.pmtiles

3. scripts/ — main grab-bag

NHD → PMTiles

• build-nhd-pmtiles.ps1 — single-state NHD GPKG → nhd_{state}.pmtiles (flowlines + waterbodies via ogr2ogr → FlatGeobuf → tippecanoe).
• nhdtiles/build-nhd-batch.py — batch driver: auto-detects all NHD_H_*_State_GPKG folders in ../nhdpipeline/, splits them into batches of 6 (--1, --2, …) and runs the PMTiles build per state.
• nhdtiles/build-nhd-pmtiles.ps1 — the per-state build invoked by the batch.
• nhdtiles/nhd.ps1 — helper PowerShell for the NHD tile workflow.

NHD → Supabase (Postgres)

• import-nhd-pa.ps1 — exports NHD attributes to CSV (ogr2ogr), then bulk loads into Supabase via psql COPY. Geometry excluded.
• import-nhd-pa.mjs — same import but over the Supabase REST API (no direct psql connection needed).
• nhd-supabase.ps1 — interactive prompt-driven version of the NHD → Supabase import (asks for GPKG path + DB credentials).
• drop-nhd-unused-columns.ps1 — backs up then drops unused columns/indexes from the nhd_flowline table to slim the database.

Terrain & contours (raster)

• build-3dep-terrain-rgb.ps1 — USGS 3DEP TIFFs → terrain-RGB raster tiles (gdalbuildvrt/gdalwarp/gdal2tiles.py/rio), zoom 7–14.
• build-pa-water-corridor-terrain.ps1 — terrain-RGB tiles clipped to a buffer around PA flowlines/waterbodies (smaller, water-corridor-only output).
• build-pa-water-corridor-contours.ps1 — contour lines (gdal_contour) clipped to the same water corridor, output as tiles.

Live cameras

• webcam.py — discovers live YouTube river/lake webcams (yt_dlp search + geopy geocoding) and writes a CSV dataset.
• import-cameras.mjs — imports that webcams CSV into Supabase with a spiderweb layout (requires SUPABASE_SERVICE_ROLE_KEY).

Upload / utility

• upload-r2-directory.mjs — recursively uploads a local directory to Cloudflare R2 (S3 API). node upload-r2-directory.mjs <localDir> [remotePrefix].
• route-har-recorder.mjs + route-har-routes.json — Playwright-driven recorder that drives the app over a set of routes and captures network HAR files (used to harvest tile URLs). The JSON defines the routes to record.
• batch_decode_tiles.js — multi-threaded decoder for XOR-obfuscated tiles (worker_threads, tunable --workers / --concurrency).

4. scripts/TileScr/ — bathymetry / depth-tile pipeline

The pipeline that scrapes nautical-chart depth tiles and turns them into depth + hillshade PMTiles. Rough flow: record HAR → extract tile URLs → download tiles → decode → extract depth points → build hillshade/DEM → pack to MBTiles → convert to PMTiles.

Harvest & download

• extract-har.py — pulls matching tile URLs out of a .har into a URL list (and a cleaned HAR).
• combine_har.js — combines multiple HAR files and bins their URLs into per-state input files.
• process-route-hars.ps1 — polling driver that watches a HAR drop folder and processes new route HARs through the pipeline.
• download_tiles.mjs / download_tiles.ps1 — download PBF tiles from a URL list into a z/x/y directory (the .mjs adds proxy/concurrency support).
• scrap.ps1 — small wrapper to kick off a named scrape run.
• clean.py — removes 0-byte files / empty dirs from a downloaded tile tree.

Decode & extract

• batch_decode_tiles.js — threaded XOR-decode of obfuscated PBF tiles.
• decode_test.js — single-tile decode sanity check.
• extract_depth_points.js — parses decoded SOUNDG vector tiles into depth_points.json (lng/lat/depth).
• pbf_to_json.js — dumps a decoded PBF tile to readable JSON.

ENC (S-57 nautical chart) path

• enc_to_depth_points.js — converts an S-57 SOUNDG GeoJSON (via ogr2ogr) into the depth_points.json format.
• enc_to_pmtiles.ps1 — extracts DEPCNT + DEPARE from S-57 ENCs and builds PMTiles (parallel ogr2ogr → tippecanoe).
• enc_pipeline.mjs / enc_pipeline.js / enc_pipeline.ps1 — convert decoded per-state tile dirs into {STATE}-depth.pmtiles and {STATE}-hillshade.pmtiles (.mjs is the maintained version; --group=N / --state=FL / --state=FL,PA).

Build rasters & pack tiles

• build_hillshade.py — interpolates depth_points.json into a per-tile bathymetric DEM, computes hillshade, writes PNG raster tiles into MBTiles.
• folder_to_mbtiles.js — packs a z/x/y.pbf directory into a vector MBTiles.
• mbtile-combo.mjs — merges multiple MBTiles and converts the result to PMTiles.
• split_zoom.mjs — splits a tile set by zoom level (--PA --zoom=14).
• serve_pmtiles.js — tiny Express server to preview a local .pmtiles.

Inspection helpers

• check_zoom.js, check_actual_zoom.js, check_pmtiles_zoom.js — report the zoom range of a PMTiles/tile set.
• inspect_pbf.js / inspect_pbf.mjs — inspect a decoded PBF tile.
• inspect_pmtiles.js, inspect_pmtiles_layers.js, inspect_layers.js, list_nhd_layers.js — list layers/metadata inside a PMTiles file.

foldertotiles/

A self-contained snapshot of the core depth-tile build scripts (build_dem.py, build_hillshade.py, enc_pipeline.js, extract_depth_points.js, folder_to_mbtiles.js, scripts/batch_decode_tiles.js) with its own package.json for running them in isolation.

---

Typical end-to-end flows

NHD layer for a state

nhdpipeline/index.js (JSON attrs)  +  scripts/build-nhd-pmtiles.ps1 (geometry PMTiles)
        → import-nhd-pa.(ps1|mjs) into Supabase
        → upload PMTiles to R2 (upload-r2-directory.mjs)

Bathymetry / depth + hillshade

route-har-recorder.mjs → TileScr/extract-har.py → download_tiles.mjs
   → batch_decode_tiles.js → extract_depth_points.js → build_hillshade.py
   → folder_to_mbtiles.js / mbtile-combo.mjs → pmtiles convert → R2

Terrain / elevation

elevationpipeline/terrain-dem-pipeline.py → pmtiles convert → R2
   (or scripts/build-3dep-terrain-rgb.ps1 for terrain-RGB raster tiles)