← all projects · project dashboard

GridPulse — EIA Grid Data Product // Portfolio build

built 2026-10-16 10:54 UTC
NEXT
🚢 MVP SHIPPED — LIVE at evanderpool.github.io/gridpulse (22 days ahead of the 09-07 deadline). Phases 0–3 all complete: 2×/day Actions cron runs ingest→transform→derive→report, commits data to the orphan branch, republishes Pages; v1.0-baseline tagged (improvement-loop baseline frozen). 72 tests, both workflows green. One live CI failure caught+fixed same-run (offline stages demanded the key). 13-MONTH BACKFILL LIVE 2026-08-16: 9,686 hours on the public page, 359k rows, 0 quarantined, bronze gzipped (132→48MB), REAL YoY (ERCOT renewables +10.1 pts). P3 review agent died on session limit — riskiest items hand-verified, 1 finding fixed+pinned (auto_window clamp). Next: stretch slate in order (dashboard → quality panel → pandas backend → weather → AI notes → suggestions) + add GridPulse to resume v2.0
Status: In Progress · 2 items overdue
92%
Progress
11/12 tasks done
4/5
Milestones
next 2026-09-30
2026-09-07
Deadline
39d overdue
0
Blocked
tasks needing action
2
Overdue
tasks + milestones
2026-08-16
Last updated
stale 61d

Overview

DescriptionPortfolio data product — hourly US electricity demand and fuel mix by balancing authority from the EIA open-data API v2, run through a bronze/silver/gold pipeline (httpx, Pydantic v2, Polars, SQLite + Parquet, DuckDB) whose centerpiece is a deterministic conversion tool with quarantine, quality flags, and a per-run metrics ledger. Published as an auto-regenerated GitHub Pages HTML report (primary) + Streamlit dashboard (stretch). Answers exactly 3 business questions (renewables share YoY, regional peak/demand curves, duck-curve depth/ramp). Deliberately NO Airflow/Docker/Postgres — right-sizing stated as a design decision. Planning completed externally 2026-08-15 (two-agent brainstorm + red-team stress test); authoritative plan and red-team report live as claude.ai artifacts — links in `projects/gridpulse/README.md`.
PriorityHigh
ClientInternal (Artificial Management)
Start2026-08-15 (planning locked; no code yet)
Deadline2026-09-07 39d overdue
Next milestoneStretch (strict order): dashboard → quality panel → pandas backend → weather join → AI notes → suggestion engine (2026-09-30)
Updated2026-08-16 stale 61d
TeamErick AI Engineering Build Agent Data Integrity Agent Documentation Agent Portfolio Agent
LinksHandoff folder · plan + red-team artifacts (see project README)
README: projects/gridpulse/README.md

Tasks by department 11/12 done · 92%

Erick
Request free EIA API key (human act — Erick's email)2026-08-16Complete
AI Engineering Build Agent
Exploratory pulls validating all 3 questions (Q3 duck-curve solar is the risky one)2026-08-18Complete
Repo creation: MIT license, LF .gitattributes, orphan data branch, CI skeleton (ruff + pytest only pre-ship)2026-08-18Complete
Phase 1 vertical slice2026-08-22Complete
Phase 2 conversion suite + metrics ledger (git SHA per run)2026-08-29Complete
Phase 3 report + cron + Pages + baseline tag2026-09-07Complete
13-month backfill (one-time, enables YoY on the report)2026-08-20Complete
Data Integrity Agent
Capture real EIA payloads (incl. malformed) as case-named test fixtures2026-08-18Complete
Phase 1 sequential reviews (logic 9 findings, code 12) — all fixed + pinned2026-08-22Complete
Phase 2 sequential reviews (logic 5 findings incl. 1 HIGH, code 3 MED + LOWs) — all fixed + pinned2026-08-29Complete
Documentation Agent
README skeleton + architecture diagram — written first, it is the spec2026-08-18Complete
Portfolio Agent
Case study + LinkedIn post2026-09-14late

Milestones 4/5 reached

Phase 0 — 8 gates: EIA key, data validation, repo hygiene, branch strategy, CI skeleton, fixtures, README skeleton2026-08-18Complete
Phase 1 — vertical slice: ingest → bronze → validate → convert → SQLite, idempotency proven by test2026-08-22Complete
Phase 2 — conversion depth: backends interface, quality flags, quarantine, structured logs, metrics ledger2026-08-29Complete
Phase 3 — gold aggregates, HTML report, Actions cron 2×/day, Pages live, v1.0-baseline tag — MVP SHIP2026-09-07Complete
Stretch (strict order): dashboard → quality panel → pandas backend → weather join → AI notes → suggestion engine2026-09-30late

Risks & blockers

  • EIA API key not yet requested — gates everything (Erick action; free registration)
  • Q3 duck-curve solar data availability unvalidated — red-team says validate all 3 questions BEFORE creating the repo
  • Data must live on an orphan `data` branch, never main (git bloat)
  • GitHub Actions cron auto-disables after ~60 days of repo inactivity — same liveness trap the AM heartbeat exists for
  • Laptop CPU-fan fault makes push-every-session mandatory

Recent changes source: logs/changes.md

2026-08-16 · github.com/evanderpool/gridpulse (app/, pandas…
2026-08-16 · github.com/evanderpool/gridpulse (main + data b…
2026-08-16 · github.com/evanderpool/gridpulse + evanderpool.…
2026-08-16 · github.com/evanderpool/gridpulse + project-trac…
2026-08-16 · github.com/evanderpool/gridpulse
2026-08-16 · github.com/evanderpool/gridpulse