Evidence Dashboard Migration — Summary of Findings
Date: 2026-02-24 Status: PoC Complete, Production Planning Documents: This summarizes the findings from 6 documents produced during the Evidence Dashboard Migration (EDM) spike.
Background
Skyledge delivers fleet analytics dashboards via Evidence, an open-source BI framework. The current pipeline works as follows: Dagster produces per-organization DuckDB files from daily telemetry data, then a separate Dagster asset clones the Evidence project, builds static HTML/JS/CSS per control room, and uploads to S3. These are served via CloudFront and embedded in the Angular app as iframes with signed cookies.
This approach has reached its limits due to iframe isolation, per-control-room build scaling, no reuse of platform components (Evidence uses Svelte), limited interactivity, and significant operational complexity in the Dagster build pipeline.
Decision
Replace Evidence static-site dashboards with in-platform Stencil web components backed by a new Python/FastAPI query service that queries the existing per-org DuckDB files. The full ADR is in adr-replace-evidence-with-in-platform-dashboards.md.
What’s Reused (No Changes)
- WidgetLibraryLoaderService — existing runtime ESM loader in the Angular app
- WidgetComponent — existing generic web component host
- WidgetSkyledgeInterface — existing contract for passing API access, auth, and entity context to widgets
- CDN pipeline —
webcomponents.prod.skyledge.comalready hosts Stencil bundles - Per-org DuckDB files — Dagster’s
organization_duck_dbasset continues unchanged - Existing chart components —
slc-bar-chart,slc-line-chart,slc-gauge-chart, etc.
What’s New
analyticsStencil library — new library inskyledge-official-web-componentswith dashboard components (slc-fleet-overview,slc-asset-utilisation, etc.)- Dashboard Query Service — Python/FastAPI service querying DuckDB, serving JSON
- SQL queries migrated from Evidence markdown into the query service as
.sqlfiles
Architecture
Angular Host App └─ WidgetLibraryLoaderService loads analytics.esm.js from CDN └─ WidgetComponent creates slc-fleet-overview / slc-asset-utilisation └─ HTTP (JWT) → Dashboard Query Service (FastAPI) └─ SQL → DuckDB (per-org file cached from S3)The query service is consumer-agnostic — it also serves a weekly email digest job via API key auth.
Dashboard Query Service Design
Full architecture in architecture-dashboard-query-service.md.
Key Decisions
| Decision | Choice | Rationale |
|---|---|---|
| Language | Python 3.11+ | Best DuckDB driver support, matches Dagster pipeline |
| Framework | FastAPI | OpenAPI docs, Pydantic integration |
| Deployment | ECS Fargate | Long-lived containers cache DuckDB files locally (Lambda rejected — cold start would re-download tens-to-hundreds MB files) |
| Semantic layer | None (direct SQL) | Cube.js evaluated and rejected — added two processes, complex queries needed raw SQL anyway, and DuckDB is fast enough on 2-3M rows |
| Query organization | .sql files in folders per dashboard |
Convention over tooling; revisit dbt when views depend on other views or dashboard count exceeds ~10 |
Query Architecture (3 Layers)
- Raw tables —
asset_data,assets,control_room_asset_map(owned by Dagster) - Views — temporary SQL views created per-request, filtering to a single control room with derived columns
- Dashboard queries — SQL files querying the views for specific dashboard data needs
Auth
Two methods supported:
- JWT (interactive users via Stencil components) — validated against Auth0 JWKS
- API key (
X-Authorizationheader, machine consumers) — uses existing platform API keys
Both resolve to the same internal identity: organization_id + permissions.
API Conventions
Follows existing Skyledge platform API conventions (camelCase, kebab-case paths). Endpoints use POST with JSON body under /controlrooms/{controlRoomId}/dashboards/{dashboardName}/v{version}. Per-dashboard versioning allows independent breaking changes.
PoC Results
Two dashboard components were built as proof-of-concept, proving the full vertical slice.
slc-fleet-overview (Overview Tab)
- Endpoint:
POST /controlrooms/{controlRoomId}/dashboards/fleet-overview/v1 - Features: 4 hero KPI cards (utilisation, fuel consumption, distance, moving hours), date range filter, vehicle type dropdown, fuel unit toggle, sortable data table with
slc-asset-namewidget column - SQL complexity: ~8 queries (simplest of the three Evidence tabs)
- Detailed plan in
poc-overview-dashboard-migration.md
slc-asset-utilisation (Asset Utilisation Tab)
- Endpoint:
POST /controlrooms/{controlRoomId}/dashboards/asset-utilisation/v1 - Features: 3 hero KPIs with week-over-week deltas, week picker, business hours / 24/7 toggle, best/worst performer panels, ECharts line chart with percentile bands, grouped performance table
- SQL complexity: ~25 queries; introduces 2 additional DuckDB views (
skyledge_dailydata_wend,skyledge_dailydata_bus_wend) - Detailed plan in
poc-asset-utilisation-dashboard-migration.md
API Reference
Full request/response schemas for both endpoints are documented in fleet-overview-api.md, including integration notes for web component consumers (asset detail links, timezone handling, fuel unit toggle behaviour, etc.).
Key Patterns & Lessons Learned
Captured in poc-implementation-guide.md. Critical findings:
Stencil Component Patterns
skyledgeprop timing — the prop is not available whencomponentWillLoadfires. Must guard all access and use@Watch('skyledge')to initialize when it arrives.- 4-file component structure —
models.ts,service.ts,utils.ts,component.tsxper dashboard. shadow: falserequired — for platform styles and widgets to work.
slc-table Integration
- Use
createClientSideDataSource(push all data via RxJS Subject). Do not usecreateServerSideDataSourcewith an empty callback — it causes an infinite re-render loop that crashes the browser tab. - Widget columns (e.g.,
slc-asset-name) require rows to haveidandnameproperties.
ECharts Integration
- Accessed via
this.skyledge.charts.echarts— not installed as a dependency. - Needs retry pattern for init (container and echarts may not be ready immediately).
- Theme-aware colors via
getChartColors(this.el)andsetupThemeChangeListener. - Container must have explicit pixel dimensions.
Store Access
- Asset types for filter dropdowns come from
this.skyledge.store.assetTypes$(RxJS observable).
Gotchas
- Empty required API params (e.g., empty
year_week) crash the FastAPI backend — compute sensible defaults client-side. - ag-grid row grouping requires Enterprise edition — use custom accordion or
slc-tablegrouping instead.
Migration Roadmap
Phase 1: Proof of Concept ✅
Build slc-fleet-overview and slc-asset-utilisation replicating Evidence tabs, stand up query service.
Phase 2: Parallel Running
Both Evidence (iframe) and new dashboards available. Validate data parity. Gather user feedback.
Phase 3: Full Migration
Migrate remaining Evidence views (Fuel Consumption). Route all control rooms to in-platform dashboards.
Phase 4: Retirement
- Remove
ff-evidence-dashboardfeature flag and iframe code path - Retire
control_room_dashboardDagster asset and Evidence project repo - Retire
reports.skyledge.comCloudFront distribution and S3 bucket - Remove
GetReportUrlForControlRoomControllerand CloudFront signer from Java backend
Production Considerations (Post-PoC)
- Query service URL — currently hardcoded to
localhost:8090; needs environment config orWidgetSkyledgeInterfaceintegration - Error handling — integrate with platform toast/notification system
- Loading states — skeleton loaders instead of spinners
- Responsive layout — handle mobile/tablet breakpoints
- Timezone handling — verify
timezone_offset_hoursrequirement with backend team - Table selection → chart overlay — wire slc-table checkbox selection to update
selectedVehicleson chart - Testing — no unit tests in PoC; add spec files following
slc-driver-performance.spec.tsxpattern - Observability — OpenTelemetry traces/metrics/logs; alerting on error rates, query latency, S3 failures
Revisit Triggers
Reconsider the architecture if:
- Dashboard count exceeds ~30-50 (may need a composition framework)
- Non-engineers require dashboard authoring (may need a low-code layer)
- Query complexity outgrows single DuckDB files (may need a dedicated analytics database)
- Cross-dashboard metric governance becomes important (may need a semantic layer)
Document Index
| Document | Description |
|---|---|
adr-replace-evidence-with-in-platform-dashboards.md |
Architecture Decision Record — problem, decision, trade-offs, migration strategy |
architecture-dashboard-query-service.md |
Full architecture for the Python/FastAPI query service — tech choices, query layers, auth, deployment, observability |
fleet-overview-api.md |
API reference for both dashboard endpoints with request/response schemas |
poc-overview-dashboard-migration.md |
Detailed PoC plan for the fleet overview tab (query service + Stencil component) |
poc-asset-utilisation-dashboard-migration.md |
Detailed PoC plan for the asset utilisation tab (ECharts, week filtering, grouped tables) |
poc-implementation-guide.md |
Lessons learned, patterns, and gotchas from PoC implementation |