Skip to content

Evidence Dashboard Migration — Summary of Findings

Date: 2026-02-24 Status: PoC Complete, Production Planning Documents: This summarizes the findings from 6 documents produced during the Evidence Dashboard Migration (EDM) spike.


Background

Skyledge delivers fleet analytics dashboards via Evidence, an open-source BI framework. The current pipeline works as follows: Dagster produces per-organization DuckDB files from daily telemetry data, then a separate Dagster asset clones the Evidence project, builds static HTML/JS/CSS per control room, and uploads to S3. These are served via CloudFront and embedded in the Angular app as iframes with signed cookies.

This approach has reached its limits due to iframe isolation, per-control-room build scaling, no reuse of platform components (Evidence uses Svelte), limited interactivity, and significant operational complexity in the Dagster build pipeline.


Decision

Replace Evidence static-site dashboards with in-platform Stencil web components backed by a new Python/FastAPI query service that queries the existing per-org DuckDB files. The full ADR is in adr-replace-evidence-with-in-platform-dashboards.md.

What’s Reused (No Changes)

  • WidgetLibraryLoaderService — existing runtime ESM loader in the Angular app
  • WidgetComponent — existing generic web component host
  • WidgetSkyledgeInterface — existing contract for passing API access, auth, and entity context to widgets
  • CDN pipeline — webcomponents.prod.skyledge.com already hosts Stencil bundles
  • Per-org DuckDB files — Dagster’s organization_duck_db asset continues unchanged
  • Existing chart components — slc-bar-chart, slc-line-chart, slc-gauge-chart, etc.

What’s New

  • analytics Stencil library — new library in skyledge-official-web-components with dashboard components (slc-fleet-overview, slc-asset-utilisation, etc.)
  • Dashboard Query Service — Python/FastAPI service querying DuckDB, serving JSON
  • SQL queries migrated from Evidence markdown into the query service as .sql files

Architecture

Angular Host App
└─ WidgetLibraryLoaderService loads analytics.esm.js from CDN
└─ WidgetComponent creates slc-fleet-overview / slc-asset-utilisation
└─ HTTP (JWT) → Dashboard Query Service (FastAPI)
└─ SQL → DuckDB (per-org file cached from S3)

The query service is consumer-agnostic — it also serves a weekly email digest job via API key auth.


Dashboard Query Service Design

Full architecture in architecture-dashboard-query-service.md.

Key Decisions

Decision Choice Rationale
Language Python 3.11+ Best DuckDB driver support, matches Dagster pipeline
Framework FastAPI OpenAPI docs, Pydantic integration
Deployment ECS Fargate Long-lived containers cache DuckDB files locally (Lambda rejected — cold start would re-download tens-to-hundreds MB files)
Semantic layer None (direct SQL) Cube.js evaluated and rejected — added two processes, complex queries needed raw SQL anyway, and DuckDB is fast enough on 2-3M rows
Query organization .sql files in folders per dashboard Convention over tooling; revisit dbt when views depend on other views or dashboard count exceeds ~10

Query Architecture (3 Layers)

  1. Raw tables — asset_data, assets, control_room_asset_map (owned by Dagster)
  2. Views — temporary SQL views created per-request, filtering to a single control room with derived columns
  3. Dashboard queries — SQL files querying the views for specific dashboard data needs

Auth

Two methods supported:

  • JWT (interactive users via Stencil components) — validated against Auth0 JWKS
  • API key (X-Authorization header, machine consumers) — uses existing platform API keys

Both resolve to the same internal identity: organization_id + permissions.

API Conventions

Follows existing Skyledge platform API conventions (camelCase, kebab-case paths). Endpoints use POST with JSON body under /controlrooms/{controlRoomId}/dashboards/{dashboardName}/v{version}. Per-dashboard versioning allows independent breaking changes.


PoC Results

Two dashboard components were built as proof-of-concept, proving the full vertical slice.

slc-fleet-overview (Overview Tab)

  • Endpoint: POST /controlrooms/{controlRoomId}/dashboards/fleet-overview/v1
  • Features: 4 hero KPI cards (utilisation, fuel consumption, distance, moving hours), date range filter, vehicle type dropdown, fuel unit toggle, sortable data table with slc-asset-name widget column
  • SQL complexity: ~8 queries (simplest of the three Evidence tabs)
  • Detailed plan in poc-overview-dashboard-migration.md

slc-asset-utilisation (Asset Utilisation Tab)

  • Endpoint: POST /controlrooms/{controlRoomId}/dashboards/asset-utilisation/v1
  • Features: 3 hero KPIs with week-over-week deltas, week picker, business hours / 24/7 toggle, best/worst performer panels, ECharts line chart with percentile bands, grouped performance table
  • SQL complexity: ~25 queries; introduces 2 additional DuckDB views (skyledge_dailydata_wend, skyledge_dailydata_bus_wend)
  • Detailed plan in poc-asset-utilisation-dashboard-migration.md

API Reference

Full request/response schemas for both endpoints are documented in fleet-overview-api.md, including integration notes for web component consumers (asset detail links, timezone handling, fuel unit toggle behaviour, etc.).


Key Patterns & Lessons Learned

Captured in poc-implementation-guide.md. Critical findings:

Stencil Component Patterns

  • skyledge prop timing — the prop is not available when componentWillLoad fires. Must guard all access and use @Watch('skyledge') to initialize when it arrives.
  • 4-file component structure — models.ts, service.ts, utils.ts, component.tsx per dashboard.
  • shadow: false required — for platform styles and widgets to work.

slc-table Integration

  • Use createClientSideDataSource (push all data via RxJS Subject). Do not use createServerSideDataSource with an empty callback — it causes an infinite re-render loop that crashes the browser tab.
  • Widget columns (e.g., slc-asset-name) require rows to have id and name properties.

ECharts Integration

  • Accessed via this.skyledge.charts.echarts — not installed as a dependency.
  • Needs retry pattern for init (container and echarts may not be ready immediately).
  • Theme-aware colors via getChartColors(this.el) and setupThemeChangeListener.
  • Container must have explicit pixel dimensions.

Store Access

  • Asset types for filter dropdowns come from this.skyledge.store.assetTypes$ (RxJS observable).

Gotchas

  • Empty required API params (e.g., empty year_week) crash the FastAPI backend — compute sensible defaults client-side.
  • ag-grid row grouping requires Enterprise edition — use custom accordion or slc-table grouping instead.

Migration Roadmap

Phase 1: Proof of Concept ✅

Build slc-fleet-overview and slc-asset-utilisation replicating Evidence tabs, stand up query service.

Phase 2: Parallel Running

Both Evidence (iframe) and new dashboards available. Validate data parity. Gather user feedback.

Phase 3: Full Migration

Migrate remaining Evidence views (Fuel Consumption). Route all control rooms to in-platform dashboards.

Phase 4: Retirement

  • Remove ff-evidence-dashboard feature flag and iframe code path
  • Retire control_room_dashboard Dagster asset and Evidence project repo
  • Retire reports.skyledge.com CloudFront distribution and S3 bucket
  • Remove GetReportUrlForControlRoomController and CloudFront signer from Java backend

Production Considerations (Post-PoC)

  1. Query service URL — currently hardcoded to localhost:8090; needs environment config or WidgetSkyledgeInterface integration
  2. Error handling — integrate with platform toast/notification system
  3. Loading states — skeleton loaders instead of spinners
  4. Responsive layout — handle mobile/tablet breakpoints
  5. Timezone handling — verify timezone_offset_hours requirement with backend team
  6. Table selection → chart overlay — wire slc-table checkbox selection to update selectedVehicles on chart
  7. Testing — no unit tests in PoC; add spec files following slc-driver-performance.spec.tsx pattern
  8. Observability — OpenTelemetry traces/metrics/logs; alerting on error rates, query latency, S3 failures

Revisit Triggers

Reconsider the architecture if:

  • Dashboard count exceeds ~30-50 (may need a composition framework)
  • Non-engineers require dashboard authoring (may need a low-code layer)
  • Query complexity outgrows single DuckDB files (may need a dedicated analytics database)
  • Cross-dashboard metric governance becomes important (may need a semantic layer)

Document Index

Document Description
adr-replace-evidence-with-in-platform-dashboards.md Architecture Decision Record — problem, decision, trade-offs, migration strategy
architecture-dashboard-query-service.md Full architecture for the Python/FastAPI query service — tech choices, query layers, auth, deployment, observability
fleet-overview-api.md API reference for both dashboard endpoints with request/response schemas
poc-overview-dashboard-migration.md Detailed PoC plan for the fleet overview tab (query service + Stencil component)
poc-asset-utilisation-dashboard-migration.md Detailed PoC plan for the asset utilisation tab (ECharts, week filtering, grouped tables)
poc-implementation-guide.md Lessons learned, patterns, and gotchas from PoC implementation