Skip to content

Functional Design for Trip-Based Fuel Consumption Service

functional_design.png


app.py

  • Main entry point to the Lambda service.
  • Accepts an SQS message, validates its content, and passes it to fuel_consumption_processor.py.

fuel_consumption_processor.py

  • Orchestrates the fuel consumption workflow by calling:
    • data_processor.py to calculate fuel consumption.
    • consumption_validator.py to validate the consumption result.
    • api_handler.py to update cycle attributes when the consumption result is valid.
    • slack_handler.py to send a message to a Slack channel if validation fails.

data_processor.py

  • Initiates fuel consumption calculation by loading and preparing data:

    • Offsets the trip start timestamp by going back 1 hour to help smooth the data.
    • Loads fuelPercent, speed, and distance metric history via api_handler.py.
    • Loads device_updates for the specified device and time period via db_handler.py.
    • Merges all datasets based on timestamps, allowing a tolerance of 1s.
  • Preprocesses the merged dataset:

    • Resamples speed, fuelPercent, canEngineRpm, externalPowersourceVoltage, engineIgnitionStatus, and distance to 1-second intervals and fills missing values.
    • Applies a median filter to fuelPercent using a dynamic kernel:
      • Kernel size is determined by trip duration:
        • 301 if duration < 30 minutes
        • 901 if 30 ≤ duration < 60 minutes
        • 1501 if duration ≥ 60 minutes
    • Trims the dataset back to the original trip time range, removing the 1-hour offset applied earlier.
  • Calls trip_segment_processor.py to derive trip segments using the preprocessed data.

  • Calculates the overall consumption_result, including:

    • consumption_idle
    • consumption_moving
    • refill_liters
    • start_fuel_level
    • end_fuel_level
    • consumption_liters_per_100km
    • consumption_km_per_liter
    • mileage

trip_segment_processor.py

  • Handles the core logic for segmenting the preprocessed dataset and calculating fuel consumption.
  • Divides the trip into stationary and moving segments; idle segments are then detected from stationary segments.

Idle Detection

  • Uses RPM idle thresholds when RPM values are available.
  • Otherwise, uses voltage thresholds.
  • Consecutive idle segments within a stationary period that are separated by less than 60 seconds are merged into a single idle segment.

Segment Consumption

  • For each moving and idle segment, fuel consumption is calculated by:
    • Identifying refill points.
    • Splitting segments into before-refill and after-refill subsegments.
    • Calculating consumption as the difference between start and end fuel levels.

consumption_validator.py

  • Once the consumption_result is calculated, it is validated here.
  • An error value for fuel consumption (in liters) is calculated.
    • fuel_consumption_error = |(start_fuel_level - end_fuel_level) + refill_liters - (consumption_idle + consumption_moving)|
    • This error was previously calculated across multiple trips to determine acceptable thresholds:
      • Threshold = 1.54 L (~2 L) for trip duration of 0–5 minutes
      • Threshold = 2.41 L (~3 L) for trip duration of 5–30 minutes
      • Threshold = 2.65 L (~3 L) for trip duration of 30–60 minutes
      • Threshold = 4.17 L (~5 L) for trip duration of 60+ minutes
  • A confidence score is calculated based on these thresholds and the error.
    • If fuel_consumption_error <= threshold:
      • confidence = 1.0 - 0.1 * (fuel_consumption_error / threshold)
      • This ensures the confidence remains high (between 0.9 and 1.0) as long as the error is within the threshold.
    • If fuel_consumption_error > threshold:
      • confidence = max(0.0, 0.9 * (1.0 - ((fuel_consumption_error - threshold) / threshold)))
      • This ensures the confidence decreases progressively to 0 as the error increases beyond the threshold.
  • Returns a consumption_validation_result containing:
    • valid — true if confidence >= 0.75, otherwise false
    • confidence

api_handler.py

  • Loads metric history by calling the Sky Ledge API.
  • Loads the tank capacity of the asset by calling the GET asset endpoint.
  • Updates cycle attributes by calling the PATCH endpoint.

db_handler.py

  • Handles loading device updates from the fleet database for a given device over a specified time period.
  • Calls AWS Secrets Manager to retrieve the database cluster URI.

s3_handler.py

  • Loads organization data from the skyledge-customer-data-backup S3 bucket for a given customer.
  • Retrieves the API URL and API key from the data and updates a global cache.
    • This helps avoid repetitive S3 requests when multiple Lambda invocations occur simultaneously without the instance shutting down.
  • Retrieves the battery capacity of an asset by loading data from the dagster/asset_battery_capacity/.. S3 location.
  • Loads the customer-specific instance from the prod/instance S3 location.
    • This is used to determine which database collection to access (flespi or flespi-mercury).
    • The result is cached globally to avoid repeated S3 calls, as described above.

slack_handler.py

  • Sends Slack messages to a given channel.

log_utils.py

  • Instantiates a global logger with structured logging capabilities that can be used across the application.

constants.py

  • Contains all DataFrame-related constants.