Data Pipeline for Aviation Mapping

Problem

An aviation mapping partner depended on data updates that were downloaded and processed by hand. Operators had to be notified whenever new data was available, then pull it and run local transformations — a step that was easy to miss and slow to repeat. The team needed updates to arrive reliably, without manual downloads, missed notifications, or fragile local processing, and without standing up servers to babysit.

System design

We rebuilt the workflow as a fully serverless pipeline on AWS, structured as six stages:

1. Extract the source data from OneDrive / SharePoint into AWS S3.
2. Convert the proprietary format into usable relational data.
3. Transform and clean the data.
4. Run feature engineering.
5. Build a geospatial index with Uber H3.
6. Orchestrate the whole run with Lambda and Step Functions.

Each stage runs as its own AWS Lambda function — nine modular Python functions in total — so that every function performs a single major task. Between stages, intermediate data is written to dedicated S3 directories rather than passed directly between functions, which sidesteps Lambda payload size and data-type limits when moving data through the pipeline.

Ordering matters: four functions run sequentially, four run in parallel, and a final function builds the geospatial index. A Step Functions state machine coordinates that order and tracks run state so each step fires at the right time, while EventBridge provides the custom triggers that decide when operations start. The result is traceable and scalable — faults can be located at a specific stage, and each run produces thorough logs.

The heaviest computation is the geospatial join. Using the Uber H3 library, we compute distances and relationships across millions of locations with hex-bin aggregations instead of pairwise calculation, which reduced that step’s processing time roughly fifty-fold.

Engineering tradeoffs

• Lambda layers over containers. Because the pipeline needed only a few custom packages (such as H3), we packaged dependencies as Lambda layers rather than container images, keeping the build and deploy path simpler.
• S3 as the hand-off medium. Staging each stage’s output in S3 traded a little extra I/O for freedom from inter-Lambda payload limits and tighter fault isolation.
• Least privilege by default. Lambda functions and the state machine were granted only the exact IAM roles and permissions they needed.
• Single-responsibility functions. Confining each Lambda to one task made the system more fault tolerant and easier to reason about, and unit tests guard data quality at each step.

The hardest part was not the compute but the ingest: extracting data from OneDrive and SharePoint into AWS took longer than expected because of firewall constraints and security review. Moving code that worked locally into Lambda was also not a clean lift-and-shift.

Results

• Geospatial processing time dropped roughly fifty-fold after rebuilding the index with H3.
• About eight hours of manual update work per week were eliminated, along with the notify-and-download step operators used to miss.
• Pipeline runs are traceable end to end, with clear fault isolation and logs.

On cost, the serverless model was inexpensive to run and low maintenance — Lambda is billed per invocation (on the order of $0.20 per million invocations at the time), and full automation removed the recurring manual labor entirely.

Lessons and considerations

• Weigh open-source Lambda templates and pre-built layers carefully; out-of-date code or libraries can be costly to untangle.
• Prefer native functionality over custom libraries where it does the job.
• Treat the local-to-Lambda transition as real work, not a guaranteed lift and shift.
• Price holistically: account for the true cost of human error and the long-term maintenance implications, not just per-invocation cost.