Project
Data Pipeline into QuickBase
A serverless AWS ETL pipeline that syncs cleaned operational data into QuickBase, cutting monthly platform costs from nearly $2,000 to under $100.
Problem
A program that builds maps for the aviation industry needed cleaned, current data available inside QuickBase, where operations teams actually work. Getting it there meant extracting data from source databases, running extensive cleaning, feature development, and transformation, and then loading the result into QuickBase — recurring work that otherwise fell to people preparing data by hand. The team needed that preparation to run on schedule, stay trustworthy stage to stage, and cost little enough to leave running, without standing up servers to maintain.
System design
We built the workflow as a fully serverless ETL pipeline on AWS, running in a secure, isolated account with customized identity and access management, permissions, policies, and safeguards. The pipeline is organized into three stages, and every stage carries its own integrity checks and stop-gap measures so a problem is caught where it happens rather than propagating downstream.
The first stage begins with an EventBridge Scheduler call that triggers a Lambda, which downloads data from the source databases and writes it into a versioned S3 bucket. We then query the new dataset to decide whether it has actually changed. If it has, a sequence of Python Lambda scripts runs the cleaning and transformation modules, coordinated by a Step Functions state machine that tracks run state and fires each step in order. If nothing has changed, the pipeline logs that fact and skips the rest of the run.
The final stage formats the dataset specifically for transfer through the QuickBase API. It breaks the data into its fundamental components and updates QuickBase records, tracking appended, updated, and deleted rows so the destination stays a faithful reflection of the source. Throughout, AWS CloudWatch tracks every run, with logging customized for two different audiences — technical engineers who need to locate faults and operational stakeholders who need to know the data is current.
Engineering tradeoffs
- EventBridge and Lambda over a VM or managed Airflow. Cost efficiency is a core pillar of AWS’s Well-Architected Framework, and it drove the central architecture decision. During design we tested three options: a custom VM, AWS Managed Airflow (MWAA), and an EventBridge/Lambda stack. The VM was difficult to maintain. MWAA worked but scaled expensively and lacked simple on/off controls. Moving to EventBridge, Step Functions, and Lambda cut the monthly cost from nearly $2,000 under MWAA to under $100.
- Change detection before compute. Querying the new dataset first, and skipping downstream processing when nothing changed, keeps the pipeline from paying to reprocess identical data — a natural fit for per-invocation serverless billing.
- Modular, single-responsibility stages. Splitting the work into discrete modules with integrity checks at each step made the pipeline more fault tolerant and easier to reason about, since a failure is isolated to one stage.
- Logging for two audiences. Customizing CloudWatch logs for both engineering and management meant the same run served debugging and operational visibility without extra reporting work.
Results
- Monthly platform costs dropped from nearly $2,000 under MWAA to under $100 on the EventBridge/Lambda stack.
- Runs that found no source changes were logged and skipped, avoiding redundant downstream compute.
- Both engineering and management stakeholders gained clear operational logs into every run.
- Recurring manual data preparation was replaced by a scheduled serverless workflow that keeps QuickBase current with far less oversight.
The pipeline is complete, with no immediate enhancements planned. Its architecture and the lessons learned from it are now being applied to additional pipeline projects across other domains.