

Moodle LMS to BigQuery: Replacing Thousands of Nested API Calls With One Pipeline
Talk to Our Data EngineersOverview
A leading dental-industry media company runs its continuing-education programs on a Moodle-based learning management system. Its 1.4M+ enrollment records held the answers to which courses drew learners, who finished them, and how completion trended over time, but none of it reached the company's analytics warehouse. The only route out was course-by-course API calls that could not run at that scale. NeenOpal designed and deployed a Python pipeline on Google Cloud Run that loaded every historical enrollment into Google BigQuery and now refreshes automatically each month. The full monthly run finishes in about 12 minutes, with no manual effort from the client's team.
- 1.4M+
- unique enrollment records loaded into BigQuery
- 3,000+
- courses loaded with full metadata
- ~12 mints
- for each automated monthly refresh
- 500
- records per batch, saved progressively so no run loses completed work
- 0
- duplicate enrollments in the warehouse after overlapping backfills
Customer Challenges
Learning data is some of the most valuable data an education business owns, and getting it out of an LMS in bulk is rarely straightforward. The client faced four connected problems.

1.4M+ Records Against a Serverless Clock
Serverless functions have a hard execution-time limit. A single run attempting to load more than 1.4 million historical records would hit that limit long before it finished, leaving a partial load behind.

Overlapping Backfills That Risked Duplicates
Loading years of history meant running several backfills across date ranges. Without safeguards, records near the edges of those ranges would land in the warehouse twice and quietly inflate every enrollment and completion count.

Thousands of API Calls to Answer One Question
Getting completion data meant looping through every course, then every enrolled learner in that course, then fetching each learner's completion status. That nested pattern generated thousands of API calls per run and went well past what low-code orchestration tools can handle reliably or affordably.

No Bulk Export for Enrollment Data
The platform's standard APIs offered no flat, paginated way to export enrollments. Without one, every option for moving the data was slow, fragile or both.
Solutions
NeenOpal recognized early that low-code orchestration alone could not carry this workload and moved it to cloud-native compute built for long-running, high-volume extraction.
A Flat Enrollment Endpoint, Built With the LMS Vendor
NeenOpal worked with the LMS vendor to expose a flat, paginated enrollment endpoint. One sequence of paged requests replaced thousands of nested course-and-learner calls.
A Cloud Run Pipeline That Loads as It Goes
A Python service on Cloud Run pages through the endpoint and writes records to BigQuery in batches of 500. Progress is saved continuously, so nothing already loaded is ever lost or reloaded.
Auto-Chaining Past the Timeout Limit
When a run approaches Cloud Run's execution limit, it queues a fresh instance through Google Cloud Tasks, and that instance picks up exactly where the last one stopped. Data volume no longer decides whether a load can finish.
Deduplication Built Into Every Load
Every batch is checked against what BigQuery already holds before it is written. Overlapping backfills and repeat runs can't create duplicate enrollments, and the warehouse holds exactly one row per enrollment.
A Self-Running Monthly Refresh
Google Cloud Scheduler triggers an incremental load each month over a rolling 31-day window, so records updated near a month boundary are never missed. A staging layer then converts raw timestamps, calculates completion rates, and joins course details, giving analysts reporting-ready tables.
Services
Python
Google BigQuery
Google Cloud Run
Google Cloud Scheduler
Google Cloud Tasks
Web Services API
Benefits

A Complete Learning History in One Place
Every historical enrollment now sits in BigQuery beside the company's marketing and CRM data. Course performance can finally be compared with how learners were acquired and engaged.

Completion Rates the Business Can Act On
Completion rates by course arrive reporting-ready, so the team can see which programs keep learners to the end and which lose them, without exporting anything by hand.

Loads That Finish at Any Volume
Auto-chaining removes the execution-time ceiling. Whether a load covers one month or the full history, it runs to completion without intervention.

Lower Running Costs Than Low-Code Orchestration
Moving the nested workload off low-code orchestration eliminated the per-operation credits that thousands of API calls would have consumed on every run.

Fresh Data With Zero Manual Effort
The monthly refresh runs on its own in about 12 minutes. No one exports files, restarts jobs or reconciles duplicates.
Conclusion
NeenOpal replaced an API pattern that could not scale with a cloud-native pipeline that loads more than 1.4 million enrollments reliably and keeps them current every month. The client now reports on learner enrollment and completion from the same warehouse as the rest of its business. This pipeline is part of the wider platform described in Unified Cross-Platform Intelligence Across 20+ Data Sources.
FAQ
Common questions about extracting Moodle LMS data into a data warehouse
How do you load more than 1.4 million records when serverless functions have a hard time limit?
By auto-chaining. When a run approaches Cloud Run’s execution limit, it queues a fresh instance through Google Cloud Tasks, and that instance resumes from exactly where the previous one stopped. Records are written to BigQuery in batches of 500 with progress saved continuously, so data volume no longer determines whether a load can finish.
How are duplicate enrollments avoided when backfills overlap?
Every batch is checked against what BigQuery already holds before it is written. Loading years of history requires several backfills across date ranges, and without that check, records near the range boundaries would land twice and quietly inflate every enrollment and completion count. The warehouse holds exactly one row per enrollment.
Why move this workload off low-code orchestration?
Low-code platforms charge per operation, and a nested pattern generating thousands of API calls consumes those credits on every single run. Moving the extraction to cloud-native compute built for long-running, high-volume work removed that per-operation cost entirely while also lifting the reliability ceiling. The full monthly refresh now completes in about 12 minutes.
Why is it so difficult to export enrollment data from Moodle in bulk?
Moodle’s standard APIs offer no flat, paginated way to export enrollments. Getting completion data means looping through every course, then every learner enrolled in that course, then fetching each learner’s completion status. That nested pattern generates thousands of API calls per run, which is well past what low-code orchestration tools handle reliably or affordably. NeenOpal worked with the LMS vendor to expose a flat, paginated enrollment endpoint instead.
Case studies
Related case studies

Zoom Events to BigQuery: Capturing the Attendee Data Standard Zoom APIs Miss
A leading dental-industry media company hosts hundreds of webinars a year on Zoom Events. Its reporting showed how many people attended each one, but not who they were. Without attendee-level data, the team couldn't follow up with the right people or connect webinar engagement to the rest of its marketing and CRM data. NeenOpal traced the problem to how Zoom Events is built and found an API route that returns the attendee records the client's own registrants generate. A Google Cloud Run pipeline now loads those records into Google BigQuery and refreshes them every month. The initial load covered more than 200 webinars and identified over 13,000 unique attendees.
Read case study
HubSpot to BigQuery: A Complete CRM History With SCD Type 2 Deal Tracking
A leading dental-industry media company runs its sales and marketing on HubSpot. More than 200,000 activity records sat across contacts, companies, deals, calls, emails, meetings, tasks, notes and SMS, with no automated way to bring them together for analysis. The team also had no way to analyze how deals moved through the pipeline over time. NeenOpal built ingestion pipelines across every major HubSpot object, linked them with 15 association tables and added SCD Type 2 tracking so deal stage history is preserved in Google BigQuery. The pipelines refresh weekly with no manual intervention.
Read case study
Building a Validated Conversational Sales Analyst on Amazon Quick for a Managed IT Provider
Our client, a managed IT services provider delivering all-inclusive computer, internet and help desk support to small and mid-sized businesses, already monitored sales opportunities across eight funnel stages in an established Power BI dashboard. Answering deeper questions about stage duration, win rates and trends still meant stacking filters and interpreting the resulting measures. NeenOpal extended that environment with a conversational AI Analyst Agent built on Amazon Quick, connected to the client's opportunity stage funnel dataset in Amazon RDS. Teams now ask questions in plain language and receive summaries, charts and the supporting SQL, cutting the time to a routine answer by 75% and the number of analysis steps by 60%. The depth sits in the foundation: a curated opportunity-stage data model, documented business rules, and answers reconciled with the existing Power BI dashboard month by month and filter by filter, which matched on every indicator tested. Power BI stays exactly where it was.
Read case study
AI-Augmented ETL for Automating Multi-Portal Retail Reporting on AWS
A consumer beauty and personal care brand needed to bring its Walmart and Amazon supplier and performance data into a single AWS RDS PostgreSQL database. Walmart's data was split across a standard supplier API and three separate reporting portals (Retail Link, Scintilla, and Supplier One), each with its own authentication, file formats, and update cadence. NeenOpal designed a hybrid ETL platform with two ingestion paths running in parallel: an orchestrated API pipeline for the data Walmart exposes directly, and an AI-driven report retrieval pipeline for the data that fell outside the scope of the client's existing API access. Both paths converge on a shared Glue transformation layer, a common S3 staging area, and a single PostgreSQL database, with a separate one-time Spark job bringing in Amazon Vendor Central's historical data. The result is a single, config-driven system rather than a set of one-off integrations.
Read case study
Automating Signed Credit Application PDFs with HubSpot Webhooks and AWS Serverless
Our client, a managed technology services provider, partnered with NeenOpal to automate the business credit application its customers complete when applying for credit terms. Until then the whole process ran over email. The team sent the customer a Word form as an attachment, the customer filled in their details and emailed it back, and someone then filed the completed form against the right account in Salesforce. Every application involved a round of back and forth correspondence, manual handling at both ends, and a record that existed only as a document attached to an account. NeenOpal designed and delivered HubSpot form to PDF automation on AWS, a serverless document automation pipeline that replaced the email exchange with a single link on the client's website. A customer opens the form, enters their details, and signs, and a branded PDF is generated automatically, emailed to the address captured on the form and to the client's own team, and archived in encrypted Amazon S3, while the submission itself is recorded field by field in HubSpot. Nobody sends a form out, chases it back, or files it by hand.
Read case study
Tableau Dashboard Development for a US Insurance Organization: 4 Data Sources Unified
Our client, an insurance organization with field teams spread across every US state, partnered with NeenOpal to replace a manual, spreadsheet-driven reporting process with a consolidated Tableau dashboard environment governed by row-level security. Operational data lived in four separate systems covering Airtable, Salesforce, Google Analytics 4, and Google Search Console, and bringing them together meant exporting and stitching the data by hand, which left leadership without a single reliable view of performance. NeenOpal consolidated all four sources into an Azure SQL Server layer, reviewed and corrected the data quality issues that surfaced along the way, and delivered three Tableau dashboards across seven screens that refresh automatically every day. Row-level security was implemented across all fifty states and the independent districts, driven by an entitlement model the client maintains through a single Google Sheet, so every representative sees the states they are responsible for and nothing more.
Read case study