Skip to content
NewNeenOpal Earns ISO 42001 Certification for AI GovernanceKnow More

Moodle LMS to BigQuery: Replacing Thousands of Nested API Calls With One Pipeline

Talk to Our Data Engineers

Talk to our team

Tell us what you’re working on. A senior consultant will follow up with next steps.

Overview

A leading dental-industry media company runs its continuing-education programs on a Moodle-based learning management system. Its 1.4M+ enrollment records held the answers to which courses drew learners, who finished them, and how completion trended over time, but none of it reached the company's analytics warehouse. The only route out was course-by-course API calls that could not run at that scale. NeenOpal designed and deployed a Python pipeline on Google Cloud Run that loaded every historical enrollment into Google BigQuery and now refreshes automatically each month. The full monthly run finishes in about 12 minutes, with no manual effort from the client's team.

1.4M+
unique enrollment records loaded into BigQuery
3,000+
courses loaded with full metadata
~12 mints
for each automated monthly refresh
500
records per batch, saved progressively so no run loses completed work
0
duplicate enrollments in the warehouse after overlapping backfills

Customer Challenges

Learning data is some of the most valuable data an education business owns, and getting it out of an LMS in bulk is rarely straightforward. The client faced four connected problems.

  • 1.4M+ Records Against a Serverless Clock

    Serverless functions have a hard execution-time limit. A single run attempting to load more than 1.4 million historical records would hit that limit long before it finished, leaving a partial load behind.

  • Overlapping Backfills That Risked Duplicates

    Loading years of history meant running several backfills across date ranges. Without safeguards, records near the edges of those ranges would land in the warehouse twice and quietly inflate every enrollment and completion count.

  • Thousands of API Calls to Answer One Question

    Getting completion data meant looping through every course, then every enrolled learner in that course, then fetching each learner's completion status. That nested pattern generated thousands of API calls per run and went well past what low-code orchestration tools can handle reliably or affordably.

  • No Bulk Export for Enrollment Data

    The platform's standard APIs offered no flat, paginated way to export enrollments. Without one, every option for moving the data was slow, fragile or both.

Solutions

NeenOpal recognized early that low-code orchestration alone could not carry this workload and moved it to cloud-native compute built for long-running, high-volume extraction.

  1. A Flat Enrollment Endpoint, Built With the LMS Vendor

    NeenOpal worked with the LMS vendor to expose a flat, paginated enrollment endpoint. One sequence of paged requests replaced thousands of nested course-and-learner calls.

  2. A Cloud Run Pipeline That Loads as It Goes

    A Python service on Cloud Run pages through the endpoint and writes records to BigQuery in batches of 500. Progress is saved continuously, so nothing already loaded is ever lost or reloaded.

  3. Auto-Chaining Past the Timeout Limit

    When a run approaches Cloud Run's execution limit, it queues a fresh instance through Google Cloud Tasks, and that instance picks up exactly where the last one stopped. Data volume no longer decides whether a load can finish.

  4. Deduplication Built Into Every Load

    Every batch is checked against what BigQuery already holds before it is written. Overlapping backfills and repeat runs can't create duplicate enrollments, and the warehouse holds exactly one row per enrollment.

  5. A Self-Running Monthly Refresh

    Google Cloud Scheduler triggers an incremental load each month over a rolling 31-day window, so records updated near a month boundary are never missed. A staging layer then converts raw timestamps, calculates completion rates, and joins course details, giving analysts reporting-ready tables.

Services

  • Python
  • Google BigQuery
  • Google Cloud Run
  • Google Cloud Scheduler
  • Google Cloud Tasks
  • Web Services API

Benefits

  • A Complete Learning History in One Place

    Every historical enrollment now sits in BigQuery beside the company's marketing and CRM data. Course performance can finally be compared with how learners were acquired and engaged.

  • Completion Rates the Business Can Act On

    Completion rates by course arrive reporting-ready, so the team can see which programs keep learners to the end and which lose them, without exporting anything by hand.

  • Loads That Finish at Any Volume

    Auto-chaining removes the execution-time ceiling. Whether a load covers one month or the full history, it runs to completion without intervention.

  • Lower Running Costs Than Low-Code Orchestration

    Moving the nested workload off low-code orchestration eliminated the per-operation credits that thousands of API calls would have consumed on every run.

  • Fresh Data With Zero Manual Effort

    The monthly refresh runs on its own in about 12 minutes. No one exports files, restarts jobs or reconciles duplicates.

Conclusion

NeenOpal replaced an API pattern that could not scale with a cloud-native pipeline that loads more than 1.4 million enrollments reliably and keeps them current every month. The client now reports on learner enrollment and completion from the same warehouse as the rest of its business. This pipeline is part of the wider platform described in Unified Cross-Platform Intelligence Across 20+ Data Sources.

FAQ

Common questions about extracting Moodle LMS data into a data warehouse

How do you load more than 1.4 million records when serverless functions have a hard time limit?

By auto-chaining. When a run approaches Cloud Run’s execution limit, it queues a fresh instance through Google Cloud Tasks, and that instance resumes from exactly where the previous one stopped. Records are written to BigQuery in batches of 500 with progress saved continuously, so data volume no longer determines whether a load can finish.

How are duplicate enrollments avoided when backfills overlap?

Every batch is checked against what BigQuery already holds before it is written. Loading years of history requires several backfills across date ranges, and without that check, records near the range boundaries would land twice and quietly inflate every enrollment and completion count. The warehouse holds exactly one row per enrollment.

Why move this workload off low-code orchestration?

Low-code platforms charge per operation, and a nested pattern generating thousands of API calls consumes those credits on every single run. Moving the extraction to cloud-native compute built for long-running, high-volume work removed that per-operation cost entirely while also lifting the reliability ceiling. The full monthly refresh now completes in about 12 minutes.

Why is it so difficult to export enrollment data from Moodle in bulk?

Moodle’s standard APIs offer no flat, paginated way to export enrollments. Getting completion data means looping through every course, then every learner enrolled in that course, then fetching each learner’s completion status. That nested pattern generates thousands of API calls per run, which is well past what low-code orchestration tools handle reliably or affordably. NeenOpal worked with the LMS vendor to expose a flat, paginated enrollment endpoint instead.

Authors

  • Madhumita GiribabuSenior Associate Consultantin

Case studies

Related case studies