Migrate large Hugging Face datasets to MongoDB with a looping subworkflow

This n8n template provides a production-ready, memory-safe pipeline for ingesting large Hugging Face datasets into MongoDB using batch pagination. It is designed as a reusable data ingestion layer for RAG systems, recommendation engines, analytics pipelines, and ML workflows.

The template includes: - A main workflow that orchestrates pagination and looping - A subworkflow that fetches dataset rows, sanitizes them, and inserts them into MongoDB safely

---

🚀 What This Template Does

- Fetches rows from a Hugging Face dataset using the datasets-server API - Processes data in configurable batches (offset + length) - Removes Hugging Face id fields to avoid MongoDB duplicate key errors - Inserts clean documents into MongoDB - Automatically loops until all dataset rows are ingested - Handles large datasets without memory overflow

---

🧩 Architecture Overview

Main Workflow (Orchestrator) - Starts the ingestion process - Defines dataset, batch size, and MongoDB collection - Repeatedly calls the subworkflow until no rows remain