Domain-specific web content crawler with depth control & text extraction

This template implements a recursive web crawler inside n8n. Starting from a given URL, it crawls linked pages up to a maximum depth (default: 3), extracts text and links, and returns the collected content via webhook.

---

🚀 How It Works

1) Webhook Trigger Accepts a JSON body with a url field. Example payload: { "url": "https://example.com" }

2) Initialization - Sets crawl parameters: url, domain, maxDepth = 3, and depth = 0. - Initializes global static data (pending, visited, queued, pages).

3) Recursive Crawling - Fetches each page (HTTP Request). - Extracts body text and links (HTML node). - Cleans and deduplicates links. - Filters out: - External domains (only same-site is followed) - Anchors (), mailto/tel/javascript links - Non-HTML files (.pdf, .docx, .xlsx, .pptx)

4) Depth Control & Queue - Tracks visited URLs - Stops at maxDepth to prevent infinite loops - Uses SplitInBatches to loop the queue

5) Data Collection - Saves each crawled page (url, depth, content) into pages[] - When pending = 0, combines results