ScrapingBee and Google Sheets integration template

This workflow contains community nodes that are only compatible with the self-hosted version of n8n.


This workflow allows users to extract sitemap links using ScrapingBee API. It only needs the domain name www.example.com and it automatically checks robots.txt and sitemap.xml to find the links. It is also designed to recursively run the workflow when new .xml links are found while scraping the sitemap.

How It Works 1. Trigger: The workflow waits for a webhook request that contains domain=www.example.com 2. It then looks for robots.txt file, if not found it checks sitemap.xml 3. Once it finds xml links, it recursively scrapes them to extract the website links 4. For each xml file, first it checks whether it's a binary file and whether it's a compressed xml 5. If it's a text response, it directly runs a code that extracts normal website link and another code to extract xml links 6. If it's a binary that is not compressed, it just extracts text from the binary and then extract webiste links and xml links 7. If it's a compressed binary, it first decompresses it and then extracts the text and then the links and xml 8. After extracting website links, it appends those links directly to a sheet 9. After extracting xml links, it scrapes them recursively until it finds all website links

When the workflow is finished, you will see the output in the links column of the Google Sheet that we added to the workflow.


Set Up Steps 1. Get your ScrapingBee API Key [here](https://www.scrapingbee.com/?utmsource=n8n&utmmedium=template) 2. Create a new google sheet with an empty column named links. Connect to the sheet by signing in using your Google Credential and add the link to your sheet. 3. Copy the webhook url, and send a GET request with domain as query parameter. Example:

curl "https://webhooklink?domain=scrapingbee.com"