Discover article URLs from any website with GPT-5-mini and Google Sheets

Automatically discover and extract article URLs from any website using AI to identify valid content links while filtering out navigation, category pages, and irrelevant content—perfect for building content pipelines, news aggregators, and research databases.

What Makes This Different: - AI-Powered Intelligence - Uses GPT-5-mini to understand webpage context and identify actual articles vs navigation pages, eliminating false positives - Browser Spoofing - Includes realistic User-Agent headers and request patterns to avoid bot detection on publisher sites - Smart URL Normalization - Automatically strips tracking parameters (utm, fbclid, etc.), removes duplicates, and standardizes URLs - Source Categorization - AI assigns logical source names based on domain and content type for easy filtering - Rate Limiting Built-In - Configurable delays between requests prevent IP blocking and respect website resources - Deduplication on Save - Google Sheets append-or-update pattern ensures no duplicate URLs in your database

Key Benefits of AI-Powered Content Discovery: - Save 10+ Hours Weekly - Automate manual link hunting across dozens of publisher sites - Higher Quality Results - AI filters out 95%+ of junk links (nav pages, categories, footers) that rule-based scrapers miss - Scale Effortlessly - Add new seed URLs to your sheet and the same workflow handles any website structure - Industry Agnostic - Works for news, blogs, research papers, product pages—any content type - Always Up-to-Date - Schedule daily runs to catch new content as it's published - Full Audit Trail - Track discovered URLs with timestamps and sources in Google Sheets

---

Who's it for

This template is designed for content marketers, SEO professionals, researchers, media monitors, and anyone who needs to aggregate content from multiple sources. It's perfect for organizations that need to track competitor blogs, curate industry news, build research databases, monitor brand mentions, or aggregate content for newsletters without manually checking dozens of websites daily or writing complex scraping rules for each source.

How it works / What it does

This workflow creates an intelligent content discovery pipeline that automatically finds and extracts article URLs from any webpage. The system: