Insurance news aggregation keyword analysis & database storage via Supabase

Automatically collect, analyze, and store industry news articles with intelligent filtering and dual-database storage

🎯 What This Workflow Does

This comprehensive news scraping automation monitors multiple industry sources every 6 hours, intelligently extracts relevant content, analyzes it for keyword relevance, and stores high-quality articles in both a content library and knowledge base. Perfect for content marketers, researchers, industry analysts, and anyone building an AI-powered knowledge system.

✨ Key Features

Multi-Source Aggregation: Collects from RSS feeds, Google News, and direct website scraping Smart Content Extraction: Uses Cheerio to parse HTML and extract clean article content AI-Ready Keyword Analysis: Automatically identifies and tags industry-specific terms Intelligent Filtering: Only stores articles meeting your relevance threshold (default: 30%) Dual Storage System: Saves to both content library (marketing) and knowledge base (AI training) Rate Limiting: Includes polite delays to respect website resources Fully Automated: Runs on schedule without manual intervention

🔧 Technical Highlights

Schedule-based trigger (every 6 hours, customizable) Handles 3 content types: RSS feeds, Google News RSS, and web pages Limits to top 10 articles per source to manage volume Tracks 17+ insurance-specific keywords (easily customizable for any industry) Includes comprehensive error handling and timeout management Supabase integration (works with any REST API database)

📚 Perfect For