Discover hidden website API endpoints using regex and AI

💡 What it is for

This workflow helps to automatically discover undocumented API endpoints by analysing JavaScript files from the website's HTML code.

When building automation for platforms without public APIs, we face a significant technical barrier. In a perfect world, every service would offer well-documented APIs with clear endpoints and authentication methods. But the reality is different.

Before we resort to complex web scraping, let's analyse the architecture of the platform and check whether it makes internal API calls. We will examine JavaScript files embedded in the HTML source code to find and extract potential API endpoints.

⚙️Key Features

To discover hidden API endpoints, we can apply two major approaches:

1. Predefined regex extraction: manually insert a fixed regex with the necessary conditions to extract endpoints. Unlike LLM, which creates a custom regex for each JS file, we provide a generic expression to capture all URL strings. We do not want to accidentally miss important API endpoints.

2. AI-supported extraction: - ask LLMs to examine the structure of the JavaScript code. The 1st model will: - capture potential API endpoints - create a detailed description of each identified endpoint with methods and query parameters - the 2nd LLM connected to the AI Agent will generate a regex for each JS file individually based on the output of the 1st model. In addition to pure endpoint extraction, we supplement our analysis with: