Is It Worth It: AI Crawler Analytics

Is It Worth It: AI Crawler Analytics

This Python script demonstrates how to fetch the visible, readable text from a webpage and extract a clean snippet—perfect for quick content analysis or monitoring site changes. It strips out scripts, styles, and HTML tags, leaving only the human-readable portion.

The core challenge breaks down into four clear steps, all tackled inside a single function:

  1. Download the page – use requests with a realistic User-Agent header to mimic a browser and avoid being blocked.
  2. Remove non-content blocks – strip <script>…</script> and <style>…</style> sections, which don’t carry readable text.
  3. Strip HTML tags – replace every remaining tag with a space, so the raw content is reduced to plain text.
  4. Clean and truncate – collapse multiple spaces, trim leading/trailing whitespace, and keep only the first 4,000 characters (enough for a meaningful snippet without storing the whole page).
import requests
import re

def fetch_text(url):
    try:
        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
        r = requests.get(url, headers=headers, timeout=15)
        r.raise_for_status()
        # simple extraction of visible text from body, remove script/style
        text = re.sub(r'<script.*?</script>', '', r.text, flags=re.DOTALL)
        text = re.sub(r'<style.*?</style>', '', text, flags=re.DOTALL)
        text = re.sub(r'<[^>]+>', ' ', text)
        text = re.sub(r'\s+', ' ', text).strip()
        return text[:4000] if len(text) > 4000 else text
    except Exception as e:
        return f"Error: {e}"

home = fetch_text('https://siteup.ai')
blog = fetch_text('https://siteup.ai/blog')

print("=== HOMEPAGE TEXT ===")
print(home)
print("\n=== BLOG TEXT ===")
print(blog)

In short, the fetch_text() function:

  • Uses requests with a browser-like User-Agent to download the page.
  • Applies a sequence of regular expressions to remove <script> and <style> blocks, strip all other HTML tags, and collapse whitespace.
  • Returns the first 4,000 characters of clean text or an error message if something goes wrong.

The key takeaway is that basic web‑page text extraction can be achieved in just a few lines of Python. For production use, however, a dedicated HTML parser like BeautifulSoup is recommended to handle more complex structures and edge cases reliably.

Frequently Asked Questions

Q: Why use regular expressions instead of a full HTML parser like BeautifulSoup?
A: Regular expressions keep the script lightweight and dependency-free, which is ideal for quick experiments or environments where installing extra libraries is not possible. However, regex can break on malformed or deeply nested HTML. For anything beyond simple extraction, a parser is the safer choice.

Q: What are the main limitations of this approach?
A:

  • It doesn’t execute JavaScript, so any content loaded dynamically won’t appear.
  • The extraction is based purely on the raw HTML string; it can’t preserve semantic structure (headings, paragraphs).
  • The User-Agent may still be blocked by some sites that require additional headers or cookies.
  • Only the first 4,000 characters are returned, which might miss important content further down the page.

Q: How can I handle pages that load content via JavaScript?
A: This script uses only requests, which fetches the static HTML. For JavaScript‑rendered pages, you’d need tools like Selenium, Playwright, or a headless browser. You could also look for the site’s underlying API calls that deliver the data in a structured format.

Q: Can I increase or remove the 4,000‑character limit?
A: Absolutely. Simply change the number 4000 in the return line to your desired limit, or return text unaltered to get the full extracted text. Be cautious with very large pages, as storing the entire text might use more memory than expected.

Q: Is this script safe for monitoring site changes?
A: It can be used as a simple monitor—compare the extracted text over time to detect changes. However, for reliable monitoring, add error handling for network timeouts, respect robots.txt rules, and consider the site’s terms of service. Repeated rapid requests may lead to your IP being blocked.