Internet scraping is the method of accumulating info from web sites robotically. A traditional scraper normally extracts uncooked textual content, HTML components, or the complete web page content material. However when you’re constructing AI brokers or giant language mannequin (LLM) purposes, sending your entire webpage to the mannequin isn’t at all times the perfect strategy.
A greater manner is to first clear the web page, convert it into Markdown, after which use an LLM to grasp the content material and return solely the reply the person wants. This makes the output cleaner, simpler to learn, and simpler to make use of in one other workflow.
It additionally helps cut back token utilization. As an alternative of passing a messy webpage filled with navigation hyperlinks, buttons, scripts, footers, and repeated content material, we solely ship the helpful web page content material to the mannequin. The LLM then returns a centered reply in Markdown as an alternative of dumping the entire web page again to the person.
On this information, we are going to construct a easy AI net scraper in Python utilizing Jupyter Pocket book. It would fetch a webpage, clear the HTML, convert it into Markdown, settle for a person question, and return a transparent Markdown reply primarily based on the web page content material.
# Setting Up
We are going to use Jupyter Pocket book for this venture. It makes it simpler to check every step first earlier than turning the scraper into a correct utility programming interface (API) or utility.
Begin by putting in the required Python packages:
!pip set up requests beautifulsoup4 markdownify openai ftfy python-dotenv
We are going to use:
Within the subsequent cell, import the required libraries:
import os
import re
import requests
from bs4 import BeautifulSoup, Remark
from ftfy import fix_text
from markdownify import markdownify as markdownify_html
from openai import OpenAI
from dotenv import load_dotenv
from IPython.show import Markdown, show
Subsequent, be sure your OpenAI API secret’s obtainable as an setting variable. The safer manner is to create a .env file in the identical folder as your pocket book and add your key there:
OPENAI_API_KEY=your_api_key_here
Then load it contained in the pocket book:
load_dotenv()
consumer = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
You can even test that the important thing was loaded appropriately:
if not os.getenv("OPENAI_API_KEY"):
elevate ValueError("OPENAI_API_KEY is lacking. Add it to your .env file first.")
Additionally be sure your OpenAI platform account has billing arrange. For brand new API accounts, you could want so as to add pay as you go credit earlier than you’ll be able to run API calls. If a mannequin isn’t obtainable in your account, use one other mannequin out of your OpenAI dashboard.
Now outline the mannequin identify:
MODEL_NAME = "gpt-5.4-nano"
We’re utilizing a smaller mannequin right here as a result of this activity doesn’t want a big reasoning mannequin. The objective is straightforward: learn the cleaned webpage content material, perceive the person question, and return a centered Markdown reply.
# Fetching the Webpage
Now we are going to create the primary perform. This perform will fetch the webpage utilizing the requests package deal and return the uncooked HTML.
def fetch_page(url: str) -> str:
"""
Obtain the HTML content material from a webpage.
"""
headers = {
"Person-Agent": "SimpleAIScraper/1.0"
}
response = requests.get(url, headers=headers, timeout=15)
response.raise_for_status()
return response.textual content
The Person-Agent header tells the web site that the request is coming from our scraper. Some web sites block requests that don’t embody a person agent, so including one makes the request a bit extra dependable.
We additionally use timeout to keep away from ready indefinitely if the web site doesn’t reply. The raise_for_status() name will cease the code if the request fails — for instance, if the web page returns a 404 or 500 error.
Now let’s take a look at the perform with an actual web site:
uncooked = fetch_page("https://www.olostep.com/")
print(uncooked[:500])
This can obtain the uncooked HTML from the webpage and print the primary 500 characters.
Uncooked HTML output | Picture by Writer
At this stage, the output will nonetheless look messy as a result of it accommodates the complete web page HTML, together with tags, scripts, format components, and different content material we don’t want.
# Cleansing the HTML
The uncooked HTML from a webpage normally accommodates a number of content material we don’t want. It may embody scripts, styling, navigation menus, buttons, varieties, headers, footers, popups, and different format components.
Earlier than sending the web page content material to the LLM, we have to clear the HTML. This helps cut back noise and makes the ultimate Markdown a lot simpler for the mannequin to grasp.
We are going to use BeautifulSoup to parse the HTML and take away pointless components.
def clean_html(html):
html = fix_text(html)
soup = BeautifulSoup(html, "html.parser")
# Take away apparent noisy tags
for tag in soup([
"script", "style", "noscript", "svg", "img", "iframe",
"nav", "header", "footer", "aside", "form", "button"
]):
tag.decompose()
noise_words = [
"cursor",
"modal",
"popup",
"floating",
"signup",
"login",
"cookie",
"banner",
"navbar",
"menu",
"footer",
"header",
"subscribe",
"newsletter",
"loading",
"wait",
"success",
"auth",
"w-nav",
"w-form"
]
# First gather noisy tags
tags_to_remove = []
for tag in soup.find_all(True):
if tag.attrs is None:
proceed
class_value = tag.get("class", [])
id_value = tag.get("id", "")
if isinstance(class_value, record):
class_text = " ".be a part of(class_value).decrease()
else:
class_text = str(class_value).decrease()
id_text = str(id_value).decrease()
if any(phrase in class_text or phrase in id_text for phrase in noise_words):
tags_to_remove.append(tag)
# Then take away them safely
for tag in tags_to_remove:
tag.decompose()
physique = soup.physique if soup.physique else soup
return str(physique)
First, we use fix_text() to wash any damaged or unusual textual content encoding points. Then BeautifulSoup parses the HTML so we are able to take away the elements we don’t want.
We take away apparent noisy tags like script, fashion, nav, header, footer, type, and button. These sections normally don’t assist reply the person question and may waste tokens.
After that, we search for noisy class names and IDs. Many web sites use phrases like popup, cookie, navbar, publication, or modal inside their HTML. If a tag accommodates these phrases, we gather it and take away it safely.
Now let’s run the perform on the uncooked HTML:
clear = clean_html(uncooked)
print(clear[:500])
As you’ll be able to see, the webpage is now a lot cleaner. It nonetheless accommodates helpful HTML tags and textual content, however many of the noisy format, scripts, navigation, and popups have been eliminated.
Cleaned HTML output | Picture by Writer
# Changing HTML to Markdown
Now we are going to convert the cleaned HTML into Markdown. Markdown is simpler to learn, simpler to save lots of, and simpler for the LLM to grasp in comparison with uncooked HTML.
This step additionally helps cut back enter tokens as a result of we take away pointless formatting, pictures, clean traces, and repeated textual content. For the conversion, we are going to use markdownify.
def html_to_markdown(html):
markdown_text = markdownify_html(
html,
heading_style="ATX",
bullets="-"
)
markdown_text = fix_text(markdown_text)
# Take away picture markdown
markdown_text = re.sub(r"", "", markdown_text)
# Take away additional areas and clean traces
markdown_text = re.sub(r"[ t]+", " ", markdown_text)
markdown_text = re.sub(r"n{3,}", "nn", markdown_text)
traces = []
skip_lines = [
"click to try",
"wait...",
"you've successfully reserved your spot.",
"thank you! your submission has been received!",
"oops! something went wrong while submitting the form.",
"product",
"resources",
"company"
]
for line in markdown_text.splitlines():
line = line.strip()
if not line:
proceed
if line.decrease() in skip_lines:
proceed
traces.append(line)
return "n".be a part of(traces)
First, we use markdownify to transform the cleaned HTML into Markdown. We set the heading fashion to ATX, which suggests headings will use commonplace Markdown syntax with #, ##, and ###.
Then we run fix_text() once more to wash any remaining encoding points. After that, we take away picture Markdown as a result of picture hyperlinks are normally not helpful for answering text-based questions.
We additionally take away additional areas and clean traces so the ultimate content material is compact. This makes the web page simpler to examine and helps cut back the variety of tokens despatched to the mannequin.
The skip_lines record removes repeated web site textual content similar to type messages, navigation labels, and small call-to-action textual content. You possibly can replace this record primarily based on the web site you might be scraping.
Now let’s run the perform:
md = html_to_markdown(clear)
print(md[:500])
As you’ll be able to see, the textual content is now a lot cleaner and nearer to the format we wish. As an alternative of uncooked HTML, we now have readable Markdown with helpful headings, paragraphs, and bullet factors.
Markdown output | Picture by Writer
# Asking a Person Question In opposition to the Web page
Now we are going to create the perform that sends the cleaned Markdown content material to the LLM. This perform takes two inputs: the webpage content material in Markdown and the person question.
As an alternative of asking the mannequin to summarize the entire web page, we ask it to reply a particular query utilizing solely the web page content material. This makes the response extra centered and helpful.
def answer_query_from_page(markdown_text, user_query):
immediate = f"""
You might be an AI net scraping assistant.
You'll obtain Markdown extracted from a webpage.
Your activity is to reply the person's question utilizing solely the helpful web page content material.
Person question:
{user_query}
Webpage Markdown:
{markdown_text}
Directions:
- Return solely clear Markdown.
- Use solely info from the webpage Markdown.
- Don't invent lacking particulars.
- Ignore navigation hyperlinks, buttons, CTAs, popups, ornamental labels, picture captions, and repeated advertising fragments.
- Ignore traces like "Begin without spending a dime", "Contact Gross sales", "Your AI Agent", and ornamental workflow examples except they instantly reply the question.
- Concentrate on headings, paragraphs, product descriptions, characteristic sections, pricing particulars, documentation textual content, and factual claims.
- If the web page doesn't include the reply, say: "The web page doesn't include this info."
- Hold the reply quick, clear, and centered.
"""
response = consumer.responses.create(
mannequin=MODEL_NAME,
enter=immediate
)
return response.output_text
The immediate is a very powerful a part of this step. It tells the mannequin what function it ought to play, what content material it might use, and how much reply it ought to return.
We additionally inform the mannequin to make use of solely the offered Markdown. That is vital as a result of we don’t need the mannequin to guess or add info that isn’t current on the webpage.
The instruction to return solely clear Markdown makes the output simpler to show in a pocket book, save to a file, or go into one other AI workflow.
This perform is the place the AI net scraper turns into genuinely helpful. We’re now not simply extracting web page textual content — we’re asking the LLM to grasp the cleaned web page and return the precise reply the person is searching for.
# Creating the Full AI Internet Scraper
Now we are going to create the ultimate perform that connects every little thing collectively.
This perform will take the URL and the person question as inputs. It would then fetch the webpage, clear the HTML, convert the content material into Markdown, and return the reply utilizing the gpt-5.4-nano mannequin.
def ai_web_scraper(url, user_query):
raw_html = fetch_page(url)
cleaned_html = clean_html(raw_html)
markdown_text = html_to_markdown(cleaned_html)
reply = answer_query_from_page(markdown_text, user_query)
return reply
That is our full AI net scraper pipeline. As an alternative of manually working every step one after the other, we are able to now name a single perform and get a clear Markdown reply from any webpage.
The move is straightforward:
- Fetch the webpage.
- Clear the HTML.
- Convert it into Markdown.
- Ask the LLM a query.
- Return the ultimate reply.
This retains the code easy and simple to reuse later in an API, chatbot, or agent workflow.
# Testing the AI Internet Scraper
Now let’s take a look at our AI net scraper. We are going to present it with an internet site URL and ask what the corporate does.
url = "https://www.olostep.com/"
user_query = "What does this firm do?"
end result = ai_web_scraper(url, user_query)
show(Markdown(end result))
In return, we get a correct Markdown response concerning the firm and its product. That is a lot better than returning the complete webpage content material as a result of the reply is targeted, readable, and instantly associated to the person question.
Scraper output for a corporation overview question | Picture by Writer
Now let’s attempt a distinct web page and ask about pricing.
url = "https://www.olostep.com/pricing"
user_query = "Assist me perceive the pricing"
end result = ai_web_scraper(url, user_query)
show(Markdown(end result))
In a couple of seconds, we get a clear response that’s straightforward to grasp. As an alternative of manually visiting the pricing web page and looking for the related info, the scraper extracts the web page, cleans it, and asks the LLM to clarify solely what issues.
Scraper output for a pricing question | Picture by Writer
We will additionally save the ultimate response as a Markdown file.
with open("ai_scraper_result.md", "w", encoding="utf-8") as file:
file.write(end result)
print("Markdown saved to ai_scraper_result.md")
Output:
Markdown saved to ai_scraper_result.md
Now the result’s saved as a Markdown file, which you’ll be able to open, edit, share, or use in one other workflow.
# Closing Ideas
Constructing your individual AI instruments is way simpler now. With a couple of traces of Python and an LLM, we turned a standard webpage right into a easy question-answering engine that may learn the web page, perceive the person question, and return a clear Markdown reply.
That is highly effective as a result of you don’t at all times want a posh system to resolve a particular downside. Typically, a small specialised answer is sufficient.
However it is usually vital to keep in mind that every little thing has a value. Working the app on a server prices cash. Calling an LLM prices cash. Sustaining the scraper, fixing damaged pages, dealing with errors, and bettering the system over time additionally prices money and time.
So earlier than constructing your individual customized answer, it’s value taking a look at present instruments like Olostep, Firecrawl, or Exa. In some circumstances, paying for a ready-made scraping or net intelligence API might make extra sense. In different circumstances — particularly if the duty is small, native, or very particular — constructing your individual light-weight answer might be the higher possibility.
Abid Ali Awan (@1abidaliawan) is an authorized information scientist skilled who loves constructing machine studying fashions. At present, he’s specializing in content material creation and writing technical blogs on machine studying and information science applied sciences. Abid holds a Grasp’s diploma in expertise administration and a bachelor’s diploma in telecommunication engineering. His imaginative and prescient is to construct an AI product utilizing a graph neural community for college students fighting psychological sickness.
