Web Scraping
Definition
Web scraping programmatically extracts content from HTML pages by parsing the DOM structure and selecting target elements using CSS selectors or XPath expressions. Unlike a crawler (which follows links across many pages), scraping often refers to the extraction logic applied to a single page or a defined set of pages. Web scraping is used to import content from a company's existing web presence — product documentation, blog posts, FAQ pages — into a knowledge base without manual data entry. The scraped content is cleaned of HTML markup and navigation elements before indexing.
Why It Matters
Web scraping enables teams to build a knowledge base from the content they have already published, rather than starting from scratch. For organizations with comprehensive websites, blog archives, or help centers, scraping provides an immediate 80% solution for knowledge base population. It also enables ongoing synchronization — regularly scraping web content ensures the knowledge base reflects the current state of published information.
How It Works
A scraper sends an HTTP GET request to the target URL, receives the HTML response, and parses it using a library (BeautifulSoup, Cheerio, or a headless browser for JavaScript-rendered pages). CSS selectors or XPath expressions identify the main content area, extracting its text while discarding navigation, headers, footers, and advertisements. The extracted text is passed to the knowledge base ingestion pipeline. For JavaScript-rendered pages, a headless browser (Puppeteer, Playwright) executes the page JavaScript before extraction.
How Web Scraping Works
Seed URLs
Crawler
Parser
Clean & Store
1,240
Pages crawled
380
New URLs found
1,180
Indexed
Real-World Example
A company wants to add their blog archive (500 posts) to their AI knowledge base. A scraper processes each post URL from the sitemap, extracts the article title and body text using CSS selectors targeting the main content div, strips HTML formatting, and passes clean text to the ingestion pipeline. 500 blog posts are ingested in 20 minutes.
Common Mistakes
- ✕Scraping content without respecting robots.txt — always check and respect crawling restrictions.
- ✕Extracting HTML including navigation, ads, and sidebars — scrapers must target only the main article content to avoid polluting the knowledge base with irrelevant text.
- ✕Not handling JavaScript-rendered content with a static HTTP client — many modern websites render content client-side and require a headless browser to extract text.
Related Terms
Content Crawler
A content crawler is an automated tool that systematically visits web pages — starting from a URL or sitemap — and extracts their content for ingestion into a knowledge base. It enables organizations to automatically populate and keep their AI knowledge base current with content published on their website or help center.
Document Ingestion
Document ingestion is the process of importing, parsing, and indexing external documents — PDFs, Word files, web pages, CSVs, and more — into a knowledge base or AI retrieval system. It transforms raw files into searchable, retrievable content that an AI can use to answer questions.
Sitemap Indexing
Sitemap indexing uses a website's sitemap.xml file — a structured list of all URLs — to systematically discover and ingest all relevant web pages into a knowledge base. It provides a more reliable and complete alternative to link-following crawls by using the site's own declared page inventory.
Knowledge Base
A knowledge base is a centralized repository of structured information — articles, FAQs, guides, and documentation — that an AI chatbot or support system uses to answer user questions accurately. It is the foundation of any AI-powered self-service experience, directly determining how accurate and comprehensive the bot's answers are.
Content Freshness
Content freshness refers to how current and up-to-date knowledge base articles are. Fresh content produces accurate AI answers; stale content produces confidently wrong answers. Maintaining freshness requires review workflows, expiry policies, and systematic audits that keep articles aligned with the current state of the product.
Ready to build your AI chatbot?
Put these concepts into practice with 99helpers — no code required.
Start free trial →