Documentation Index#

Fetch the complete documentation index at: https://docs.crewai.com/llms.txt Use this file to discover all available pages before exploring further.

The ScrapeWebsiteTool is designed to extract and read the content of a specified website.

ScrapeWebsiteTool#

We are still working on improving tools, so there might be unexpected behavior or changes in the future.

Description#

A tool designed to extract and read the content of a specified website. It is capable of handling various types of web pages by making HTTP requests and parsing the received HTML content. This tool can be particularly useful for web scraping tasks, data collection, or extracting specific information from websites.

Installation#

Install the crewai_tools package

pip install 'crewai[tools]'

Example#

from crewai_tools import ScrapeWebsiteTool

# To enable scrapping any website it finds during it's execution
tool = ScrapeWebsiteTool()

# Initialize the tool with the website URL, 
# so the agent can only scrap the content of the specified website
tool = ScrapeWebsiteTool(website_url='https://www.example.com')

# Extract the text from the site
text = tool.run()
print(text)

Arguments#