Scrapy is a Python-based web crawling platform that is free and open-source. It was created with web scraping in mind, but it may also be used to collect data via APIs or as a general-purpose web crawler.
The architecture of the Scrapy project is based on “spiders,” which are self-contained crawlers that are given a set of instructions.

It allows developers to reuse their code, making it easier to construct and grow huge crawling applications. Scrapy also comes with a web-crawling shell that developers may use to test their assumptions about a site’s behavior.

Project Background

Applications