Scrapy is a Python-based web crawling platform that is free and open-source. It was created with web scraping in mind, but it may also be used to collect data via APIs or as a general-purpose web crawler.
The architecture of the Scrapy project is based on “spiders,” which are self-contained crawlers that are given a set of instructions.

It allows developers to reuse their code, making it easier to construct and grow huge crawling applications. Scrapy also comes with a web-crawling shell that developers may use to test their assumptions about a site’s behavior.

Project Background

  • Project: Scrappy
  • Author: Zyte
  • Initial Release: 2008
  • Type: Web Crawler
  • License: BSD License
  • Contains: web crawling shell
  • Language:  Python 
  • GitHub: scrapy/scrapy with 42k stars and 435 contributors
  • Runs On: Windows, Linux, MacOS
  • Twitter:/scrapyproject


  • Data mining 
  • Monitoring and automated testing
Scroll to Top