Installation guide

This document describes how to install Scrapy on Linux, Windows and Mac OS X.

Requirements

Install Python

First, you need to install Python, if you haven’t done so already.

Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/

Install Scrapy

There are many ways to install Scrapy. Pick the one you feel more comfortable with.

Download and install an official release

Download Scrapy from the Download page. Scrapy is distributed in two ways: a source code tarball (for Unix and Mac OS X systems) and a Windows installer (for Windows). If you downloaded the tarball, you can install it as any Python package using setup.py:

tar zxf Scrapy-X.X.X.tar.gz
cd Scrapy-X.X.X
python setup.py install

If you downloaded the Windows installer, just run it.

Warning

In Windows, you may need to add the C:\Python25\Scripts (or C:\Python26\Scripts) folder to the system path by adding that directory to the PATH environment variable from the Control Panel.

Installing with easy_install

You can install Scrapy using setuptools‘s easy_install with:

easy_install -U Scrapy

Installing with pip

You can install Scrapy using pip with:

pip install Scrapy

Platform specific instructions

Linux

Ubuntu 9.10 or above

If you’re running Ubuntu 9.10 (or above), use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

Debian or Ubuntu (9.04 or older)

If you’re running Debian Linux, run the following command as root:

apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson

Then:

easy_install -U w3lib

And then follow the instructions in Install Scrapy.

Arch Linux

If you are running Arch Linux, run the following command as root:

pacman -S twisted libxml2 pyopenssl python-simplejson

Then:

easy_install -U w3lib

And then follow the instructions in Install Scrapy.

Other Linux distros

The easiest way to install Scrapy in other Linux distros is through easy_install, which will automatically install Twisted, w3lib and lxml as dependencies. See Installing with easy_install.

Another way would be to install dependencies, if you know the packages in your distros that meets them. See Requirements.

Mac OS X

The easiest way to install Scrapy on Mac is through easy_install or pip, which will automatically install Twisted, w3lib and lxml dependencies.

See Installing with easy_install.

Windows

There are two ways to install Scrapy in Windows: