How to install Scrapy on Windows 10

It took a lot of time for me to install scrapy in my windows pc. I have tried the Installation Guide by scrapy, and tried the tutorials from YouTube and always ended up with having errors. And i tried for weeks installing and uninstalling components, always got different errors. And finally, with lots of research, I successfully installed Scrapy. So, this is how i did it.

Step 1: Install Python 2.7

You can download Python 2.7 from here. Please make sure that you are downloading and installing Python 2.7, because scrapy don’t support the Python 3 versions. But, scrapy is working on making it compatible with Python 3. If you have already installed Python 3, uninstall it before installing Python 2.7.

Now you need to add C:\Python27 and C:\Python27\Scripts to your Path environment variable. To do this open your command prompt and type the following and hit enter:

c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py

To do it manually, go to your system properties, either through My Computer or Control Panel.

Then Select Advanced Settings , then Environment Variables from Advanced Tab.

Then select path from system variables. Then click edit.

Then copy-paste C:\Python27\;C:\Python27\Scripts; to the end of the path after the semicolon(if there is no semicolon in the end add it before pasting the path.). Then OK button, you have successfully added the path.

To check whether Python have installed properly, go to command prompt and type python and hit enter. If you have installed it properly you can see a message like in the below image showing the version and other details of Python.

Note: The versions of the softwares shown in the image will be different in the website (except python 2.7), since constant updating is happening.

Step 2: Install Pip and Setup tools

All the Python versions come after 2.7.9 already have pip and setup tools installed. To check whether pip have installed correctly and to check the version use the following command in cmd.

pip --version

If your pip is outdated, you can update pip using the command:

pip install --upgrade pip

If your python doesn’t have setuptools, download ez_setup.py from here to your python folder. To save the file Right click on the page and then select Save as. Then go to cmd and change the path to python. Then use the following command to install setuptools.

python ez_setup.py

Step 3: Install Microsoft Visual C++ for Python 2.7

Microsoft Visual C++ Compiler for Python 2.7 is really important to install before installing scrapy. Download it from here.

Step 4: Install OpenSSL

For windows, it doesn’t come pre-installed with the os, we have to install it separately. Download the OpenSSL from here to your C:\Python27 folder.

Right click on the .whl file and select Save link as to download the file to your python folder.

To install OpenSSL, open cmd. Then change the path using
cd C:\Python27

Then use the following command:
python -m pip install pyOpenSSL-16.0.0-py2.py3-none-any.whl

Once its installed completely, you can see the following message in the end.

Step 4: Install lxml

Installing lxml is really important. Download lxml from here. We need to download and install the latest version which supports our Python 2.7. Download the file to your Python 2.7 folder, which is C:\Python27.

To install the lxml, open cmd and change path to python as we did in previous step. Then use the following command to install lxml.
python -m pip install lxml-3.6.0-cp27-cp27m-win32.whl

Step 5: Install Pywin32

Download Pywin32 from here and install it. Please make sure that you are downloading the right version for the Python installed in your system.

Step 6: Install Scrapy

Finally!  It needs a lot of effort to install scrapy without any errors. After installing all the above components successfully, now, we can install Scrapy. Open a cmd window and type

pip install Scrapy

and hit enter. It will take few mins to complete the installation. And finally we completed installation of scrapy successfully.

Note: When you copy-paste commands to your cmd window, don’t use keyboard shortcuts since it wont work, use mouse to do it.

How to install Scrapy in Visual Studio?

How to use Scrapy in python?

How to install Scrapy in colab?