How to Use Rotating Proxies in Python (Requests & Scrapy)

Introduction to Python Web Scraping & Proxies

Python is the most popular language for data extraction, web scraping, and API automation. With libraries like Requests, Scrapy, Selenium, and Playwright, developers can fetch and parse web pages at massive scales.

However, high-frequency scraping commands will quickly result in IP bans, rate limit HTTP status codes (such as 429 Too Many Requests), or captcha screens. To scrape web pages at scale, you must distribute your requests across a massive, clean pool of IP addresses.

Using **Turbo Proxy**'s SOCKS5 and HTTP backconnect gateways with Python allows your scrapers to route queries through 7M+ real residential and mobile connections, bypassing CDN firewalls and bot filters.

Setup Requirements

Make sure you have installed the necessary Python packages. To support SOCKS5 proxies in the standard requests library, you must install the PySocks dependency:

1. Using Proxies with Python Requests

The requests library accepts a proxies dictionary. We can define our authenticated proxy URL for both http and https.

HTTP Proxy Example

Configure HTTP proxy gateways with user authentication:

import requests

# Define credentials
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
PROXY_HOST = "gate.turboproxy.online"
PROXY_PORT = "7000"

# Format the proxy dictionary
proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

try:
    # Query an IP checker website
    response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
    print("Response Status Code:", response.status_code)
    print("Routed IP Address info:", response.json())
except Exception as e:
    print("Connection failed:", e)

SOCKS5 Proxy Example

SOCKS5 is faster and prevents local DNS resolution leaks. To route requests via SOCKS5, prepend the proxy protocol format with socks5h:// (using the 'h' option ensures DNS name resolution is resolved on the proxy side instead of locally):

import requests

# Format the proxy dictionary with socks5h://
proxies = {
    "http": "socks5h://user-zone-us:[email protected]:7000",
    "https": "socks5h://user-zone-us:[email protected]:7000"
}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print("SOCKS5 Routed IP info:", response.json())

2. Integrating Proxies in Scrapy

For large scale crawling, **Scrapy** is the go-to framework. We can configure proxies globally in Scrapy's settings.py or inside a custom middleware.

Setting up Scrapy Middlewares

Define your Turbo Proxy connection within Scrapy's default request metadata in your spider code or middlewares:

# settings.py configuration
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
    'myproject.middlewares.TurboProxyMiddleware': 410,
}

Then, create your custom middleware class to handle basic authentication headers:

import base64

class TurboProxyMiddleware(object):
    def process_request(self, request, spider):
        # Configure host and port
        request.meta['proxy'] = "http://gate.turboproxy.online:7000"
        
        # Add Proxy Authorization Header
        proxy_user_pass = "username:password"
        encoded_user_pass = base64.b64encode(proxy_user_pass.encode('utf-8')).decode('utf-8')
        request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

Best Practices for Programmatic Scraping

• Always set time-out values: Web crawlers can hang indefinitely if a request gets stuck. Always pass a timeout=10 or timeout=15 parameter to your request methods.
• Implement User-Agent Rotation: Even with rotating proxy IPs, sending requests with a default Python header (User-Agent: python-requests/2.X.X) will trigger bot detection. Always rotate headers:

  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
  }
  response = requests.get(url, proxies=proxies, headers=headers)
  

• Handle Retry Codes: If a request receives a 502 Bad Gateway or 503 Service Unavailable error, implement a retry policy with exponential backoff.