How to Web Scraping with Selenium and Python

Web scraping is the process of extracting data from websites. As the web evolves, so do the techniques required to scrape information from it. JavaScript-heavy sites that render content dynamically are becoming increasingly common, so you need advanced tools like Selenium to scrape them.

This comprehensive guide will teach you how to leverage Selenium for web scraping in Python step-by-step.

Why Use Selenium for Web Scraping

Selenium is a popular open-source tool that allows controlling browsers through code. Here are some key advantages of using it for web scraping:

• Headless browser capabilities: Selenium can run browsers in headless mode. This means no graphical interface, ideal for server scraping.
• JavaScript execution: Sites relying on JS are scraped seamlessly. The browser engine runs all scripts.
• Bypass anti-bot protections: Humans use browsers. So using one helps masquerade your scraper as a real user.
• Interact with pages: Click buttons, scroll, fill forms… Just like a person would do!
• Access advanced browser features: Cookies, caching, proxies, user-agents… Useful to avoid blocks.

This makes Selenium one of the best tools for scraping complicated sites. Let's see how to set it up for Python web scraping.

Getting Started with Selenium in Python

These are the prerequisites to follow this Selenium Python tutorial:

• Python 3.x installed. Verify with python --version.
• PIP to install packages. Comes by default with Python.
• Chrome web browser. The most popular option.
• ChromeDriver: Matches your specific Chrome version.

First, create a virtual environment and install the Selenium Python package there:

python -m venv selenium_venv
source selenium_venv/bin/activate
pip install selenium

Now create a Python file called scraper.py and add the following code:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://scrapingclub.com/')

# scraping logic...

driver.quit()

This initializes a ChromeWebDriver instance that gets redirected to ScrapingClub. Let's understand it better.

How the Selenium WebDriver Works

The Selenium WebDriver is the main interface to control a browser. It exposes methods to:

• Navigate to pages.
• Locate elements on a page.
• Interact with elements by clicking, entering text, etc.
• Execute JavaScript code in the browser.
• Capture screenshots of the page or specific elements.

The most popular browser driver is ChromeDriver, which allows controlling Google Chrome. But WebDriver offers support for all major browsers like Firefox, Edge, Safari, etc.

When you call webdriver.Chrome(), it starts a Chrome process controlled by WebDriver. This automated browser doesn't open any windows.

To test it, run your script:

python scraper.py

You'll notice that Chrome opens briefly and then closes immediately. Let's keep the browser open to see it in action.

Headless Chrome Configuration

The headless browser mode allows running Chrome without a GUI. To enable it, pass a chrome_options argument to ChromeDriver:

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)

Now Chrome won't pop up when running the script. The headless configuration is ideal for web scraping servers.

Locating Page Elements for Scraping

To extract data from a page, you need to identify the HTML elements that contain it.

Selenium offers two main methods for that:

• find_element(): Returns one WebElement.
• find_elements(): Returns multiple WebElements in a list.

You can pass different element location strategies to them:

Method	Description
XPATH	Locates by XPath expression
CSS_SELECTOR	Locates by CSS selector
CLASS_NAME	Locates by HTML class name
TAG_NAME	Locates by HTML tag name
LINK_TEXT	Locates anchor elements by link text
PARTIAL_LINK_TEXT	Locates anchors by partial link text

For example, to find an element with XPath:

element = driver.find_element(By.XPATH, '//h1')

I recommend using XPath or CSS selectors, as they allow identifying any element on a page.

Using Browser DevTools to Craft Selectors

The best way to create selectors for scraping is through the browser's developer tools:

Right click any element, choose Inspect, and analyze its HTML in the Elements panel.

You can also right-click an element and select Copy > Copy XPath to get its XPath. This is very useful!

Try it on ScrapingClub to learn how it works.

Interacting with Page Elements

The WebElement objects returned by location methods allow interacting with DOM elements as a user would.

Some common interactions include:

• Entering text: Send text to inputs with element.send_keys('text')
• Clicking: Click buttons or links with element.click()
• Scraping text: Get element text with element.text
• Getting attributes: Extract attributes like src with element.get_attribute('attribute')

Let's see an example that interacts with a login form:

username = driver.find_element(By.ID, 'username')
password = driver.find_element(By.ID, 'password')
login_btn = driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]')

username.send_keys('john')
password.send_keys('1234') 
login_btn.click()

This allows logging into pages programmatically, which is very useful for scraping sites protected behind a login wall.

Waiting For Elements to Load

Modern sites rely heavily on JavaScript to load content dynamically.

Elements rendered by JavaScript won't exist immediately after loading the page. You have to wait for the browser to execute the JS code first.

Here are two ways to wait in Selenium:

Time.sleep()

This pauses execution for a given number of seconds:

time.sleep(5)
# Wait for 5 seconds

Simple but inefficient. You have to guess how long to wait.

Implicit Waits

An implicit wait tells WebDriver to wait up to a certain number of seconds when finding elements:

driver.implicitly_wait(10) 
# Wait up to 10 seconds

This way, Selenium waits as needed before throwing an error if the search fails. You don't have to sleep for a fixed time.

The drawback is that it waits for all element searches. If that's not necessary, explicit waits work better.

Explicit Waits

An explicit wait makes WebDriver wait until a condition is met before proceeding.

For example, wait until an element contains specific text:

WebDriverWait(driver, 10).until(EC.text_to_be_present_in_element(element, 'Text'))

Some other expected conditions you can use:

• title_contains(): Title contains text
• presence_of_element_located(): Element is present
• element_to_be_clickable(): Element is clickable

Explicit waits are the most flexible and reliable option.

Executing JavaScript in the Browser

Since Selenium controls a real browser, you can execute JavaScript code directly:

driver.execute_script('alert("Hello World");')

This opens an alert popup with “Hello World”.

Why is JS execution useful?

• Get page data not exposed through Selenium APIs.
• Scroll elements into view before interacting with them.
• Scroll pages with infinite scrolling.
• Bypass anti-scraping protections.

Some examples of useful JS scripts:

• Get title: return document.title;
• Scroll page: window.scrollTo(0, Y)
• Get inner HTML: return document.body.innerHTML

JavaScript support makes Selenium extremely powerful for scraping complicated sites.

Taking Screenshots

Selenium allows taking screenshots of web pages:

• Screenshot of full page:

driver.save_screenshot('page.png')

• Screenshot of a single element:

element = driver.find_element(By.ID, 'myElement')
element.screenshot('element.png')

This is handy for:

• Debugging your scraper and visualizing the effects of actions.
• Collecting graphical data from sites.

Configuring Browser Settings

Here are some browser configurations that are useful for web scraping:

• Headless mode: Launches browser without GUI.
• User agent: Masquerade as a specific device or browser.
• Window size: Emulate different devices with different screen sizes.
• Mobile mode: Make Selenium think it's a mobile browser.
• Disable images/JavaScript: Speed up page loads.
• Custom headers: Change request headers like cookies.
• Proxy: Mask scraper IP and bypass IP blocks.

For example, to set a custom user agent:

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('user-agent=CustomAgent')

driver = webdriver.Chrome(options=options)

These capabilities are what make Selenium so powerful compared to simple HTTP requests or parsers. You have the full functionality of a browser.

Avoiding Bot Detection and Blocks

Websites don't want their data scraped. So they implement anti-bot protections to detect and block scraping bots.

Some ways sites try to stop scrapers:

• Analyzing request patterns.
• Checking common scraping user-agents.
• Fingerprinting browsers.
• Honeypot traps.
• Requiring JS or cookies.
• CAPTCHAs.
• IP blocking.

Since Selenium mimics a real browser, it can bypass many of these protections. However, sites are getting smarter at fingerprinting browser automation tools like Selenium.

Here are some tips to avoid blocks:

• Use a headless browser but configure it to mimic a real one accurately.
• Set a custom realistic user agent string.
• Use proxies and rotate IPs frequently.
• Add random delays between page visits.
• Disable images and JS if not needed.
• Scroll pages and click elements to mimic human behavior.

The best way to avoid issues is to act like a real user as much as possible.

Speeding Up Web Scraping with Selenium

Although Selenium is very powerful, it can also be slow and resource intensive. Here are some tricks to improve performance:

• Enable headless mode to avoid rendering the browser UI.
• Disable images, CSS, fonts, and JS if not required.
• Use explicit waits instead of time.sleep() to prevent unnecessary delays.
• Close browser windows you aren't using anymore to release resources.
• Limit DOM access by storing scraped elements instead of re-querying.
• Parallelize operations by running multiple browser instances.
• Throttle page visits to avoid flooding servers.
• Use caching to avoid repeating expensive operations.

With a well-optimized scraper, you can extract thousands of items per hour.

Scraping JavaScript Sites with Selenium

One of the biggest advantages of Selenium is scraping pages that heavily rely on JavaScript.

Let's see how to scrape two common JS patterns:

Infinite Scroll

Many sites use infinite scrolling to load content continuously as the user scrolls down.

To scrape all pages, simulate scrolling with Selenium:

last_height = driver.execute_script('return document.body.scrollHeight')

while True:
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')
    
    time.sleep(2)

    new_height = driver.execute_script('return document.body.scrollHeight')

    if new_height == last_height:
        break
    
    last_height = new_height

# Extract data from new elements

This scrolls to the bottom of the page until no more content loads.

Content Loaded by XHR Requests

Modern sites use AJAX requests to update content.

For example, clicking a button sends a request that injects new data into the DOM.

To scrape this, simulate clicks and wait for the XHR response before extracting information:

load_btn = driver.find_element(By.ID, 'loadBtn')
load_btn.click() 

WebDriverWait(driver, 10).until(AJAX_COMPLETED) 

# Extract data from new elements

There are many other patterns like pagination, single-page apps, etc. that you'll commonly encounter.

The key is to reverse engineer how the site works and then reproduce user actions with Selenium.

Scraping Techniques to Avoid Getting Blocked

Even with Selenium, some sites are tricky to scrape due to advanced anti-bot methods.

Here are some tips to deal with difficult sites:

• Use proxies – Rotate IPs to prevent blocks.
• Randomize delays – Don't scrape too fast to appear human.
• Fake user actions – Scroll, hover, click, etc. to seem real.
• Solve CAPTCHAs – Decrypt automatic challenges when encountered.
• Monitor blocks – Check for 403 errors and solve captchas.

However, these techniques can get very complex. The easiest solution? Use a web scraping API.

For example, ScrapingBee handles CAPTCHAs, proxies, and blocks automatically so you can scrape without hassles.

APIs abstract away these challenges and let you focus on extracting data.

Advanced Usage with Selenium Wire

Selenium Wire is a nifty Python package that extends Selenium's capabilities.

It allows you to do things like:

• Mock API calls made by the browser.
• Inspect requests and override headers.
• Set up proxies for scraping.
• Block URLs and resource types (images, media, etc.).
• Modify response bodies before they reach the browser.

This makes it possible to bypass protections that would be difficult with regular Selenium.

For example, here's how to use it to intercept requests:

from seleniumwire import webdriver 

options = webdriver.ChromeOptions()
options.headless = True

# Enable request interception 
driver = webdriver.Chrome(options=options, seleniumwire_options=opts)

def interceptor(request):
    if request.url.endswith('track'):
        # Block tracking requests
        request.abort()

driver.request_interceptor = interceptor

driver.get("http://www.example.com")

This allows blocking any requests containing track in the URL. Powerful!

Selenium Wire makes your scrapers extremely versatile.

Wrapping Up

Let's summarize the key takeaways about web scraping with Selenium:

• It launches a real browser that can execute JS, handle cookies, etc. This allows scraping complex sites.
• The WebDriver API exposes methods to interact with page elements just like a user.
• Locating elements efficiently is critical to build a reliable scraper. Use browser DevTools to craft CSS and XPath selectors.
• Selenium offers many configurations to mimic browsers. This helps bypass anti-scraping systems.
• Key challenges are dealing with dynamic page content, avoiding detection, and managing resources.
• For ultimate scraping power and performance, use it with tools like Selenium Wire and ScrapingBee.

Web scraping with Selenium requires some effort. But the data extraction possibilities are endless if you master it.

This guide should have provided you with a comprehensive overview of using Selenium for web scraping in Python. The next step is to starting scraping some sites!