How to Make Selenium Faster: Improving Page Load Performance

Do you ever feel like banging your head against the desk while waiting for a Selenium script to finish running? Page load times slow to a crawl as you watch the browser renderer spin endlessly…

We've all been there. Dealing with a slow Selenium is one of the biggest pain points in test automation.

But don't pull your hair out just yet! There are proven techniques to optimize and speed up Selenium scripts.

Why is Selenium So Darn Slow?

Before we can fix Selenium's sluggishness, we need to understand the root causes behind it.

There are 5 primary reasons why your Selenium scripts feel slow:

1. Loading Unnecessary Page Resources

Modern web pages are packed full of assets – images, scripts, stylesheets, media files, and more. By default, Selenium will load everything without discretion.

This table shows the breakdown of resources on average webpages today (source: Web Almanac).

<table> <tbody> <tr> <td>Images</td> <td>1,026 KB</td> </tr> <tr> <td>JavaScript</td> <td>509 KB</td> </tr> <tr> <td>CSS</td> <td>72 KB</td> </tr> <tr> <td>HTML</td> <td>31 KB</td> </tr> </tbody> </table>

As you can see, images alone make up over 1 MB per page on average.

All of those extra assets result in:

• More network round trips to fetch resources
• Additional bandwidth consumption
• More computations to render the page visually

These add up to significantly longer page load times.

2. Using Inefficient Locator Strategies

The locators you use to find elements on the page have a dramatic impact on speed.

locator selectionXPath selectors require traversing the entire DOM structure. Meanwhile, ID locators instantly pinpoint a single element.

Studies show CSS selectors are approximately 3x faster than XPath (source: Capybara).

Using sub-optimal locator strategies is a common reason for slow element selection in Selenium scripts.

3. Executing Requests Sequentially

Browser automation tasks often involve iterating through a loop to load hundreds or thousands of pages:

for url in urls:
  driver.get(url)
  # scrape page

By default, this will make requests one-by-one. On a site taking 5+ seconds to fully load, it would finish 1 URL per 5 seconds at best.

Running requests asynchronously across multiple browser instances in parallel is far more efficient.

4. Failing to Properly Release Resources

Long running Selenium tests tend to leak memory over time as browsers fail to release resources properly.

The gradual buildup of unused objects degrades performance and causes things to progressively slow down.

5. Getting Blocked by Anti-Bot Measures

Many sites try to detect and throttle bots with things like:

• CAPTCHAs
• reCAPTCHAs
• email / SMS verification
• IP blocking

These present obstacles that hinder automation and introduce delays as Selenium tries to overcome them.

12 Tips to Drastically Speed Up Selenium Page Load Times

Now that you know the common sources of slow Selenium scripts, let's dig into specific techniques to avoid these pitfalls.

Follow these tips to slash page load times and get your browser automation running smoothly again:

1. Block Unnecessary Page Resources

Don't load assets you don't need. Stop images, CSS, fonts, and scripts from being fetched:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", {
  "profile.managed_default_content_settings.images": 2 # block images
})

driver = webdriver.Chrome(options=options)

You can block any resource type. This saves bandwidth, eliminates rendering overhead, and avoids dozens of unnecessary network requests.

2. Use More Efficient Locator Strategies

CSS selectors are approximately 3x faster than XPath expressions (source: Capybara).

ID and name locators are even faster by instantly pinpointing unique elements on the page.

# Fast
driver.find_element(By.ID, 'submit-button')

# Slow 
driver.find_element(By.XPATH, '//button[@id="submit-button"]')

Avoid complex XPath and traversal-heavy CSS selectors when possible.

3. Leverage Parallel Execution

Sequential scripts execute requests one after another. Parallel execution divides up the workload concurrently across multiple browser instances.

Here's an example using Python's threading module:

from threading import Thread

def scrape_page(url):
  driver = webdriver.Chrome()
  driver.get(url)
  # scrape page
  driver.quit() 

threads = []
for url in urls:
  thread = Thread(target=scrape_page, args=[url])
  threads.append(thread)
  thread.start() 

for thread in threads:
  thread.join()

This runs each page load asynchronously on a separate thread. Much faster than loading pages one-by-one!

4. Use Headless Browsers

Headless Chrome and Firefox elimininate the overhead of rendering a visible UI. Without a GUI, they use less memory and operate faster.

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True 

driver = webdriver.Chrome(options=options)

For even better performance, run headless browsers on a Linux server rather than locally on your machine.

5. Restart Browsers and Servers Frequently

Having automation scripts run continuously can cause resource leaks over time. Browsers fail to release objects, servers accrue processes, etc.

Restarting fresh periodically provides a clean slate and prevents gradual slowdowns:

import os, time

while True:
  # run selenium script
  time.sleep(3600) # sleep for 1 hour
  os.system("sudo reboot") # reboot server

Aim for restarting every few hours when running long-living Selenium processes.

6. Employ Smart Waits

Avoid naive time.sleep() calls that pause execution arbitrarily.

Instead, use smart waits that pause dynamically until elements are ready:

from selenium.webdriver.support.ui import WebDriverWait

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'submit')))

This waits specifically for the condition you want, rather than forcing an absolute sleep.

7. Recycle Browser Instances

Creating new browser instances is expensive. It's faster to reuse the same browser window:

driver.get('page1.html')
# scrape page 1

driver.get('page2.html')  
# scrape page 2

# etc...
driver.quit()

This avoids the overhead of constructing new browsers repeatedly.

8. Keep Selenium and Drivers Updated

Use the latest versions of Selenium and the browser drivers. New releases contain speed enhancements and performance fixes.

For example, Selenium 4 reduced webdriver initialization time by up to 75% compared to v3 (source: Selenium 4.0).

9. Tweak Browser Settings

Experiment with browser capabilities and prefs to enable faster page loads:

capabilities = webdriver.DesiredCapabilities.CHROME.copy()
capabilities['pageLoadStrategy'] = 'eager' # load all resources in parallel

Tweaking these low-level knobs can produce big optimizations.

10. Disable Browser Extensions

Extensions add overhead that can negatively impact page load times.

Launch Chrome using a clean profile with addons disabled:

from selenium.webdriver.chrome.options import Options

options = Options()  
options.add_argument("--disable-extensions")

driver = webdriver.Chrome(options=options)

Disable other browser features like pop-up blockers too.

11. Limit WebDriver Scope

The more pages and actions performed in each WebDriver instance, the more memory it consumes.

Close browsers promptly once their work is done to free up system resources:

driver = webdriver.Chrome()

driver.get(url1) 
# work on page 1
driver.quit() # close browser

driver = webdriver.Chrome() 

driver.get(url2)
# work on page 2
driver.quit() # close browser

This limits how much memory any given browser session can accumulate.

12. Upgrade System Resources

If all else fails, throw more computing power at it!

Upgrading to a system with:

• More CPU cores for true parallelism
• Increased RAM capacity
• SSD rather than HDD drives
• More bandwidth to handle requests

…will universally speed up Selenium scripts. But it ain't cheap!

Takeaways to Speed Up Your Selenium Scripts

Dealing with slow page loads in Selenium can make you want to tear your hair out strand-by-strand.

But there are solutions! Here are the key takeaways:

• Block unnecessary resources – Only load page assets you actually need.
• Use efficient locators – Favor ID, name, and simple CSS over complex XPaths.
• Execute in parallel – Open multiple browser instances concurrently.
• Employ headless browsing – Eliminate the performance drag of GUI rendering.
• Restart often – Prevent gradual memory leaks and bloat.
• Leverage smart waits – Avoid naive hard-coded sleeps.
• Recycle browsers – Don't create new ones unnecessarily.
• Keep Selenium updated – Take advantage of speed improvements in new versions.
• Simplify scripts – Don't make browsers work harder than they need to.

Apply these tips to overcome common bottlenecks. Take control of your test automation by finally taming Selenium's slow speeds.

And if it's still too slow even after optimization, don't shy away from external services that handle the browser automation for you. They provide an easy path to faster script execution.

No more waiting around endlessly as Selenium crawls page by page. Take these lessons and soon you'll be scraping dynamic pages with lightning speed!