How to Handle the User Agent in Node.js for Web Scraping

The user agent string identifies the software making HTTP requests to servers. When web scraping, setting the proper user agent is crucial to avoiding blocking. In this comprehensive guide, you'll learn how to configure the user agent in Node.js using the Axios library.

We'll cover:

• What the user agent is
• Viewing your default user agent
• Setting a custom user agent
• Implementing user agent rotation
• Challenges of web scraping at scale
• Anti-scraping techniques sites use
• BrightData for easy scraping with auto user agent handling

What is the User Agent in Node.js?

The user agent header is sent with all HTTP requests made by client software like browsers and scraping tools built with Node.js.

It identifies information about the software to servers. For example:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36

This reveals it's Chrome browser version 74 on Windows 10 64-bit. Sites can use this to detect bots making too many requests.

In Node.js, libraries like Axios or Selenium WebDriver are used to make HTTP requests for web scraping. Next, let's use Axios to view the default user agent.

Viewing Your Default User Agent

First, install Axios:

npm install axios

Then we'll make a request to https://httpbin.org/headers and log the user agent.

Create an index.js file and add:

const axios = require('axios');

axios.get('https://httpbin.org/headers')
  .then(response => { 
    const headers = response.data.headers;
    console.log(headers['User-Agent']);
  })
  .catch(error => {
    console.log(error);  
  });

Run it with:

node index.js

This prints your default user agent, which identifies Axios making the requests:

axios/0.21.1

This default user agent is easily detected as a bot, so next let's set a custom one.

Setting a Custom User Agent

To replace the default user agent, we need to pass a custom header.

First, construct a headers object with your new user agent:

const headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36' 
};

Next, make the request while passing the headers as the 2nd parameter:

axios.get('https://httpbin.org/headers', { headers })
  .then(response => {
    //...
  })

Now your scraped websites see requests coming from a real Chrome browser.

However, using the same static user agent for all requests can still get detected. Next let's learn how to rotate user agents.

Rotating User Agents

Rotating through different user agents is crucial for avoiding blocking while web scraping. Sending too many requests from what appears as the same browser triggers bot protection systems.

To implement user agent rotation, we can create an array of different user agents:

const userAgents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 
  'Mozilla/5.0 (iPhone; CPU iPhone OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/109.0.0.0 Mobile/15E148 Safari/604.1',
];

Then each request, randomly select one to use:

const randomUserAgent = userAgents[Math.floor(Math.random() * userAgents.length)];
 
const options = {
   headers: {
     'User-Agent': randomUserAgent
   }
}
 
axios.get(url, options);

This makes your traffic appear more human since different user agents are used per request.

Challenges of Web Scraping At Scale

Configuring the user agent properly is the first step, but not the only challenge with web scraping. As you scrape more pages, here are other protections sites use:

• IP Blocking – blocking too many requests from the same IP.
• CAPTCHAs – requiring manual human verification.
• JavaScript Rendering – sites rely heavily on JavaScript to load page data.

And those are just a few common challenges. Dealing with all these requires significant development work and infrastructure.

Luckily, there are easier solutions to handle these instead of building it all yourself…

BrightData – Web Scraping API with Automatic User Agent Rotation

BrightData provides a reliable web scraping API that handles rotating user agents automatically under the hood.

Rather than worrying about low-level challenges like bot prevention and infrastructure, you can focus on extracting the data you need.

Some key features:

• Pre-Rotated User Agents – Over 17 million user agents rotated per request.
• Proxy Rotation – Global proxy pool rotated each request to prevent IP blocks.
• JavaScript Rendering – Supports full page rendering.
• Captcha Solving – Bypasses CAPTCHAs automatically for high uptime.

Let's see how easy it is to start scraping with auto user agent handling using their API.

Automatic User Agent Rotation with BrightData

Then navigate to the Request Builder page and make a GET request with the generated code:

Let's send requests to https://httpbin.io/user-agent to view the changing user agents:

As you can see, each request uses a different user agent from BrightData's pool of millions!

No need to handle rotating user agents manually anymore. Their infrastructure handles it automatically in the background.

And that's just one of the many scraping challenges BrightData deals with for you, including:

• Proxies
• JavaScript Rendering
• CAPTCHAs
• Infrastructure

This frees you to focus on extracting and transforming the data you need.

Conclusion

Configuring the user agent properly is the first step for evading bot detection while web scraping. By mimicking real browser user agents, sites find it harder to distinguish your scraped traffic.

However handling user agents is just the start – web scrapers also face many other challenges like proxies, captchas, and more.

Tools like BrightData handle all of these tedious scraping challenges automatically:

• User agents rotated per request
• Global proxies rotated each request
• Full JavaScript rendering
• Automatic captcha solving

With an API like BrightData, you can skip handling the tedious scraping challenges and infrastructure. This leaves you free to focus on your data extraction use case.