How to Use a Proxy with Axios in 2023

Axios is a powerful JavaScript library for making HTTP requests. With just a few lines of code, developers can send GET requests to a web page and retrieve its HTML content in various formats, including JSON and CSV.

However, many websites block bot traffic, which can hinder your Axios web scraper. Fortunately, combining Axios with a proxy to route your requests can help circumvent website detection.

This tutorial will show you how to do that using real-world examples. We'll explore free and paid proxies and some handy methods to avoid getting blocked.

Ready? Let's get started!

Prerequisites

Axios is a JavaScript library, so we'll be web scraping using NodeJS.

To get started, make sure you have NodeJS and npm installed. Then, create a new folder for your JavaScript file and initialize a new project using the following command:

mkdir scrapeaxios
cd scrapeaxios
npm init -y

This creates a package.json file with the default values.

Next, install Axios and its dependencies:

npm i axios

Now we're ready to make requests with Axios!

How to Use Proxies with Axios

Let's look at an example of setting a proxy in Axios using HTTPBin as a target page and an IP from Free Proxy List:

IP: '149.129.239.170', 
port: 8080

First, import Axios and add the proxy details:

const axios = require('axios');

axios.get('https://httpbin.org/ip', {
  proxy: {
    protocol: 'http',
    host: '149.129.239.170', 
    port: 8080,
  },
})
.then(res => {
  console.log(res.data)
})
.catch(err => console.error(err))

Since the httpbin endpoint returns the public IP address of the requesting client, your code should return your proxy's IP address.

Run the code, and your result should look like this:

{ "origin": "149.129.239.170" }

Great! Now let's look at parsing the response as JSON, which often comes in handy:

const axios = require('axios');

axios.get('https://httpbin.org/ip', {
  proxy: {
    protocol: 'http',
    host: '149.129.239.170',
    port: 8080,
  },
})
.then(res => {
  const data = JSON.parse(res.data);
  console.log(data);
}) 
.catch(err => console.error(err))

The code above parses the response data into a JSON object so you can access it more easily in your script.

Proxy Authentication with Axios

Paid proxies often require authentication with a username and password. Axios offers an auth property that lets you pass them in to use your proxy server:

auth: {
  username: '',
  password: '' 
}

Here is how to add the auth property to your script:

const axios = require('axios');

axios.get('https://httpbin.org/ip', {
  proxy: {
    protocol: 'http',
    host: 'proxy_host',
    port: portNumber,
    auth: {
      username: '',
      password: ''  
    },
  },
})
.then(res => {
  console.log(res.data);
})
.catch(err => console.error(err))

Replace the username and password with your actual credentials.

Environment Variables for an Axios Proxy

Environment variables allow you to specify proxy settings and credentials without passing them directly in the Axios request.

To use them, first define the proxy details in your environment variables. For example:

export HTTP_PROXY=http://<Proxy_host>:<port>
export HTTPS_PROXY=https://<Proxy_host>:<port>

Then make your Axios request normally, and it will pick up the environment variable values automatically:

const axios = require('axios');

axios.get('https://httpbin.org/ip')  
.then(res => {
  console.log(res.data); 
})
.catch(err => console.error(err))

This approach keeps your credentials secure and separate from your code.

Use a Rotating Proxy with Axios

Websites can detect and block requests from specific IP addresses. To avoid this, you need to distribute requests across multiple proxy IPs.

Here are some options for creating a proxy rotator with Axios:

Manually Switching Proxies

You can manually switch proxies after a number of requests, for example:

// Array of proxies 
const proxies = [
  {ip: '123.45.67.89', port: 8080},
  {ip: '98.76.54.32', port: 8080}
];

// Counter to track requests
let counter = 0; 

// Make request
axios.get(url, {proxy: proxies[0]})

// Increment counter and switch proxy if limit reached 
if(++counter >= 10){
  counter = 0;
  // Reset index to switch to next proxy
  proxies.push(proxies.shift()) 
}

This isn't very efficient, but works for smaller projects.

Automate Proxy Rotation

For larger projects, automate the proxy rotation process. The key is to shift the proxies in a queue, so each request gets the next available proxy.

// Array of proxies
const proxies = [
  {ip: '123.45.67.89', port: 8080},
  {ip: '98.76.54.32', port: 8080}  
];

// Function to rotate proxies
const getProxy = () => {
  // Pop first proxy from queue
  const proxy = proxies.shift();
  
  // Add proxy back to end of queue 
  proxies.push(proxy);
  
  // Return current proxy
  return { 
    host: proxy.ip,
    port: proxy.port
  }
}

// Make request using rotating proxy
axios.get(url, {proxy: getProxy()})

This ensures each request uses the next proxy in queue.

Use a Proxy Pool Service

For best results, use a paid proxy solution like Bright Data which provides reliable residential IPs and takes care of rotating proxies for you automatically.

Bright Data offers a pool of 72+ million residential IPs from 195 countries to route your requests through.

To use it with Axios:

1. Sign up and get your account credentials
2. Make a request using the Bright Data hostname, port, and your username/password:

const axios = require('axios');

axios.get(url, {
  proxy: {
    host: 'zproxy.luminati.io',
    port: 22225,
    auth: {
      username: 'customer-<customerId>',  
      password: '<password>'
    }
  }
})  
.then(res => {
  // Your response
})
.catch(err => {
  // Handle errors 
});

The residential IPs will automatically rotate to avoid getting blocked.

You can also target a specific country or city by adding flags like -country US or -city 'New York' to the username.

Conclusion

Using proxies with Axios is crucial to avoid blocks while web scraping. We explored free proxies but those often fail or get detected.

For best results, use a reliable paid proxy solution like Bright Data. It provides rotating residential IPs from around the world to mimic real users.

With just a few lines of code, you can send requests through Bright Data proxies and scrape any website effectively using Axios.