How to Retry Requests Using Axios [2023]

HTTP requests can temporarily fail, so it's essential to handle those to ensure reliability. In this comprehensive tutorial, you'll learn how to implement the Axios retry feature using Bright Data Proxy: which failed requests can be retried, how different retry mechanisms work, as well as the best strategies to use with examples.

What You Need to Know to Build Axios Retry Logic

There are some common questions that come to mind when considering building an Axios retry logic for your web scraper in NodeJS. For example:

  • Should I always retry failed requests or just in certain cases?
  • When should I retry failed requests?
  • How many retry attempts should I make?

We'll answer these key questions in this section to set the foundation for building a robust Axios retry logic.

Types of Failed Requests

When a request fails, there are generally two scenarios:

  1. Timeouts with no response from the server
  2. Errors returned from the server

Let's explore both scenarios to better understand the available strategies for retrying with Axios.

Timeouts

Issues like overloaded servers or unstable internet connectivity can result in delayed or no responses from a server, leading to timeouts.

In case of a timeout, first double check that your own internet connection is stable.

If your internet connectivity checks out, the issue likely lies with the server you're requesting from. To prevent unnecessarily long wait times, it's best practice to set a timeout in milliseconds to abort the request using Axios' timeout configuration option.

// Set a 15 second (15000ms) timeout
axios.get('https://example.com', {
  timeout: 15000 
})
.then(response => {
  // Handle response
})
.catch(error => {
  // Handle error  
});

To detect a timeout, watch for timeout related errors with the ECONNABORTED status code and implement an Axios retry mechanism conditionally or using backoff strategies (more later).

Here's how to handle timeouts with Axios:

axios.get('https://example.com', {
  // 10ms timeout 
  timeout: 10  
})
.catch(error => {

  // Check if timeout error
  if (error.code === 'ECONNABORTED') {
    
    // Implement retry logic here
    
  }

});

Returned an Error

A standard error response for a failed request contains a status code and descriptive error message:

  • Status code: Indicates what went wrong
  • Error message: Provides context on why the request failed

To address these cases, first check if the response contains an error. Next, inspect the status code and message to understand the failure reason. You can then retry with Axios if the error suggests a temporary server issue.

Relevant Error Status Codes

The relevant client-server communication error status code ranges are 4xx and 5xx.

The most common ones to retry with Axios when web scraping include:

Status Code Meaning
403 Forbidden Server understands request but won't fulfill due to lack of permissions
429 Too Many Requests Rate limiting – too many requests from client's IP
500 Internal Server Error Generic error occurred processing request on server
502 Bad Gateway Invalid response received from upstream server
503 Service Unavailable Server overloaded or undergoing maintenance
504 Gateway Timeout Upstream server took too long to respond

See MDN docs for more on HTTP status codes.

Number of Retries

It's wise to set an Axios retry limit to prevent performance issues and blocking from endless retries.

Also consider the error type, desired response time and other factors to determine the optimal retry count cap for a failed request.

For temporary issues like rate limiting errors, more retries may be suitable with additional measures like rotating IPs. Though there's no singular best retry count, 3-5 retries are reasonable starting points.

Delay Between Retries

Introducing delays between retries helps prevent overloading servers and APIs which can trigger rate limiting protections. Next we'll explore types of delays and how to implement them.

Fixed or Random Delays

The JavaScript setTimeout() method is useful for setting a fixed delay between retries. For random delays, combine setTimeout() and Math.random().

There's no universally ideal retry delay length. But reasonable values typically range from 300ms onwards. Experiment to find optimal delays for your system.

Exponential Backoff

The exponential backoff strategy steadily increases delays between retries instead of fixed intervals. The backoff is calculated by:

backoff_delay = backoff_factor * (2 ** (number_of_retries - 1))

For example, with a backoff factor of 1000ms:

Retry 1: 1000ms 
Retry 2: 2000ms
Retry 3: 4000ms

This strategy allows temporary issues a chance to resolve while controlling server load.

Tutorial: Retrying Axios Requests with Interceptors

The axios-retry plugin provides interceptors to easily retry failed Axios requests. It handles retries automatically based on configuration options.

Start by installing axios-retry:

npm install axios-retry

1. Retry on Error with Axios

Import the axiosRetry interceptor and pass the Axios instance and optional configuration object:

import axiosRetry from 'axios-retry';
import axios from 'axios';

// Configure axios retries
axiosRetry(axios, {
  retries: 3 
});

axios.get('https://example.com') // Will retry if fails
  .then(/* ... */) 
  .catch(/* ... */);

By default, axios-retry retries failed GET, HEAD, OPTIONS and idempotent requests with 5xx status codes. To expand the retry rules, use the retryCondition option:

axiosRetry(axios, {
  
  retries: 3,

  retryCondition(error) {
    
    // Retry on 404 or 429 status codes
    if (error.response.status === 404 || error.response.status === 429) {
      return true;
    }

    return false;

  }  
});

2. Implement Exponential Backoff

Use the exponentialDelay method for an exponential backoff retry delay:

import { exponentialDelay } from 'axios-retry';

axiosRetry(axios, {

  retries: 3, 
  
  retryDelay: exponentialDelay 
});

The default backoff factor is 100ms. To configure, add the factor as a parameter:

axiosRetry(axios, {

  retries: 3, 
  
  retryDelay: () => exponentialDelay(1000) // Backoff of 1 second

});

3. axios-retry Options

Name Description Type Default
retries Number of retries before failing Number 3
retryCondition Determines if request should retry Function Retry on 5xx errors
shouldResetTimeout Resets timeout before retries Boolean false
retryDelay Controls delay between retries in ms Function No delay
onRetry Triggered before each retry Function

Avoid Getting Blocked with Bright Data Proxy

A common pitfall when crawling is getting blocked by bot protections when your requests seem automated.

Let's demonstrate by scraping a G2.com page:

axios.get('https://www.g2.com/products/notion/reviews')
.then(response => {
  console.log(response.data);  
})
.catch(error => {
  console.log(error); 
});

This gives a 403 Forbidden error:

{
 "error": "Access denied", 
 "code": 403 
}

Proxies help avoid blocks, but aren't enough for advanced bot detection systems.

Web scraping APIs like Bright Data simplify avoiding blocks through easy to use APIs that handle proxies, browsers, and bot protections for you.

Bright Data's data extraction API has useful anti-bot features like automatic proxy rotation, browser emulation, and real-device crawling.

Get started with some free Bright Data requests from your dashboard, then navigate to Integrations > SDKs to get sample code for Node.js and Axios like below:

const { Bright } = require('@brightdata/bright');
const axios = require('axios');

const bright = new Bright('<YOUR_BRIGHT_DATA_API_KEY>');

bright.inject({ axiosInstance: axios });

const { data } = await axios.get('https://www.g2.com/products/notion/reviews');

This seamlessly proxies Axios requests through Bright Data to avoid blocks.

Note: Replace <YOUR_BRIGHT_DATA_API_KEY> with your actual API key.

Bright Data Axios Proxy

Under the hood, Bright injects a proxy agent and user agent into Axios for bot protection and real browser mimicking:

const proxyAgent = BrightDataProxyAgent(); 

axios.get('https://example.com', {
  
  httpsAgent: proxyAgent,
  
  headers: {
    'User-Agent': bright.userAgent 
  }
  
}).then(/* ... */)

With the Axios retry interceptor, you can seamlessly retry through fresh Bright Data proxies and IPs on each request.

Conclusion

Handling failed requests is vital for resilient web scraping. In this guide, we covered:

  • Common failed request scenarios
  • When and how to retry requests with Axios
  • Configuring exponential backoff
  • Avoiding bot blocks with Bright Data proxy rotation

Bright Data simplifies proxy management through easy APIs. Sign up and integrate it along with Axios interceptors to dodge blocks and build reliable scrapers.

Frequently Asked Questions

Does Axios have built-in retry capabilities?

No, Axios doesn't have default retries, but libraries like axios-retry add them.

How do I retry failed Axios requests?

Use axios-retry, configure it with options like retry count, then make requests as usual. The interceptor will handle retries on failures.

How can I increase the request timeout limit in Axios?

Use the retryDelay option in axios-retry to implement an increasing timeout, such as incrementing the delay by 1 second per retry.

What's the easiest way to avoid getting blocked while scraping?

Web scraping APIs like Bright Data handle proxy rotation, browsers, and bot mitigation automatically through simple REST APIs, removing lots of complexity.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *