cURL is an extremely powerful tool for transferring data over network protocols like HTTP, FTP, SMTP and more. With cURL, you can automate requests, download files, test APIs and much more – all from the command line interface.

In this comprehensive 3,000+ word guide, we will cover installing cURL on Ubuntu 18.04 and then dive deep into some of its most advanced capabilities:

  • Automating complex cURL workflows
  • Uploading files via HTTP
  • SSL Configuration
  • HTTP/2 Support
  • Obscure but useful options
  • Troubleshooting common issues
  • And more!

We will also explore some cURL best practices, see where it excels over alternative tools, and provide numerous code snippets you can incorporate into your own projects.

Let‘s get started!

Installing cURL on Ubuntu

First a quick install recap. cURL comes pre-bundled on most Linux distros, but if needed you can install it with: 

sudo apt update
sudo apt install curl

Verify everything worked:

curl --version 
# curl 7.68.0 (x86_64-pc-linux-gnu)

Next let‘s do a quick GET request test drive before diving into more advanced operations:

curl https://www.example.com

Excellent – your environment is ready start leveraging cURL scripts!

cURL Automation Workflows

While running ad-hoc cURL commands is handy, the tool really shines when applied to automated workflows.

Some example use cases:

Continuous Integration Pipelines: Run API integration tests by scripting cURL. For instance:

curl -i -d ‘{ "key": "value" }‘ https://api.example.com/endpoint

crontab Scheduled Jobs: Check for updated data/reports from external services. Or ping health check endpoints routinely.

Bootstrap Scripts: Programmatically download release artifacts during product deployments.

Data Science: Fetch experiment results from remote servers. Or populate datasets.

Let‘s walk through a sample automation flow…

Say we needed to check for new users added to a SaaS platform each morning. We can setup a cronjob calling a check-users.sh script:

#!/bin/bash

DATE=$(date +%F)
LOG_FILE="users-$DATE.log" 

curl https://api.example.com/users > $LOG_FILE

This fetches the user data to a dated log file allowing analysts to lookup changes. We could enhance with email alerts, Slack notifications, etc.

As you can see, scripting cURL leads to all sorts of automation possibilities!

It tends to be simpler and more portable than compiling custom applications in many cases.

POSTing Data and Uploading Files

So far we have focused on GET requests. But cURL supports other key HTTP verbs like POST, PUT, DELETE.

This allows uploading data directly to APIs.

For instance, to create a new user by POSTing a JSON payload:

curl -d ‘{"name":"John Smith", "email":"john@example.com"}‘ -H "Content-Type: application/json" -X POST https://api.example.com/users

Pretty slick!

You can also upload files using the --data-binary parameter instead of -d: 

curl --data-binary @myfile.zip -X POST https://api.example.com/upload

This encodes the local file contents directly into the request body.

Or you could read from stdin by piping like:

cat myfile.zip | curl -X POST https://api.example.com/upload

Uploading via cURL may seem simplistic, but it gets the job done without heavyweight dependencies.

SSL Certificates Configuration

When connecting to HTTPS sites, cURL needs to verify the server‘s SSL certificate is valid and trusted.

By default, it uses the certificate bundle provided by your Linux distro (e.g. Ubuntu root certs).

You can override and provide your own certificates if needed:

curl --cacert /path/to/ca-bundle.pem https://example.com

Alternatively, you can disable SSL verification but this is insecure: 

curl -k https://example.com

Other SSL options like client certificates are available too:

curl --cert client.pem --key key.pem https://site.com

See man curl for additional capabilities.

Warning: Disabling SSL certificate validation compromises security and should only be done for testing purposes on non-sensitive connections!

Getting SSL right is finicky – thankfully the cURL docs cover these scenarios in-depth.

HTTP/2 Support

Did you know cURL supports the next-gen HTTP/2 protocol?

By default, cURL still uses HTTP/1.1 for compatibility‘s sake.

But if a server is configured for HTTP/2, you can leverage it for performance gains:

curl --http2 https://nghttp2.org

The HTTP/2 spec aims to solve several bottlenecks around latency and bandwidth utilization compared to HTTP/1.1.

Benefits include:

  • Fully multiplexed connections avoid blocking on responses
  • Header compression reduces overhead
  • Server push capabilities

So while not widely adopted yet, it is useful to know cURL speaks HTTP/2 as more APIs start leveraging it.

Obscure but Handy cURL Options

cURL has support for an enormous number of parameters and use cases.

We already touched on the big ones like -O, --progress, etc.

Let‘s highlight some unsung options I personally rely on for simplified scripting:

Retry Downloads (-C -)

We mentioned built-in retry handling for partial downloads before. What if connectivity dies mid-transfer?

Just rerun the same command and cURL will attempt resume downloading thanks to -C -:

# Initial failed attempt
$ curl -O https://example.com/bigfile

# Retry handling 
$ curl -C - -O https://example.com/big_file

This works where tools like wget often fail.

Follow Redirects (-L)

By default, cURL does NOT follow HTTP redirects from 3xx responses.

Add -L to recursively follow Location headers: 

$ curl -L https://example.com

Saving a bit of logic handling chains of temporary redirects.

GET If Modified Since (-z)

Want a simpler alternative to REST API timestamp queries?

Use conditional GETs via the If-Modified-Since header!

$ curl -z "Sat, 29 Oct 2022 19:43:31 GMT" https://example.com

The server will return the requested content only if it changed since the passed date. Handy for keeping local copies in sync.

There are 100s more useful options like this to explore via curl --help.

Debugging Issues with cURL

While cURL is generally highly reliable, you may occasional run into problems.

Let‘s discuss how to diagnose some common failure scenarios:

Connection Errors

Can‘t seem to reach a site? Double check for DNS or connection timeouts:

$ curl -v https://invalidsite
* Rebuilt URL to: https://invalidsite/
* Could not resolve host: invalidsite
* Closing connection 0
curl: (6) Could not resolve host: invalidsite

If it works via browser but not cURL, may be a TLS or firewall issue.

Certificate Issues

SSL handshakes involve multiple steps, so failures may not be obvious at first glance.

If seeing errors establishing trust, inspect closely:

$ curl -v https://expiredcert.com
* Rebuilt URL to: https://expiredcert.com/
* Trying 93.184.216.34...
* TCP_NODELAY set
* Connected to expiredcert.com (93.184.216.34) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

Pay attention to handshake failures and error reasons.

Authorization Failures

Apis normally respond with 4xx client errors on auth problems:

$ curl -i https://api.example.com/private
HTTP/1.1 401 Unauthorized

{"error": "Authentication credentials invalid"}

If you expect 200 OK, double check API keys/secrets match or required headers are sent.

Hopefully these give you ideas on reducing headaches!

For much more comprehensive troubleshooting advice, the cURL debugging bible is a must-read resource.

Introducing the All Powerful curl.sh Script

To simplify running cURL repeatedly while handling inputs/outputs, auth, etc. it is advisable to wrap functionality into a reusable script.

Here is an example curl.sh shell wrapper that:

  • Accepts the Request URL as a parameter
  • Saves response to a dated log file
  • Authenticates via API token header
  • Validates SSL certificates
  • Provides request diagnostics 
#!/bin/bash

URL="$1"
DATE=$(date +%F)
TOKEN="fde4d11a-9d11-4999-9ea4-9d1afd9d1114" 

LOG="logs/resp-${DATE}.txt"

curl -w "\nRequest Diagnostics\nHTTP Code: %{http_code}\nTotal Time: %{time_total} s\nSize Downloaded: %{size_download} bytes\nAverage Speed: %{speed_download} bytes/s\n" \
  --header "Authorization: Bearer ${TOKEN}" \
  --cacert ca-bundle.pem \
  -o "${LOG}" \
  "$URL"

You could enhance to parse special codes, handle failures gracefully, output formats, etc.

But this shows how wrapping cURL leads to easy customization.

Now we can just invoke ./curl.sh https://api.example.com/data without worrying about lower-level details each run.

cURL vs Alternatives

We have established cURL is a versatile tool for transferring data programmatically. But how does it compare to alternatives?

Let‘s contrast it to a few popular options:

Feature cURL Postman HTTPie Selenium
CLI usage ✅ ✅
Graphical interface ✅ ✅
Browser automation ✅
File uploads ✅ ✅
Authentication helpers ✅
Syntax terseness ✅
Extensibility ✅ ✅ ✅
Interoperability ✅ ✅
Widespread availability ✅

As you can see, cURL excels for scripted uses from the terminal or CI/CD systems. It also has great support for legacy platforms.

The tradeoff is less hand-holding and helpers compared to visual tools like Postman and Selenium.

In my experience, cURL hits the sweet spot nicely between simplicity, power and portability.

But there are cases trading some capabilities for more user-friendly interfaces and workflows is worthwhile.

Hopefully this comparison helps frame strengths of each approach.

Expanding Your cURL Knowledge

We have really just scratched the surface of cURL‘s immense set of options and use cases.

Here are a few parting resources if you would like to continue mastering cURL scripting:

Let me know what custom cURL tricks you end up incorporating into your own projects!

Summary

That wraps up our deep dive into maximizing cURL for advanced HTTP workflows. Here are some key highlights:

  • Automate all kinds of tasks with simple cURL scripting 🚀
  • Master file uploads, SSL config, HTTP/2 and more 🔧
  • Debug issues quicker thanks to built-in diagnostics 🐞
  • Simplify common operations by wrapping with helpers ✨
  • Achieve things not easily possible through browsers alone 🖥

Hopefully you now feel empowered to unleash cURL across projects old and new no matter their scope and scale. Happy scripting! 🙌

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *