Continuous Integration/Continuous Delivery (or Deployment, depending on your point of view) pipelines are at the core of many successful software projects. When designing your pipelines you sooner or later end up using REST calls to perform certain tasks. cURL is a popular command line tool to invoke REST APIs, and is commonly used in pipelines. Before you start using cURL in that capacity I’d like to draw your attention to a potential pitfall you can run into.
CI/CD Pipelines
A CI/CD pipeline typically consists of a series of tasks executed after a (git) commit is pushed to the remote registry. The idea is to ensure compliance with coding standards, formatting, and code quality, amongst a great wealth of other things. A pipeline is typically sub-divided into stages such as “build”, “lint”, “deploy” or anything else you can think of. Each stage consists of one or more tasks.
Whether or not the pipeline progresses to the next stage depends on the success or failure of tasks. Return codes are usually used to determine success or failure: a return code of 0 implies success, everything else usually terminates the pipeline’s execution.
Experimenting with cURL Exit Codes
In order to use cURL effectively in a CI pipeline it’s important to understand its error codes. Consider the following simulated API using node and express.js:
import express from 'express'
const app = express()
const port = 8080
const host = '0.0.0.0'
// allow for a successful test
app.get('/', (req, res) => {
res.set('Content-Type', 'text/plain')
res.send('test successful')
})
// invoke this URL to provoke a HTTP 400 (bad request) error
// see https://expressjs.com/en/4x/api.html#res.set for details
app.get('/failure', (req, res) => {
res.set('Content-Type', 'text/plain')
res.status(400).send('Bad Request')
})
app.listen(port, host, () => {
console.log(`Simulated API server available on ${host}:${port}!`)
})
I created a small container image with the above code using the node:lts image (that’s node 18.14.2 and express 4.18.2, the most current versions at the time of writing, Feb 25th) and ran it.
“But what about security?” I hear you ask. You will undoubtedly have noted that this isn’t production code, it lacks authentication and other security features, logging, and basically everything apart from returning a bit of text and a HTTP status code. I’m also going to use HTTP calls for the API – enabling HTTPS would have been overkill for my example. In the real world you wouldn’t run APIs without TLS protection, would you? Since this post is about HTTP status codes and cURL in CI pipelines none of the extra bells and whistles are necessary, and crucially they’d probably distract from the actual problem. If you’re coding your APIs you should always adhere to industry best practices!
Starting the container
I stated the container as follows:
podman run --rm -it --name some-api --publish 8080:8080 api:0.5
The CMD directive in the project’s Dockerfile starts node and passes the api.mjs file to it. The API is now ready for business:
Simulated API server available on 0.0.0.0:8080!
Scenario 1: normal, successful completion
Let’s start with the successful invocation of the simulated API:
$ curl http://localhost:8080 test successful $ echo $? 0
OK, nothing to see here, moving on… This is what was expected and shown for reference ;)
Scenario 2: Bad Request
I’m pointing curl to http://localhost:8080/failure next:
$ curl http://localhost:8080/failure Bad Request $ echo $? 0
Hmm, so that’s odd, curl‘s return code is 0 (= success) despite the error? Let’s dig a little deeper by using the verbose option and returning the headers
$ curl -iv http://localhost:8080/failure * Trying ::1:8080... * Connected to localhost (::1) port 8080 (#0) > GET /failure HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.74.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 400 Bad Request HTTP/1.1 400 Bad Request < X-Powered-By: Express X-Powered-By: Express < Content-Type: text/plain; charset=utf-8 Content-Type: text/plain; charset=utf-8 < Content-Length: 11 Content-Length: 11 < ETag: W/"b-EFiDB1U+dmqzx9Mo2UjcZ1SJPO8" ETag: W/"b-EFiDB1U+dmqzx9Mo2UjcZ1SJPO8" < Date: Sat, 25 Feb 2023 11:34:16 GMT Date: Sat, 25 Feb 2023 11:34:16 GMT < Connection: keep-alive Connection: keep-alive < Keep-Alive: timeout=5 Keep-Alive: timeout=5 < * Connection #0 to host localhost left intact Bad Request
So it’s pretty clear that the HTTP status code is 400 (Bad Request). But that’s not reflected in the return code. Let’s fix this!
Instructing cURL to fail
A look at the cURL manual page reveals this interesting option:
-f, --fail
(HTTP) Fail silently (no output at all) on server errors. This
is mostly done to enable scripts etc to better deal with failed
attempts. In normal cases when an HTTP server fails to deliver a
document, it returns an HTML document stating so (which often
also describes why and more). This flag will prevent curl from
outputting that and return error 22.
This method is not fail-safe and there are occasions where non-
successful response codes will slip through, especially when au‐
thentication is involved (response codes 401 and 407).
Which looks like exactly what I need. Let’s try this option:
$ curl --fail http://localhost:8080/failure curl: (22) The requested URL returned error: 400 Bad Request $ echo $? 22
Well that’s better! There’s a non-zero return code now.
Summary
The --fail option in curl (or --fail-with-body if your version of curl is 7.76 or later) allows DevOps engineers to architect their pipelines with greater resilience. Rather than manually parsing the cURL output checking for errors you can now rely on the REST API call’s return code to either proceed with the pipeline or stop execution. Please note that the –fail option isn’t fail-safe, as per the above comment in the man-page. Neither does it protect you from an API returning a HTTP-200 code if in fact an error occurred. But it’s definitely something I’ll use from now on by default.
Martins Blog 