At adjust we recently tried to replace the Go standard http
library with fasthttp
. Fasthttp is a low allocation, high performance HTTP library, in synthetic benchmarks the client shows a 10x performance improvement and in real systems the server has been reported to provide a 3x speedup. The service we wanted to improve makes a very large number of HTTP requests and so we were very interested in using the fasthttp
client.
In the course of making the switch we encountered a number of difficulties. First the fasthttp
library presents a very different interface to the programmer which must be adjusted to. Second there were a number of quirks in the implementation which made progress rather slow.
Making a simple request
To begin with we would like to learn how to perform a simple HTTP request using the fasthttp
client. Below is a very simple request using the Go standard library, error handling has been omitted for brevity.
For all the code snippets below the test server writes the request’s “User-Agent” header value and body into the response on separate lines. We write the actual output of the snippet in comments beneath each print statement.
1 2 3 4 5 6 7 8 9 10 11 |
|
A fasthttp
request can be written, also without error handling, very similarly:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
Getting the response body
The body of an http.Response
is exposed as an exported io.ReadCloser
field. The body of a fasthttp.Response
is exposed via the Body()
method call which returns a []byte
. The implication of this is that the entire body must be read and a sufficiently large []byte
allocated before the body can be processed. This is a surprising feature of a library which prioritises performance and low memory allocations.
One curious aspect of the Body()
method is that it returns no error, in contrast to reading from an io.ReadCloser
. It would be interesting to see how that method is implemented to get a better idea of how fasthttp
works.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
|
The Body()
method operates on two unexported fields body
and bodyStream
. It first checks if bodyStream
is non-nil, and if it is, reads from the bodyStream
into the body
field. Finally the contents of the body
field are returned to the caller.
This is pleasantly straightforward, but there is one odd wrinkle, this method will silently eat errors.
Looking at line 15 in the example above we can see that any errors encountered while reading from bodyStream
are written into the body
field and the original error is not returned. An error could occur, but we would never find out about it. Lets look further into our simple HTTP request example to see how the Body()
method would actually execute.
If we trace the execution of our simple request above we find the following execution path:
1 2 3 4 5 |
|
Inside ReadLimitBody(...)
we find this critical piece of code
1 2 3 4 5 6 7 |
|
We can see, on line 3, that the call to readBody(...)
sets the bytes bodyBuf.B
to be the result of reading from the connection. So the stream reader field will be nil. We can see that errors are being returned from the readBody(...)
method call. That’s good, but we have only covered one simple case. From further analysis I do believe that errors are not swallowed by the fasthttp
client, but I am not certain. There is a potential execution path which results in errors being silently swallowed.
Making POST requests
Our existing application performs both GET and POST requests. We ran into a small problem making POST requests. We will start with a simple POST example using fasthttp
. Here we set our method to POST and fill in the body with some form-encoded values. Now we see both the “User-Agent” and the non-empty request body in the response.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
Setting your “User-Agent”
Next we want to set our “User-Agent” header manually, but there is a small problem.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
While the standard library http.Client
does provide a default “User-Agent” header value, this value is overridden when any other value is provided. Fasthttp is still sending it’s default “fasthttp” and our “Test-Agent” value is not being picked up.
We wanted to get a better look at the headers that were being set, so we added a single debug line println(req.Header.String())
. Now we can no longer ignore errors in our code, because that innocent looking req.Header.String()
causes client.Do(...)
to fail.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
When we print the request headers we get to see the preloaded “User-Agent: fasthttp” header value is still stored, and in particular ahead of our “Test-Agent” value. This certainly explains why we aren’t seeing our value. We will look into why this is happening after we deal with the request error.
Why do our requests return an error?
After adding a simple println
statement we now get the error “Error: non-zero body for non-POST request. body=”p=q”“. The client now seems to believe that our request is not a POST. The critical call path here is
1 2 3 4 5 |
|
We can look into the IsGet()
method to see some interesting caching behaviour.
1 2 3 4 5 6 7 8 |
|
The method IsGet() reads the RequestHeader.method
field and sets the RequestHeader.isGet
cache field, to speed up future method calls. Unfortunately at this point we haven’t set our method and in the absence of any value it defaults to GET. So RequestHeader.isGet
is set to true, which causes future calls to IsGet()
to return true regardless of the value the RequestHeader.method
field. Critically this method is also called inside HostClient.doNonNilReqResp(...)
to test whether the request should have an empty body or not, causing the error we see above.
It’s worth noting that the call path contains 4 exported methods, any one of which would create the same confusing behaviour. You must be very careful to call req.Header.SetMethod(...)
early if you intend to make POST requests.
Why is fasthttp
sending its default “User-Agent”
Interestingly it looks like the unexpected “fasthttp” user-agent is a bug that is also caused by caching. If we look at the RequestHeader.AppendBytes(...)
, which builds the header args, it performs the following check
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
We can see, on line 13, that the userAgent
value is taken from the field RequestHeader.userAgent
, we could quickly confirm that our preferred header value “Test-Agent” was held inside a field RequestHeader.h
but is completely missed by the call to h.parseRawHeaders
which looks inside RequestHeader.rawHeaders
. This complex arrangement of headers and various cached values makes interacting with headers a true minefield of unexpected behaviour.
Should You Use Fasthttp
It’s a difficult question to answer. Fasthttp
does reduce allocations and I have no doubt it will bring significant benefits to some systems, particularly those performing high volume HTTP requests and not much else. Garbage collection is not free, and fasthttp
could bring real performance improvements, and potentially reduce your hardware requirements. But, fasthttp
is not simple and it appears that fasthttp
has been built primarily for use on servers.
The client implementation reuses the data structures used on the server, this means, for example, that the fasthttp.Response
used by the client contains a very large amount of code which is only useful to a server. This makes understanding the codebase and debugging any problems much harder.
The high level of complexity and the likelihood that the fasthttp
client has not been extensively used in production means that you would need to expect a very large benefit to justify the adoption of fasthttp
today.
Thanks
We would like to thank Valyala and other contributors for making a high performance http library available for Go. We know it is no small task.