How can I support unsafe URIs?

[srabraham]

Hi there,
We're in a silly situation with our web services. Some of our clients call our APIs with unsafe (unencoded) URIs. For example, we'll get something like

GET /?a=This is gross HTTP/1.1 rather than something sane like
GET /?a=This%20is%20better HTTP/1.1

We were able to make that work in our fasthttp-based server, up until this PR #1710. Before then, header.go looked backward from the end of the line for the last " " to find the HTTP version. That meant it was fine for the URI to contain spaces. Now, the code finds the HTTP version by seeking the second " " in the line, so if your URI contains a space, the code will error out.

The new behavior from that PR may be more in line with the RFCs, so I'm not asking you to backtrack. I am wondering how you'd suggest we handle this absurd situation though, assuming I'm stuck with my clients sending dirty URIs.

As an example of what I'm talking about, here's a test that passes before that PR (at a8cb5d5), but fails after that PR (at master). Thanks!

// insert this test into repo root's header_test.go
func TestRequestHeaderUnencodedParam(t *testing.T) {
	t.Parallel()

	s := "GET /?a=This has unescaped spaces HTTP/1.1\r\n" +
		"\r\n"
	var h RequestHeader
	br := bufio.NewReader(bytes.NewBufferString(s))
	if err := h.Read(br); err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
	if string(h.requestURI) != "/?a=This has unescaped spaces" {
		t.Fatalf("unexpected request URI: %q. Expecting %q", h.requestURI, "/?a=This has unescaped spaces")
	}
}

[erikdubbelboer]

I would suggest you stay with an old version of fasthttp in that case. Of course the best is if your clients would fix their issues.

[srabraham]

Yeah, I guess that's it...there's seemingly nowhere I can intercept the connection before it gets into the fasthttp code. Sigh. Thanks for the quick response.

I would add though that I think the error is misleading in my example.

If someone sends GET /?a=This has unescaped spaces HTTP/1.1\r\n then fasthttp gives an error of unsupported HTTP version "has unescaped spaces HTTP/1.1". I can see why it's doing that, but I'd argue that if there's an extra space somewhere in the first line, it's almost always going to be in the URI rather than in the HTTP version. I'd be inclined to split the first line by the first " " to find the method and the final " " to find the version, then use the middle bit as the URI. The code could still error out if the URI contains a " " of its own, but the error could be more direct about it.

[srabraham]

Also, a space is currently the only URI character that makes the parsing code choke, even though there are a bunch of other reserved/illegal characters. e.g. this currently passes the first-line parsing:
GET /?a=:/?#[]@!$&'()*+,;= HTTP/1.1

I think I'm digging my own grave here as someone needing to support invalid URIs 😆

[slicingmelon]

Hi @srabraham,

I've struggled with this for months, and finally, I managed to send true raw requests with a custom client and several patches on fasthttp.

Take a look at the codebase of my tool.

1. HTTP Client settings:

https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/core/engine/rawhttp/client.go#L121
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/core/engine/rawhttp/client.go#L173

2. Building raw requests:

https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/core/engine/rawhttp/request.go#L93
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/core/engine/rawhttp/request.go#L271
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/core/engine/rawhttp/request.go#L281

3. Patches on fasthttp

header.go

https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/header.go#L258
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/header.go#L2746
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/header.go#L2769
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/header.go#L2855
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/header.go#L2978

http.go

https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/http.go#L1625

uri.go

https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/uri.go#L339
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/uri.go#L710
https://github.com/slicingmelon/gobypass403/blob/9c698b48a98fa188aaff44335d9075c580e7066b/pkg/fasthttp-client-1.64.0/uri.go#L926

These are just some of them. You could scroll through the patched fasthttp code here: https://github.com/slicingmelon/gobypass403/tree/main/pkg/fasthttp-client-1.64.0

No HTTP client in Go supports such malformed requests.

[srabraham]

Thanks a bunch for that, @slicingmelon.

I did get this all working for our needs via a gross forked repo. I'm not proud of hacking around the HTTP standard, but oh well.

Here's what I did on the server side: VideoAmp@58ac611

and on the client side, in order to actually test such nastiness, here's a demonstration, which should print the full URI

func TestHTTP(t *testing.T) {
	ln := fasthttputil.NewInmemoryListener()
	defer ln.Close()
	go fasthttp.Serve(ln, func(r *fasthttp.RequestCtx) {
		fmt.Println("Got request for path", string(r.RequestURI()))
	})
	conn, _ := ln.Dial()
	defer conn.Close()
	conn.Write([]byte("GET /my/path?key=eww this has literal spaces HTTP/1.1\r\n\r\n"))
	io.ReadAll(conn)
}

[slicingmelon]

Hi @srabraham,

I got it, thanks for the follow-up. I originally read your question as being more about the client side, which is why I focused on raw requests and client patches.

Totally agree that once you drop down to a raw net.Conn, wiring a raw request is simple. Glad you managed to get it working with a fork.