Git LFS GET request malformed

[x-8523]

Description

Hello,

We have updated our gitea to version 1.17.3. We were coming from version 1.15.8.
We then had an issue where we were not able to fetch LFS files anymore.
On our test server, we have been able to reproduce the issue with gitea v1.16.0. After a downgrade to v1.15.8, we think that it was working fine.

Important note: Git LFS clients are directly communicate with our s3. (means gitea is configured with SERVE_DIRECT: true)

After investigation, we saw that the git lfs client was receiving from gitea invalid credentials for the LFS requests.
The LFS client was receiving a correct presigned url to download the file but it was doing the query to this url with an HTTP "Authorization" header which contains a bearer token.
Please note that this token is exactly the same as the token used to authenticate to gitea for LFS POST requests.

With manual tests, we were able to download the file but as soon as we were adding this famous HTTP header, the request was failing with HTTP 400 with the following response from our S3:
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><RequestId>tx000000000000000000792-006351619a-1101ec-default</RequestId><HostId>1101ec-default-default</HostId></Error>%

To sumup:
The presigned URL received by gitea is ok and is usable but then the git lfs client is adding this HTTP header. The addition of this HTTP header in the request is making it failing.
We are wondering why the token to authenticate to gitea is used in the request to s3.
It may come from this part of the code:

gitea/cmd/serv.go

Lines 279 to 283 in 6a03309

	tokenAuthentication := &git_model.LFSTokenResponse{
	Header: make(map[string]string),
	Href: url,
	}
	tokenAuthentication.Header["Authorization"] = fmt.Sprintf("Bearer %s", tokenString)

As a workaround, we have switched the "SERVE_DIRECT" to "false" but we would like to come back to true because it means that it is our gitea machine which is getting the whole "LFS load" instead of our s3.

The git trace of what we have seen is here:

17:25:07.043266 trace git-lfs: tq: retrying object 17a7b90cfd72782acc4e10aca32c8db33ea9190f6c26f26c7e56953629e2190d: LFS: Client error: https://s3.xxx/gitea/lfs/17/a7/b90cfd72782acc4e10aca32c8db33ea9190f6c26f26c7e56953629e2190d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=M6BZZLLIDG9SXXX4YL60%2F20221020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221020T152507Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D%2217a7b90cfd72782acc4e10aca32c8db33ea9190f6c26f26c7e56953629e2190d%22&X-Amz-Signature=43d70dc8e2c90ed95d0b42eae6c5e3272e79f6592dd9502d45862ed9fe4dabf4
17:25:07.043339 trace git-lfs: tq: enqueue retry #8 after 10.00s for "17a7b90cfd72782acc4e10aca32c8db33ea9190f6c26f26c7e56953629e2190d" (size: 28284)
17:25:07.043403 trace git-lfs: tq: sending batch of size 1
17:25:07.043607 trace git-lfs: ssh cache: gitea@git.xxx git-lfs-authenticate org/repository.git download
17:25:07.043676 trace git-lfs: api: batch 1 files
17:25:07.044030 trace git-lfs: HTTP: POST https://git.xxx/org/repository.git/info/lfs/objects/batch
> POST /org/repository.git/info/lfs/objects/batch HTTP/1.1
> Host: git.xxx
> Accept: application/vnd.git-lfs+json
> Authorization: Bearer XXX
> Content-Length: 228
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.2.0 (3.2.0; linux amd64; go 1.17.11)
>
{"operation":"download","objects":[{"oid":"92f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1","size":27966}],"transfers":["basic","ssh","lfs-standalone-file"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}17:25:07.050910 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 876
< Cache-Control: no-store, no-transform
< Connection: keep-alive
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 20 Oct 2022 15:25:07 GMT
< Server: nginx/1.20.1
< Set-Cookie: i_like_gitea=0d8c63bbe708cd4f; Path=/; HttpOnly; SameSite=Lax
< Set-Cookie: _csrf=K3sBnLRybfTFKAL8s9CJrx1dsuw6MTY2NjI3OTUwNzA0NTE3MzI2Mg; Path=/; Expires=Fri, 21 Oct 2022 15:25:07 GMT; HttpOnly; SameSite=Lax
< Set-Cookie: macaron_flash=; Path=/; Max-Age=0; HttpOnly; SameSite=Lax
< X-Frame-Options: SAMEORIGIN
<
17:25:07.051440 trace git-lfs: HTTP: {"objects":[{"oid":"92f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1","size":27966,"actions":{"download":{"href":"https://s3.xxx/gitea/lfs/92/f9/15acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=M6BZZLLIDG9SXXX4YL60%2F20221020%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20221020T152507Z\u0026X-Amz-Expires=300\u0026X-Amz-SignedHeaders=host\u0026response-content-disposition=attachment%3B%20filename%3D%2292f915acd3c5a232
{"objects":[{"oid":"92f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1","size":27966,"actions":{"download":{"href":"https://s3.xxx/gitea/lfs/92/f9/15acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=M6BZZLLIDG9SXXX4YL60%2F20221020%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20221020T152507Z\u0026X-Amz-Expires=300\u0026X-Amz-SignedHeaders=host\u0026response-content-disposition=attachment%3B%20filename%3D%2292f915acd3c5a23217:25:07.051614 trace git-lfs: HTTP: d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1%22\u0026X-Amz-Signature=85f4aa9aa408c63d71b4a8619ec99542aa1450c9662a82ffcf2c940293bda45e","header":{"Authorization":"Bearer XXX"}}}}]}
d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1%22\u0026X-Amz-Signature=85f4aa9aa408c63d71b4a8619ec99542aa1450c9662a82ffcf2c940293bda45e","header":{"Authorization":"Bearer XXX"}}}}]}
17:25:07.051765 trace git-lfs: xfer: adapter "basic" End()
17:25:07.051802 trace git-lfs: xfer: adapter "basic" worker 0 stopping
17:25:07.051820 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
17:25:07.051834 trace git-lfs: xfer: adapter "basic" worker 4 stopping
17:25:07.051847 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
17:25:07.051859 trace git-lfs: xfer: adapter "basic" worker 2 stopping
17:25:07.051872 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
17:25:07.051885 trace git-lfs: xfer: adapter "basic" worker 6 stopping
17:25:07.051885 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
17:25:07.051910 trace git-lfs: xfer: adapter "basic" worker 5 stopping
17:25:07.051896 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
17:25:07.051958 trace git-lfs: xfer: adapter "basic" worker 1 stopping
17:25:07.051870 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
17:25:07.052018 trace git-lfs: xfer: adapter "basic" worker 3 stopping
17:25:07.051891 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
17:25:07.052105 trace git-lfs: xfer: adapter "basic" worker 7 stopping
17:25:07.052159 trace git-lfs: xfer: adapter "basic" stopped
17:25:07.052261 trace git-lfs: tq: starting transfer adapter "basic"
17:25:07.052295 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
17:25:07.052336 trace git-lfs: xfer: adapter "basic" started
17:25:07.052343 trace git-lfs: xfer: adapter "basic" worker 1 starting
17:25:07.052382 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
17:25:07.052370 trace git-lfs: xfer: adapter "basic" worker 5 starting
17:25:07.052409 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
17:25:07.052405 trace git-lfs: xfer: adapter "basic" worker 7 starting
17:25:07.052391 trace git-lfs: xfer: adapter "basic" worker 6 starting
17:25:07.052456 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
17:25:07.052432 trace git-lfs: xfer: adapter "basic" worker 0 starting
17:25:07.052445 trace git-lfs: xfer: adapter "basic" worker 3 starting
17:25:07.052508 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
17:25:07.052370 trace git-lfs: xfer: adapter "basic" worker 4 starting
17:25:07.052607 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
17:25:07.052452 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
17:25:07.052418 trace git-lfs: xfer: adapter "basic" worker 2 starting
17:25:07.052705 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
17:25:07.052485 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "92f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1"
17:25:07.053740 trace git-lfs: HTTP: GET https://s3.xxx/gitea/lfs/92/f9/15acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1
> GET /gitea/lfs/92/f9/15acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=M6BZZLLIDG9SXXX4YL60%2F20221020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221020T152507Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D%2292f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1%22&X-Amz-Signature=85f4aa9aa408c63d71b4a8619ec99542aa1450c9662a82ffcf2c940293bda45e HTTP/1.1
> Host: s3.xxx
> Authorization: Bearer XXX
> User-Agent: git-lfs/3.2.0 (3.2.0; linux amd64; go 1.17.11)
>
17:25:07.065242 trace git-lfs: HTTP: 400
< HTTP/1.1 400 Bad Request
< Content-Length: 192
< Accept-Ranges: bytes
< Content-Type: application/xml
< Date: Thu, 20 Oct 2022 15:25:07 GMT
< X-Amz-Request-Id: tx00000000000000000afc5-0063516853-10b420-default
<
17:25:07.065804 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "92f915acd3c5a232d9b199b74e0e80d4998e22419de48b6b6bf80ec80667fec1"

Gitea Version

1.17.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.28.0

Operating System

Centos 7.9.2009

How are you running Gitea?

We are running the binary downloaded from the gitea website through a systemd service.

Database

MySQL

[lunny]

Looks like the LFS server implementation hasn't followed SERVE_DIRECT configuration. @KN4CK3R

[KN4CK3R]

From the git history it looks like serve direct was added in #16731 and that is Gitea 1.16. So it's a bit surprising it worked for you in 1.15.8.

[lunny]

Could you get the generated s3 link and try to download it manually?

[x-8523]

From the git history it looks like serve direct was added in #16731 and that is Gitea 1.16. So it's a bit surprising it worked for you in 1.15.8.

Indeed. We suspect it worked for us in 1.15.8 because serve direct didn't apply to LFS at the time. The update to 1.17.3 effectively enabled serve direct for LFS and revealed the bug.

We have temporarily disabled serve direct to work around this issue and everything looks okay so far. We would like to re-enable serve direct at some point, though.

Could you get the generated s3 link and try to download it manually?

We tried the generated signed URLs and they are valid. We were able to download the files manually. However, when used through git-lfs, git-lfs sends this Authorization header and our S3 (ceph+radosgw) returns a 400 InvalidArgument.

We used curl to try the signed URLs with and without the Authorization header. Without the Authorization header we receive the file and a 200. With the Authorization header we receive a 400 InvalidArgument.

[KN4CK3R]

Ok, then the fix should be something like this:

@@ -438,13 +438,14 @@ func buildObjectResponse(rc *requestContext, pointer lfs_module.Pointer, downloa
                }
 
                if download {
-                       rep.Actions["download"] = &lfs_module.Link{Href: rc.DownloadLink(pointer), Header: header}
                        if setting.LFS.ServeDirect {
                                // If we have a signed url (S3, object storage), redirect to this directly.
                                u, err := storage.LFS.URL(pointer.RelativePath(), pointer.Oid)
                                if u != nil && err == nil {
-                                       rep.Actions["download"] = &lfs_module.Link{Href: u.String(), Header: header}
+                                       rep.Actions["download"] = &lfs_module.Link{Href: u.String()}
                                }
+                       } else {
+                               rep.Actions["download"] = &lfs_module.Link{Href: rc.DownloadLink(pointer), Header: header}
                        }
                }
                if upload {

(removed the "headers to send" field)

[lunny]

But when get LFS url failed, we need fallback

[KN4CK3R]

Fix in #21531