Crates.io returns 404 when not specifying `Accept: text/html`

[kud1ing]

• awesome-rust uses awesome_bot to check links. Unfortunately links to crates.io return 404.
• curl https://crates.io/keywords/cassandra receives a 404, too.

Current summary

• This is confirmed as a bug that still exists as long as this issue is open. We do not need any further confirmations of the issue at this time.
• What we do need is a PR fixing the problem, those are quite welcome
• The workaround for this issue is to pass an Accept header of text/html.

[carols10cents]

I think we are doing something nonstandard, but I'm not sure what. Until we figure this out, a workaround is specifying HTML as the content type you want:

curl -v 'https://crates.io/crates/assert_approx_eq' -H 'Accept: text/html'

returns a 200 for me. Hope that helps!

[kud1ing]

crates.io was whitelisted in rust-unofficial/awesome-rust#310
I'd like to revert this in the end so that we can verify links to crates, categories and keywords.

[Nemo157]

Yeah, this is caused by

crates.io/src/dist.rs

Lines 38 to 52 in a20eb96

	// Second, if we're requesting html, then we've only got one page so
	// serve up that page. Otherwise proxy on to the rest of the app.
	let wants_html = req.headers()
	.find("Accept")
	.map(|accept| accept.iter().any(|s| s.contains("html")))
	.unwrap_or(false);
	if wants_html {
	self.dist.call(&mut RequestProxy {
	other: req,
	path: Some("/index.html"),
	method: None,
	})
	} else {
	self.handler.as_ref().unwrap().call(req)
	}

Interestingly requesting any URL with Accept: text/html will return 200 since the server doesn't know what the valid routes are (e.g. curl -H 'Accept: text/html' -I https://crates.io/foo/bar)

I don't think there's any way to have a bot validate URLs on crates.io without executing the javascript and checking if it loads the page successfully. At least, while crates.io does client-side only rendering, if server-side rendering of at least the initial page is ever added that should then know whether the URL being requested is valid or not.

[locks]

FYI, you can track server-side rendering efforts at #819.

[AaronFriel]

This is still not fixed.

[carols10cents]

@AaronFriel Yes, that's why this issue is still open.

[AaronFriel]

Would a commit altering the wants_html test to include wildcards be accepted? I'm just not sure why this has languished and I'm guessing there is something I'm missing.

[carols10cents]

Yes, I am not sure of the scope of the solution needed or the effects of various solutions on the frontend and backend-- please give it a try and let me know if you have questions!

[mark-i-m]

Any progress on this?