403 links causing CI failures

[SethTisue]

so e.g. https://travis-ci.org/scala/scala-lang/builds/412090629 has

- ./_site/documentation/your-first-lines-of-scala.html
  *  External link http://www.oracle.com/technetwork/java/index.html failed: 403 No error
- ./_site/news/2.12.0/index.html
  *  External link http://www.oracle.com/technetwork/java/javase/downloads/index.html failed: 403 No error
htmlproofer 3.9.1 | Error:  HTML-Proofer found 2 failures

previously: #933, I guess #939 wasn't enough?

[ashawley]

I've never seen "403 No erorr". Usually, 403 is "Forbidden" message, but the message is "No error". That message is something Oracle's web servers are giving us, not something that html-proofer is doing, see https://github.com/gjtorikian/html-proofer/blob/a9bff12/lib/html-proofer/url_validator.rb#L166

[ashawley]

If we take it literally, it sounds like it is not an error and we just ignore? Perhaps, there is nothing to change? Most redirects are 3xx level responses.

[SethTisue]

from https://github.com/gjtorikian/html-proofer/blob/master/bin/htmlproofer it looks like we could do --http-status-ignore 403, then

[SethTisue]

idk, maybe it's better to just exclude www.oracle.com from checking, since it sounds like they're doing something unusual with 403

[jvican]

idk, maybe it's better to just exclude www.oracle.com from checking, since it sounds like they're doing something unusual with 403

let's do this, it's better than ignoring all 403 👍

[ashawley]

Yeah, it's better to not ignore 403s. Best to just ignore their domain for the time being. Maybe Oracle's web hosting is throttling the html-proofer user-agent string or considers the Travis server IPs as malicious.

[ashawley]

No 403s today:

Checking 4655 external links...
Ran on 279 files!
HTML-Proofer finished successfully.

[SethTisue]

this has failed the last three builds in a row at https://travis-ci.org/scala/scala-lang/builds

[jvican]

I will try to port this to a cron job

[SethTisue]

perhaps #960 will help

[SethTisue]

no such luck, https://travis-ci.org/scala/scala-lang/builds/437654814 has

- ./_site/documentation/your-first-lines-of-scala.html
  *  External link https://www.oracle.com/technetwork/java/index.html failed: 403 No error
- ./_site/download/2.12.7.html
  *  External link http://www.oracle.com/technetwork/java/javase/downloads/index.html failed: 403 No error
- ./_site/news/2.12.0/index.html
  *  External link https://www.oracle.com/technetwork/java/javase/downloads/index.html failed: 403 No error

[ashawley]

Well, thanks for humoring me. I'm just going to move that they be ignored in #961.