crates.io is down (application error)

[klingtnet]

See the following screenshot:

[stefanpenner]

seems to be up for me: https://crates.io/

[klingtnet]

I'm still getting the application error.

[stefanpenner]

although it does appear to be "Stuck" loading

[klingtnet]

doc.crates.io works, seems to be only the package index.

[steveklabnik]

Can you refresh? Looks like it's up again

[steveklabnik]

.... and back down. maybe not.

[klingtnet]

Back online, great work!

[klingtnet]

@steveklabnik I will close this issue, but could you please give a quick summary on what caused the outage?

[alexcrichton]

I've just restarted the app servers, still in my phone so I haven't had a
chance to investigate yet. Appears to be tons of timeouts though which are
probably related to db connections. I'll investigate when I get to a
computer

On Wednesday, June 1, 2016, Andreas Linz notifications@github.com wrote:

Closed #348 #348.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#348 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAD95G3tGLSzka50LnDIiFEcBYNdc4Ceks5qHcfGgaJpZM4Irv_d
.

[steveklabnik]

@alexcrichton @brson figured it out. Safe flight <3

@klingtnet it will be on the internals thread.

[brson]

Here's my description of what happened.

A bunch of queries deadlocked in the postgres server bumping download statistics, locking tables and blocking connections. Running heroku pg:killall cleared the connections out and restored operation. It's not clear why they deadlocked.

[brson]

Further info from Heroku support.

[alexcrichton]

Oh wow, thanks for the investigation @brson! Looks like my "restart the dynos and pray" was timed around when you must have killed the connections, so I mistakenly thought it actually fixed something!