Figure out what to do about MIT's docs

[steveklabnik]

http://web.mit.edu/rust-lang_v0.8/doc/std/vec/index.html

These rate really high on Google, which is gonna cause us problems.

[nathantypanski]

Looks like Alex Prengel, judging by the Athena cluster readme from that page.

• MODIFY HISTORY:
  • 10/29/13 - Alex Prengel
    • Created this README.athena, initial locker setup
  • 10/31/13 - Alex Prengel
    • Added etc directory (not installed by default)

A quick search wasn't enough to track down his email, but it's probably not too hard to get in touch.

[mdinger]

A people search of the mit.edu seems to have found his info.
http://web.mit.edu/bin/cgicso?query=alex+prengel

[steveklabnik]

I will send him an email. UPDATE: I have sent him an email. Let's see.

[steveklabnik]

Alex wrote me back, and it is he. He, understandably, wants to make sure that anything done won't harm any students, which I reassured him I certainly wouldn't want to do. I suggested a robots.txt, based on a friend's advice. (This friend is one of the few good SEO guys.)

This raises a question of what to do when every university decides to use Rust to teach systems programming, though. 😉 Maybe the docs should generate a robots.txt by default, and then we can just remove it when deploying the official docs?

What do you think, @brson?

[alexcrichton]

I don't think robots.txt works unless it's at the root of a web server, so it'd have to be at web.mit.edu/robots.txt, which may not help with doc generation, sadly.

[alexcrichton]

See http://www.robotstxt.org/robotstxt.html under "Where to put it"

[brson]

We could put <meta name="robots" content="noindex" /> in rustdoc's output by default, and only turn it off for official builds.

[nathantypanski]

This would need to only affect building the official docs - not all rustdoc output. Surely we don't want everyone who uses rustdoc to be unable to build indexable documentation.

[steveklabnik]

Ahhh yes. I forgot it needs to be at the root. @brson , my friend suggested that those two are equivalent, basically, so yes, that would work too.

Surely we don't want everyone who uses rustdoc to be unable to build indexable documentation.

Yes, we do. Why would we want unofficial docs to rank higher than the official ones? And it has a built-in search index.

[thehydroimpulse]

By rustdoc, @nathantypanski meant (I'm assuming) other projects (i.e, not the rust project) that build their documentation using rustdoc.

[nathantypanski]

Yes, we do. Why would we want unofficial docs to rank higher than the official ones? And it has a built-in search index.

I mean for their own code. When Rust gets more popular, people will want to use Rustdoc to document their libraries. See e.g. how the Sphinx docs for this Python package are indexed by Google - that's useful, and certainly not detrimental to Python the language.

[steveklabnik]

Oh. Yes, sorry, I'm talking about for the language docs only. It's a good point that we have to think about that case though.

[steveklabnik]

So, Alex is just the sysadmin for the cluster. It's for anyone to use, so it's not even his responsibility that this is up here. So I thanked him for his time, and we'll just have to make future docs do the right thing. Hopefully we can out-SEO these old ones.