API docs: project overview & weekly status updates

[emidoots]

In this issue I will summarize my work weekly on API docs.

Product Design, RFCs, and other public memos (most recent last)

• PD 24: Sourcegraph for Public Code
• RFC 349 ABANDONED: The google-for-code team
• RFC 369 APPROVED: API docs implementation plan
• RFC 370 APPROVED: API docs LSIF data requirements / extensions
• LSP: proposal: workspace/documentation support
• RFC 371 APPROVED: API docs pivoting LSIF/LSP approach
• LSIF: proposal: documentation extension to LSIF
  • Discussion & Go implementation of proposed protocol

[emidoots]

Week of Mar 26, 2021

I have a document detailing our data requirements from LSIF, exploring how LSIF/LSP meets that today, and discussing some LSIF/LSP extensions I plan to propose in the future: https://docs.google.com/document/d/1NcNYXOnprknfHkSTMOh1ST8een-NQUVuWCg5DbMmksM/edit?usp=sharing

I have a much clearer picture today about what we need to get to a usable Go MVP today (mostly Go indexer work, no new LSIF extensions) than I did e.g. even yesterday.

I have pushed various stakeholders to commit to specific high-level goals, and have gotten much clearer direction on that and general agreement on the implementation plan, etc.

[emidoots]

Week of Apr 2, 2021

This was a short week due to me being out a day for my vaccine, plus 20% of my time going towards some unexpected customer incidents and helping others internally.

I discussed with Jean and Malo requirements from Nick, and got agreement on why focusing on search right off the bat (before hitting the MVP) would not make sense. In response to all of this, I have put together and initial "User problem" that we are solving, and worked with Jean and Malo to refine it. - you can find it in the implementation plan

2d was spent looking into lsif-go; I merged my first PR there and learned about how the codeintel test suite works. I continue to work on https://github.com/sourcegraph/sourcegraph/issues/19591 and am not blocked on anything.

[emidoots]

Week of Apr 9, 2021

Also a short week due to Easter holiday (out Mon & Tue.)

I made good progress on lsif-go support for textDocument/documentSymbols. I ran into a really nasty issue and burned a full day debugging what was going on, and am still waiting on Eric to get confidence I'm right about the cause (sourcegraph/lsif-go#142) before I merge a fix. There was a lot of time spent making sure that how I build this out will accommodate the future LSIF expansions we'll need, and that I'm faithfully following the LSIF/LSP spec (which is super underspecified in general, and was implemented incorrectly in Beyang's original PR.)

I am about 2-3 work days away from finishing the lsif-go work, at which point I will be onto the next challenge: getting Sourcegraph to store that data https://github.com/sourcegraph/sourcegraph/issues/19592 and enabling us to query it https://github.com/sourcegraph/sourcegraph/issues/19593 - it's unclear yet how long those issues will take, but I'm optimistic generally and believe we're only a few weeks away from breaking ground on the actual webapp frontend for this.

Unrelated to API docs, I helped resolve an ops issue (segfault) with Cloud SQL, discussed some customer support things and other random stuff in about 15% of my time.

[emidoots]

Week of Apr 16, 2021

Lots has been learned this week, and I finally feel like we're on a clear path to having the data we need from LSIF.

Early this week, and after several conversations with codeintel team, it became clear that my current approach of using a handful of existing LSP methods to build out API docs would not suit us well in the long run. In light of that, I pivoted the direction to using a custom LSIF/LSP extension for API docs.

This new direction will better serve us long-term, ensure we aren't wasting our time building the MVP only to find we need to redo a lot of stuff to win the hearts and minds of Go/etc developers when we want to get into the more advanced functionality. It better enables us to support advanced functionality, makes it easier to parallelize work on multiple languages if we want to, and is generally a Whole Lot Better™. It doesn't materially impact our timeline, because I knew something like this would happen and planned ahead so we had the needed time to get this right.

Working in that direction:

• I published a draft of the extension to the LSP spec and got lots of great feedback from the codeintel team.
• I sent a draft implementation of the new LSIF extension and am actively working to propose the revised spec based on feedback I got.
• I will begin repurposing the old implementation to match this new spec.

I believe we're 1 week away from working on LSIF storage aspects, and 2-3 weeks away from breaking ground on the webapp frontend.

Additionally:

• I fixed the issue with code insights on k8s.sgdev.org and sourcegraph.com
• I did a quick ~3h fix to expose backend progress information to the frontend for code insights, something that has been sorely needed: https://github.com/sourcegraph/sourcegraph/pull/20105/files#r615161445
• I revamped this issue to include a list of all API docs RFCs/docs, and made all "memo" docs into public RFCs.

[emidoots]

Week of Apr 23, 2021

API docs

• Based on all the feedback I got on the draft LSIF+LSP spec, I wrote and sent out a brand new LSIF extension spec and corresponding Go protocol implementation.
• I discussed the new LSIF extension in-depth with the codeintel team, and feel we've basically got buy-in and are finished with defining this.
• Some work remains to actually land the spec extension, but it doesn't block my other work at all. This is the first LSIF extension we've ever done/taken seriously, and so I've discussed at length with the team how we want to manage these going forward etc. and they're working on a single "Sourcegraph LSIF spec" document which will serve as an authoritative place for our version of LSIF basically.

By my count, I should have been done with the lsif-go indexer implementation last Friday. That didn't happen, and I consider us behind-schedule by approx. 4 days.

I still have confidence in hitting our May 19th "Go barebones/internal implementation" target.

Outside API docs

• Discussing how we do LSIF extensions generally (not really related to API docs) took a lot more time than I thought it would.
• I spent 1 day on things I have been historically roped into:
  • reviewing search team's syntect_server PRs
  • helping with hiring plan for code insights team
  • suggesting improvements to how we write job descriptions
  • helping Distribution with Grafana relicensing challenges
  • helping with https://app.hubspot.com/contacts/2762526/company/407948923/ contract negotiation wording, and a P1 incident.
• I just wasn't very productive Friday.

[emidoots]

The last two weeks I have been on a "focus vacation", as I described in Slack:

I will be trying an experiment I'm calling a "focus vacation" over the next two weeks.

What is it?

The idea is that Slack conversations, meetings, interviews, etc. take up "only ~20% of my time" but can be very disruptive to my focus, often coming up at the worst of times - and regardless of me scheduling "focus time" blocks on my calendar I interrupt myself. I will be taking a 2 week "working vacation" where I will be focusing really heads-down on coding API docs in a hyper-focused/hyper-productive state, since I typically find when I am away from Slack/meetings for extended periods my motivation and drive goes up substantially.

Here's how it will look:

• I will be in offline-mode starting tomorrow, until May 14th (2 weeks, back in time to help onboard Coury for code insights)
• I will update my calendar and Slack statuses to reflect I am on a "vacation" to avoid interrupts
• I will still be keeping general tabs on things: watching company meeting recordings, checking Slack occasionally just for really urgent things I might need to be aware of, to direct people to someone else, etc.
• You'll still find me regularly submitting PRs, chatting with others to push my PRs along. I may decide to post an update if something really cool happens, otherwise you'll get one "big" update at the end. 🙂 The goal here is to be hyper-focused and more productive than usual.
• I won't be attending any meetings during this time, and will ask e.g. the hiring team to swap me with others such as for interviewing the "Director of Product - Global Code Graph" candidates.

This is an experiment, I have some strong evidence this will be helpful (especially at this point in API docs' history where the planning dust has settled and all that is left is for me to "implement it") - this post is for me to refer others to in case they are interested in this idea themselves or have questions 🙂

How it went

Overall, it went really well! I was able to really go heads-down and just focus on the code, which helped me make a ton of progress.

One thing that really helped here was that if I was "in the zone", I was able to just continue. I didn't need to account for the next meeting I had scheduled, pan to wake up at a reasonable time the next morning, get interrupted by helping someone over Slack, etc.

It also didn't feel like overwork to me, it felt like a nice break from my regular day-to-day. There were some days I didn't feel very productive, so I just stepped away entirely for that day. I was only able to do that because I didn't have meetings, the expectation of being on Slack, etc.

I am going to wait to share a more specific update about the progress I've made on API docs from a technical POV, but we may well be ahead of schedule a bit thanks to this.

[emidoots]

Week of Apr 30, 2021 (past)

[emidoots]

Week of May 21, 2021

[emidoots]

Week of June 12

Back from a (real) two week vacation.

• Caught up with Coury and discussed code insights work in-depth, and more.
• Kickstarted conversations around: requirements for making API docs public, exploring search integration complexity, redesigning the repo landing page, etc.
• Began work on adding usage examples to API docs, starting with emitting the required LSIF data: API docs: make lsif-go emit data for documentation -> definition/references/hover #21945
• Aligned with Jean and Malo on direction: first make it public / indexed by Google, then add usage examples.
• Switched focus to making API docs public (https://github.com/sourcegraph/sourcegraph/milestone/117), updated project board etc.
• Interview training, 1:1 with Jean, customer call/sync, etc.

[emidoots]

Week of June 18

This week, I started out quite focused on our goal of making API docs public (no feature flag) ASAP. A lot of progress has been made, and I think we can have this finished next week.

I want to stress that because of the way LSIF works, if we want to expand/alter our generated data format to support more features in the future (whether that be search integration, jumping from API docs <-> code, or something else) we'll need to either maintain backwards compatibility for the old and new format while it rolls out (complex and time consuming) or wipe all data and start from scratch (losing all docs pages we currently have indexed, and potentially getting severely ding'd by Google.)

Because of that, I took this week to solidify the data format as much as possible. This will make it easier to roll out the features we think we want in the future (search integration, API docs <-> tooltip integration, etc.) with less risk of getting slowed down by having to deal with old and new data formats.

In practice, this means I've fixed a lot of issues but none of them have rolled out yet. Next week, all data will be wiped and we'll start fresh and begin public indexing by Google.

What's happened behind the scenes?

• We now emit (but don't yet store/expose/consume) the LSIF data needed for us to in the future support..
  • "Jump to API docs" in code views, right alongside find-references/jump-to-definition in the hover tooltips.
  • "View code" in API docs pages - i.e. jump from API docs right to definition in code.
  • Usage examples in API docs pages - i.e. find-references on whatever symbol you're looking at.
  • Search integration - we now emit some logical search keys that could be used for search integration in the future.
• A new test suite for LSIF indexers implementing the documentation extension is in progress. This makes it really easy to emit static Markdown from the LSIF data, so implementing the Sourcegraph LSIF documentation extension for other languages will be less tedious in the future.

Lots of smaller bug fixes:

• URL hashes are now much nicer, e.g. #Mocks.CreateUserAndSave instead of #ypeMocksCreateUserAndSave -- the language indexer gets to choose and uses what makes sense for the language.
• URL paths are now much nicer, e.g. /-/docs/cmd/frontend/auth instead of /-/docs/cmd-frontend-auth
• Error handling: we now differentiate between "No LSIF data" and "LSIF data, but no documentation"
• Useless Go blank identifier assignments var _ = ... are no longer included.
• Symbols defined within e.g. Go functions are no longer incorrectly included.
• We no longer sometimes emit an empty e.g. "Functions" section

[emidoots]

Week of June 25, July 2

Outside API docs

• Created onboarding material for CSE team members: https://drive.google.com/file/d/1Veat9m5gb8O0fL37b-lD5rl5fKToTmb6/view?usp=sharing
• Joined podcast w/Andrew Kelley & Beyang
• Paired with Coury to solve authorization issues with designing code insights backend
• Chatted with Dan/Distribution about Kubernetes/Docker-Compose tradeoffs and challenges in enterprise settings, brainstormed solutions

API docs

See CHANGELOG: https://github.com/sourcegraph/sourcegraph/blob/main/CHANGELOG.md#api-docs-experimental

video: https://drive.google.com/file/d/126LLrQanXH7rHr0d8d_qvmnSpdefBa2V/view?usp=sharing

[emidoots]

Key highlights recently for posterity:

• July 22nd, authored a sneak-peak blog post with the marketing team - much of the frontend work for MVP done.
• July 28th, usage examples landed: https://twitter.com/slimsag/status/1420584631993180160
• Aug 12: https://twitter.com/slimsag/status/1425998256039813123
  • Now indexed over 71 million Go symbols.
  • Working on redesigned UI with Quinn Keast.
  • Implemented major parts of redesigned UI.
  • Fixed issues with using it on Go standard library and Sourcegraph codebase.
  • Massive page performance improvements
  • The sidebar is MUCH better now, it follows as you scroll on the page, etc.

Notable mentions outside of API docs:

• Helping code insights team to beta launch while Coury was OOO (1/2 week).
• Being proactive by helping our Security team and Search team: https://github.com/sourcegraph/sourcegraph/pull/23949
  • I've taken some time over my weekend to do a full audit of our CSRF threat model, documented it extensively so anyone (even those who aren't experts with CSRF) can reasonably understand it, and made some key reccomendations that we can employ to ensure that we continue to have good security as more people work on Sourcegraph and we extend our API usage to more third-party sites, etc.
  • This also includes reccomendations for fixing the core issue where, currently, it is impossible for third-parties to leverage the Sourcegraph.com API on their own website. I will be looking into implementing that reccomendation to fixing this issue, time permitting, in the coming weeks.
• Assisting Marketing in setting up GitHub sponsors
• Assisting Search team in syntax highlighting stream highlighted results #24097
• Documenting problems in our GraphQL API and advising next steps: Remediate the severely messed up "GitBlob implements TreeEntry" inheritence structure Remediate the severely messed up "GitBlob implements TreeEntry" inheritence structure #23588
• Advising Core team on: why our syntax highlighter is in bad shape / has a lot of "tech debt", and what we should do about it why our syntax highlighter is in bad shape / has a lot of "tech debt", and what we should do about it #21942