Public API for Sketch management

[andrewn]

This is a proposal for a public-facing API that would allow users to automate the upload of sketches into the editor, replacing the scripts that talk to mongodb directly. (See #542)

Supported actions

The key actions for this API are:

• creating a sketch as a user
• adding files to the sketch

The scripts deleting existing sketches, but I've also proposed a way to replace the contents of a sketch too.

I've added the proposed API documentation as a markdown file so that we can comment on the file directly.

Files instead of IDs

The key change (as discussed in #1073) is representing trees hierarchically and referencing files/directories via their path. This seems more intuitive for API users, but some investigation is needed figure out if we still need to reference files by ID. And if so, perhaps the filename approach is only used as a convenience for sketch creation.

Further work

• File storage: I need to dig in and understand a little better how file storage currently works as the files returned by the API might need to be redirects to S3 or other hosting. And this would impact how the files are specified also.
• Conflicts and timestamps: The editor uses a updatedAt timestamp to protect against overwriting stale projects. These should probably be added everywhere.

[catarak]

Overall I like this! Some thoughts/questions/etc..:

• Should there be a route to add a new file to a sketch? This could possibly also function as a route to upload media files.
• What's your thinking behind having the PUT and PATCH routes for completely replacing a sketch vs updating it? What's a scenario in which you think this would be useful? Is it so a user can update a sketch without changing its url? There's a secret feature of the editor which doesn't have a frontend UI yet, in which you can access a sketch via a slug url. I added this for the release of the Generative Design book, (e.g. https://editor.p5js.org/generative-design/sketches/P_1_0_01 works!). I don't know if this would add to the API at all, but just throwing that information out there.
• With respect to media file uploads, my suggestion is:
  POST /:user/files/upload, which accepts

{
  name: "dog.jpg",
  content: <file_content>
}

and returns

{
  name: "dog.jpg",
  url: <new_s3_url>
}

This could also accept a query parameter that's uploadType=media|multipart for multipart uploads. The URL could then be used as an attribute on a file. This gets a little hairy if a user then decides to use the URL for multiple files/sketches—right now, if a user deletes a sketch, then the file gets deleted from S3. This makes sense, since in the web editor, you can't manually match a file to a url, it's all done behind the scenes for a user.

Another solution would be to only allow media uploads after a sketch has been created, and then use a new file route to upload the media file to the sketch itself.

• Now, timestamps: it feels like asking users to keep track of updatedAt dates is a lot. I looked at how the Google Docs API handles this, and it uses transactional add text/delete text commands. Maybe, at a certain point, we can't protect users from themselves. I don't know. I wonder how other APIs handle this.

[andrewn]

• Should there be a route to add a new file to a sketch? This could possibly also function as a route to upload media files.

Yeah, there should be. Something like PUT /:user/sketches/:id/files/:path would create a new file/directory if it didn't exist. Or replace it if it did. I wonder whether this route should just be replaced with PUT?

[andrewn]

• What's your thinking behind having the PUT and PATCH routes for completely replacing a sketch vs updating it? What's a scenario in which you think this would be useful? Is it so a user can update a sketch without changing its url?

Yep, that's part of it. Both PATCH and PUT would maintain the URL of the sketch being changed. The difference is PUT would remove everything first. And PATCH would add new things, keeping what's there, possibly changing the file contents. I do wonder if supporting these two methods for modifying the file tree is a bit confusing though and would be quite challenging to implement.

The way the current import scripts work, they delete all sketches and then re-upload them, creating new sketch IDs. This means all existing sketch links are broken. Maybe that's desired, but it might be nice for the author to choose which behaviour they want.

Another use case, is being able to change sketch metadata without having to delete the sketch. Right now, this is just the sketch name.

There's a secret feature of the editor which doesn't have a frontend UI yet, in which you can access a sketch via a slug url. I added this for the release of the Generative Design book, (e.g. https://editor.p5js.org/generative-design/sketches/P_1_0_01 works!). I don't know if this would add to the API at all, but just throwing that information out there.

I think the slug URL is a great feature and would remove the need for supporting sketch ID stability above. For v1 of the public API, shall we just support specifying a slug, and not support PUT/PATCH for sketches? This would be much simpler.

I think PATCH might be necessary if we want to eventually support clients like the Editor. Although the lower-level Files endpoints would allow better control over file operations anyway.

[andrewn]

For file uploads, I think having files' belong to sketches makes things easier as it means we can be certain when a sketch is deleted then all the associated files can be safely removed from S3. And also, when a file is deleted from a sketch, it won't be in use by other sketches so can again be deleted from S3.

I'll prototype this out in code so we can see how it works with the upload scripts.

[andrewn]

Now, timestamps: it feels like asking users to keep track of updatedAt dates is a lot. I looked at how the Google Docs API handles this, and it uses transactional add text/delete text commands. Maybe, at a certain point, we can't protect users from themselves. I don't know. I wonder how other APIs handle this.

I agree that a user managing timestamps seems a bit much. I'll do some more research in how different APIs handle this. For the bulk-import use case, we can assume that a user creating or modifying a resource within the sketch—either files or the sketch itself—updates an updatedAt timestamp returned by the API.

[andrewn]

Thanks for the comments!

We can keep this open for more discussion and I'll start writing some code to implement a basic version of these endpoints.

[catarak]

I think the slug URL is a great feature and would remove the need for supporting sketch ID stability above. For v1 of the public API, shall we just support specifying a slug, and not support PUT/PATCH for sketches? This would be much simpler.

great! i'll make sure there's an index in the db for the slug key.

I think PATCH might be necessary if we want to eventually support clients like the Editor. Although the lower-level Files endpoints would allow better control over file operations anyway.

do you mean if folks want to use the API in the context of another web application/GUI? like maybe we don't even want to suggest that could be possible right now, as it adds a huge layer of complication. i think we want to focus on moving sketches to/from the web editor, rather than adding a new UI for editing sketches.

[andrewn]

I think PATCH might be necessary if we want to eventually support clients like the Editor. Although the lower-level Files endpoints would allow better control over file operations anyway.

do you mean if folks want to use the API in the context of another web application/GUI? like maybe we don't even want to suggest that could be possible right now, as it adds a huge layer of complication. i think we want to focus on moving sketches to/from the web editor, rather than adding a new UI for editing sketches.

I was thinking that ultimately we might just have one API for the editor and public uses. But you're right, that overcomplicating things for now.

[andrewn]

I've realised that the API needs to support referencing files hosted elsewhere on the internet. For example, the ml5 examples reference models hosted in GitHub via jsdelivr.

I've split the file into two to support this. What do you think @catarak?

[andrewn]

I've just split this into two documents, public_api.md and public_api_proposed.md to make it clear what is being implemented in v1 and what is just ideas.

Will merge this into the feature/public-api branch now. We can have further discussions in new issues or PRs.