Awesome Endeavour: Async Iterators #1670
Description
JS IPFS supports two types of stream at the API level, but uses pull streams for internals. If I was working on js-ipfs at the time I'd have made the same decision. Since then, async/await became part of the JS language and the majority of JavaScript runtimes now support async/await, async iterators and for/await/of (i.e. no need to transpile). These tools give us the power to stream data without needing to rely on a library.
Just because there are new language features available doesn't mean we should switch to using them. It's a significant upheaval to change the core interface spec and its implementations (js-ipfs, js-ipfs-api etc.) without good reason.
That said, it has become apparent that there are a growing number of good reasons to do this:
- Reduction in bundle size - no need to bundle two different stream implementations, and their eco-system helper modules, no need for the
asyncmodule. - Reduce
npm installtime - fewer dependencies to install. - Allows us to remove a bunch of plumbing code that converts Node.js streams to pull streams and vice versa.
- Reduces API surface area, no
addPullStream,addReadableStream. - Building an
interface-ipfs-corecompatible interface becomes a whole lot easier, no dual promise/callback API and no multiple stream implementation variations of the same function. It would also reduce the number of tests in theinterface-ipfs-coretest suite for the same reasons. - Node.js readable streams are now async iterators thanks to #17755!
- Of note, it is trivial to convert from pull stream to (async) iterator and vice versa.
- Unhandled throws that cannot be caught will no longer be a problem
- Better stack traces, stacks no longer clipped at async boundaries,
awaitstack traces better than promise stack traces - A modern, up to date and cutting edge API will aid community contributions and adoption.
The rough plan is:
- Drop support for dual callback/promise based APIs
- Expose only APIs that return promises or iterators for async actions
- Use async/await over then/catch when dealing with promises
This will require significant discussion and coordination from the JS teams. We'll need to reach agreement on the best API to expose for each module and manage releases carefully.
Below is a table documenting the multiformats, libp2p, ipld and ipfs modules that will likely need work. I suspect that some of these modules can be removed as they do not expose an async API. Likewise there's probably modules that got missed. If you notice either way then please edit the table or comment below.
If you'd like to own this enhancement task for a module then please comment below (or add yourself to the table if you know what you are doing). Please open a PR against the module asap (does not have to be anywhere near complete!) so we can add it here also and track progress.
- 🍎 = Not started
- 🍊 = In progress
- 🍏 = Complete
Core
Multiformats
- Current progress:
- 🍏 2/2
- 🍊 0/2
- 🍎 0/2
| Module | PR | Owner | Status | Priority |
|---|---|---|---|---|
| multihashing-async | multiformats/js-multihashing-async#37 | @hugomrdias | 🍏 | P0 |
| multistream-select | https://github.com/multiformats/js-multistream-select/releases/tag/v0.15.0 | @alanshaw | 🍏 | P1 |
libp2p
- Current progress:
- 🍏 25/30
- 🍊 3/30
- 🍎 2/30
IPLD
Please read #1670 (comment) before contributing.
- Current progress:
- 🍏 8/8
- 🍊 0/8
- 🍎 0/8
| Module | PR | Owner | Status | Priority |
|---|---|---|---|---|
| ipld | ipld/js-ipld#190 | @vmx | 🍏 | P3+ |
| ipld-bitcoin | ipld/js-ipld-bitcoin#48 | @vmx | 🍏 | P3+ |
| ipld-dag-pb | ipld/js-ipld-dag-pb#124 | @vmx | 🍏 | P1 |
| ipld-dag-cbor | ipld/js-ipld-dag-cbor#107 | @vmx | 🍏 | P1 |
| ipld-ethereum | ipld/js-ipld-ethereum#51 | @vmx | 🍏 | P1 |
| ipld-git | ipld/js-ipld-git#51 | @vmx | 🍏 | P1 |
| ipld-raw | ipld/js-ipld-raw#32 | @vmx | 🍏 | P1 |
| ipld-zcash | ipld/js-ipld-zcash#39 | @vmx | 🍏 | P1 |
IPFS
- Current progress:
- 🍏 19/19
- 🍊 0/19
- 🍎 0/19
Dependents
These modules use IPFS and fall under the ipfs/ipfs-shipyard umbrella so should also be updated.
Current progress:
- 🍏 1/44
- 🍊 2/44
- 🍎 40/44
| Module | PR | Owner | Status | Priority |
|---|---|---|---|---|
| awesome-ipfs | 🍎 | |||
| benchmark-js.ipfs.io | 🍎 | |||
| cid-utils-website | 🍎 | |||
| demo-ipfs-todo | 🍎 | |||
| ipfs-companion | @lidel | 🍎 | ||
| ipfs-desktop | 🍎 | |||
| ipfs-fuse | @alanshaw | 🍎 | ||
| ipfs-geoip | ipfs-shipyard/ipfs-geoip#67 | @nijynot | 🍊 | P0 |
| ipfs-iiif-db | 🍎 | |||
| ipfs-level | 🍎 | |||
| ipfs-blob-store | ipfs-shipyard/ipfs-blob-store#26 | @niinpatel | 🍊 | P3+ |
| ipfs-locations | 🍎 | |||
| ipfs-performance-profiling | 🍎 | |||
| ipfs-pubsub-peer-monitor | 🍎 | |||
| ipfs-pubsub-room | 🍎 | |||
| ipfs-pubsub-room-demo | 🍎 | |||
| ipfs-redux-bundle | 🍎 | |||
| ipfs-registry-mirror | 🍎 | |||
| ipfs-postmsg-proxy | @alanshaw | 🍎 | ||
| ipfs-pubsub-1on1 | 🍎 | |||
| ipfs-service-worker | @vasco-santos | 🍎 | ||
| ipfs-share-files | 🍎 | |||
| ipfs-stats | 🍎 | |||
| ipfs-webui | 🍎 | |||
| ipfsd-ctl | ipfs/js-ipfsd-ctl#353 | @achingbrain | 🍏 | |
| ipld-explorer | 🍎 | |||
| ipld-explorer-cli | @alanshaw | 🍎 | ||
| ipld-explorer-components | 🍎 | |||
| ipscend | 🍎 | |||
| npm-on-ipfs | @achingbrain | 🍎 | ||
| peer-crdt-textarea-binding | 🍎 | |||
| peer-flipchart | 🍎 | |||
| peer-pad-core | 🍎 | |||
| peer-star-app | 🍎 | |||
| peer-star-network-vis | 🍎 | |||
| peer-star-network-vis-react | 🍎 | |||
| peer-star-peer-color | 🍎 | |||
| peer-star-react | 🍎 | |||
| peerpad-peer-crdt | 🍎 | |||
| service-worker-gateway | 🍎 | |||
| tevere | 🍎 | |||
| window.ipfs-fallback | @alanshaw | 🍎 | ||
| y-ipfs-connector | 🍎 | |||
| ipld-graph-builder | 🍎 | P3+ |