Should we dial the modularization a bit down? Do we all see the value on having everything as a module?

[daviddias]

Disclaimer: This proposal is not about going monorepo. Please read the full thread :)

I've been thinking a lot on building quality, maintainability and robustness on software and taking a peek at other languages and ecosystems to drive inspiration. One of the thoughts out of that journey is the simple question: Do we all see the value on having everything as a module?

For context, I'm one of the main culprits to which IPFS is so modularized. In fact, when I joined the project in 2015, it was pretty much a massive go-ipfs repo, with the exception of multiformats. I then took on js-ipfs as a greenfield project and applied the best of what I understood on modularization as a value which I believed to add a lot to a protocol stack that prioritizes interoperability, future proofing and maintainability. Looking backwards, I'm happy we all onboarded the modularization train as it enables IPFS to run on Node.js and the Browser, use a multitude of transports, some that hadn't been invented yet, and overall sets IPFS and libp2p on the track to iterate and adapt as systems evolve.

All of that said, I do think that there are some artifacts of this modularization goal that actually add cost without any gains. This is specially true when we focus on parts of the "engine" that are unique and integral to the system as a whole and non usable elsewhere.

One of the clear examples is js-libp2p and js-libp2p-switch in which we spent countless hours doing sync releases, duplicating code (the top level API is pretty much the same), duplicating documentation efforts, duplicating tests and adding overhead to the developers that had to tinker with both modules (and repos) at the same time. js-libp2p-switch is an integral part to the js-libp2p dialing engine and would be better off living inside js-libp2p itself.

There are other examples and looking at things such as #1670, we can quickly gauge how much it costs to have this over-modularization.

If this resonates with others, I'll go ahead and make a short proposal of where I believe we could head to reduce overhead/admin work:

IPFS side

• Merge into js-ipfs, the following modules:
  • ipfs-multipart
  • js-ipfs-mfs
  • js-ipfs-repo
  • js-ipfs-block-service
  • js-ipfs-unixfs-exporter
  • js-ipfs-unixfs-importer

libp2p side

• Merge into js-libp2p, the following modules:
  • js-libp2p-connection-manager
  • js-libp2p-identify
  • js-libp2p-keychain
  • js-libp2p-nat-mngr
  • js-libp2p-pnet
  • js-libp2p-ping
  • js-libp2p-switch
  • (potentially, need to think more) js-libp2p-circuit
• Merge into libp2p-websocket-star, the following repo:
  • libp2p-websocket-star-rendezvous
• Coalesce the multiple libp2p interfaces into js-libp2p-interfaces, these include:
  • interface-connection
  • interface-transport
  • interface-stream-muxer
  • interface-peer-discovery
  • interface-peer-routing
  • interface-content-routing

Looking forward to learn how people think about such transition.

[jacobheun]

I'm highly in favor of coalescing the core modules for each of these. Even if we still released them individually with something like lerna so that developers who want fine grained control can get it, testing, development and releasing would be significantly easier. I currently use VSCode workspaces to group the modules like this anyway, it just doesn't have the benefit of actually being combined.

Doing that would also potentially enable us to create and more easily deploy up-to-date, preconfigured versions of js-libp2p, etc with each release, to help users get running more quickly.

(potentially, need to think more) js-libp2p-circuit

Switch already includes it, it's a core module imo. People can always turn it off in config, but being able to ship libp2p and ensure it's already dialable over any circuit relay is nice. I think we can have the module discussion in a separate ticket if this moves forward so we don't veer too far from the intended conversation.

[daviddias]

Switch already includes it, it's a core module imo. People can always turn it off in config, but being able to ship libp2p and ensure it's already dialable over any circuit relay is nice.

Good to know, then it makes total sense then!

@jacobheun feel empowered to move independently in libp2p land. I believe that libp2p can move lot faster and reduce a ton of overhead by adopting this strategy.

For any of these updates. It would be ideal to merge the gittrees so that contributors are preserved.

[dirkmc]

I agree that having a lot of different modules adds to maintenance and release overhead, and it can be hard for a new person trying to dig through the code to understand how it works.

On the other hand I think we can do a better job of harnessing the good will of the community to help us out with bug fixes and improvements, and modularization helps us in a couple of important ways:

• currently it takes around one hour to run the js-ipfs tests.
  I'm concerned that if we include modules like mfs, repo and the importer / exporter it will take even longer.
• IPFS is really a grab bag of several different technologies. I think it's simpler for someone new to the project to fix the bug for the particular thing they're working on if they can make a change to one smaller part of the code, and run and test just that part.

Ideally we would have

• tests in js-ipfs that are generalized and run fast
• tests in the module itself that are specific and more thorough
• a way for community members to propose additions/updates to documentation without having to do PRs (and go through the build process)

[daviddias]

Fair points. Here are some thoughts

currently it takes around one hour to run the js-ipfs tests.

This is both a good and a bad sign:

• It suggests that there are probably a ton of opportunities to optimize how tests get set up and how assertions are done. For example, why make everything sequential and not run each suite in parallel?
• That we probably should rely on CI more to run the multiple test suites.
• That we have a ton of tests. One should expect a P2P File System that works both on the browser to be tested thoroughly and that translates for it to take time.

Btw, you can also use mocha to select a subset of the tests with test filters:

Test Filters
  --fgrep, -f   Only run tests containing this string                   [string]
  --grep, -g    Only run tests matching this string or regexp           [string]
  --invert, -i  Inverts --grep and --fgrep matches                     [boolean]

Another thing to consider is to take a peek at the latest developments of Jest -- https://jestjs.io --, it would minimize test time by just running the tests that touch code paths that have been changed.

I think it's simpler for someone new to the project to fix the bug for the particular thing they're working on if they can make a change to one smaller part of the code, and run and test just that part.

It's kind of a double edge sword. In one side it is easy because one has to parse less code, at the same time, one might spend hours hacking on something that misses another part of the codebase that will break with that change (it has happened many times before), then putting even a greater amount of stress on the maintainers to not only review and understand how the change affects the module but also how the change will affect the entire stack.

[hugomrdias]

Merging some repos would be extremely helpful for developer productivity.

[daviddias]

This proposal is not about going monorepo, that is at the other end of the spectrum. I recommend not using that word otherwise multiple bikesheds will get painted ad eternum :)

All that this proposal is about is about coalescing some of the modules/repos that are pretty much integral (i.e. not swapable) to the stack.

[hugomrdias]

Bikeshed averted phew 🧐

[jacobheun]

fwiw, go-libp2p did this recently with their core libp2p modules, https://github.com/libp2p/go-libp2p-core. While I haven't worked in the code base, searching through the code when I've been trying to find certain behavior has become significantly easier.

[JustMaier]

Super happy to see this discussion happening. As someone that jumped into the community pretty recently, having so many separate micro-repos definitely made digging through how things worked more difficult.

[vasco-santos]

Totally in favor of getting all mentioned libp2p modules into the main repo, getting all the core pieces for libp2p together, which will be helpful for maintenance, releasing and ease the way for new contributors.

Also heavily in favor of having libp2p-interfaces with all our current interfaces. go-libp2p already went on that way, and I believe this will allow us to present the interaction between each piece of libp2p in a much clear way, improving our docs, at the same time as easing the maintenance cost