Bundling js-ipfs-api

[haadcode]

We should explore the options and solutions to reducing the size of the distributed build of js-ipfs-api (dist/index.min.js).

Originally started here: ipfs/js-ipfs#398.

As it stands today, the dist file is at ~900kb. How far can we go? :)

[haadcode]

ConsenSys is using https://github.com/pelle/browser-ipfs for their IPFS wrapper. Their reason is simply the size: 900kb (js-ipfs-api) vs. 2kb (browser-ipfs).

@diasdavid @dignifiedquire this begs the question: why is js-ipfs-api so large? What are the dependencies that make it so large? If js-ipfs-api can't use less dependencies, should we talk about a light api lib, much like browser-ipfs?

[haadcode]

@diasdavid said:

@haadcode Thank you for bringing that up. It should go, however, into a js-ipfs-api issue, we will probably get a lot of savings from the dedup on js-ipfs-api.

This makes me thing that it would be pretty dope if js-ipfs-api was modular inside like async, so that devs can do require('ipfs-api/cat') and get just the bit they need.

[haadcode]

This makes me thing that it would be pretty dope if js-ipfs-api was modular inside like async, so that devs can do require('ipfs-api/cat') and get just the bit they need.

That would be sweeeeeet!

[haadcode]

Another comment that was related to this:

Some feedback from elsewhere (cc @pelle). @diasdavid @dignifiedquire make sure to note the comment re. dependencies.

Primary reason for using browser-ipfs:

Lots of unnecessary dependencies cause fragility and huge integration problems both in browser and ReactNative.

Secondary reason is size, which isn't an issue in ReactNative but a very big issue on the web in particular mobile web:

js-ipfs-api

99% 2016-08-18 11:38:01 ☆  |2.3.0| Big-Corn-Island in ~/code/js-ipfs-api
± |master ✓| → ls -l dist/
total 8600
-rw-r--r--  1 pelleb  staff  1550487 Aug 18 11:36 index.js
-rw-r--r--  1 pelleb  staff  1925637 Aug 18 11:36 index.js.map
-rw-r--r--  1 pelleb  staff   920487 Aug 18 11:36 index.min.js
browser-ipfs

± |master → origin {2} ✓| → ls -l dist/
total 8
-rw-r--r--  1 pelleb  staff  1592 Apr 25 21:53 ipfs.min.js

[dignifiedquire]

It is important to note, that browser-ipfs does not nearly implement the full api, where as js-ipfs-api aims to implement the full spectrum. That said I believe there is a lot we can do to reduce the size, but probably not down to 2kb :)

[dignifiedquire]

Dependency analysis with the latest master of aegir:

elliptic: 304.63 KB (14.2%)
bn.js: 170.29 KB (7.97%)
readable-stream: 146.03 KB (6.83%)
asn1.js: 126.35 KB (5.91%)
split2: 100.53 KB (4.70%)
  readable-stream: 94.3 KB (93.8%)
  through2: 4.18 KB (4.16%)
  isarray: 120 B (0.117%)
  <self>: 1.92 KB (1.91%)
hash.js: 80.62 KB (3.77%)
core-js: 70.75 KB (3.31%)
stream-http: 64.34 KB (3.01%)
  readable-stream: 49.61 KB (77.1%)
  <self>: 14.73 KB (22.9%)
browserify-aes: 61.74 KB (2.89%)
lodash._baseiteratee: 56.03 KB (2.62%)
sha.js: 52.65 KB (2.46%)
ndjson: 50.02 KB (2.34%)
  readable-stream: 47.15 KB (94.3%)
  through2: 2.09 KB (4.18%)
  isarray: 120 B (0.234%)
  <self>: 675 B (1.32%)
buffer: 47.47 KB (2.22%)
des.js: 40.69 KB (1.90%)
glob: 35.94 KB (1.68%)
diffie-hellman: 35.13 KB (1.64%)
browserify-sign: 29.14 KB (1.36%)
hoek: 28.21 KB (1.32%)
minimatch: 24.67 KB (1.15%)
parse-asn1: 23.97 KB (1.12%)
url: 23.08 KB (1.08%)
lodash: 22.44 KB (1.05%)
protocol-buffers: 20.93 KB (0.979%)
create-hash: 20.75 KB (0.971%)
tar-stream: 20.18 KB (0.944%)
  end-of-stream: 2.31 KB (11.5%)
  <self>: 17.87 KB (88.5%)
protocol-buffers-schema: 19.54 KB (0.914%)
lodash._stringtopath: 18.16 KB (0.849%)
wreck: 17.63 KB (0.825%)
public-encrypt: 17.62 KB (0.824%)
async: 16.37 KB (0.766%)
ipfs-merkle-dag: 16.24 KB (0.759%)
util: 15.4 KB (0.720%)
assert: 15.08 KB (0.705%)
qs: 14.96 KB (0.700%)
punycode: 14.33 KB (0.670%)
lodash._baseeach: 14.24 KB (0.666%)
multiaddr: 12.23 KB (0.572%)
ripemd160: 11.91 KB (0.557%)
is-property: 10.76 KB (0.503%)
boom: 10.1 KB (0.472%)
ip: 9.96 KB (0.466%)
fs.realpath: 9.58 KB (0.448%)
events: 8.13 KB (0.380%)
string_decoder: 7.61 KB (0.356%)
create-ecdh: 7.49 KB (0.350%)
lodash.map: 7.37 KB (0.345%)
miller-rabin: 6.45 KB (0.302%)
browserify-cipher: 6.25 KB (0.292%)
cipher-base: 6.25 KB (0.292%)
path-browserify: 6.04 KB (0.283%)
core-util-is: 5.9 KB (0.276%)
pbkdf2: 5.54 KB (0.259%)
bl: 5.23 KB (0.244%)
crypto-browserify: 5.12 KB (0.239%)
querystring-es3: 5.06 KB (0.237%)
process: 4.4 KB (0.206%)
constants-browserify: 4.33 KB (0.202%)
brace-expansion: 4.23 KB (0.198%)
lodash._basetostring: 4.16 KB (0.194%)
create-hmac: 4.03 KB (0.188%)
evp_bytestokey: 3.88 KB (0.181%)
vm-browserify: 3.71 KB (0.174%)
browserify-des: 3.63 KB (0.170%)
multihashes: 3.55 KB (0.166%)
stream-browserify: 3.54 KB (0.166%)
browserify-rsa: 3.53 KB (0.165%)
base64-js: 3.24 KB (0.152%)
buffer-shims: 3.16 KB (0.148%)
util-deprecate: 3.15 KB (0.147%)
stable: 2.94 KB (0.137%)
lodash.filter: 2.79 KB (0.131%)
brorand: 2.54 KB (0.119%)
sandwich-stream: 2.5 KB (0.117%)
babel-runtime: 2.47 KB (0.115%)
timers-browserify: 2.06 KB (0.0962%)
process-nextick-args: 2.03 KB (0.0950%)
base-x: 2.02 KB (0.0945%)
randombytes: 2.01 KB (0.0942%)
ieee754: 2.01 KB (0.0938%)
promisify-es6: 1.91 KB (0.0892%)
is-ipfs: 1.89 KB (0.0885%)
signed-varint: 1.84 KB (0.0863%)
  varint: 1.42 KB (77.0%)
  <self>: 435 B (23.0%)
builtin-status-codes: 1.71 KB (0.0800%)
varint: 1.67 KB (0.0779%)
multipart-stream: 1.42 KB (0.0664%)
inherits: 1.31 KB (0.0614%)
generate-function: 1.27 KB (0.0592%)
balanced-match: 1.15 KB (0.0539%)
multihashing: 1.14 KB (0.0532%)
webpack: 1.03 KB (0.0481%)
lodash._basefilter: 996 B (0.0455%)
ipfs-block: 934 B (0.0427%)
os-browserify: 927 B (0.0423%)
wrappy: 905 B (0.0413%)
inflight: 897 B (0.0410%)
to-arraybuffer: 881 B (0.0402%)
is-stream: 800 B (0.0365%)
xtend: 768 B (0.0351%)
streamifier: 685 B (0.0313%)
path-is-absolute: 599 B (0.0274%)
isstream: 588 B (0.0269%)
minimalistic-assert: 504 B (0.0230%)
once: 417 B (0.0190%)
buffer-xor: 412 B (0.0188%)
flatmap: 382 B (0.0174%)
concat-map: 345 B (0.0158%)
https-browserify: 325 B (0.0148%)
generate-object-property: 287 B (0.0131%)
isarray: 264 B (0.0121%)
bs58: 206 B (0.00941%)
detect-node: 202 B (0.00923%)
indexof: 199 B (0.00909%)
webcrypto: 154 B (0.00703%)
<self>: 50.22 KB (2.35%)

[pelle]

My general beef with most of the JS code in the ethereum world and I guess also the IPFS world is the reliance on node infrastructure and many, many dependencies.

I think the reason being is that it's easy to develop libraries on this, but also the IMHO mistaken belief that having code that works on both node and in the browser is a good thing.

Size is the most obvious problem here, for example instead of using the browsers built in networking libraries much of what is implemented in these many dependencies are things that already exist. They may not be as good, but they do the trick for pretty much all browser cases, because in the end thats what is used under all the dependencies and fancy facades anyway.

So if the primary reason for creating js-ipfs-api is to run in node. Thats fine and it's awesome that it can in a pinch run in the browser. But if we want web apps interacting with ipfs we need to implement things in a way thats efficient for the browser.

The less obvious problem is fragility, due to so many dependencies, duplicated dependencies, multiple libraries that do the same thing. Each of these libraries causing problems with browserify, webpack what have you.

So I know and understand thats how node work and it works well in a simple uniform environment like the v8. But with all the different browsers and now also things like React Native it is a literal nightmare interacting with these libraries.

The most important thing I think the IPFS community can do is fully document the http api well. Interacting with http is the lowest common denominator and any good web developer understands how to do this. My browser-ipfs shows that for most cases it's not necessary to be particularly complex.

Secondly different classes of js api's need to be created. This project I only just learnt about yesterday, so I'm learning about it now and may not understand much about it yet.

But the needs of people interacting from the browser and from doing ipfs server integrations are very different.

Anyway I'm happy to help out and will attempt to make browser-ipfs as compliant as possible without adding to the dependency list and add more features as needed.

[jbenet]

cc @RichardLitt for http api docs comments above

[jbenet]

@dignifiedquire im not sure why these are necessary. You could prob drop 30% on crypto libraries alone!

# okay why is there an impl of (what i can only guess is) ECC
elliptic: 304.63 KB (14.2%)

#  i imagine this is here because of the crypto libs
bn.js: 170.29 KB (7.97%)

readable-stream: 146.03 KB (6.83%)  # wow that's big.
asn1.js: 126.35 KB (5.91%)  # why do we need asn1.js encoding? TLS?

# more crypto stuff... why? TLS?
des.js: 40.69 KB (1.90%)
diffie-hellman: 35.13 KB (1.64%)
browserify-sign: 29.14 KB (1.36%)
ripemd160: 11.91 KB (0.557%)
create-ecdh: 7.49 KB (0.350%)

[dignifiedquire]

@jbenet yes this is a known issue with our current set of crypto libraries, neither forge nor browserify-crypto are modular. This means using a single crypto function (in this case sha2 from multihashing) imports the complete! crypto library.

In this specific case we could replace https://github.com/multiformats/js-multihashing/blob/master/src/index.js#L4 with direct imports to sha.js but to properly fix these issues we need a modular crypto library, plus the things I'm currently drafting a proposal for, which will improve a lot of these issuse.

[pelle]

I was looking through the code trying to find out where the digest code is called and it seems just from my skimming around to be wherever the DAGNode and DAGLink is called. I'm still learning about how these things work so forgive my ignorance. But do they ever actually need to create digests themselves in this particular use case?

If they do then perhaps a lightweight version of the library can be generated using webpack or browserify with only the higher level apis which would solve the majority of these issues.

I agree a good clean sha implementation with only a few dependencies is important. I'll see if I can help with that, since at least that part I do understand :-).

[pelle]

Sorry I just realized that sha.js will do exactly that. @dignifiedquire replacing that line as you suggest will absolutely be a huge help. That may even let me use it on ReactNative.