[feedback wanted] Removing the webworker code from sql.js

[lovasoa]

I feel like today, providing a custom, limited webworker API, based on message-passing instead of promises is not the best choice, and is not useful to many. On the contrary, it creates confusion to have so many different assets available for download. When needed, it is very easy to turn the main library code into a web worker, in an automated way, without loosing any of the original API, using libraries like comlink.

What I propose :

• remove the worker-specific code from the main sql.js repository
• add a note in the readme about how to run the code in a web worker
• if someone can commit to maintaining the old WebWorker code (@kaizhu256 ?) : create a second repository in the sql-js organization, containing the webworker, publishing separate release assets. The repository could use continuous integration to automatically build assets without duplicating any code from the main sql.js repository. I can create the initial setup for the repository if someone is willing to maintain it in the long term.

[kaizhu256]

i have an idea to provide a minimal-maintennance promise-based worker in ~100 lines of code in src/api.js. will provide a pull-request of it to see what you think. been using my simple-hack for awhile in production. its usage is as follows:

<script src="dist/sql-wasm.js"></script>
<script>
// example-usage
var sql = await initSqlJs();
// src/api.js now has a new Worker class (~100 lines of extra code)
// with one promise-method Worker.prototype.postMessage
var worker = new sql.Worker(
    "https://foo.com/dist/worker.sql-wasm.js"
);
try {
    var result = await worker.postMessage({
        action: "exec",
        sql: "SELECT * FROM myTable"
    });
} catch (sqlError) {
    console.error(sqlError);
}
/*
result = {
    messageId: 1,
    results: [...]
};
 */
</script>

[brody4hire]

it is very easy to turn the main library code into a web worker [...] using libraries like comlink.

comlink definitely looks nice, maybe add an example to the documentation?

minimal-maintennance promise-based worker in ~100 lines of code in src/api.js

Definitely a nice idea. I wonder if we could put this in a separate module in this org and give a pointer. I am personally biased towards packages in small, focused modules whenever possible https://blog.sindresorhus.com/small-focused-modules-9238d977a92a

[lovasoa]

@kaizhu256 Let's try to stay focused. What do you think about separating the worker code from the main library ? If you have your separate library, you can then make the changes you want to it.

[kaizhu256]

admit i'm biased against splitting worker-code into separate module.

as product-developer, i've had years of poor-experience translating modular-based design-patterns into deliverable UX-features. and i feel like web-workers is integral to sql.js.

workers are mandatory for common use-case of manipulating data > 20mb in sql.js (and still have satisfactory UX). so i feel strongly it should be a builtin feature (if i can make it easy-to-maintain).

[lovasoa]

I do agree that most users of the library will want to use sql.js inside a web worker. But I don't think most want the library itself to provide the web worker. Modern javascript stacks almost always use a compiler, and people will want something that integrates well with their existing stack, not a separate file with a separate poorly-documented API.

[kaizhu256]

@lovasoa how about getting rid of worker.sql-xxx.js as follows:

1. make src/worker.js isomorphic (like - make worker.sql-wasm.js isomorphic (will load in web-worker, nodejs, and browser without errors). #376) and merge it into src/api.js

2. for legacy-compatibility, update Makefile to alias worker.sql-xxx.js from sql-xxx.js (eventually will deprecate and remove the alias).

3. update src/api.js with a no-frills SQL.Worker class that exposes a single promise-based method, SQL.Worker.prototype.postMessage

all-in-all we've gotten rid of maintaining the separate worker-file, and only need to document SQL.Worker.prototype.postMessage

[lovasoa]

The goal here was to remove the web worker, not to make it the default!
People using another web worker strategy or no web worker don't want to pay for other unrelated web worker code. I'm not sure I understand what would be the inconvenient of having your custom web worker interface distributed as a separate project. Why would it be more difficult for you to integrate and use ?

[brody4hire]

While I would favor moving the web worker code out, I wonder if we should ask ourselves what kind of an API we want to export in the first place. I can think of the following quick alternatives:

• SQL.js exports a lower-level, synchronous API and have users use a wrapper if they want a higher-level asynchronous API that could be used more directly from a worker
• SQL.js exports a slightly higher-level asynchronous API and would just work from a worker or a very simple wrapper
• export both synchronous and asynchronous APIs

While I would favor the first and downvote the last, I do think all of these alternatives should be valid for discussion. For example, Node.js fs API does export both synchronous and asynchronous API variants.

[kaizhu256]

because then the sqli.js online demo will depend on the external worker-github-repo. it creates more maintainability-issues for the demo.

[kaizhu256]

@brodybits, yes SQL.Worker.prototype.postMessage is a low-level, no-frills async worker-api (in 100 lines of code or less). its for product-developers like me who don't want to deal with yet-another-module to get basic-functionality out of sql.js.

[lovasoa]

You could have a demo using your api on your repo, and there would be a different demo using a more traditional stack on the main sql.js repo. What do you think ?

[kaizhu256]

so make the online-demo externally depend on experimental comlink? that sounds like more maintainnence headaches :`(