State of support for `net/http` on `wasm` targets running within browsers on commodity hardware (AMD64/ARM64 etc)

[owenwaller]

Hi,

Could some one please be clear on the current level of support offered by tinygo for the net/http package when building for a wasm target, but where the wasm binary will be executed in a browser on common hardware/OS combinations e.g. linux/AMD64, linux/ARM64, macos/ARM64, windows/AMD64 etc?

I understand the situation for embedded hardware platforms is very different, where there may not be an OS as such and the API's can vary wildly between different devices. But I am not asking about wasm net/http support on those platforms. Nor, am I asking about support for other non-browser JS run-times (e.g NodeJS).

The standard Go compiler has supported net/http for wasm within a browser since wasm support was first released. As far as I am aware this was achieved by mapping the net.http.Get(...) and net.http.Client.Get(...) methods to the fetch api provided by the browser.

Could the tinygo project not take a similar approach, even if this means specifying a new target e.g -target=wasmbrowser?

I am aware of other 3rd party libraries (for example https://pkg.go.dev/marwan.io/wasm-fetch) that already do support the browser's fetch API. But these in general are not appropriate for two reasons:

1. All of the Go programmer knowledge about how to use the `net.http.Client is lost.
2. The programmer may not be calling net.http.Client.Get(...) directly in their code. It could be being called via a 3rd party package the programmer is using. In this case, short of PR to the 3rd party package authors, or a fork of the 3rd party package, it is impractical to change the 3rd party package over to use an alternative to the net/http package API's.

tinygo therefore needs to support as much of the net/http package is as appropriate when the wasm is executed within a browser.

Having the net/http package available would then enable tinygo as suitable compiler for wasm based web UI's. These UIs are typically making http.Get(...) calls to exchange JSON payloads with a server and then processing the response to update the UI. But at the minute this doesn't look like its possible.

Two very simple examples:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
       // use the default http client in net/http
	_, err := http.Get("http://www.tinygo.org/") // <--------- runtime panic here AFAIK
	if err != nil {
		err := fmt.Errorf("http.Get fails with: %w", err)
		log.Fatal(err)
	}
}

and

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	c := http.Client{}
	_, err := c.Get("http://www.tinygo.org/") <----------- runtime panic here AFAIK
	if err != nil {
		err := fmt.Errorf("http.Get fails with: %w", err)
		log.Fatal(err)
	}
}

These both result in the runtime panic of:

main.wasm:0x20f8d 
Uncaught (in promise) 
RuntimeError: unreachable
    at main.runtime.runtimePanicAt (main.wasm:0x20f8d)
    at main.runtime.nilPanic (main.wasm:0x4e6a)
    at main.(:8888/*net/http.Client).Get (http://localhost:8888/main.wasm)
    at main.runtime.run$1 (main.wasm:0x29e5b)
    at main.runtime.run$1$gowrapper (main.wasm:0x202e5)
    at main.tinygo_launch (main.wasm:0x6b5)
    at main.(:8888/*internal/task.Task).Resume (http://localhost:8888/main.wasm)
    at main.runtime.scheduler (main.wasm:0x205c4)
    at main._start (main.wasm:0x20219)
    at main._start.command_export (main.wasm:0x6a07c)

Is there some sort of update available on support for net/http when the wasm is executed by a browser?

If this isn't supported (as appears to be the case) is there a timeline available for this support or a list of actions that have to be taken?

Many thanks

Owen

[bradleypeabody]

In terms of approaches that can be used to fix this - can anyone on the TinyGo team comment on the approach of basically copying this file from the stock Go repo https://cs.opensource.google/go/go/+/refs/tags/go1.23.0:src/net/http/roundtrip_js.go into the appropriate TinyGo src dir and hacking on it until it works? net/http as a package is seems like it unlikely to ever work as intended on microcontrollers, but for Wasm this client use case is extremely common and useful. So I'm thinking just bringing over that subset into the TinyGo code so the sorts of examples shown above by @owenwaller can function, tuning it until it works - it's probably the best approach for the net/http package overall.

So I think a key question is, is this something the TinyGo team would entertain a PR for?

[atdiar]

release 0.26 seemed to include a net package. But it disappeared with the next release. Does anyone have a clue what happened?

cc @deadprogram

edit: the answer is that the net package seems to be ported (at least partially), using netdev.

[owenwaller]

@gedw99 @deadprogram

Ah, but this actually shows the problem I am trying to get the tinygo team to fix.

The problem is, in the code that was referenced it uses this import:

    "github.com/syumai/workers/internal/jshttp"

See here.

If we poke around a bit we'll see the code base is littered with calls like this that use the Call(...) function to call into JS.

Specifically the Fetch(...) function copied below.

func (s *DurableObjectStub) Fetch(req *http.Request) (*http.Response, error) {
	jsReq := jshttp.ToJSRequest(req)

	promise := s.val.Call("fetch", jsReq)
	jsRes, err := jsutil.AwaitPromise(promise)
	if err != nil {
		return nil, err
	}

	return jshttp.ToResponse(jsRes)
}

So here, the author is calling the JS fetch API directly from Go via the syscall/js package. There are other similar examples (not always on the fetch API) in the rest of the code base, including the internal packages.

I have never said that this can't be done.

My point is, and the reason for this request (or plea!) is that this wrapping is totally and utterly unnecessarily on a GOOS=js GOARCH=wasm browser target. The Go standard library already does all of this for us. It's been tested to death and just works. We just need the tinygo team to offer the same support as the standard Go compiler for this specific platform case.

I understand that it will be (almost has to be) different for micro-controller targets and that is fine. But these are not my target platform and aren't the target platform for this request.

This lack of support for net/http on a GOOS=js GOARCH=wasm browser target is leading to a fragmentation of the tinygo/Go compiled as wasm ecosystem.

Every project now has to use their own wrapper over the JS fetch API. See here for another totally different example.

This helps no one in the end.

What we want is for everyone targeting a GOOS=js GOARCH=wasm browser target to be able to use the Go standard library net/http packages. In the same way as they use the other standard library packages that tinygo already supports.

If we can have that, then both of the above wrappers (and all the others) are no longer required.

Critically, we can run the same code, compiled with the standard Go compiler. So we can immediately test using a simple go test on the native platform. We would then have (identical) code reuse between the native and tinygo target platforms.

A reduction in the amount off fragmentation can only be a good thing. If we can fix this, I think it would make a difference to the community who use the standard Go compiler to compile to a GOOS=js GOARCH=wasm browser target. It could increase the adoption of tinygo as a tool chain as it would substantially improve an area where tinygo is currently weak (IMHO).

Note: For the avoidance of any doubt. I have code that is used on both a command line tool and a web UI (via the vugu project) (the application is closed source...sorry). The code makes http.Get(...) calls. At the minute I cannot use tinygo to compile this code. As a result I can't use tinygo to build the web based UI. Given the binary size difference that using tinygo would make I'd like to use tinygo, but I can't. The lack of support for the standard Go net/http package then becomes an absolute blocker.

[joeblew999]

Thanks for the thorough explanation.

I share your frustration