proposal: embed: add -embed=compress flag to go

[twmb]

The design proposal for embed contains a section on compression to reduce binary size. This has come up for me a few times recently and I wonder if there is any appetite for the final sentence of this section,

"Future work might be to add -embed=compress as a go build option for use in limited environments."

Having written the same decompression logic a few times now in a few different ways (decompress one file, decompress a tarball, gunzip vs. zstd), it'd be super convenient to have compression of embedded files or dirs built directly into Go.

Another benefit of of allowing compression at the build stage (and on-the-fly decompression once) is for code repositories. Currently, I am checking in a tiny compressed tarball and have tests against the sha256 of the checkin. Tarballs aren't really great for GH repositories, but considering how infrequently this tarball changes, it's an alright tradeoff right now.

This isn't a super pressing need, but I couldn't find a previous open nor closed proposal, and in the original proposal, the only comment that mentions compression is this one (the point of which is that TinyGo rarely targets RAM heavy systems).

[ianlancetaylor]

When are the embedded files decompressed? If compression is signaled by an option like -embed=compress then presumably the code must work correctly whether the files are compressed for not. When using //go:embed with a string or []byte there is no step where we can decompress the file. So do we need to decompress all embedded files at program startup? That doesn't seem ideal for all use cases.

For an embedded io/fs.FS we can presumably store an embedded archive/zip file. The archive/zip package already implements io/fs.FS. Then compression should happen automatically?

[twmb]

Agree, this seems hard to impossible to do with []byte and string. Would it be feasible to add special embed.String or embed.Bytes types that can have optimizations such as this?

Opting into embed.FS also being compressed is another good option.

[bcmills]

Would it be feasible to add special embed.String or embed.Bytes types that can have optimizations such as this?

I don't think we should have embed.String or embed.Bytes types that look like ordinary string or []byte but decompress data at run-time.

But maybe we could support go:embed on func() []byte or func() string declarations instead? That is, this declaration:

//go:embed bigfile.bin
func content() string

could be a shorthand for something like:

//go:embed bigfile.bin
var contentFS embed.FS

var content = sync.OnceValue(func() string {
	f, err := content.Open("bigfile.bin")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	b := new(strings.Builder)
	if _, err := io.Copy(b, f); err != nil {
		panic(err)
	}
	return b.String()
})

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[hherman1]

This sounds a bit premature to me. Unless there is more demand than I realize

[tux21b]

We use Brotli to compress all bundled UI artifacts (HTML, JS, CSS, etc.) with our app. A special HTTP handler either serves those files directly (if the client supports Brotli) or decompresses the bundled Brotli files and serves them uncompressed.

The proposal seems to completely ignore this use cases:

• zstd is nice, but has no browser support - other algorithms like brotli or gzip might be more useful
• compressing files individually might be preferred
• accessing the compressed data might be useful in some scenarios
• configuring the compression algorithm on a per file basis might be needed

I think it's ok to not support all those features upfront, but the proposal in its current form can not be easily extended later.

[rsc]

This seems too difficult and fraught with problems and there doesn't seem to be much demand for it.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group