cmd/go: no easy way to use modules fully privately outside of GOPATH

[marystern]

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

What did you do?

I'm trying out the new finalized modules feature in 1.14 and doing everything outside of GOPATH.
I want all my code to never be publicly visible (i.e. it's a fully private codebase and always will be).

I had an existing codebase with a main package and a separate "utility" package.
Under ~/go/src, everything works fine. But, to try out modules locally, I'm putting my code elsewhere (eg ~/src for the sake of example).

However, to migrate to using modules I'm now having to add a compulsory preceding domain to use the utility package from the main package, so eg, before I had this in my main program:

import "util/log"

which has to change to:

import "x.x/util/log"

"x.x" was the minimum domain name that I could think of that:
a) Would not accidentally resolve to a real domain
b) satisfies the "dot required" restriction in the tools.

My Issues:

1. Inside the utility package itself, I cannot use relative imports, so I have a package util/email that uses util/log, and has now to be changed to import from x.x/util/log to allow "go test" to work (even though it's inside the same package!)
2. I had set GOPRIVATE, but had changed the package name, so go was now trying to resolve and access domain x.x (I realize this is "user error", but is easy to get wrong accidentally, and leaks info very easily (there's another issue somewhere that mentions this).
3. x.x is ugly, other ideas such as example.com, etc are all fine, but have the problem of looking like real places, and hence the leaking issue is a real security concern.
4. Code is not portable from within GOPATH to outside (ie without adding doamin prefixes inside the codebase). (It's possible that this is "by design"?)
5. The docs don't really help users to understand how to work in this very simple (fully local and private) case! My guess is that you would recommend using ~/go (and hence GOPATH), but I think I read somewhere that this is no longer the case?

Proposals

Two thoughts comes to mind (apologies for using this issue also for a prooposal):

1. It might be fairly easy to just say "if the first part of an import does not have a dot in it, assume it's local and private and hence do none of the public module stuff at all". This has the advantage of being simple; not having to remember to set GOPRIVATE and keeping the code GOPATH-compatible.

2. Another idea might be to use "_" as a fully private, local, never-publicly used domain name to indicate such usage in a Module World. So in this case, I could use

import "_/util/log"

Maybe there would be problems with these two ideas, but I'm throwing them out there, not as fully formed proposals, but rather to see if you can see the problem and come up with a good solution.

Anyway, I've been a golang user for many years, but Modules seem to be making my life 100 times harder for zero gain (in this particular case; of course I realize it's all brilliant for public code and dependency/version management, etc, so thanks, and keep up the good work :) )!

[DisposaBoy]

Not a comment on the proposal -- even though I agree 100% and am disappointed that they've highjackingreserved non-domain module names for std lib usage...

I just wanted to recommend the use of x.localhost instead of x.x.

As you've seen, using x.x will result in a name lookup and depending on what you've used instead of literal x.x, you might suddenly find yourself pulling in code from some random, possibly malicious, server on the internet. Think the recent-ish drama around Google now owning the .dev TLD. I've also seen legit requests coming into golag.org and other domains I've registered for security research purposes.

[marystern]

Hi again, sorry, but this is an addendum to the above.

I was under the impression that having a local module and doing a "go install" would build and install that module into the local module-cache, but it apparently does not: is that expected? It seems that I'm possibly supposed to use a "replace" directive in my go.mod so that the main module can find the util module. If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare! (Maybe I'm just missing something here?)

(Would it not be a good idea for "go install" to simply install the module into my local module cache so that it can be easily found by any other local modules that use the utils? Is this a bug?)

[marystern]

Not a comment on the proposal -- even though I agree 100% and am disappointed that they've highjackingreserved non-domain module names for std lib usage...

I just wanted to recommend the use of x.localhost instead of x.x.

As you've seen, using x.x will result in a name lookup and depending on what you've used instead of literal x.x, you might suddenly find yourself pulling in code from some random, possibly malicious, server on the internet. Think the recent-ish drama around Google now owning the .dev TLD. I've also seen legit requests coming into golag.org and other domains I've registered for security research purposes.

Thanks DisposaBoy, I'm not sure, but I thought that top-level domains had to be a minimum of 3-chars in length, so I chose "x" assuming that it could never be resolved to a real server (also to avoid import-namespace cruft), but it seems that the name still tries to get resolved anyway (apparently even when it's listed in GOPRIVATE which I find very strange) :(

[bcmills]

A module's path only has to have a dot in the first component if you want to be able to fetch it automatically (such as using go mod download or go get -u).

You can, today, replace a module with a dotless prefix, and in fact we do that a lot in the cmd/go tests themselves. (For example, see https://github.com/golang/go/blob/52fdd624a4087342f529cd7e52c92f44adf2e2bf/src/cmd/go/testdata/script/mod_indirect.txt.)

Note that we still reserve those names in general for the standard library (#32819), but you can use them at your own risk.

[DisposaBoy]

You can't tell people that it's fine to use names without a dot, and then immediately take it back by claiming them as reserved...

[bcmills]

@DisposaBoy, we don't want to limit the names available to the standard library, but it seems reasonable to reserve a specific path prefix (or set of path prefixes) for local, user-defined modules. Want to draft a proposal?

[bcmills]

@marystern, go install intentionally does not add anything to the module cache.

(Unlike GOPATH, the module cache is a cache: given a complete go.mod file, and assuming that the upstream sources still exist or are still mirrored in a GOPROXY, the module cache can be rebuilt from the network, starting from scratch, in a way that does not depend on any local state.)

See also #28835.

[bcmills]

@marystern

If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare!

Could you explain more about your use-case? (Why would 'replace' directives — or a single dotless root module — not scale, and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)

[marystern]

@marystern

If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare!

Could you explain more about your use-case? (Why would 'replace' directives — or a single dotless root module — not scale, and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)

@bcmills
The use case is this:
I have two main programs which use a shared, separate package at 2 different version levels. These are all held in 3 separate, private, local git repos. All code is private and will always be, and my Company doesn't want external network requests where they are not necessary (or open to abuse).

Also individual team members should be free to clone the 3 repos anywhere they like on their local filesystems, and work on any or all projects.

So, user Alice checks out

~/src/main1/..
~/src/main2/...
~/src/util/...

but Bob likes to use this arrangement:

~/main1/src/..
~/main2/src/...
~/shared_libs/special/util/src/...

So, from the above you can see that both Alice and Bob might need to edit the go.mod replace directives to accommodate their individual workspace preferences. This is what I was referring to as "un-scalable", i.e. across disparate team members or other users of the code (i.e. without mandating a scheme such as "put all your code in ~/go/src so that all relative go.mod replace directives don't need to be edited", i.e. they all have to be under a shared parent dir much like the old GOPATH approach).

re: and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)
if I use a domain that is currently invalid, it's possible for it to be made valid in the future, in which case that domain's server gets requests from our organization even though this wasn't our intention, so potentially leaks some info about our codebase accidentally (I'm not sure what would actually be sent, but you get the idea). Hope this helps!