proposal: stdlib: provide a standard implementation of the Optional / Maybe generic type

[glenjamin]

I was able to find similar discussions on stdlib generic Slice and Map packages, but I was unable to find anything about Optional. If this was just a case of poor searching (or poor indexing), then my apologies - please close this.

---

At the moment, there are two ways to handle optionality - either through the type's zero value (eg Time), or by using a pointer type.

However, using a pointer type creates a degree of ambiguity. Sometimes a pointer is chosen for mutability reasons, and nil is never intended to be present. Likewise sometimes a pointer is chosen only for performance reasons.

It has been noted in many places that with the new Go generics system, creating an Optional[T any] type to represent a value that may or may not be filled is rather trivial.

However, if the creation and use of such types is left up to the ecosystem, it is likely that at least a handful of incompatible implementations may surface - whereas if the stdlib held the main implementation then everyone else could build on top of it as needed.

[ianlancetaylor]

Can you propose a package and a specific API? Thanks.

[glenjamin]

I don't personally have a strong preference for a specific API, but I'm happy to propose a strawman. I have not yet given this serious thought - I'd mostly expected to find an already active thread on the topic, but this should start the ball rolling.

Package is an interesting one, it feels too small to need its own package, but I don't see an obvious alternative.

package optional

type Value[T any] struct {
  value T
  ok bool
}

func Ok[T any](value T) Value[T] {
  return Value{value: value, ok: true}
}
func (v Value[T]) Empty() bool {
  return !v.ok
}
func (v Value[T]) Get() (T, bool) {
  return v.value, v.ok
}
func (v Value[T]) Default(fallback T) T {
  if v.Empty() {
    return fallback
  }
  return v.value
}

Edit: It would probably also be useful to implement some common stdlib interfaces like fmt, json and sql

[DeedleFake]

Whatever API is decided on, encoding/json needs to have good support for the new type. json.Marshaler implementation isn't good enough for this because it can't be used to programmatically completely omit a field, so encoding/json itself would have to have support for it without API changes. The current solution for this is to use pointers and nil for optionality, but that doesn't allow for flexibility in terms of JSON's null and complete omission when unmarshalling, and for marshalling omitempty has to be specified at compile-time. This could allow fairly easy differentiation, though, via an optional pointer, such as optional.Value[*string].

it feels too small to need its own package

I don't think that that's a problem. errors was, and honestly still is, a very small package for quite a long time, with only a single exported function and one unexported type with a single method. It's gotten a bit bigger over the years, but not by much.

In terms of actual API, I'd like a Must() or Unwrap() method that panics if ok is false or just returns value. I'm also not partial to either, per se, but Rust's equivalent of the proposed Default() method, with Rust calls or(), takes and returns another optional type, not the underlying type directly. In other words, func (v Value[T]) Or(other Value[T]) Value[T] instead. Returning an optional wrapper instead of the underlying type also allows it to couple with various ways of extracting it, such as the aforementioned Must(): v.Or(optional.Ok(0)).Must().

On the other end of things, a top-level function, maybe Of(), to wrapping existing two-return functions into an optional.Value could be useful:

func Of[T any](v T, ok bool) Value[T] {
  return Value[T]{v: v, ok: ok}
}

// In another package:

someEnvVariable := optional.Of(os.LookupEnv("SOME_ENV_VARIABLE")).Default("default value").Must()

[deanveloper]

I'm honestly unsure how useful this is compared to the typical , ok pattern. One of the things that I like about Go is that complexity typically moves vertically, rather than horizontally. For instance, I personally find

envVar, ok := os.LookupEnv("SOME_ENV_VARIABLE")
if !ok {
    envVar = "default value"
}

much more readable than

envVar := optional.Of(os.LookupEnv("SOME_ENV_VARIABLE")).Default("default value").Must()

I guess what I am getting at is, what are the benefits of Optional[T] over (T, bool)? Are these benefits worth introducing 2 patterns that (possibly) do the same thing?

[glenjamin]

My rough thinking was that it's better to stick to a minimal API that allows those other functions to be written in packages, than to include them all initially.

As long as the underlying type is consistent in the ecosystem and exposes enough information to write the other functions, I don't think there's a big downside to having those be 3rd party functions.

[ernado]

Somehow related: #11939 (proposal: encoding/json, encoding/xml: support zero values of structs with omitempty)

[ernado]

@deanveloper

I found optionals extremely useful for encoding/decoding:

// optional package
type Value[T any] struct {
	Value T
	Set   bool
}

func (v Value[T]) Get() (T, bool) {
	return v.Value, v.Set
}

// code usage:

type Error struct {
	Type        string
	Description string
}

type Response struct {
	Name  optional.Value[string] `json:"name,omitempty"`
	Error optional.Value[Error]  `json:"error,omitempty"`
}

func parse(data []byte) {
	var r Response
	json.Unmashal(data, &r)
	
	// Using fields:
	r.Name.Value // access to value, will be zero value of not set
	if r.Name.Set {
		// Explicit check for name to be set.
		fmt.Println("Name:", r.Name.Value)
	}
	
	// Using methods:
	if v, ok := r.Error.Get(); ok {
		fmt.Println("Got error:", v.Type)	
	}
}

This will eliminate a whole class of boilerplate, like aws.String() and friends.
Every complex REST (and even gRPC, all of them) API binding in go uses pointers for optionals, because concept of zero value is not so universal in practice.

aws/aws-sdk-go#363
kubernetes/enhancements#1173
graph-gophers/graphql-go#308

I've personally implemented code generation for optionals and will be very happy not to do so again, it is too ad-hoc and looks like footgun.

[glenjamin]

I'm honestly unsure how useful this is compared to the typical , ok pattern.

@deanveloper is this in response to the proposal as a whole, or the Of function proposed in #48702 (comment) ?

I agree that for map-like APIs the existing pattern works fine.

[DeedleFake]

@deanveloper

One of the things that I like about Go is that complexity typically moves vertically, rather than horizontally.

I agree, generally, and, for something like this, most languages would wrap the calls from line to line, but Go's semicolon insertion rules make that look kind of strange:

envVar := optional.Of(os.LookupEnv("SOME_ENV_VARIABLE")).
	Default("default value").
	Must()

Compare that with something like Rust:

let envvar = env::var_os("SOME_ENV_VARIABLE")
	.or(Some("default value"))
	.unwrap();

Unfortunately, I get the feeling that this awkwardness will come up a lot more with generics than it did before, as generics enable an entire class of method chaining patterns that were infeasible previously.

Edit: Also, like @glenjamin said, I think that for normal functions like LookupEnv, there's little reason to use this in this way. I was just using it as an easy example. Sometimes conversion might be useful, though, like when setting up a struct with an optional.Value in it for, for example, encoding:

data := PredefinedEncodableStruct{
	AnOptionalString: optional.Of(SomethingThatUsesTheOKBooleanPattern("arg")).Default("default"),
}
err := enc.Encode(data)
// ...

Edit 2: Something else that should be noted: optional.Of() will only work for functions that return exactly two values. That is by far the most common case, but it's definitely not the only one.

[AndrewHarrisSPU]

An observation on Option[T] vis-a-vis record-oriented encoding/decoding I wonder about: the common, proven-to-be-useful schemes all tend to leak a bit about underlying state machines. Option[T] is not as precise as something like Omittable[T] or Nullable[T] would be for JSON. There are similar problems for any scheme - e.g. NA[T], NaN[T], Inf[T] for "missing" values in R. I think a similar observation is made where Rust's Serde warns:

Users tend to have different expectations around the Option enum compared to other enums.

I agree that there should be some opportunities to realize better APIs with generics around encoding. Record-to-record is one sense of an API direction ... maybe a Swiss Army Record type? I feel like the sense of state-machine-to-state-machine is just fundamentally hard to capture generically ... I can see Option[T] making sense for an elaborate solution that is generic-over-state-machines rather than generic-over-records here but not escaping library code.

[zephyrtronium]

The only "real" examples so far of where Option[T] would be helpful are for encoding and decoding. Does it make sense to put it in package encoding? I think that would help to illustrate that the intent is not to replace the , ok idiom.

[DeedleFake]

@AndrewHarrisSPU

An observation on Option[T] vis-a-vis record-oriented encoding/decoding I wonder about: the common, proven-to-be-useful schemes all tend to leak a bit about underlying state machines. Option[T] is not as precise as something like Omittable[T] or Nullable[T] would be for JSON.

This is the problem with the current pointer-based approach as well. Pointer nullability is a side effect of the way pointers work, not the primary purpose, and yet a lot of times pointers are used simply to signify optionality. The is a particular issue because dereferencing can lead to crash bugs, meaning that extra handling has to be done for dealing with incoming data in particular and there's no compile-time safety around it. Optional values provide that safety, as you're really dealing with two completely separate types with no real equiavalance of the automatic dereference of the . operator. A method like Must() or Unwrap() could still cause a crash, but it's a lot more explicit. I think it's actually in a pretty similar vein to the way that error handling is explicit in Go, at least for functions with more than one return. If you want that other return value, you have to explicitly ignore the error with _.

@zephyrtronium

The only "real" examples so far of where Option[T] would be helpful are for encoding and decoding. Does it make sense to put it in package encoding? I think that would help to illustrate that the intent is not to replace the , ok idiom.

I can definitely see the argument for this, but I'm a bit worried that if it winds up being useful elsewhere than it'll be quite awkward for it to be in there, as it can't really be cleanly moved later.