@stream and @defer: Giving the server control of incremental delivery

[Keweiqu]

@stream and @defer are collaborative efforts between the client and server to improve the application’s time-to-interact metric. This issue discusses some important reasons to enable the server to choose whether to deliver data asynchronously.

Eagerly fulfill @defer and @stream

In the case that a piece of less important data happens to be expensive, @defer and @stream enables the application to improve the TTI without making the expensive data slower. However, these two criteria are not always aligned.

If the deferred data are cheap and fast to retrieve & if the important (non-deferred) data is the bottleneck: the server may decide to include this data in the initial payload as the data is already ready. In this situation unnecessarily delaying delivery can negatively affect the user experience by triggering multiple renders.
(Caveats: although eagerly fulfill @defer and @stream may be considered “free” from the server side, there is a client side cost of processing more data for the initial payload in terms of JSON deserialize. There are potential optimizations through braces matching that seeks to the deferred data and skip the entire subtree.)

Fast Experimentation and Iteration

Often times, it is hard to tell the true cost of a piece of data upfront:

• Shared memoization may make certain fields appear cheaper.
• Per field level attribution is hard to achieve as most GraphQL resolvers are async in practice.
  Therefore, when using @defer and @stream it’s useful to set up an experiment and compare the result. If a client has a long release cycle, turning @defer and @stream on and off on the server side is much faster.

Additional benefits of server side configuration:

• Peak time versus off-peak time: the peak and off-peak optimization is also described in @stream and @defer: Asynchrony, laziness, or both? #691 @stream and @defer: Asynchronous, lazy or both?
• Personalized adaptive behavior: Further optimizations may be achieved through personalized configuration with the help of ML prediction. As an example, @defer and @stream in the context of prefetch comes at certain cost when the user didn’t go to the prefetched surface. With personalized ML prediction, the server can decide whether to respect @defer and @stream directives based on the likelihood of the person visiting the surface.

Proposed standardization:

For the list reasons above, it is essential to allow server side control of whether to respect @defer and @stream. Therefore, we’d like to propose to specify the requirement in the spec that:

• Client side should be able to properly handle eagerly fulfilled @defer and @stream payloads.
• An optional boolean argument “if” to control whether to turn on and off @defer and @stream. While it’s not strictly required to enable server side control, this argument expresses control as a configuration mapped to a variable. This turns out to be a convenient abstraction for experimentation and adaptive configuration.

[kdawgwilk]

A use case I am trying to solve where this would help is integration LLMs into GraphQL which work by streaming tokens (a chunk of characters) as an async streamed response. This is more driven by the server than the client. It feels like the server should have the power to dictate if a field needs to be streamed in some use cases

[benjie]

I think it's critical that the client opts-in to streamed responses, however I think the schema should definitely be able to indicate which fields should be streamed, such that tools like GraphiQL will automatically add the @stream directive when those fields are auto-completed. I think this could be done via metadata as in #300 (and https://github.com/graphql/graphql-wg/blob/main/rfcs/AnnotationStructs.md)

[Keweiqu]

@benjie my understanding was that client can mark any list type field with @stream. So similiar to @if and @Skip we don't need to include it in the schema. But totally open to alternative suggestions.

[benjie]

@Keweiqu indeed; the client can ask for any list field to be streamed via the client-to-server @stream directive. What I’m suggesting is that the schema could indicate which fields would benefit from being streamed. Maybe the server knows that streaming friends is pointless because all the data comes from an SQL JOIN anyway, but streaming LLM tokens would be much more useful since they take a fair while to generate. Server hinting which things are stream-worthy seems valuable to me. (We could also extend this to “fields that should probably be deferred because they are slow, but it’s still up to you, dear client” too.)

[Keweiqu]

@benjie I see your point. I was thinking in general any composite type list would have latency benefits from streaming since resolving subfields are also delayed. Even if resolving the list itself doesn't have any latency difference, subfields would make a difference, and also parsing a smaller payload on the client side?

[benjie]

@Keweiqu Indeed, that could certainly add up in some circumstances. I may not have worded my previous comment very well, what I'm actually proposing is that some fields (such as LLM token streams or other lists that are produced in a slow item-by-item basis) should almost always be streamed, and we should give a way for schema authors to indicate this as a hint to their clients so that when they auto-complete the field in GraphiQL or their IDE it maybe comes with the @stream directive or something like that. Similarly we should indicate fields that should be deferred (because they are known to be slow) - for example fetching a list of your previous invoice URLs from your payment provider might be worthy of defer (rather than stream) since the data arrives all at once, it's just URLs (so no value streaming item by item), but it takes a while to arrive in the first place. Schema authors are in a good position to indicate these hints, and changing the hints over time would be non-breaking so they can always reflect the latest trends/implementation details.

[felipe-gdr]

@benjie schema hints for defer/stream is a very interesting suggestion.
Consumers will often not be in the best position to decide whether deferring/streaming a certain field will result in significant latency improvements. This type of knowledge is usually detained by the schema owners.