OpenAPI Schema generation stackoverflow when converting Flags enum to a string array & is used in API models with default value

[wsloth]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Hi, I'm not 100% sure this is an aspnetcore or a user issue, but it might be a bit of both.

I have a flags enumeration defined like this:

[Flags]
[JsonConverter(typeof(FlagsEnumAsArrayConverter<FlagsEnum>))]
enum FlagsEnum
{
    None = 0,
    Flag1 = 1,
    Flag2 = 2,
    Flag3 = 4,
    Flag4 = 8,
    Flag5 = 16
}

And I want to use it in an API model similarly to this:

/// <summary>
/// Use the flags in any model and use a default value instead of a null value.
/// This is where things go wrong -- if it is a nullable property it works.
/// </summary>
record FlagsEnumModel(FlagsEnum Flags = FlagsEnum.Flag1);

To have a better developer experience when using this, I want the client to be able to send flags as arrays of strings, instead of an integer or concatenated string. So Example: ["Flag1", "Flag2"] instead of "Flag1, Flag2".

To do this, I created the following JsonConverter below. I've tested this in other places (Kafka (de)serialization, unit tests outputting JSON snapshots), and it works fine.

However, when I want to view my OpenAPI document, it fails to generate, crashing the process with a stack overflow.

/// <summary>
/// This converter is used to serialize and deserialize flags enums as arrays.
/// Note: the default enum value (Unknown/None/0) is not included in the output array.
/// Example: ["Flag1", "Flag2"] instead of "Flag1, Flag2".
/// </summary>
/// <typeparam name="T">An enum decorated with [Flags]</typeparam>
public class FlagsEnumAsArrayConverter<T> : JsonConverter<T>
    where T : struct, Enum
{
    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType != JsonTokenType.StartArray)
        {
            throw new JsonException($"Expected StartArray token, got {reader.TokenType}");
        }

        T value = default;
        reader.Read();
        while (reader.TokenType != JsonTokenType.EndArray)
        {
            if (reader.TokenType != JsonTokenType.String)
            {
                throw new JsonException($"Expected string in array, got {reader.TokenType}");
            }

            var flagName = reader.GetString();
            if (!Enum.TryParse<T>(flagName, out var flag))
            {
                throw new JsonException($"Invalid flag value: {flagName}");
            }

            value = (T)Enum.ToObject(typeof(T), Convert.ToInt64(value) | Convert.ToInt64(flag));
            reader.Read();
        }

        return value;
    }

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        var names = value.ToString().Split(", ", StringSplitOptions.RemoveEmptyEntries);
        writer.WriteStartArray();
        foreach (var name in names)
        {
            writer.WriteStringValue(name);
        }

        writer.WriteEndArray();
    }
}

Expected Behavior

I would expect the OpenAPI document to generate correctly as outputting an array of enum values. Or at the very least, an array of strings.

Steps To Reproduce

I created a reproducable sample repository here: https://github.com/wsloth/aspnetcore-flagsenum-arrayconverter-issue

It contains both a .NET 9 and a .NET 10 Preview 4 project. They both contain the issue, but it seems like the .NET 9 executable crashes entirely, while the .NET 10 executable keeps running (but does output "stack overflow" in the console).

I was diving into the internals on the .NET 9 version, and saw that it went wrong in OpenApiJsonSchema, specifically in ReadOpenApiAny. It would see the array start token, but it had no value at all in the schema (it looked like [ ]) before reaching the end token.

I'm not sure if there should be a value in this array, or if the issue is that if there is a value, it's another instance of the FlagsEnum, which enters another (de)serialization loop which generates yet another array.

Exceptions (if any)

... continues forever ...
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef, System.String ByRef)
   at OpenApiJsonSchema.ReadOpenApiAny(System.Text.Json.Utf8JsonReader ByRef)
   at OpenApiJsonSchema.ReadProperty(System.Text.Json.Utf8JsonReader ByRef, System.String, Microsoft.OpenApi.Models.OpenApiSchema, System.Text.Json.JsonSerializerOptions)
   at OpenApiJsonSchema+JsonConverter.Read(System.Text.Json.Utf8JsonReader ByRef, System.Type, System.Text.Json.JsonSerializerOptions)
   at System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TryRead(System.Text.Json.Utf8JsonReader ByRef, System.Type, System.Text.Json.JsonSerializerOptions, System.Text.Json.ReadStack ByRef, System.__Canon ByRef, Boolean ByRef)

.NET Version

10.0.100-preview.4.25258.110

Anything else?

No response

[BrennanConroy]

Can you please share what request you tried?

Wasn't able to reproduce any issues with 9.0 or 10.0-preview4 using body: JSON.stringify({"Flags": ["Flag1", "Flag3"]}

Oh, when you access openapi/v1.json.