Support NRT and MSV in required and nullable (#1185)

* Added tests for nullable and required properties in schema generation

* Added handling of modelstate validation in setting required attributes

* Not all swagger documents need to be saved to disk; changes in OpenApiTestContext to allow for this

* Added OpenApi client tests for nullable and required properties

* Use NullabilityInfoContext for nullability information

* Post-merge fixes

* Post-merge fixes

* Fixed: do not share NullabilityInfoContext, it is not thread-safe

* Review feedback

OpenApiTests/SchemaProperties test collection:
* Allow for usage of OpenApiStartup directly
* Remove unneeded adding of JsonConverter
* Replace null checks with .ShouldHaveCount() calls
* Adjust test names
Misc:
* Reverted .editorconfig changes and fixed ApiException constructor instead
* Remove Debug statement

* remove redundant new lines in eof added by cleanupcode

* Improved naming in OpenApiTests/SchemaProperties

* Review feedback: NullabilityTests

* Improved JsonApiClient and testing

SchemaProperty Tests:
* More rigorous test suite, see PR description

IJsonApiClient:
* Renamed registration method to a more functionally descriptive name.
* Improved documentation to contain most relevant information on top instead of at the end, and removed ambiguigity in wording.

JsonApiClient
* Fix bug: disallow omitting members that are explicitly required by the OAS description
* Renamed AttributeNamesContainer to AttributesObjectContext because it was more than just a container
* Misc: better variable naming

* Fix test: should not omit required field in test request body

* Temp enable CI buid for current branch

* Rename test files: it no longer only concerns required attributes, but more generally request behaviour

* Changes and tests for support of nullable and required for relationships

* - Rename placeholder model names and properties to examples consisent with existing test suite
- Use existing DbContext instead of temporary one

* Move into consistent folder structure, remove bad cleanupcode eof linebreaks

* Organise tests such that they map directly to the tables in #1231 and #1111

Organise tests such that they map directly to the tables in #1231 and #1111

* Add two missing 'Act' comments

* More elaborate testing

-> in sync with latest version of nullability/required table
-> introduces ResourceFieldValidationMetadataProvider
-> Fix test in legacy projects
-> Reusable faker building block for OpenApiClient related concerns

* Remove non-sensical testcases. Add caching in ObjectExtensions.

* Remove overlooked code duplication in OpenApiTests, revert reflection caching in object extension

* Make AutoFakers deterministic; generate positive IDs

* Fix nameof

* Use On/Off naming, shorten type names by using Nrt+Msv

* Renamed EmptyResource to Empty to further shorten FK names

* Fixed invalid EF Core mappings, resulting in logged warnings and inability to clear optional to-one relationship when NRT off; fixed wrong public names

* Move misplaced Act comments

* Optimize and clarify ResourceFieldValidationMetadataProvider

* Rename method, provide error message

* Refactor JsonApiClient: simplified recursion by using two converters, clearer naming, separation of concerns, improved error message

* Add relationship nullability assertions in OpenAPI client tests

* Cleanup JsonElementExtensions

* Cleanup ObjectExtensions

* Make base type abstract, remove redundant TranslateAsync calls, inline relationship Data property name

* Simplify usings

* Sync up test names

* Fix invalid tests

* Fix assertion messages

* Sync up tests

* Revert change to pass full options instead of just the naming policy

* Fix casing in test names

* Simplify Cannot_exclude_Id tests

* Rename base type to avoid OpenApiClientTests.OpenApiClientTests

* Adapt to existing naming convention

* Remove redundant assertions, fix formatting

* Correct test names

* Centralize code for property assignment in tests

* Apply Resharper hint: convert switch statement to expression

* Simplify expressions

* Simplify exception assertions

* Use string interpolation

* Corrections in openapi documentation

* Simplify code

* Remove redundant suppression

* Combine OpenAPI client tests for create resource with null/default attribute

* Fixup OpenAPI example and docs

* Revert "Merge branch 'master' into openapi-required-and-nullable-properties"

This reverts commit 66a2dc4, reversing
changes made to c3c4844.

* Workaround for running OpenAPI tests on Windows

* Address failing InspectCode

* Remove redundant calls

* Remove redundant tests

* Move types out of the wrong namespace

* Remove redundant suppressions in openapi after update to CSharpGuidelinesAnalyzer v3.8.4

---------

Co-authored-by: Bart Koelman <10324372+bkoelman@users.noreply.github.com>

Author: maurei

docs/usage/openapi-client.md

@@ -2,46 +2,56 @@
 
 You can generate a JSON:API client in various programming languages from the [OpenAPI specification](https://swagger.io/specification/) file that JsonApiDotNetCore APIs provide.
 
-For C# .NET clients generated using [NSwag](https://github.com/RicoSuter/NSwag), we provide an additional package that introduces support for partial PATCH/POST requests. The issue here is that a property on a generated C# class being `null` could mean "set the value to `null` in the request" or "this is `null` because I never touched it".
+For C# .NET clients generated using [NSwag](https://github.com/RicoSuter/NSwag), we provide an additional package
+that introduces support for partial PATCH/POST requests. The concern here is that a property on a generated C# class
+being `null` could mean "set the value to `null` in the request" or "this is `null` because I never touched it".
 
 ## Getting started
 
 ### Visual Studio
 
-The easiest way to get started is by using the built-in capabilities of Visual Studio. The next steps describe how to generate a JSON:API client library and use our package.
+The easiest way to get started is by using the built-in capabilities of Visual Studio.
+The next steps describe how to generate a JSON:API client library and use our package.
 
 1.  In **Solution Explorer**, right-click your client project, select **Add** > **Service Reference** and choose **OpenAPI**.
 
 2.  On the next page, specify the OpenAPI URL to your JSON:API server, for example: `http://localhost:14140/swagger/v1/swagger.json`.
-    Optionally provide a class name and namespace and click **Finish**.
-    Visual Studio now downloads your swagger.json and updates your project file. This results in a pre-build step that generates the client code.
+    Specify `ExampleApiClient` as class name, optionally provide a namespace and click **Finish**.
+    Visual Studio now downloads your swagger.json and updates your project file.
+    This adds a pre-build step that generates the client code.
 
-    Tip: To later re-download swagger.json and regenerate the client code, right-click **Dependencies** > **Manage Connected Services** and click the **Refresh** icon.
-3.  Although not strictly required, we recommend to run package update now, which fixes some issues and removes the `Stream` parameter from generated calls.
+    > [!TIP]
+    > To later re-download swagger.json and regenerate the client code,
+    > right-click **Dependencies** > **Manage Connected Services** and click the **Refresh** icon.
 
-4.  Add some demo code that calls one of your JSON:API endpoints. For example:
+3.  Although not strictly required, we recommend to run package update now, which fixes some issues.
+
+4.  Add code that calls one of your JSON:API endpoints.
 
     ```c#
     using var httpClient = new HttpClient();
     var apiClient = new ExampleApiClient("http://localhost:14140", httpClient);
 
-    PersonCollectionResponseDocument getResponse =
-        await apiClient.GetPersonCollectionAsync();
+    PersonCollectionResponseDocument getResponse = await apiClient.GetPersonCollectionAsync();
 
     foreach (PersonDataInResponse person in getResponse.Data)
     {
-        Console.WriteLine($"Found user {person.Id} named " +
-            $"'{person.Attributes.FirstName} {person.Attributes.LastName}'.");
+        Console.WriteLine($"Found person {person.Id}: {person.Attributes.DisplayName}");
     }
     ```
 
 5.  Add our client package to your project:
 
-   ```
-   dotnet add package JsonApiDotNetCore.OpenApi.Client
-   ```
+    ```
+    dotnet add package JsonApiDotNetCore.OpenApi.Client
+    ```
+
+6.  Add the following glue code to connect our package with your generated code.
 
-6.  Add the following glue code to connect our package with your generated code. The code below assumes you specified `ExampleApiClient` as class name in step 2.
+    > [!NOTE]
+    > The class name must be the same as specified in step 2.
+    > If you also specified a namespace, put this class in the same namespace.
+    > For example, add `namespace GeneratedCode;` below the `using` lines.
 
     ```c#
     using JsonApiDotNetCore.OpenApi.Client;
@@ -56,6 +66,9 @@ The easiest way to get started is by using the built-in capabilities of Visual S
     }
     ```
 
+    > [!TIP]
+    > The project at src/Examples/JsonApiDotNetCoreExampleClient contains an enhanced version that logs the HTTP requests and responses.
+
 7.  Extend your demo code to send a partial PATCH request with the help of our package:
 
     ```c#
@@ -66,30 +79,43 @@ The easiest way to get started is by using the built-in capabilities of Visual S
             Id = "1",
             Attributes = new PersonAttributesInPatchRequest
             {
-                FirstName = "Jack"
+                LastName = "Doe"
             }
         }
     };
 
-    // This line results in sending "lastName: null" instead of omitting it.
-    using (apiClient.RegisterAttributesForRequestDocument<PersonPatchRequestDocument,
-        PersonAttributesInPatchRequest>(patchRequest, person => person.LastName))
+    // This line results in sending "firstName: null" instead of omitting it.
+    using (apiClient.WithPartialAttributeSerialization<PersonPatchRequestDocument, PersonAttributesInPatchRequest>(patchRequest,
+        person => person.FirstName))
     {
-        PersonPrimaryResponseDocument patchResponse =
-            await apiClient.PatchPersonAsync("1", patchRequest);
+        await TranslateAsync(async () => await apiClient.PatchPersonAsync(1, patchRequest));
 
         // The sent request looks like this:
         // {
         //   "data": {
         //     "type": "people",
         //     "id": "1",
         //     "attributes": {
-        //       "firstName": "Jack",
-        //       "lastName": null
+        //       "firstName": null,
+        //       "lastName": "Doe"
         //     }
         //   }
         // }
     }
+
+    static async Task<TResponse?> TranslateAsync<TResponse>(Func<Task<TResponse>> operation)
+        where TResponse : class
+    {
+        try
+        {
+            return await operation();
+        }
+        catch (ApiException exception) when (exception.StatusCode == 204)
+        {
+            // Workaround for https://github.com/RicoSuter/NSwag/issues/2499
+            return null;
+        }
+    }
     ```
 
 ### Other IDEs
@@ -100,12 +126,12 @@ Alternatively, the next section shows what to add to your client project file di
 
 ```xml
 <ItemGroup>
-  <PackageReference Include="Microsoft.Extensions.ApiDescription.Client" Version="3.0.0">
+  <PackageReference Include="Microsoft.Extensions.ApiDescription.Client" Version="7.0.11">
     <PrivateAssets>all</PrivateAssets>
     <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
   </PackageReference>
-  <PackageReference Include="Newtonsoft.Json" Version="12.0.2" />
-  <PackageReference Include="NSwag.ApiDescription.Client" Version="13.0.5">
+  <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
+  <PackageReference Include="NSwag.ApiDescription.Client" Version="13.20.0">
     <PrivateAssets>all</PrivateAssets>
     <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
   </PackageReference>

docs/usage/openapi.md

@@ -16,25 +16,26 @@ JsonApiDotNetCore provides an extension package that enables you to produce an [
     ```c#
     IMvcCoreBuilder mvcCoreBuilder = builder.Services.AddMvcCore();
 
+    // Include the mvcBuilder parameter.
     builder.Services.AddJsonApi<AppDbContext>(mvcBuilder: mvcCoreBuilder);
 
-    // Configures Swashbuckle for JSON:API.
+    // Configure Swashbuckle for JSON:API.
     builder.Services.AddOpenApi(mvcCoreBuilder);
 
     var app = builder.Build();
 
     app.UseRouting();
     app.UseJsonApi();
 
-    // Adds the Swashbuckle middleware.
+    // Add the Swashbuckle middleware.
     app.UseSwagger();
     ```
 
 By default, the OpenAPI specification will be available at `http://localhost:<port>/swagger/v1/swagger.json`.
 
 ## Documentation
 
-Swashbuckle also ships with [SwaggerUI](https://swagger.io/tools/swagger-ui/), tooling for a generated documentation page. This can be enabled by installing the `Swashbuckle.AspNetCore.SwaggerUI` NuGet package and adding the following to your `Program.cs` file:
+Swashbuckle also ships with [SwaggerUI](https://swagger.io/tools/swagger-ui/), which enables to visualize and interact with the API endpoints through a web page. This can be enabled by installing the `Swashbuckle.AspNetCore.SwaggerUI` NuGet package and adding the following to your `Program.cs` file:
 
 ```c#
 app.UseSwaggerUI();

src/Examples/JsonApiDotNetCoreExampleClient/ExampleApiClient.cs

@@ -2,6 +2,8 @@
 using JsonApiDotNetCore.OpenApi.Client;
 using Newtonsoft.Json;
 
+// ReSharper disable UnusedParameterInPartialMethod
+
 namespace JsonApiDotNetCoreExampleClient;
 
 [UsedImplicitly(ImplicitUseTargetFlags.Itself)]
@@ -11,6 +13,50 @@ partial void UpdateJsonSerializerSettings(JsonSerializerSettings settings)
     {
         SetSerializerSettingsForJsonApi(settings);
 
+        // Optional: Makes the JSON easier to read when logged.
         settings.Formatting = Formatting.Indented;
     }
+
+    // Optional: Log outgoing request to the console.
+    partial void PrepareRequest(HttpClient client, HttpRequestMessage request, string url)
+    {
+        using var _ = new UsingConsoleColor(ConsoleColor.Green);
+
+        Console.WriteLine($"--> {request}");
+        string? requestBody = request.Content?.ReadAsStringAsync().GetAwaiter().GetResult();
+
+        if (!string.IsNullOrEmpty(requestBody))
+        {
+            Console.WriteLine();
+            Console.WriteLine(requestBody);
+        }
+    }
+
+    // Optional: Log incoming response to the console.
+    partial void ProcessResponse(HttpClient client, HttpResponseMessage response)
+    {
+        using var _ = new UsingConsoleColor(ConsoleColor.Cyan);
+
+        Console.WriteLine($"<-- {response}");
+        string responseBody = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
+
+        if (!string.IsNullOrEmpty(responseBody))
+        {
+            Console.WriteLine();
+            Console.WriteLine(responseBody);
+        }
+    }
+
+    private sealed class UsingConsoleColor : IDisposable
+    {
+        public UsingConsoleColor(ConsoleColor foregroundColor)
+        {
+            Console.ForegroundColor = foregroundColor;
+        }
+
+        public void Dispose()
+        {
+            Console.ResetColor();
+        }
+    }
 }

src/Examples/JsonApiDotNetCoreExampleClient/OpenAPIs/swagger.json

@@ -2051,6 +2051,10 @@
         "additionalProperties": false
       },
       "personAttributesInResponse": {
+        "required": [
+          "displayName",
+          "lastName"
+        ],
         "type": "object",
         "properties": {
           "firstName": {
@@ -2341,6 +2345,9 @@
         "additionalProperties": false
       },
       "tagAttributesInResponse": {
+        "required": [
+          "name"
+        ],
         "type": "object",
         "properties": {
           "name": {
@@ -2715,6 +2722,10 @@
         "additionalProperties": false
       },
       "todoItemAttributesInResponse": {
+        "required": [
+          "description",
+          "priority"
+        ],
         "type": "object",
         "properties": {
           "description": {
@@ -2970,6 +2981,9 @@
         "additionalProperties": false
       },
       "todoItemRelationshipsInResponse": {
+        "required": [
+          "owner"
+        ],
         "type": "object",
         "properties": {
           "owner": {

src/Examples/JsonApiDotNetCoreExampleClient/Program.cs

@@ -1,26 +1,47 @@
-namespace JsonApiDotNetCoreExampleClient;
+using JsonApiDotNetCoreExampleClient;
 
-internal static class Program
-{
-    private const string BaseUrl = "http://localhost:14140";
+using var httpClient = new HttpClient();
+var apiClient = new ExampleApiClient("http://localhost:14140", httpClient);
 
-    private static async Task Main()
-    {
-        using var httpClient = new HttpClient();
+PersonCollectionResponseDocument getResponse = await apiClient.GetPersonCollectionAsync();
 
-        ExampleApiClient exampleApiClient = new(BaseUrl, httpClient);
+foreach (PersonDataInResponse person in getResponse.Data)
+{
+    Console.WriteLine($"Found person {person.Id}: {person.Attributes.DisplayName}");
+}
 
-        try
-        {
-            const int nonExistingId = int.MaxValue;
-            await exampleApiClient.DeletePersonAsync(nonExistingId);
-        }
-        catch (ApiException exception)
+var patchRequest = new PersonPatchRequestDocument
+{
+    Data = new PersonDataInPatchRequest
+    {
+        Id = "1",
+        Attributes = new PersonAttributesInPatchRequest
         {
-            Console.WriteLine(exception.Response);
+            LastName = "Doe"
         }
+    }
+};
 
-        Console.WriteLine("Press any key to close.");
-        Console.ReadKey();
+// This line results in sending "firstName: null" instead of omitting it.
+using (apiClient.WithPartialAttributeSerialization<PersonPatchRequestDocument, PersonAttributesInPatchRequest>(patchRequest, person => person.FirstName))
+{
+    await TranslateAsync(async () => await apiClient.PatchPersonAsync(1, patchRequest));
+}
+
+Console.WriteLine("Press any key to close.");
+Console.ReadKey();
+
+// ReSharper disable once UnusedLocalFunctionReturnValue
+static async Task<TResponse?> TranslateAsync<TResponse>(Func<Task<TResponse>> operation)
+    where TResponse : class
+{
+    try
+    {
+        return await operation();
+    }
+    catch (ApiException exception) when (exception.StatusCode == 204)
+    {
+        // Workaround for https://github.com/RicoSuter/NSwag/issues/2499
+        return null;
     }
 }

src/JsonApiDotNetCore.OpenApi.Client/Exceptions/ApiException.cs

@@ -14,7 +14,7 @@ public sealed class ApiException : Exception
 
     public IReadOnlyDictionary<string, IEnumerable<string>> Headers { get; }
 
-    public ApiException(string message, int statusCode, string? response, IReadOnlyDictionary<string, IEnumerable<string>> headers, Exception innerException)
+    public ApiException(string message, int statusCode, string? response, IReadOnlyDictionary<string, IEnumerable<string>> headers, Exception? innerException)
         : base($"{message}\n\nStatus: {statusCode}\nResponse: \n{response ?? "(null)"}", innerException)
     {
         StatusCode = statusCode;

src/JsonApiDotNetCore.OpenApi.Client/IJsonApiClient.cs

@@ -5,17 +5,19 @@ namespace JsonApiDotNetCore.OpenApi.Client;
 public interface IJsonApiClient
 {
     /// <summary>
-    /// Ensures correct serialization of attributes in a POST/PATCH Resource request body. In JSON:API, an omitted attribute indicates to ignore it, while an
-    /// attribute that is set to "null" means to clear it. This poses a problem because the serializer cannot distinguish between "you have explicitly set
-    /// this .NET property to null" vs "you didn't touch it, so it is null by default" when converting an instance to JSON. Therefore, calling this method
-    /// treats all attributes that contain their default value (<c>null</c> for reference types, <c>0</c> for integers, <c>false</c> for booleans, etc) as
-    /// omitted unless explicitly listed to include them using <paramref name="alwaysIncludedAttributeSelectors" />.
+    /// Ensures correct serialization of JSON:API attributes in the request body of a POST/PATCH request at a resource endpoint. Properties with default
+    /// values are omitted, unless explicitly included using <paramref name="alwaysIncludedAttributeSelectors" />
+    /// <para>
+    /// In JSON:API, an omitted attribute indicates to ignore it, while an attribute that is set to <c>null</c> means to clear it. This poses a problem,
+    /// because the serializer cannot distinguish between "you have explicitly set this .NET property to its default value" vs "you didn't touch it, so it
+    /// contains its default value" when converting to JSON.
+    /// </para>
     /// </summary>
     /// <param name="requestDocument">
-    /// The request document instance for which this registration applies.
+    /// The request document instance for which default values should be omitted.
     /// </param>
     /// <param name="alwaysIncludedAttributeSelectors">
-    /// Optional. A list of expressions to indicate which properties to unconditionally include in the JSON request body. For example:
+    /// Optional. A list of lambda expressions that indicate which properties to always include in the JSON request body. For example:
     /// <code><![CDATA[
     /// video => video.Title, video => video.Summary
     /// ]]></code>
@@ -28,9 +30,10 @@ public interface IJsonApiClient
     /// </typeparam>
     /// <returns>
     /// An <see cref="IDisposable" /> to clear the current registration. For efficient memory usage, it is recommended to wrap calls to this method in a
-    /// <c>using</c> statement, so the registrations are cleaned up after executing the request.
+    /// <c>using</c> statement, so the registrations are cleaned up after executing the request. After disposal, the client can be reused without the
+    /// registrations added earlier.
     /// </returns>
-    IDisposable RegisterAttributesForRequestDocument<TRequestDocument, TAttributesObject>(TRequestDocument requestDocument,
+    IDisposable WithPartialAttributeSerialization<TRequestDocument, TAttributesObject>(TRequestDocument requestDocument,
         params Expression<Func<TAttributesObject, object?>>[] alwaysIncludedAttributeSelectors)
         where TRequestDocument : class;
 }