Discovery does not discover injectables not co-located with resources

[ThomasBarnekow]

DESCRIPTION

Context. I was trying to leverage resource hooks to implement dynamic, role-based authorization for resources. For example, the API should only return a subset of resources (e.g., Document) to which the authenticated user has been granted read access (Update 2021-05-12: based on @bart-degreed's feedback on resource hooks, I then played with resource services for the same purpose).

Issue. I've tried to mimic the JsonApiDotNetCoreExample in my solution. However, my resource hooks definition, which is derived from ResourceHooksDefinition<Document> (Update 2021-05-12: but not in the same assembly as the Document resource), is not registered by the following call:

services.AddJsonApi<MyDbContext>(ConfigureJsonApiOptions, discovery => discovery.AddCurrentAssembly());

Update 2021-05-12. The same issue applies to resource services (derived from JsonApiResourceService<TResource, TId>) and resource definitions (derived from JsonApiResourceDefinition<TResource, TId>) that are not co-located with the resources (e.g., Document) referenced by the type argument TResource. The expected and actual behavior descriptions reflect this new insight.

EXPECTED BEHAVIOR

Resource definitions, services, repositories, and hooks (called "injectables" in the ServiceDiscoveryFacade) should be discoverable even if they are not co-located with the resources.

ACTUAL BEHAVIOR

"Injectables" are not automatically discovered if they are not co-located with the resources, in which case they must be registered manually.

VERSIONS USED

• JsonApiDotNetCore version: 4.1.1
• ASP.NET Core version: 5.0
• Entity Framework Core version: 5.0
• Database provider: SQL Server

[bart-degreed]

Some background: Resource hooks were added as an experimental feature and it was never properly finished, tested and documented. The implementation is recursive, complex, inefficient for large tables and the code doesn't meet the quality standards the rest of the project does. If you dive deep enough, you'll likely find a ton of issues. We've disabled resource hooks by default and they solely exist in the example project because we have a few tests that still depend on it. We've decided to no longer invest in the feature (other than fix critical bugs, but the bar is very high).

Instead we're taking a simpler approach, currently being reviewed at #977. It covers most capabilities that resource hooks provides, but it is non-recursive, doesn't execute extra queries and has a cleaner API surface with async support built-in.

If you're eager to try it, please do so and let us know how it works out. This is going to be the big feature for our next release.

[ThomasBarnekow]

@bart-degreed, thanks for the quick reply. I've had a quick look at #977 but am unsure as to whether or not this provides everything I need (e.g., because I did not see a way to filter resources). Further, I am not sure whether resource hooks or the feature provided by #977 are even the best way to achieve what I want to achieve. So, therefore, I would be really thankful if you could point me in the right direction for implementing dynamic, role-based authorization (or access control) in JADNC.

In my application, users, roles (e.g., reader, contributor, editor), etc. will be part of the data model. There will be role-related relationships between users and the access-controlled resources (or resource "containers") based on which the authorization to read, create, or modify a resource could be determined. For example, in the context of a request like GET /documents, the set of documents returned only includes the subset that should be visible to the requestor based on certain attributes or relationships of the documents.

[ThomasBarnekow]

@bart-degreed, while the issue was originally about resource hooks not being discovered, the same issue exists for resource definitions (and likely other types as well). They are also not discovered. I have updated the issue description accordingly.

[wayne-o]

I’d be interested in the response to your follow up question

…

On Mon, 10 May 2021 at 17:58, Thomas Barnekow ***@***.***> wrote: @bart-degreed <https://github.com/bart-degreed>, while the issue was originally about resource hooks not being discovered, the *same issue exists for resource definitions* (and likely other types as well). They are also not discovered. I have updated the issue description accordingly. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#995 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLPDGPGMVEHW4JOTLYMVTTNAGCZANCNFSM44RL4EUA> .

[bart-degreed]

@bart-degreed, while the issue was originally about resource hooks not being discovered, the same issue exists for resource definitions (and likely other types as well). They are also not discovered. I have updated the issue description accordingly.

I am unable to reproduce this. Added a resource definition (not hooks) to the example project, which is discovered and invoked as it should. Can you provide a full repro project that I can debug into?

About authentication, there are numerous ways to do that. In its simplest form, scope attributes are added to controller methods that validate against the incoming set of claims (based on resource type). On the other end of the spectrum is a full-blown permission subsystem that validates inheritance hierarchies of permissions, similar to NTFS security descriptors. I don't intend to prescribe how any of such systems should be built, because it is complex, a lot of work and highly depends on individual requirements. In general, I would recommend to use some form of caching though and evaluate in-memory, to prevent numerous joins taking down performance. In theory, such a system can be built on top of the new (and existing) resource definition callbacks, but it all depends on the complexity of requirements. The closest to inferring what effectively has changed I can point you are the examples for microservices. It gets tricky with implicit side effects of relationship changes. For example, when assigning another author to a blog resource, does the user have permission to detach the original author? You'd need to run extra queries on the side to determine things like that.

Overall, for well-defined authentication/authorization requirements, along with debuggable working examples that are covered by integration tests, provided by community members, I'm willing to think along or review and give pointers, But I personally have no interest in building such a system myself, so the mass of the work needs to be done by others.

[ThomasBarnekow]

@bart-degreed, would it be possible for you to provide an example project that doesn't require a PostgreSQL database but rather SQL Server Local DB, which every Visual Studio user would have installed? It would be easier to test those things. I can't unfortunately provide a full repro project so quickly.

[ThomasBarnekow]

@bart-degreed, I think I found out what might lead to the problem. In my solution, the models and the API are not in the same project. In fact, three projects/assemblies would be relevant:

• The project/assembly containing the models (deriving from Identifiable<TId>);
• the project/assembly containing the DbContext; and
• the ASP.NET Core project/assembly (the "current" assembly) containing the Program and Startup classes as well as the controllers and the IResourceService<TResource, TId> and IResourceDefinition<TResource, TId> implementations.

The only things I wanted to automatically discover were the hooks (now removed), services, and definitions in the ASP.NET Core project/assembly. So I used the following call, thinking it would discover the services in the current assembly:

services.AddJsonApi<MyDbContext>(ConfigureJsonApiOptions, discovery => discovery.AddCurrentAssembly());

That doesn't work, because it seems the code tries to find the resources first. Since none can be found in the current assembly, the services are not discovered.

I've now added all relevant assemblies as follows (where Document is a class in the assembly containing the models):

services.AddJsonApi<MyDbContext>(
    ConfigureJsonApiOptions,
    discovery =>
        discovery
            .AddCurrentAssembly()
            .AddAssembly(typeof(Document).Assembly)
            .AddAssembly(typeof(MyDbContext).Assembly))

However, that doesn't work either (and I've also tried ordering the assemblies differently). So it seems there are some assumptions as to how those classes are co-located in assemblies.

How would I have to structure it to make it work? Does everything have to be co-located in one assembly?

[ThomasBarnekow]

I've tested my hypothesis by co-locating the resource definitions (implementing IResourceDefinition<TResource, TId>) with the resources (inheriting from Identifiable<TId>). In that case, the resource definitions are automatically discovered.

Separating models, repository, and the web API is not uncommon in my mind. What is the reason for constraining discovery in this way?

[ThomasBarnekow]

@bart-degreed, I updated the issue description to more precisely point out the issue, i.e., the requirement to co-locate resources and the related definitions, services, repositories, and hooks for those to be discoverable.

[bart-degreed]

Separating models, repository, and the web API is not uncommon in my mind. What is the reason for constraining discovery in this way?

I don't suspect the limitation was intentional. I've created a PR to fix this. Order of assemblies is irrelevant, they are queued for processing at a later stage.

@bart-degreed, would it be possible for you to provide an example project that doesn't require a PostgreSQL database but rather SQL Server Local DB, which every Visual Studio user would have installed? It would be easier to test those things. I can't unfortunately provide a full repro project so quickly.

Long story short: unlikely, but you're free to do so for your own experiments. Why? For an internal project, I've gone from InMemory to Sqlite to SQLLocalDB to SQL Server in a docker container for running tests. At each stage, something did not work the same as in production (multiple schema's, stored procedure support and a few more). SQLLocalDB is especially painful as it seems to work with files on disk, but in reality it depends on a running service. So deleting the .mdf file on disk (or corrupting it when stopping the debugger) requires special commands to recover. In the end, we spent several days trying to make mssql in docker perform better (it was disastrous) and were able to gain about 30% speedup with tricks using SQL backup/restore, global locks, copying .mdf/.ldf files inside the container etc. Even with that, running 65 tests takes 40 sec. Compare that with Postgres in JADNC, which runs 950 tests in 1 minute. That's 10x faster! The reason is because we create a clean database for each test class, which is lightning fast in Postgres, but takes about 7 seconds in SQL Server.

I've found the overall dev-experience using Postgres to be superior, compared to mssql. All it takes for you is running /run-docker-postgres.ps1 after you've installed Docker Desktop. It you want a clean slate, just re-run the script. pgAdmin is a nice GUI, similar to SQL Server Management Studio but web-based. I've used it a lot in the past, but now I rarely need it.

[wayne-o]

I think this is probably where I have been tripped up to :) I wasn;t aware of the namespace issue - looking forwards to trying this out!

…

On Wed, 12 May 2021 at 10:36, Bart Koelman ***@***.***> wrote: Separating models, repository, and the web API is not uncommon in my mind. What is the reason for constraining discovery in this way? I don't suspect the limitation was intentional. I've created a PR to fix this. Order of assemblies is irrelevant, they are queued for processing at a later stage. @bart-degreed <https://github.com/bart-degreed>, would it be possible for you to provide an example project that doesn't require a PostgreSQL database but rather SQL Server Local DB, which every Visual Studio user would have installed? It would be easier to test those things. I can't unfortunately provide a full repro project so quickly. Long story short: unlikely, but you're free to do so for your own experiments. Why? For an internal project, I've gone from InMemory to Sqlite to SQLLocalDB to SQL Server in a docker container for running tests. At each stage, something did not work the same as in production (multiple schema's, stored procedure support and a few more). SQLLocalDB is especially painful as it *seems* to work with files on disk, but in reality it depends on a running service. So deleting the .mdf file on disk (or corrupting it when stopping the debugger) requires special commands to recover. In the end, we spent several days trying to make mssql in docker perform better (it was disastrous) and were able to gain about 30% speedup with tricks using SQL backup/restore, global locks, copying .mdf/.ldf files inside the container etc. Even with that, running 65 tests takes 40 sec. Compare that with Postgres in JADNC, which runs 950 tests in 1 minute. That's 10x faster! The reason is because we create a clean database for each test class, which is lightning fast in Postgres, but takes about 7 seconds in SQL Server. I've found the overall dev-experience using Postgres to be superior, compared to mssql. All it takes for you is running /run-docker-postgres.ps1 after you've installed Docker Desktop <https://www.docker.com/products/docker-desktop>. It you want a clean slate, just re-run the script. pgAdmin <https://www.pgadmin.org/> is a nice GUI, similar to SQL Server Management Studio but web-based. I've used it a lot in the past, but now I rarely need it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#995 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABLPDDSUAPE3T6KTF4WWSDTNJD3HANCNFSM44RL4EUA> .

[ThomasBarnekow]

@bart-degreed, thanks for the insights on Postgres. I have worked with multiple relational databases (e.g., Oracle, DB/2, SQL Server) but not yet Postgres. To use that, I'd have to look out for a PaaS offering in Azure.

For my current project, I'm using Azure SQL Database in production and MSSQLLocalDB for development. For a simple demo or example, MSSQLLocalDB should be totally fine.