Skip to content

Add SpelIndexResolver #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

rwinch
Copy link
Member

@rwinch rwinch commented Dec 15, 2015

This request adds support for resolving indexes using SpEL expressions.

NOTE It should be noted that there is a FIXME in the code Details are in the comment. Please let me know if you see something I should do differently here, or if it is a large problem that needs fixed.

christophstrobl and others added 24 commits December 14, 2015 21:57
…ed by Hashes and Sets.

Prepare issue branch.
…ed by Hashes and Sets.

We now enable storing domain object as a flat Redis 'HASH' and maintain additional 'SET' structures to enable finder operations on simple properties.
  
  @keyspace("persons");
  class Person {

    @id String id;
    @indexed String firstname;
    String lastname;

    Map<String, String> attributes;

    City city;

    @reference Person mother;
  }

The above is stored in the HASH with key 'persons:1' as

  _class = org.example.Person
  id = 1
  firstname = rand
  lastname = al’thor
  attributes.[eye-color] = grey
  attributes.[hair-color] = red
  city.name = emond's field
  city.region = two rivers
  mother = persons:2

Complex types are flattened out to their full property path for each of the values provided. If the properties actual value type does not match the declared one the '_class' type hint is added to the entry.

  city._class = CityInAndor.class
  city.name = emond's field
  city.region = two rivers
  city.country = andor

Map and Collection like structures are stored with their key/index values as part of the property path. If the map/collection value type does not match the actutal objects one the '_class' type hint is added to the entry.
  
  list.[0]._class = DomainType.class
  list.[0].property1 = ...

  map.[key-1]._class = DomainType.class
  map.[key-1].property1 = ...

Properties marked with '@reference' are stored as semantic references by just storing the key to the referenced object 'HASH' instead of embedding its values. 

   mother = persons:2

Please note that referenced objects are not transitively updated/saved and that lazy loading of references will be part of future development.

A 'save' operation therefore executes the following:

  # flatten domain type and add as hash
  HMSET persons:1 id 1 firstname rand …

  # add the newly inserted entry to the list of all entries of that type
  SADD persons 1

  # index the firstname for finder lookup
  SADD persons.firstname:rand 1

Simple finder operation like 'findByFirstname' use 'SINTER' to find matching 

  SINTER persons.firstname:rand
  HGETALL persons:1

Besides resolving an index via the '@Index' annotation we also allow to add custom configuration via the 'indexConfiguration' attribute of '@EnableRedisRepositories'.

  @configuration
  @EnableRedisRepositories(indexConfiguration = CustomIndexConfiguration.class)
  class Config { }

  static class CustomIndexConfiguration extends IndexConfiguration {

    @OverRide
    protected Iterable<RedisIndexDefinition> initialConfiguration() {
      return Arrays.asList(
        new RedisIndexDefinition("persons", "lastname"),
      );
    }
  }
We need to have a dedicated RepositoryFactory to be able to tweak query creation. Now derived queries are freshly instantiated new for every execution.
…ontext.

Added "redisTemplateRef" to @EnableRedisRepositories. The default points to "redisTemplate".
- Introduce Bucket to not operate directly on a Map of byte[].
- Introduce IndexData to not operate directly on Map of byte[] for indexes.
- Move index updates to IndexWriter.
- Additional tests.
- Add hamcrest matcher for newly introduced Bucket.
- Renamed IndexedDataWriter to IndexWriter.
- Introduced IndexResolver.
- Added unit tests for writing index values.
- Align keys for indexes with OHM.
- Introduced RedisHash allowing to define TTL for types.
- Add RedisKeySpaceEvents.
- Add RedisMessageListeners for Keyspace Events
- Trigger helper structures clean up based upon Redis events.
- Store phantom key to be able to load expired value after key is actually already gone.
- Introduce RedisPersistentEntity.
- Use MappingContext for setting up template and Adapter.
- Allow programatic TTL configuration.
Enable custom conversions for more influence on conversions.

Still needs some polishing aka tests, factoryBeans, documentation and the such but gives a glimpse on what’s the purpose of it.

Important: we also need to check for potential index structures on custom converted objects, which has not been implemented so far.
Introduce annotation that allows to mark a single numeric property on aggregate root level to hold a dynamic timeout value. That supersedes any other configured timeout. This allows to define timeouts dynamically on every put/save operation.

@RedisHash
class Person {

  @id String id;
  @timetolive Long ttl;
}
Rename TimeToLiveResolver -> TimeToLiveAccessor and expose the accessor via RedisPersistentEntity instead of using it only internally for getting time to live values.
Change visibility and add as default in setup.
- Allow Converters to read/write Map<String,byte[]> for more influence on actual mapping.
- Use provided type passed down from template to not require to have an explicit _class field on root level.

This one requires DATAKV-117
Remove map key and list index values for secondary index structures. Paths like 'map.[key].property' are normalized to map.key.property. Same applies for lists 'list.[index].property' is converted to 'list.property'.

To allow better control over index resolution the entire logic has been moved to IndexResolver now inspecting the whole type instead of single properties.
RedisCallbacks now be directly used for eg. accessing index values by key to retrieve a set of domain types by ids.
Spellchecking, missing references, additionally formatting,...
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.
This commit adds support for adding an index using Spel
expressions.
@christophstrobl christophstrobl self-assigned this Dec 22, 2015
@christophstrobl
Copy link
Member

@rwinch looking at SpelIndexResolver I was wondering if it would make sense to include the SpelExpression evaluation as a special kind of index instead of having an explicit resolver for it.

Something like:

class SpelIndex extends RedisIndex {

    private String expression;

    public SpelIndex(String keyspace, String indexName, String expression) {
        super(keyspace, indexName);
        this.expression = expression;
    }

    public String getExpression() {
        return expression;
    }
}

//...

config.addIndexDefinition(new SpelIndex("spring:session:sessions", "principalName", "getAttribute('SPRING_SECURITY_CONTEXT')?.authentication?.name"));

The above can be optionally enhanced with a path to evaluate on, if that one is not present it's only applied on root level.
How much influence do you actually need on the EvaluationContext?

@rwinch
Copy link
Member Author

rwinch commented Jan 7, 2016

@christophstrobl Thanks for the feedback. I'm certainly open to implementation suggestions. My understanding of the current design was that the RedisIndex contains meta information to resolve the index, but no specific state. The IndexResolver allows some sort of state to be passed in (i.e. the object being persisted). Since the SpEL expression requires a reference to the object being persisted and there is no way to communicate that to the RedisInstance (or subclasses), I felt it was best to to do this in an IndexResolver.

In short, how would you communicate the persisted object to the index so it could be populated on the EvaluationContext?

@christophstrobl
Copy link
Member

@rwinch the seperation still exists so there's the RedisIndex holding meta info and the IndexedData in between is the resolver that checks the settings and property values. Using sort of a IndexedDataCreator via a factory would produce the state holding IndexedData out of the metadata and the actual value. The thing is that the EvaluationContext could not be configured as easily as you did in the PR.

@rwinch
Copy link
Member Author

rwinch commented Jan 8, 2016

Would I have reference to the object (i.e. HttpSession) itself? How would the object be passed into the index so the index could create the EvaluationContext with the object?

@christophstrobl
Copy link
Member

having somthing like:

 class SpelIndexCreator implements IndexedDataCreator {

        private final SpelIndex index;
        private final SpelExpressionParser parser;

        public SpelIndexCreator(SpelIndex index, SpelExpressionParser parser) {
            this.index = index;
            this.parser = parser;
        }

        @Override
        public IndexedData createFor(Object value) {

            Expression expression = parser.parseExpression(index.getExpression());
            StandardEvaluationContext context = new StandardEvaluationContext();
            context.setRootObject(value);
            context.setVariable("this", value);
            // context.setBeanResolver(beanResolver);
            Object indexValue = expression.getValue(context);


            return new SimpleIndexedPropertyValue(index.getKeyspace(), index.getIndexName(), indexValue);
        }

    }

that is accessible via an IndexedDataCreatorFactory.

IndexData acutalIndexedValue = dataCreatorFactory.get(spelIndexDefinition).createFor(value);

@rwinch
Copy link
Member Author

rwinch commented Jan 8, 2016

Thanks for the response. I'm not familiar with IndexedDataCreator...is this new? Would it be injected into the IndexResolverImpl? If IndexedDataCreator is new what is the advantage of this over a distinct IndexResolver?

NOTE I'm ok with whatever design you want so long as I can resolve SpEL expressions. I'm just not sure if I understand how you want me to modify the PR. I'm also curious to learn what advantages you see with your suggested approach.

@christophstrobl
Copy link
Member

@rwinch no need to modify the PR. Advantage would be that you can have the normal indexing features along the SpEL based ones by just telling the IndexResolver that you want a specific expression to be evaluated.

@rwinch
Copy link
Member Author

rwinch commented Jan 11, 2016

Advantage would be that you can have the normal indexing features along the SpEL based ones by just telling the IndexResolver that you want a specific expression to be evaluated.

Thanks. I'd imagine this could also be achieved by providing a CompositeIndexResolver which iterates over a collection of IndexResolver implementations. This might provide more flexibility in the end while still allowing the IndexResolver implementations the ability to be very focused on their responsibilities. Thoughts?

@christophstrobl
Copy link
Member

@rwinch yeah thought about it - no reason why not have a CompositeIndexResolver. Still the current IndexResolverImpl is not quite the was I want it to be. But this is something to be changed in another ticket.

christophstrobl pushed a commit that referenced this pull request Jan 12, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Jan 12, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 4, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 4, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 8, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 8, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 9, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 9, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 11, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 11, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
@mp911de mp911de force-pushed the issue/DATAREDIS-425 branch from cdc5f6c to 384cfb2 Compare February 19, 2016 13:10
christophstrobl pushed a commit that referenced this pull request Feb 23, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 23, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 26, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Feb 26, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Mar 14, 2016
This commit adds support for distinguishing between
the property path that is being indexed and the
name of the index that should be used.

This is useful when you do not want the property
path and index name to be the same. For example,
if you want to use Spel expressions, it may not be
possible to use the path as the name of the index.

Original PR #160
christophstrobl pushed a commit that referenced this pull request Mar 14, 2016
This commit adds support for adding an index using SpEL expressions.

Original PR #160
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants