MongoObservationCommandListener OutOfMemory

[JonathanHel]

Hi,

When we iterating over a collection with a large number of documents (> 1 mio) our pod ran out of Memory.
Even, when we just used the MongoCurser without any other actions this happens.

void republishFromDatabase(Consumer<StoreVehicleOperations> onNext, boolean changeStream, boolean stateStore) {
   try (var cursor = mongoTemplate.getCollection(Vehicle.COLLECTION_NAME)
        .withReadPreference(ReadPreference.secondaryPreferred())
        .find()
        .batchSize(batchSize)
        .cursor()) {
        while (cursor.hasNext()) {
            var document = cursor.next();
        }
    }
}

With this configuration:

@Bean        
public MongoClientSettingsBuilderCustomizer mongoCustomizer()  {
        return builder -> {
            builder
                    .contextProvider(ContextProviderFactory.create(observationRegistry))
                    .uuidRepresentation(UuidRepresentation.JAVA_LEGACY)
                    .retryWrites(false)
                    .addCommandListener(new MongoObservationCommandListener(observationRegistry));

        };
    }

I found out hat this happens duo the usage of the MongoObservationCommandListener.
Since this listener will add a getMore observation for each getMore and will link them:

getMore
  getMore
    getMore
      ...

After I removed
.addCommandListener(new MongoObservationCommandListener(observationRegistry))
from my MongoClientSettingsBuilderCustomizer, it works fine.

This configuration works:

           return builder -> {  builder
                    .uuidRepresentation(UuidRepresentation.JAVA_LEGACY)
                    .retryWrites(false);
                    };

I tried this with:
spring-boot-starter-mongodb-reactive with 3.1.0
spring-boot-starter-mongodb with 3.1.0

[mp911de]

Observations are kept in memory until they are finished. Have you tried installing a ObservationPredicate to exclude observation for the large iteration? Alternatively, you can increase the fetch size to create less observations. /cc @marcingrzejszczak

[JonathanHel]

Yes that would work, but might be hard to always assume beforehand which observation should be skipped.
And if this happens on production the code needs first extended to solve issues.
Does it make sense to only store a defined number of observations?

[marcingrzejszczak]

I mean if you fetch a gigantic number of data (let's say X) we will create at least another gigantic number of observations (at least also X). We can't know this upfront either.

[JonathanHel]

OK, a framework can not know everything before hand.
Maybe, a hint within the documentation i.e. https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongodb.observability

Might be helpful for other developers in the future.