Old data not visible for some users

[amruth-movano]

Describe the bug

Hi Team,
some of our users are not able to see their old data which is older than a week.

For some users if i fetch weekly data i.e. if i fetch data for whole week at one time, it's giving me result but when I fetch data for single day in past, it's not giving me any result.

Below is my schema for one table. I have also attached my logic files that how are we configuring amplify. Can you please check and let us know why amplify is behaving like this way?
amplify-issue.zip

extension SleepSession2 {
  // MARK: - CodingKeys 
   enum CodingKeys: String, ModelKey {
    case id
    case user_id
    case sleep_start_time
    case sleep_end_time
    case sleep_data
    case avg_heart_rate
    case avg_temperature
    case avg_breathing_rate
    case avg_sp_o2
    case avg_hrv
    case health_score
    case data_point_score
    case sleep_jobid
    case ring_serial_no
    case sleep_window_start
    case sleep_window_end
    case owner
    case createdAt
    case updatedAt
  }
  
  static let keys = CodingKeys.self
  //  MARK: - ModelSchema 
  
  static let schema = defineSchema { model in
    let sleepSession2 = SleepSession2.keys
    
    model.authRules = [
      rule(allow: .owner, ownerField: "owner", identityClaim: "cognito:username", provider: .userPools, operations: [.create, .update, .delete, .read])
    ]
    
    model.listPluralName = "SleepSession2s"
    model.syncPluralName = "SleepSession2s"
    
    model.attributes(
      .index(fields: ["user_id", "sleep_window_end"], name: nil),
      .primaryKey(fields: [sleepSession2.user_id, sleepSession2.sleep_window_end])
    )
    
    model.fields(
      .field(sleepSession2.id, is: .required, ofType: .string),
      .field(sleepSession2.user_id, is: .required, ofType: .string),
      .field(sleepSession2.sleep_start_time, is: .required, ofType: .dateTime),
      .field(sleepSession2.sleep_end_time, is: .required, ofType: .dateTime),
      .field(sleepSession2.sleep_data, is: .optional, ofType: .embeddedCollection(of: Int.self)),
      .field(sleepSession2.avg_heart_rate, is: .optional, ofType: .double),
      .field(sleepSession2.avg_temperature, is: .optional, ofType: .double),
      .field(sleepSession2.avg_breathing_rate, is: .optional, ofType: .double),
      .field(sleepSession2.avg_sp_o2, is: .optional, ofType: .double),
      .field(sleepSession2.avg_hrv, is: .optional, ofType: .double),
      .field(sleepSession2.health_score, is: .optional, ofType: .double),
      .field(sleepSession2.data_point_score, is: .optional, ofType: .string),
      .field(sleepSession2.sleep_jobid, is: .optional, ofType: .string),
      .field(sleepSession2.ring_serial_no, is: .optional, ofType: .string),
      .field(sleepSession2.sleep_window_start, is: .required, ofType: .dateTime),
      .field(sleepSession2.sleep_window_end, is: .required, ofType: .dateTime),
      .field(sleepSession2.owner, is: .optional, ofType: .string),
      .field(sleepSession2.createdAt, is: .optional, isReadOnly: true, ofType: .dateTime),
      .field(sleepSession2.updatedAt, is: .optional, isReadOnly: true, ofType: .dateTime)
    )
    }
}

Steps To Reproduce

It is reproducible for some users all the time.
Login with user.
Check the data for olders days
Datastore not giving any data though the data is present in dynamodb

Expected behavior

Amplify datastore should give us the data, if it present in dynamodb

Amplify Framework Version

2.33.6

Amplify Categories

DataStore

Dependency manager

Swift PM

Swift version

5.9.2

CLI version

12.11.1

Xcode version

15.0

Relevant log output

<details>
<summary>Log Messages</summary>


INSERT LOG MESSAGES HERE

```

Is this a regression?

Yes

Regression additional context

No response

Platforms

iOS

OS Version

iOS 17.0

Device

iPhone 13

Specific to simulators

No response

Additional context

No response

[amruth-movano]

Please see attached verbose log file -

https://drive.google.com/file/d/12c5dvjpmoHZIYtNEkdGePfmvpd5SDwV4/view?usp=sharing

[lawmicha]

From the attached amplify-issue.zip, I can see the sync expression being used

let syncExpressions = [
            DataStoreSyncExpression.syncExpression(UserProfile2.schema, where: {
                UserProfile2.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(ExerciseSession2.schema, where: {
                ExerciseSession2.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(TimedData.schema, where: {
                if let dateLimit = dateLimit {
                TimedData.keys.user_id.eq(userId).and(TimedData.keys.time_stamp.gt(dateLimit.temporalDateTime))
                } else {
                    TimedData.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(MenstrualLog.schema, where: {
                if let dateLimit = dateLimit {
                MenstrualLog.keys.user_id.eq(userId).and(MenstrualLog.keys.time_stamp.gt(dateLimit.temporalDateTime))
                } else {
                    MenstrualLog.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(MobileToken2.schema, where: {
                MobileToken2.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(SleepSession2.schema, where: {
                if let dateLimit = dateLimit {
                SleepSession2.keys.user_id.eq(userId).and(SleepSession2.keys.sleep_window_end.gt(dateLimit.temporalDateTime))
                } else {
                    SleepSession2.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(SleepStage.schema, where: {
                if let dateLimit = dateLimit {
                SleepStage.keys.user_id.eq(userId).and(SleepStage.keys.data_timestamp.gt(dateLimit.temporalDateTime))
                } else {
                    SleepStage.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(SpotCheck.schema, where: {
                if let dateLimit = dateLimit {
                SpotCheck.keys.user_id.eq(userId).and(SpotCheck.keys.data_timestamp.gt(dateLimit.temporalDateTime))
                } else {
                    SpotCheck.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(UserDailyMedians2.schema, where: {
                if let dateLimit = dateLimit {
                UserDailyMedians2.keys.user_id.eq(userId).and(UserDailyMedians2.keys.time_stamp.gt(dateLimit.temporalDateTime))
                } else {
                    UserDailyMedians2.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(UserGoals.schema, where: {
                UserGoals.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(UserPreference.schema, where: {
                UserPreference.keys.user_id.eq(userId)
            }),

            DataStoreSyncExpression.syncExpression(DeviceTelemetry.schema, where: {
                if dateLimit != nil {
                DeviceTelemetry.keys.user_id.eq(userId).and(DeviceTelemetry.keys.time_stamp.ge((Date() - .hours(2)).temporalDateTime))
                } else {
                    DeviceTelemetry.keys.user_id.eq(userId)
                }
            })
        ]
        
        try Amplify.add(plugin: AWSDataStorePlugin(
            modelRegistration: DataStoreModels(),
            configuration: .custom(errorHandler: onError, syncMaxRecords: maxLimit, syncExpressions: syncExpressions)
        ))

The data to be synchronized is filtered down to the userId or when dateLimit is non nil then both the userId and the dateLimit which is used to control syncing weekly or last day data of the particular user.

DataStore performs a sync[Model] query to retrieve the data and apply the syncExpression on the data client-side, reconcile the data to the local database and then reconciles the subscription events from the subscription connection as an on-going basis. The subscription is established with the syncExpression, applying the filter on the server side.

The synchronization is going to be drastically improved if we applied the syncExpression with the sync query call on the server side. This is pending proiritization of this PR #3550 . Some challenges we faced with getting the server side filter released is due to supporting the edge case of supporting customers that do not have AppSync backends that support server side filter (as that feature was launched after the client side filter implementation was put in place).

The problems you see may be solved if your AppSync backend is provisioned with filter parameter on the sync query operations, which means it supports server side filtering, and taking on the latest version of Amplify after we ship the library changes to pass the syncExpression as the filter parameter, since it would no longer evaluate the expression against the response data from the sync request locally.

There could be a bug in the client side evaluation such that given a dateLimit and the response model, it is incorrectly discarding it.

The verbose file is over 100M, could you narrow down the logs to just the output from the result of calling DataStore.start() on a fresh install.

Also could you share code snippet of how and what the dateLimit is set to? We could reproduce this with that information.

[lawmicha]

can you give us the full code of the extensions used by this line inside startAndPopulateDataStore?

.now.advanced(by: -7, byAdding: .day).startOfDay

[amruth-movano]

@lawmicha this is the code used by extension -

public func advanced(
by value: Int,
byAdding unit: Calendar.Component,
calendar: Calendar = .current
) -> Date {
calendar.date(byAdding: unit, value: value, to: self)!
}

[amruth-movano]

@lawmicha for datelimit, we had set it while login, you can check the sessionmanager class attached in zip file, inside signin function you will get a call to startAndPopulateDataStore with the dateLimit

[5d]

@amruth-movano ,

The verbose log you provided is too extensive for us to investigate. Could you please narrow it down to the specific issue where fetching 1 week data is successful, but fetching single day data is not?

Additionally, in the verbose log, there should be generated GraphQL queries. Could you verify whether the single-day query, which is supposed to retrieve the data, is actually fetching the data in the AWS AppSync console?

[amruth-movano]

Hi @5d
I am attaching 4 files. So I have data on 22nd, 23rd 24th may in backend and in week table it is showing but not in daily page.
Also after doing logout and relogin again, facing same issues in data store.
Can you please check & let us know the issue?

verbose.zip

[lawmicha]

Hi @amruth-movano, I checked the code and I misspoke earlier, it is passing the syncExpression as the filter input for sync query. It is applying the syncExpression client side only on the subscription events. As @5d mentioned, can you add Amplify.Logging.logLevel = .verbose and launch the app. Have dataLimit set to weekly and initiate a DataStore.start()? The logs will then have the GraphQL queries. Repeat this with daily, and you should be able to find two sync query requests with different filter inputs.

Can you provide us with these logs that show the GraphQL queries, and you should also be able to replay them in the AppSync console to see the response which will be reconciled to the local database

[amruth-movano]

HI @lawmicha
I have ran app by setting dateLimit to week and here are the verbose logs. And I am new to amplify, can you please explain in detail if there is any issue in our log?

Archive.zip