Android wipes app data on some devices

[kashmiry]

Current behavior

Currently async storage works great on all iOS devices and most of Android devices.
The issue has been reporting by android users of my app since I released the app on android about 6 months ago.
The users report that when they make changes in the app, and reopen the app again all their saved setting & data would reset and clear. (I use async storage as the main storage for the app)
This is really frustrating and I can't detect exactly what causes it.
I am noticing most users report this from Huawei phones some from Samsung phones.
I have a Vivo Y15 phone and I seem to get this issue once a month the app would clear the storage, though theres alot of free space on the phone.
How ever on another One plus phone I tested on, it never happens. I would say it happen in about 20% of Android phones.
How can I debug this and know what causes it?
I produce apps in AAB format could this be an issue? and I am not enabling hermes.

Expected behavior

The app won't wipe the data on android with no reason.

Repro steps

I am testing more and was able to reproduce it.
The issue happens when data stored increase to around up to 4 ~ 6 mb. After that I close the app and open it again, and everything is wiped.
I use asyncStorage with redux-persist and some posts and data that the user loads get stored with asyncStorage though this delete all data not only on redux but also on data that I used setItem with.
again this issue does not happen with most Android phones, some phones even with lots of ram up to 6gb and free space it still happens, not sure why!

Log error:
Logged error in android studio when I launch the app and a wipe happens:
E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=0, totalRows=1; query: SELECT key, value FROM catalystLocalStorage WHERE key IN (?)

I tried to increase the storage, adding this to my android/gradle.properties:
AsyncStorage_db_size_in_MB=50
but still the wipe occurs on reopen when storage is between 4 ~ 6 mb

Environment

• Platforms tested:
  • Android
  • iOS
  • macOS
  • Windows
• AsyncStorage version: 1.13.3
• Environment:
  System:
  OS: macOS 11.1
  CPU: (4) x64 Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz
  Memory: 961.64 MB / 8.00 GB
  Shell: 3.2.57 - /bin/bash
  Binaries:
  Node: 15.2.0 - /usr/local/bin/node
  Yarn: 1.22.0 - ~/.yarn/bin/yarn
  npm: 7.0.10 - /usr/local/bin/npm
  Watchman: 4.9.0 - /usr/local/bin/watchman
  Managers:
  CocoaPods: 1.9.3 - /usr/local/bin/pod
  SDKs:
  iOS SDK:
  Platforms: iOS 14.4, DriverKit 20.2, macOS 11.1, tvOS 14.3, watchOS 7.2
  Android SDK: Not Found
  IDEs:
  Android Studio: 4.1 AI-201.8743.12.41.7042882
  Xcode: 12.4/12D4e - /usr/bin/xcodebuild
  Languages:
  Java: 11.0.2 - /usr/bin/javac
  Python: 2.7.16 - /usr/bin/python
  npmPackages:
  @react-native-community/cli: Not Found
  react: 16.13.1 => 16.13.1
  react-native: ^0.63.4 => 0.63.4
  react-native-macos: Not Found
  npmGlobalPackages:
  react-native: Not Found

[imarsiglia]

I was able to solve this issue modifying the onCreate method in MainApplication.java for android
Just add this at the end of the method

try {
Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
field.setAccessible(true);
field.set(null, 100 * 1024 * 1024); //100MB
} catch (Exception e) {
if (BuildConfig.DEBUG) {
e.printStackTrace();
}
}

[krizzu]

@kashmiry

Hey, sorry for late response. Thanks for providing all those details!
From the log message you provided, it looks like they data you try to put/read into storage is too big to fit in one go (due to cursor being limited size), which has nothing to do with storage size. SQLite has limits, where in some cases can vary between device vendors.

As a fix, I can suggest to either chop data into smaller chunks, on come with some sort of data pagination.

[kashmiry]

I was able to solve this issue modifying the onCreate method in MainApplication.java for android
Just add this at the end of the method

try {
Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
field.setAccessible(true);
field.set(null, 100 * 1024 * 1024); //100MB
} catch (Exception e) {
if (BuildConfig.DEBUG) {
e.printStackTrace();
}
}

@imarsiglia Thanks 🙏🏼 This fixed it for me now.
Does that mean when storage reaches 100mb the issue will occur again? is there any setbacks for setting such a large data size or is it fine?

@kashmiry

Hey, sorry for late response. Thanks for providing all those details!
From the log message you provided, it looks like they data you try to put/read into storage is too big to fit in one go (due to cursor being limited size), which has nothing to do with storage size. SQLite has limits, where in some cases can vary between device vendors.

As a fix, I can suggest to either chop data into smaller chunks, on come with some sort of data pagination.

@krizzu Thank you for the response the solution above fixed it for me, is it suitable and safe to apply?
Also as I mentioned I use redux and redux-pressist, and it seems to store the whole store in one async storage item and read it later.

• Are you familiar with it? if yes whats the best approach here to divide the data?
• Do I have to divide data in the objects and reducers inside the store, Or I should have multiple stores in some way?

[iBotPeaches]

Thanks for this thread. We are hitting same issue and the gradle.properties override does not help.

# https://react-native-async-storage.github.io/async-storage/docs/advanced/db_size
AsyncStorage_db_size_in_MB=1024

I would expect to get around that weird 6mb limit, but I may be dumping XML blobs ranging from 100kb to 500kb into the db and right around when it hits 6mb - it breaks bad.

                         E  Failed to read row 0, column 0 from a CursorWindow which has 0 rows, 2 columns.
    unknown:ReactNative  W  Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
                         W  java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
                         W      at android.database.CursorWindow.nativeGetString(Native Method)
                         W      at android.database.CursorWindow.getString(CursorWindow.java:438)
                         W      at android.database.AbstractWindowedCursor.getString(AbstractWindowedCursor.java:66)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$1.doInBackgroundGuarded(AsyncStorageModule.java:179)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$1.doInBackgroundGuarded(AsyncStorageModule.java:146)
                         W      at com.facebook.react.bridge.GuardedAsyncTask.doInBackground(GuardedAsyncTask.java:36)
                         W      at com.facebook.react.bridge.GuardedAsyncTask.doInBackground(GuardedAsyncTask.java:20)
                         W      at android.os.AsyncTask$2.call(AsyncTask.java:304)
                         W      at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$SerialExecutor$1.run(AsyncStorageModule.java:63)
                         W      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                         W      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
                         W      at java.lang.Thread.run(Thread.java:761)

Then I noticed I was out of date. So I tried again on latest (1.14.1). Submitting 30 xml files knowing I was over 6mb probably. Restarted app and...

    unknown:ReactNative  W  Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
                         W  java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
                         W      at android.database.CursorWindow.nativeGetString(Native Method)
                         W      at android.database.CursorWindow.getString(CursorWindow.java:438)
                         W      at android.database.AbstractWindowedCursor.getString(AbstractWindowedCursor.java:66)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$1.doInBackgroundGuarded(AsyncStorageModule.java:179)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$1.doInBackgroundGuarded(AsyncStorageModule.java:146)
                         W      at com.facebook.react.bridge.GuardedAsyncTask.doInBackground(GuardedAsyncTask.java:36)
                         W      at com.facebook.react.bridge.GuardedAsyncTask.doInBackground(GuardedAsyncTask.java:20)
                         W      at android.os.AsyncTask$2.call(AsyncTask.java:304)
                         W      at java.util.concurrent.FutureTask.run(FutureTask.java:237)
                         W      at com.reactnativecommunity.asyncstorage.AsyncStorageModule$SerialExecutor$1.run(AsyncStorageModule.java:63)
                         W      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
                         W      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
                         W      at java.lang.Thread.run(Thread.java:761)

So to answer the questions above also.

it looks like they data you try to put/read into storage is too big to fit in one go (due to cursor being limited size)

I'm not. I'm submitting arguably not db normal text (xml payloads), but they are small and insert fine until enough of them collect. The reloading the db once it exceeds 6mb on Android (even with that max value) leads to crashes and then it empties/corrupts the table and I lose the data.

I'm trying the hackish fix for increasing cursor size now and will edit back.

EDIT: Hacky fix worked! Thanks imarsiglia

[krizzu]

Sorry again for late response

@kashmiry

@krizzu Thank you for the response the solution above fixed it for me, is it suitable and safe to apply?

I cannot say how "safe" it is to use, because I have no idea about possible implications. From what I see, it modifies the default cursor size for whole class - might affect other dependencies which are using it.

Also as I mentioned I use redux and redux-pressist, and it seems to store the whole store in one async storage item and read it later.

• Are you familiar with it? if yes whats the best approach here to divide the data?

Not really familiar with it, sorry. Maybe there's a way to modify the default behavior of dumping whole data into AsyncStorage?

• Do I have to divide data in the objects and reducers inside the store, Or I should have multiple stores in some way?

Maybe if there's no way to do it via modifying default behavior, maybe chopping down state into smaller objects? redux-persist is storing data based on it (dumping whole reducer's state into AsyncStorage)

@iBotPeaches
So in your case, increasing the size of DB will not do the trick, because it's again, related to the cursor size (the buffer that transfers data to/from db), so the same concerns I mentioned above applies here.

Just FYI, I'm working on adding a different storage mechanism as opt-in feature for now, which uses recommended way of storing data on Android. I'm yet to test if those kind of scenarios are persisting on it, but overall it's less error prone than current solution.

[iBotPeaches]

Thanks!

It appears I was chasing the wrong repo/issues anyway. I didn't realize that redux-persist just stores the entire store in one json file. I thought it acted like a db backed storage engine and set/get a group of items at a time and merged them at runtime.

That was of course wrong - thanks for the help. We stopped the bleeding for now and investigating a proper way to persist a redux store with large amounts of data which may just be a filesystem based driver.

My references for other googlers: rt2zz/redux-persist#199 rt2zz/redux-persist#284

[kashmiry]

I tested the next storage solution, but does not seem to solve it for me.
Now what worked best for me is increasing cursor limit in MainApplication.java.

• this solution to separate state in nested reducers in redux-persist. This does not wipe all data from all reducers. works for me, for now.

[sasweb]

@krizzu any updates on this topic? Especially in terms of how the next solution handles this?

I think this is very critical since it will mostly affect production apps with large amounts of data. I can also imagine that many devs won't notice it in the first place :/

[github-actions]

This issue has been marked as stale due to inactivity. Please respond or otherwise resolve the issue within 7 days or it will be closed.

[shehzadosama]

I was able to solve this issue modifying the onCreate method in MainApplication.java for android Just add this at the end of the method

try { Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize"); field.setAccessible(true); field.set(null, 100 * 1024 * 1024); //100MB } catch (Exception e) { if (BuildConfig.DEBUG) { e.printStackTrace(); } }

Thanks, @imarsiglia, it worked for me.
Also whoever trying this solution, please import these 2 lines at the top of MainApplication.java

import java.lang.reflect.Field;
import android.database.CursorWindow;

[cjhines]

Experiencing this in 1.12.1. Only seems to be on certain Android devices.