LBRYCommunity/lbry-desktop

Fork 0

Cache state in IndexedDB #194

Closed

6ea86b96 wants to merge 5 commits from redux-persist into master

6ea86b96 commented

2017-06-04 19:42:23 +02:00

(Migrated from github.com)

We can cache parts of the state in local storage to achieve almost instantaneous load times.

We can also compress that data nice and small:

It looks a bit like this:

We can cache parts of the state in local storage to achieve almost instantaneous load times. ![fast-load](https://cloud.githubusercontent.com/assets/20863631/26763955/aaf6d70e-4987-11e7-9377-31032a31c67d.gif) We can also compress that data nice and small: ![2017-06-05 at 00 35](https://cloud.githubusercontent.com/assets/20863631/26763939/22292dbe-4987-11e7-90e4-ca9c3826720d.jpg) It looks a bit like this: ![2017-06-05 at 00 37](https://cloud.githubusercontent.com/assets/20863631/26763943/45972620-4987-11e7-8f00-b51056f26f93.jpg)

kauffj commented

2017-06-05 01:08:53 +02:00

(Migrated from github.com)

This is exciting! One thing to consider is what this behavior should be like as the landing page gets more dynamic.

6ea86b96 commented

2017-06-05 09:00:44 +02:00

(Migrated from github.com)

Remaining to do here is investigate cache expiry. I'm looking at adding at time based expiration to individual claims. I'm not too sure what happens if the cache size hits the localStorage 10MB limit. I couldn't find much information on how redux-persist handles that case. Even compressed, the cache does grow fairly rapidly. I was able to grow it to 230KB after 30 minutes of searching.

We will also need to refresh claim data while navigating to ensure updated claims are synced to the store/cache.

Yes, I was trying to cache the minimum necessary data to speed everything up. This branch was forked from the claim ids branch. I'm actually only caching a very small subset of the state. Just the `byId`, `claimsByUri` and `myClaims` properties of the `claims` store property. This way we still lookup featured content, uploaded/downloaded files, etc, and just load the claim from cache. Hopefully we can also minimize situations where we change the structure of the data and it's different in the cache like this too. Remaining to do here is investigate cache expiry. I'm looking at adding at time based expiration to individual claims. I'm not too sure what happens if the cache size hits the localStorage 10MB limit. I couldn't find much information on how `redux-persist` handles that case. Even compressed, the cache does grow fairly rapidly. I was able to grow it to 230KB after 30 minutes of searching. We will also need to refresh claim data while navigating to ensure updated claims are synced to the store/cache.

kauffj commented

2017-06-05 18:30:03 +02:00

(Migrated from github.com)

A few comments:

Rather than worrying about date based time outs, we could just always refresh when serving from a cached state, possibly with some minor indicator that potentially stale data is being rendered in the interim.
We can invalidate cached data via the build process as well. For example, it could be directly tied to lbryschema or lbry version numbers.
The slowness of file_list and claim_list_mine and other API calls is a relatively high priority for improvements to the daemon. This doesn't really affect anything here, just letting you know.

A few comments: 1) Rather than worrying about date based time outs, we could just always refresh when serving from a cached state, possibly with some minor indicator that potentially stale data is being rendered in the interim. 2) We can invalidate cached data via the build process as well. For example, it could be directly tied to `lbryschema` or `lbry` version numbers. 3) The slowness of `file_list` and `claim_list_mine` and other API calls is a relatively high priority for improvements to the daemon. This doesn't really affect anything here, just letting you know.

6ea86b96 commented

2017-06-06 07:49:28 +02:00

(Migrated from github.com)

Those daemon improvements will be great. claim_list_mine results are actually already in the cache and refreshed as you navigate. file_list did seem like a good caching target too. Not sure how much optimization can be done when you need to read from the file system? I didn't add those to the cache yet though, just to keep it simple.

I think the problem is more about users using the app for long enough that localstorage limits are reached, which seems possible within around 1 full day of heavy usage. Of course this also makes date based timeouts not particularly useful. Seems like a FIFO strategy would be great here, but not particularly easy to implement. 100% agree we should refresh when serving from cache. Unless I'm missing something I don't think invalidating from builds is going to help that much in this case, at least once releases become a bit less frequent. I'm kind of hoping `redux-persist` just handles storage full out of the box. Going to look into that today. Those daemon improvements will be great. `claim_list_mine` results are actually already in the cache and refreshed as you navigate. `file_list` did seem like a good caching target too. Not sure how much optimization can be done when you need to read from the file system? I didn't add those to the cache yet though, just to keep it simple.

kauffj commented

2017-06-06 15:19:32 +02:00

(Migrated from github.com)

Most slow API commands are driven by remote calls to validate signatures. File list is slow not from disk reads but because it is re-validating the claim info every time it is fetched.

Yes, this is dumb.

Most slow API commands are driven by remote calls to validate signatures. File list is slow not from disk reads but because it is re-validating the claim info every time it is fetched. Yes, this is dumb.

6ea86b96 commented

2017-06-07 10:36:01 +02:00

(Migrated from github.com)

Ok @kauffj, so there are 2 separate issues here.

localStorage filling up. I've fixed this by switching to localForage which uses indexedDB under the covers. Electron has much larger size limits for this and when it does hit them indexedDB handles eviction itself on a LRU basis.
Making sure that the claims/prices/fileInfo/etc are updated. This is a general issue now that everything is in redux and we need to figure out a nice way to sync data. I think we tackle that separately.

Ok @kauffj, so there are 2 separate issues here. 1) localStorage filling up. I've fixed this by switching to `localForage` which uses `indexedDB` under the covers. Electron has much larger size limits for this and when it does hit them `indexedDB` handles eviction itself on a LRU basis. 2) Making sure that the claims/prices/fileInfo/etc are updated. This is a general issue now that everything is in redux and we need to figure out a nice way to sync data. I think we tackle that separately.

6ea86b96 commented

2017-06-07 17:17:13 +02:00

(Migrated from github.com)

Just rebased on master and squashed into one commit.

6ea86b96 commented

2017-06-08 08:14:07 +02:00

(Migrated from github.com)

Realized that this would cause some problems on the discover page. Cached claim data wouldn't get updated so people wouldn't see updated community content.

I've added some code to force claims to be refreshed on the discover page and on the show page when it's mounted. Search already seems to resolve claim data so I just made that a little more efficient with batch actions.

@kauffj

Realized that this would cause some problems on the discover page. Cached claim data wouldn't get updated so people wouldn't see updated community content. I've added some code to force claims to be refreshed on the discover page and on the show page when it's mounted. Search already seems to resolve claim data so I just made that a little more efficient with batch actions. @kauffj

kauffj commented

2017-06-09 00:53:50 +02:00

(Migrated from github.com)

I'd like to wait to merge into master until after we release 0.12.

👍 1

6ea86b96 commented

2017-06-11 12:47:28 +02:00

(Migrated from github.com)

Closing this in favour of https://github.com/lbryio/lbry-app/pull/225