Phase out localdb for claim tables; Just use host filesystem and/or sdk. #879

New issue

Open

opened 2019-01-24 03:29:50 +01:00 by jessopb · 2 comments

jessopb commented

2019-01-24 03:29:50 +01:00

(Migrated from github.com)

It was suggested that we store file names using sdhash.

The last question about the file table is whether we store the results of ffprobe and exif transformations. We might want to keep the file table to store file dimension data that's not in chainquery by sdhash, but it should not be authoritative that the file actually exists.

There has been a lot of discussion about cutting ties with localdb, at least for file and claim information. Reasons for this: Chainquery handles most requests very well across instances - even pending/mempool claims. Localdb complicates the code Localdb falsely reports about file availability if you enable delete files manually or using SDK to manage space and storage. Manually keeping sdk's sqlite and local mysql consistent is not ideal. It was suggested that we store file names using sdhash. The last question about the file table is whether we store the results of ffprobe and exif transformations. We might want to keep the file table to store file dimension data that's not in chainquery by sdhash, but it should not be authoritative that the file actually exists.

jessopb commented

2019-03-01 09:32:25 +01:00

(Migrated from github.com)

We'll keep the file table for dimension information, but we won't use the file table for whether a file exists.
lbrynet file list is now implemented by outpoint.

We'll keep the file table for dimension information, but we won't use the file table for whether a file exists. lbrynet file list is now implemented by outpoint.

jessopb commented

2019-03-25 19:53:27 +01:00

(Migrated from github.com)

create a publish cache
create a file_process cache
publish.js
add to publish cache and process cache
don't create file record
fetchClaimData
get chainquery and file_list cached outpoint responses rather than localdb
getClaimData
assemble claim data from file_list instead of db
getClaimId (gets claimId from name and shortId or name and channelName/Id
instead of LocalDB, match shortId in cache, check name, return longId.
claimShortId (gets shortId)
if chainquery fails, return longId.
getClaimIdAndServeAsset
don't resolve with localdb, resolve with lbrynet and cache

- [x] create a publish cache - [x] create a file_process cache - [ ] publish.js - add to publish cache and process cache - don't create file record - [x] fetchClaimData - get chainquery and file_list cached outpoint responses rather than localdb - [x] getClaimData - assemble claim data from file_list instead of db - [ ] getClaimId (gets claimId from name and shortId or name and channelName/Id - instead of LocalDB, match shortId in cache, check name, return longId. - [ ] claimShortId (gets shortId) - if chainquery fails, return longId. - [ ] getClaimIdAndServeAsset - don't resolve with localdb, resolve with lbrynet and cache

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/spee.ch#879

No description provided.