LBRYCommunity/hub
1
0
Fork 0
Code Issues 35 Pull requests 6 Projects Releases Packages Wiki Activity

Precomputed hashx status #17

Merged
jackrobison merged 7 commits from precomputed-hashx-status into master 2022-04-07 18:06:18 +02:00
Conversation 0 Commits 7 Files changed 10 +372 -56
jackrobison commented 2022-04-05 20:12:44 +02:00 (Migrated from github.com)
Copy link

fixes https://github.com/lbryio/scribe/issues/18

Adds the following column families:

  • HashXStatusPrefixRow: updated every block for all touched hashXs
  • HashXMempoolStatusPrefixRow: updated with every refresh of mempool for all of the newly touched hashXs, cleared upon new block.

This requires a database migration, it takes about 2 hours currently and will run automatically on first run from a version 7 database. To make this as fast as possible, make sure you have ~20gb of memory to use and run scribe with the CACHE_ALL_TX_HASHES setting on, without this setting ~4gb of ram is needed to migrate and it takes about 6 hours. The other services (scribe-hub and scribe-elastic-sync) don't need to be run while migrating, this saves on memory.

After running the migration the cache_all_tx_hashes setting can be turned back off if desired.

fixes https://github.com/lbryio/scribe/issues/18 Adds the following column families: - `HashXStatusPrefixRow`: updated every block for all touched hashXs - `HashXMempoolStatusPrefixRow`: updated with every refresh of mempool for all of the newly touched hashXs, cleared upon new block. This requires a database migration, it takes about 2 hours currently and will run automatically on first run from a version 7 database. To make this as fast as possible, make sure you have ~20gb of memory to use and run `scribe` with the `CACHE_ALL_TX_HASHES` setting on, without this setting ~4gb of ram is needed to migrate and it takes about 6 hours. The other services (scribe-hub and scribe-elastic-sync) don't need to be run while migrating, this saves on memory. After running the migration the cache_all_tx_hashes setting can be turned back off if desired.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area: database

   
area: documentation

Improvements or additions to documentation

  
area: elasticsearch

   
area: herald

   
area: packaging

   
area: scribe

   
consider soon

Discuss this issue at the next planning meeting, then remove this label

  
critical

   
dependencies

Pull requests that update a dependency file

  
good first issue

Good for newcomers

  
hacktoberfest

Welcome to Hacktoberfest

  
help wanted

Extra attention is needed

  
improvement

   
needs: repro

Needs a way to reproduce

  
new feature

   
priority: blocker

Issue is blocking release, do ASAP

  
priority: high

Work needs to be moved into sprint ASAP

  
priority: low

Work should be done but can stay in the backlog for now

  
priority: medium

Work needs to be done within 2-3 sprints

  
type: bug

Existing functionality is wrong or broken

  
type: bug-fix

   
type: discussion

A conversation. No specific changes requested

  
type: feature request

   
type: improvement

Existing (or partially existing) functionality needs to be changed

  
type: new feature

New functionality that does not exist yet

  
type: refactor

Minimal user-visible changes, but significant internal work

  
type: task

Either work that's not related to the code, or a small chore that does not fit into other categories

No labels
area: database
area: documentation
area: elasticsearch
area: herald
area: packaging
area: scribe
consider soon
critical
dependencies
good first issue
hacktoberfest
help wanted
improvement
needs: repro
new feature
priority: blocker
priority: high
priority: low
priority: medium
type: bug
type: bug-fix
type: discussion
type: feature request
type: improvement
type: new feature
type: refactor
type: task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/hub#17
No description provided.
Delete branch "precomputed-hashx-status"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?