Commit graph

1860 commits

Author SHA1 Message Date
Jack Robison 2138e7ea33
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 32f8c9e59f
renormalization 2021-10-05 16:44:49 -04:00
Jack Robison 3a16edd8a6
fix trending overflow 2021-10-05 16:44:49 -04:00
Jack Robison 165f3bb270
refactor trending 2021-10-05 16:44:49 -04:00
Jack Robison 0ba75153f3
trending fixes 2021-10-05 16:44:49 -04:00
Jack Robison db2789990f
make app backward compatible with trending_score
-update trending decay function to zero out low trending score values faster
2021-10-05 16:44:49 -04:00
Jack Robison acaf299bcb
log time to update and decay trending in elasticsearch 2021-10-05 16:44:49 -04:00
Jack Robison 1940301824
skip integrity errors for trending spikes 2021-10-05 16:44:49 -04:00
Jack Robison 34576e880d
update trending in elasticsearch
-add TrendingPrefixSpike to leveldb
-expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings
2021-10-05 16:44:49 -04:00
Brendon J. Brewer 65c0668d40
constants 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 53bd2bcbfe
Put trending score into ES 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 388724fccb
Mark claims as touched 2021-10-05 16:44:49 -04:00
Jack Robison 231eabb013
fix non normalized canonical urls 2021-10-05 16:44:49 -04:00
Jack Robison 54903fc2ea
handle unicode error for unnormalized names 2021-10-05 16:44:49 -04:00
Jack Robison 3a1baf0700
prefix db 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 0c0e36b6f8
trending 2021-10-05 16:44:49 -04:00
Jack Robison 234c03db09
fix claims not having non-normalized names 2021-10-05 16:44:49 -04:00
Jack Robison 28aa7da349
merge conflicts 2021-10-05 16:44:49 -04:00
Jack Robison c51e344b87
fix missing fields in reposts 2021-10-05 16:44:49 -04:00
Jack Robison 54461dfa75
fix merge conflicts and simplify extract_doc 2021-10-05 16:44:49 -04:00
Jack Robison 2d48e93f74
fix bulk es sync 2021-10-05 16:44:49 -04:00
Jack Robison af22646322
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 722b42a93e
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 8f9e7f77a7
handle invalid claim update 2021-10-05 16:44:49 -04:00
Jack Robison 09bb1ba494
fix keeping claim_hash_to_txo and txo_to_claim in sync 2021-10-05 16:44:49 -04:00
Victor Shyba d4137428ff
implement blocking and filtering 2021-10-05 16:44:49 -04:00
Jack Robison b4d6c4f5b7
fix _get_pending_claim_name 2021-10-05 16:44:49 -04:00
Jack Robison ffbe59ece5
fix applying expiration fork 2021-10-05 16:44:49 -04:00
Jack Robison fab9c90ccb
update iterators to use pack_partial_key 2021-10-05 16:44:49 -04:00
Jack Robison fb1a774bc4
delete lbry/wallet/server/storage.py
-expose leveldb lru cache size as `CACHE_MB` hub param
2021-10-05 16:44:49 -04:00
Jack Robison 98bc7d1e0e
remove dead code 2021-10-05 16:44:49 -04:00
Jack Robison f7622f24b2
non blocking mempool loop 2021-10-05 16:44:49 -04:00
Jack Robison f0a195a6d4
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison 180ba27d84
run advance_block in threadpool 2021-10-05 16:44:49 -04:00
Jack Robison f944671f86
use claim_to_txo cache 2021-10-05 16:44:49 -04:00
Jack Robison def2903f7d
faster _cached_get_active_amount for claims
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison 0273a4e839
fix claim search by fee for claims without fees 2021-10-05 16:44:49 -04:00
Jack Robison f8d2f02c5d
clear claim_to_txo cache before reading 2021-10-05 16:44:49 -04:00
Jack Robison 25147d8897
handle claims that dont exist in ES sync 2021-10-05 16:44:49 -04:00
Jack Robison 0fb6f05fba
in memory claim_to_txo and txo_to_claim dictionaries 2021-10-05 16:44:49 -04:00
Jack Robison 4e4e899356
fix spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison 5a01dbf269
split flush from advance_block 2021-10-05 16:44:49 -04:00
Jack Robison 30b923b283
rename extend_ops 2021-10-05 16:44:49 -04:00
Jack Robison 73ba381d20
faster spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison 1a5912877e
faster get_future_activated 2021-10-05 16:44:49 -04:00
Jack Robison 813e506b68
threadpool 2021-10-05 16:44:49 -04:00
Jack Robison 077ca987f7
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison c632a7a6a5
fix getting block hash during reorg 2021-10-05 16:44:49 -04:00
Jack Robison e33e767510
fix test 2021-10-05 16:44:49 -04:00
Jack Robison ac82617aa9
fix spends in address histories 2021-10-05 16:44:49 -04:00
Jack Robison a35dfd1fd1
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison c28aae9913
fix expiring channels 2021-10-05 16:44:49 -04:00
Jack Robison c26a99e65c
fix abandoning signed claims in the same tx as their channel
-fix canonical/short url in es
2021-10-05 16:44:49 -04:00
Jack Robison ca57dcfc2f
handle failure to generate a short id 2021-10-05 16:44:49 -04:00
Jack Robison df5662dd69
fix resolve by short id 2021-10-05 16:44:49 -04:00
Jack Robison 1ac7831f3c
move MemPool into BlockProcessor 2021-10-05 16:44:49 -04:00
Jack Robison 292d272a94
combine MemPool and Notifications classes 2021-10-05 16:44:49 -04:00
Jack Robison a6ee8dc66e
fix touched hashXs notifications 2021-10-05 16:44:49 -04:00
Jack Robison 496f89f184
reorg claims in the search index 2021-10-05 16:44:49 -04:00
Jack Robison 7a56eff1ac
small fixes 2021-10-05 16:44:49 -04:00
Jack Robison 07e182aa16
rename 2021-10-05 16:44:49 -04:00
Jack Robison 7de06aa1e0
delete stale code 2021-10-05 16:44:49 -04:00
Jack Robison 3955b64405
simplify advance and reorg 2021-10-05 16:44:49 -04:00
Jack Robison 2bb55d681d
update limited_history 2021-10-05 16:44:49 -04:00
Jack Robison f94e6ac527
update lookup_utxos 2021-10-05 16:44:49 -04:00
Jack Robison b344f17b86
update RevertableOpStack 2021-10-05 16:44:49 -04:00
Jack Robison 677b8cb633
add remaining db prefixes 2021-10-05 16:44:49 -04:00
Jack Robison a1ddd762e0
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 68474e4057
skip es sync during initial hub sync, halt the hub upon finishing initial sync 2021-10-05 16:44:49 -04:00
Jack Robison a84b9ee396
fix es sync 2021-10-05 16:44:49 -04:00
Jack Robison b9c2ee745a
fix non localhost elasticsearch 2021-10-05 16:44:49 -04:00
Jack Robison 615e489d8d
fix stream_update --clear_channel flag 2021-10-05 16:44:49 -04:00
Jack Robison c68f9f6f16
fix signed claim invalidation corner cases 2021-10-05 16:44:49 -04:00
Jack Robison 229cb85a6a
extra deletes
-the channel_to_claim/claim_to_channel entries already get deleted when the claim txo is spent
2021-10-05 16:44:49 -04:00
Jack Robison e5c22fa665
fix has_no_source for reposts 2021-10-05 16:44:49 -04:00
Jack Robison 8bcfff05d7
update channel_to_claim and claim_to_channel at the same time 2021-10-05 16:44:49 -04:00
Jack Robison 6416ee8151
typing and fix error string 2021-10-05 16:44:49 -04:00
Jack Robison f8eceb48e6
update staged txo_to_claim after invalidating channel sig
-fixes abandon of claim with invalidated signature and an update in same block
2021-10-05 16:44:49 -04:00
Jack Robison 310c483bfa
missing channel_to_claim delete 2021-10-05 16:44:49 -04:00
Jack Robison a8f20361aa
fix RepostKey 2021-10-05 16:44:49 -04:00
Jack Robison 290be69d99
typing 2021-10-05 16:44:49 -04:00
Jack Robison 3b96bd7ea0
fix 2021-10-05 16:44:49 -04:00
Jack Robison dc2f22f5fa
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 821be29f41
rename effective_amount prefix 2021-10-05 16:44:49 -04:00
Jack Robison 52ff1a12ff
fix undeleted claim_to_channel record 2021-10-05 16:44:49 -04:00
Jack Robison 814699ef11
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 0c30838b25
fix mismatch in claim_to_txo<->txo_to_claim 2021-10-05 16:44:49 -04:00
Jack Robison cf66c2a1ee
rename things
-fix effective amount integrity error
2021-10-05 16:44:49 -04:00
Jack Robison 2ee419ffca
fix 2021-10-05 16:44:49 -04:00
Jack Robison bfb9d696d7
pretty print 2021-10-05 16:44:49 -04:00
Jack Robison bb2a34dd6b
fix duplicate activate 2021-10-05 16:44:49 -04:00
Jack Robison ed652c0c56
fix updating resolve by effective amount after abandoning support 2021-10-05 16:44:49 -04:00
Jack Robison 1dc961d6eb
use RevertableOpStack in _get_takeover_ops 2021-10-05 16:44:49 -04:00
Jack Robison d119fcfc98
remove debug prints 2021-10-05 16:44:49 -04:00
Jack Robison 4d3573724a
add RevertableOpStack to verify consistency of ops as they're staged 2021-10-05 16:44:49 -04:00
Jack Robison 8b37a66075
fix fee amount overflow in es 2021-10-05 16:44:49 -04:00
Jack Robison ba4f32075a
faster claim producer
-make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits
2021-10-05 16:44:49 -04:00
Jack Robison 218be22576
imports 2021-10-05 16:44:49 -04:00
Jack Robison 7688293716
close db in sync script 2021-10-05 16:44:49 -04:00
Jack Robison 458f8533c4
try default block size 2021-10-05 16:44:49 -04:00
Jack Robison 34502752fc
update elastic sync 2021-10-05 16:44:49 -04:00
Jack Robison d6758fd823
invalidate channel signatures upon channel abandon 2021-10-05 16:44:49 -04:00
Jack Robison 65700e790e
_prepare_claim_for_sync generators 2021-10-05 16:44:49 -04:00
Jack Robison 7c34e4bb96
logging 2021-10-05 16:44:49 -04:00
Jack Robison d0d6e3563b
use default sync=False during write_batch 2021-10-05 16:44:49 -04:00
Jack Robison a2619f8c78
genesis_bytes attribute 2021-10-05 16:44:49 -04:00
Jack Robison 42d07fd2f0
fix 2021-10-05 16:44:49 -04:00
Jack Robison 8bea10960f
disable es (revert) 2021-10-05 16:44:49 -04:00
Jack Robison 9cbb19c304
_cached_get_active_amount 2021-10-05 16:44:49 -04:00
Jack Robison 1b94dfd712
fix removing unactivated support 2021-10-05 16:44:49 -04:00
Jack Robison 9f3604d739
debug 2021-10-05 16:44:49 -04:00
Jack Robison 4a1b2be269
leveldb tuning 2021-10-05 16:44:49 -04:00
Jack Robison 962dc1b55b
debug 2021-10-05 16:44:49 -04:00
Jack Robison 07c86502f6
refactor ClaimToTXO prefix 2021-10-05 16:44:49 -04:00
Jack Robison adb188e5d0
filter abandoned claims from those considered for early activation 2021-10-05 16:44:49 -04:00
Jack Robison ce031dc6b8
only do early takeover on a larger amount (fix case where they're equal) 2021-10-05 16:44:49 -04:00
Jack Robison 18b5f03247
filter supported claim hashes for claims that dont exist from early takeover/activations 2021-10-05 16:44:49 -04:00
Jack Robison 8a555ecf1c
remove extra open functions 2021-10-05 16:44:49 -04:00
Jack Robison 1b325b9acd
fix flush id 2021-10-05 16:44:49 -04:00
Jack Robison 1bdaddb319
fix clearing pending_support caches upon abandon 2021-10-05 16:44:49 -04:00
Jack Robison 7896e177ef
fix putting spent unactivated supports in removed_active_support 2021-10-05 16:44:49 -04:00
Jack Robison ce8e659008
fix syncing claim to es where channel is in the same block 2021-10-05 16:44:49 -04:00
Jack Robison 27be5deeb2
ignore activation for headless supports 2021-10-05 16:44:49 -04:00
Jack Robison 515f270c3a
faster get_future_activated 2021-10-05 16:44:49 -04:00
Jack Robison ffff3bd334
debugging 2021-10-05 16:44:49 -04:00
Jack Robison f493f13b25
prints 2021-10-05 16:44:49 -04:00
Jack Robison e605c14b13
flush count 2021-10-05 16:44:49 -04:00
Jack Robison 338488f16d
tests 2021-10-05 16:44:49 -04:00
Jack Robison 2abc67c3e8
reposts 2021-10-05 16:44:49 -04:00
Jack Robison eb1ba143ec
fix updating the ES search index
-update search index to use ResolveResult tuples
2021-10-05 16:44:49 -04:00
Jack Robison 6f5bca0f67
bid ordered resolve, feed ES claim data from block processor 2021-10-05 16:44:49 -04:00
Jack Robison 407cd8dd4b
fix duplicate update op for early activating claim 2021-10-05 16:44:49 -04:00
Jack Robison 62a4f0fc04
fix early takeovers by not-yet activated claims 2021-10-05 16:44:49 -04:00
Jack Robison 3eb9d23108
require previous_winning arg for get_takeover_name_ops 2021-10-05 16:44:49 -04:00
Jack Robison 410d4aeb21
fix takeover edge case
if a claim with a higher value than that of a claim taking over a name exists but isn't yet activated, activate it early and have it take over the name
2021-10-05 16:44:49 -04:00
Jack Robison 0a28d216fd
comments 2021-10-05 16:44:49 -04:00
Jack Robison b69faf6920
bid ordered resolve (WIP) 2021-10-05 16:44:49 -04:00
Jack Robison e77f9981df
DBError 2021-10-05 16:44:49 -04:00
Jack Robison d27c2cc1e9
remove unused COIN file 2021-10-05 16:44:49 -04:00
Jack Robison 586b19675e
claim takeovers 2021-10-05 16:44:49 -04:00
Jack Robison f2907536b4
move get_expiration_height and claimtrie constants to Coin class 2021-10-05 16:44:49 -04:00
Jack Robison 9a11ac06bf
claim activations and takeovers (WIP) 2021-10-05 16:44:49 -04:00
Jack Robison aa3b18f848
advance_blocks -> advance_block 2021-10-05 16:44:49 -04:00
Jack Robison 103bdc151f
dead code 2021-10-05 16:44:49 -04:00
Jack Robison 6d4c1cd879
LBRYBlockProcessor -> BlockProcessor
- temporarily disable claim_search
2021-10-05 16:44:49 -04:00
Jack Robison cacbe30871
rebase 2021-10-05 16:44:49 -04:00
Jack Robison bfeeacb230
tests 2021-10-05 16:44:49 -04:00
Jack Robison 04bb7b4919
add wrapper for getnamesintrie
-used for verifying db state against lbrycrd
2021-10-05 16:44:49 -04:00
Jack Robison b7df277a5c
db state struct
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison c681041b48
claim expiration 2021-10-05 16:44:49 -04:00