Madiator2011/lbry-sdk
1
0
Fork 0
forked from LBRYCommunity/lbry-sdk
Code Pull requests Activity
8769 commits 111 branches 641 tags 30 MiB
Commit graph

538 commits

Author SHA1 Message Date
Jack Robison
f01b8c849d use RevertableOpStack in _get_takeover_ops 2022-03-15 15:34:07 -03:00
Jack Robison
6a46f50a35 remove debug prints 2022-03-15 15:34:07 -03:00
Jack Robison
468ed91ee3 add RevertableOpStack to verify consistency of ops as they're staged 2022-03-15 15:34:07 -03:00
Jack Robison
b856e2120a fix fee amount overflow in es 2022-03-15 15:34:07 -03:00
Jack Robison
fd7bfbea78 faster claim producer 
-make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits
 2022-03-15 15:34:07 -03:00
Jack Robison
a74434e269 imports 2022-03-15 15:34:07 -03:00
Jack Robison
92be63ff5b close db in sync script 2022-03-15 15:34:07 -03:00
Jack Robison
4ce16b6509 try default block size 2022-03-15 15:34:07 -03:00
Jack Robison
2341667d04 update elastic sync 2022-03-15 15:34:07 -03:00
Jack Robison
89e7c8582e invalidate channel signatures upon channel abandon 2022-03-15 15:34:07 -03:00
Jack Robison
da4e4ecd23 _prepare_claim_for_sync generators 2022-03-15 15:34:07 -03:00
Jack Robison
c85648d43b logging 2022-03-15 15:34:07 -03:00
Jack Robison
c90331af98 use default sync=False during write_batch 2022-03-15 15:34:07 -03:00
Jack Robison
e5461f6d4f genesis_bytes attribute 2022-03-15 15:34:07 -03:00
Jack Robison
ac82d6b27b fix 2022-03-15 15:34:07 -03:00
Jack Robison
01db974040 disable es (revert) 2022-03-15 15:34:07 -03:00
Jack Robison
6ae6cf733d _cached_get_active_amount 2022-03-15 15:34:07 -03:00
Jack Robison
ad1d25d945 fix removing unactivated support 2022-03-15 15:34:07 -03:00
Jack Robison
69ed47fc22 debug 2022-03-15 15:34:07 -03:00
Jack Robison
7c62654534 leveldb tuning 2022-03-15 15:34:07 -03:00
Jack Robison
531e6c1a61 debug 2022-03-15 15:34:07 -03:00
Jack Robison
066f797ad4 refactor ClaimToTXO prefix 2022-03-15 15:34:07 -03:00
Jack Robison
9f0611f3d9 filter abandoned claims from those considered for early activation 2022-03-15 15:34:07 -03:00
Jack Robison
b0d2efd613 only do early takeover on a larger amount (fix case where they're equal) 2022-03-15 15:34:07 -03:00
Jack Robison
073283a433 filter supported claim hashes for claims that dont exist from early takeover/activations 2022-03-15 15:34:07 -03:00
Jack Robison
5be04448ea remove extra open functions 2022-03-15 15:34:07 -03:00
Jack Robison
5541b80179 fix flush id 2022-03-15 15:34:07 -03:00
Jack Robison
35232b3650 fix clearing pending_support caches upon abandon 2022-03-15 15:34:07 -03:00
Jack Robison
f4ca3ea66b fix putting spent unactivated supports in removed_active_support 2022-03-15 15:34:07 -03:00
Jack Robison
306efa17cc fix syncing claim to es where channel is in the same block 2022-03-15 15:34:07 -03:00
Jack Robison
83107ad877 ignore activation for headless supports 2022-03-15 15:34:07 -03:00
Jack Robison
b0cf25bf5c faster get_future_activated 2022-03-15 15:34:07 -03:00
Jack Robison
6330424bcd debugging 2022-03-15 15:34:07 -03:00
Jack Robison
5f3850bfa2 prints 2022-03-15 15:34:07 -03:00
Jack Robison
82e6658483 flush count 2022-03-15 15:34:07 -03:00
Jack Robison
49f4add8d1 tests 2022-03-15 15:34:07 -03:00
Jack Robison
6ea96e79bd reposts 2022-03-15 15:34:07 -03:00
Jack Robison
9ad31008a5 fix updating the ES search index 
-update search index to use ResolveResult tuples
 2022-03-15 15:34:07 -03:00
Jack Robison
966f47a5b1 bid ordered resolve, feed ES claim data from block processor 2022-03-15 15:34:07 -03:00
Jack Robison
8711ece274 fix duplicate update op for early activating claim 2022-03-15 15:34:07 -03:00
Jack Robison
c0e2d56f55 fix early takeovers by not-yet activated claims 2022-03-15 15:34:07 -03:00
Jack Robison
2ba044ee4e require previous_winning arg for get_takeover_name_ops 2022-03-15 15:34:07 -03:00
Jack Robison
d69180da71 fix takeover edge case 
if a claim with a higher value than that of a claim taking over a name exists but isn't yet activated, activate it early and have it take over the name
 2022-03-15 15:34:07 -03:00
Jack Robison
b6e4cb9102 comments 2022-03-15 15:34:07 -03:00
Jack Robison
99d16fcb5a bid ordered resolve (WIP) 2022-03-15 15:34:07 -03:00
Jack Robison
96c318ee88 DBError 2022-03-15 15:34:07 -03:00
Jack Robison
c1ac4d8261 remove unused COIN file 2022-03-15 15:34:07 -03:00
Jack Robison
e678df86e0 claim takeovers 2022-03-15 15:34:07 -03:00
Jack Robison
6aa124592d move get_expiration_height and claimtrie constants to Coin class 2022-03-15 15:34:07 -03:00
Jack Robison
53ee3a5f80 claim activations and takeovers (WIP) 2022-03-15 15:34:07 -03:00
Jack Robison
8af410b184 advance_blocks -> advance_block 2022-03-15 15:34:07 -03:00
Jack Robison
8cb3fe8831 dead code 2022-03-15 15:34:07 -03:00
Jack Robison
2e92f3acad LBRYBlockProcessor -> BlockProcessor 
- temporarily disable claim_search
 2022-03-15 15:34:07 -03:00
Jack Robison
4e58094e4b rebase 2022-03-15 15:34:07 -03:00
Jack Robison
eb0eacd404 add wrapper for getnamesintrie 
-used for verifying db state against lbrycrd
 2022-03-15 15:34:07 -03:00
Jack Robison
1ff3ab4b83 db state struct 
-remove dead code
 2022-03-15 15:34:07 -03:00
Jack Robison
06841a4fde claim expiration 2022-03-15 15:34:07 -03:00
Jack Robison
d57cd5acd7 get_claim_by_claim_id 2022-03-15 15:34:07 -03:00
Jack Robison
b40cda78ee claims db 
-move all leveldb prefixes to DB_PREFIXES enum
-add serializable RevertableOp interface for key/value puts and deletes
-resolve urls from leveldb
 2022-03-15 15:34:07 -03:00
Jack Robison
0a833f5f83 transaction_num_mapping 2022-03-15 15:34:07 -03:00
Jack Robison
ef3bab16d3 disable sqlite in block processor 2022-03-15 15:34:07 -03:00
Jack Robison
3e826d0a5d named tuples 2022-03-15 15:34:07 -03:00
Jack Robison
bcd2c7d90b consolidate flush_backup 2022-03-15 15:34:07 -03:00
Jack Robison
eda1b0b3fc remove lbry.wallet.server.history 2022-03-15 15:34:07 -03:00
Jack Robison
bc7fe680c0 consolidate leveldb block advance/reorg 
-move methods from History to LevelDB
 2022-03-15 15:34:07 -03:00
Jack Robison
3ed748f2fd atomic flush_dbs 2022-03-15 15:34:07 -03:00
Jack Robison
9634753efd combine leveldb databases 2022-03-15 15:34:07 -03:00
Jack Robison
31df4f0bb5 Merkle staticmethods 2022-03-15 15:34:07 -03:00
Victor Shyba
c2dcc4c898 avoid [''] on peers list 2021-09-13 15:57:21 -03:00
Victor Shyba
b64aa51c0c fix stream_types being an integer 2021-09-10 18:57:20 -03:00
Victor Shyba
9e81dd2360 refactor arguments fixup 2021-09-10 18:57:20 -03:00
Victor Shyba
1c31ec66f2 simplify operator handling 2021-09-10 18:57:20 -03:00
Victor Shyba
21c71bfac1 update sync utility 2021-08-09 18:33:47 -03:00
Victor Shyba
6baaed3581 refactor query with new fields 2021-08-09 18:33:47 -03:00
Victor Shyba
152dbfd5d1 reflect fee_currency, fee_amount and duration on repost searches 2021-08-09 18:33:47 -03:00
Victor Shyba
a56d14086b reflect media_type on repost searches 2021-08-09 18:33:47 -03:00
Victor Shyba
aee87693f8 reflect stream_type on repost searches 2021-08-09 18:33:47 -03:00
Victor Shyba
e222b6ad9c log that a invalid query happened 2021-08-09 15:07:44 -03:00
Victor Shyba
19b17374e8 throttle instead of disconnecting 2021-08-09 15:07:44 -03:00
Victor Shyba
43989122bb add error type and message to error readme and update code 2021-08-09 15:07:44 -03:00
Victor Shyba
72712d6047 raise and disconnect if too many parameters are used on search 2021-08-09 15:07:44 -03:00
Victor Shyba
c591792de9 has_source is a special case 2021-07-22 16:25:55 -03:00
Victor Shyba
3108543ae5 3 missing fields 2021-07-22 16:25:55 -03:00
Victor Shyba
1eb221c743 translate reposted, signature_valid and normalized 2021-07-22 16:25:55 -03:00
Alex Grin
bebf6bc2e7 Update constants.py 2021-07-22 16:25:55 -03:00
Alex Grin
9e91cc2138 Update constants.py 2021-07-22 16:25:55 -03:00
Victor Shyba
5bd411ca27 filtering hash->id 2021-07-22 16:25:55 -03:00
Victor Shyba
a533cda6f0 ES: all _hash to _id 2021-07-22 16:25:55 -03:00
Victor Shyba
c2184fb3bf run migration on history db open 2021-07-07 18:39:38 -03:00
Victor Shyba
919c09fcb0 add migration 2021-07-07 18:39:38 -03:00
Victor Shyba
1d9dbd40ec increase flush counter to 32 bits 2021-07-07 18:39:38 -03:00
Lex Berezhny
485b958599 revert release 2021-06-22 12:50:11 -04:00
Lex Berezhny
7d49b046d4 added support to config for determining if value is set and implemented hub selection logic 2021-06-22 11:07:58 -04:00
Lex Berezhny
5f0426c840 country jurisdiction added to hub UDP protocol 2021-06-22 11:07:58 -04:00
Lex Berezhny
73e239cc5f client side hub discovery pub/sub and hub metadata stored, removed old peers implementation 2021-06-22 11:07:58 -04:00
Lex Berezhny
ad670f721a working client peer hub 2021-06-22 11:07:58 -04:00
Lex Berezhny
028a4a70cf wallet server federation, client portion 2021-06-22 11:07:58 -04:00
Lex Berezhny
e457b2f0d6 fix trending to use built-in sqlite instead of apsw 2021-06-15 18:22:42 -04:00
Lex Berezhny
c9cf7fd4d4 drop apsw in wallet.server.db.elasticsearch.sync 2021-06-15 18:22:42 -04:00
Lex Berezhny
25e16c3565 dropping apsw 2021-06-15 18:22:42 -04:00
Victor Shyba
3974df4a62 fix interaction between two modes 2021-05-27 20:14:12 -03:00
Victor Shyba
ca28de02d8 test and implementation for remove_duplicates on post-search filtering 2021-05-27 20:14:12 -03:00
Victor Shyba
bfc15ea029 handle limit being 0 and skip reordering if 0/none 2021-05-27 20:14:12 -03:00
Victor Shyba
6e8b8a5920 always call search_ahead 2021-05-27 20:14:12 -03:00
Victor Shyba
09f6637fe0 remove unused multiprocessin.Manager 2021-05-17 15:07:32 -03:00
Victor Shyba
ccadd88af5 fix cache call 2021-05-13 22:40:21 -03:00
Victor Shyba
cc02a0efc2 fix es migration bug, expand test case 2021-05-13 19:00:53 -03:00
Victor Shyba
43a1385b79 test sync helper 2021-05-13 19:00:53 -03:00
Jack Robison
eba0c9be34
fix typo 2021-05-07 16:51:19 -04:00
Jack Robison
81ebde88db
resync ES search index on version bumps 
-bump ES search index to version 1
 2021-05-07 14:36:53 -04:00
Victor Shyba
a4058b84ce clean out unused sharding 2021-05-07 15:03:37 -03:00
Victor Shyba
7bf211a52b apply reposted_claim_type on es sync 2021-05-07 15:03:37 -03:00
Victor Shyba
d5f722792f fix and test has_source for channel reposts 2021-05-07 15:03:37 -03:00
Victor Shyba
0f02906c9b fix has_source for reposted channels 2021-05-07 15:03:37 -03:00
Jack Robison
b5dfce7861
Revert "finished switch from using hash # in URLs to colon :" 
This reverts commit 888aa558
 2021-05-07 11:31:28 -04:00
Jack Robison
2ca5a65544
Revert "FindShortestID updated" 
This reverts commit 8f04a50c
 2021-05-07 11:30:31 -04:00
Jack Robison
32f7ecb261
fix invalid claim fees breaking the block processor 2021-05-06 11:18:58 -04:00
Jack Robison
21d0038ff2
add timestamps to hub log 2021-04-28 16:47:00 -04:00
Jack Robison
c094d8f2e8
add ALLOW_LAN_UDP hub setting 2021-04-28 16:47:00 -04:00
Jack Robison
c465d6a6c2
ignore udp packets with low source ports 2021-04-28 16:47:00 -04:00
Lex Berezhny
8f04a50ce1 FindShortestID updated 2021-04-28 16:21:01 -04:00
Lex Berezhny
888aa5586b finished switch from using hash # in URLs to colon : 2021-04-28 16:21:01 -04:00
Victor Shyba
f0d334d3e2 refactor from review 2021-04-28 13:28:38 -03:00
Victor Shyba
5f829b048f use separator to avoid cache key conflicts 2021-04-27 22:57:04 -03:00
Victor Shyba
1a961e66ff invalidate short_id cache on new block 2021-04-27 22:57:04 -03:00
Victor Shyba
fdb0e22656 cache search_ahead 2021-04-27 22:57:04 -03:00
Victor Shyba
cc2852cd48 new implementation for limit_claims_per_channel 2021-04-14 18:32:16 -04:00
shubhendra
601f99ac16 Remove unnecessary generator 
Signed-off-by: shubhendra <withshubh@gmail.com>
 2021-04-06 21:22:27 -04:00
shubhendra
68399ca31c Iterate dictionary directly 
Signed-off-by: shubhendra <withshubh@gmail.com>
 2021-04-06 21:22:27 -04:00
shubhendra
4725f510d8 Remove unnecessary use of comprehension 
Signed-off-by: shubhendra <withshubh@gmail.com>
 2021-04-06 21:22:27 -04:00
Victor Shyba
46dd389d0d add elasticsearch service to gitlab 2021-03-30 17:00:15 -04:00
Jack Robison
7832c62c5d
add ELASTIC_HOST and ELASTIC_PORT settings to hub 2021-03-30 12:48:13 -04:00
Victor Shyba
aa727cb9b1 show channels regardless of no_source 2021-03-30 09:47:08 -04:00
Victor Shyba
b8c9a99f20 fix no_source for reposts 2021-03-30 09:47:08 -04:00
Lex Berezhny
aff995b0d0 temporary fix for mempool sync failing during reorg 2021-03-29 16:11:03 -04:00
Victor Shyba
5235a150b1 add prog name to sync arg parser 2021-03-24 17:07:17 -03:00
Victor Shyba
7df4cc44c4 fixes from review 2021-03-24 16:30:33 -03:00
Victor Shyba
7f5d88e95c remove dead/broken/unused API 2021-03-19 19:58:13 -03:00
Victor Shyba
d09663c066 remove flush call 2021-03-19 19:58:13 -03:00
Victor Shyba
ef97c9b69f torba-server -> hub 2021-03-19 19:58:13 -03:00
Victor Shyba
d855e6c8b1 move elasticsearch things into its own module 2021-03-19 19:58:13 -03:00
Victor Shyba
cd66f7eb43 if not no_totals, use default page size 2021-03-19 19:58:13 -03:00
Victor Shyba
a3e146dc68 sort on index time 2021-03-19 19:58:13 -03:00
Victor Shyba
b81305a4a9 index and allow has_source 2021-03-19 19:58:13 -03:00
Victor Shyba
73884b34bc apply no_totals 2021-03-19 19:58:13 -03:00
Victor Shyba
6166a34db2 check cache item before locking 2021-03-19 19:58:13 -03:00
Victor Shyba
6fa7da4b1c less slices 2021-03-19 19:58:13 -03:00
Victor Shyba
c3e426c491 fix search by channel for invalid channel 2021-03-19 19:58:13 -03:00
Victor Shyba
21e023f0db fix search by channel 2021-03-19 19:58:13 -03:00
Victor Shyba
063be001b3 cache inner parsing 2021-03-19 19:58:13 -03:00
Victor Shyba
5dff02e8bc on resolve, get all claims at once 2021-03-19 19:58:13 -03:00
Victor Shyba
60a59407d8 cache the encoded output instead 2021-03-19 19:58:13 -03:00
Victor Shyba
20a5aecfca fix lib exception to asyncio TimeoutError 2021-03-19 19:58:13 -03:00
Victor Shyba
c2e7b5a67d restore some of the interrupt metrics 2021-03-19 19:58:13 -03:00
Victor Shyba
8f32303d07 apply search timeout 2021-03-19 19:58:13 -03:00
Victor Shyba
891b1e7782 track results up to 200 2021-03-19 19:58:13 -03:00
Victor Shyba
f26394fd3b report deletions on docs that doesnt exist, but dont raise 2021-03-19 19:58:13 -03:00
Victor Shyba
4d83d42b4c fix equality instead of mod 2021-03-19 19:58:13 -03:00
Victor Shyba
57f1108df2 fix query being json serializable 2021-03-19 19:58:13 -03:00
Victor Shyba
2641a9abe5 make better resolve cache 2021-03-19 19:58:13 -03:00
Victor Shyba
6b193ab350 make indexing cooperative 2021-03-19 19:58:13 -03:00
Victor Shyba
b1bb37511c use right key on cache 2021-03-19 19:58:13 -03:00
Victor Shyba
319187d6d6 log mempool task exceptions 2021-03-19 19:58:13 -03:00
Victor Shyba
02eb789f84 caching for resolve 2021-03-19 19:58:13 -03:00
Victor Shyba
eb6924277f round time to 10 minutes and fetch referenced by id 2021-03-19 19:58:13 -03:00
Victor Shyba
bd8f371fd5 bump referenced rows query limit up 2021-03-19 19:58:13 -03:00
Victor Shyba
1783ff2845 dont delete claims on reorg 2021-03-19 19:58:13 -03:00
Victor Shyba
d388527ffa log indexing errors 2021-03-19 19:58:13 -03:00
Victor Shyba
19494088bd generate from queue 2021-03-19 19:58:13 -03:00
Victor Shyba
920dad524a simplify sync and use asyncio Queue instead 2021-03-19 19:58:13 -03:00
Victor Shyba
ec89bcac8e improve sync script for no-downtime maintenance 2021-03-19 19:58:13 -03:00
Victor Shyba
a916c1f4ad check if db file exists before sync 2021-03-19 19:58:13 -03:00
Victor Shyba
a9a0ac92d7 ignore unset flag 2021-03-19 19:58:13 -03:00
Victor Shyba
da8a8bd1ef filter+fts and tests for edge cases 2021-03-19 19:58:13 -03:00
Victor Shyba
67817005b5 check ES synced without a process and wait for ES 2021-03-19 19:58:13 -03:00
Jack Robison
24d11de5a7 torba-elastic-sync 2021-03-19 19:58:13 -03:00
Victor Shyba
e12fab90d1 docker compose update 2021-03-19 19:58:13 -03:00
Victor Shyba
0a194b5b01 claim_ids query 2021-03-19 19:58:13 -03:00
Victor Shyba
8d028adc53 be a writer by default 2021-03-19 19:58:13 -03:00
Victor Shyba
dfca15395e claim id is also a keyword 2021-03-19 19:58:13 -03:00
Victor Shyba
e21f2362fe apply reorg deletion as well 2021-03-19 19:58:13 -03:00
Victor Shyba
038a5f999f cache encoded headers 2021-03-19 19:58:13 -03:00
Victor Shyba
5d3704c7ea reader mode 2021-03-19 19:58:13 -03:00
Victor Shyba
87037c06c9 remove reader code 2021-03-19 19:58:13 -03:00
Victor Shyba
dd412c0f50 delete sqlite fts 2021-03-19 19:58:13 -03:00
Victor Shyba
bf44befff6 backport fixes from server 2021-03-19 19:58:13 -03:00
Victor Shyba
e61874bb6f only repeat search if it has blocked items 2021-03-19 19:58:13 -03:00
Victor Shyba
ec9a3a4f7c do not page filtered 2021-03-19 19:58:13 -03:00
Victor Shyba
e439a3a8dc advanced resolve 2021-03-19 19:58:13 -03:00
Victor Shyba
19f70d7a11 create changelog trigger 2021-03-19 19:58:13 -03:00
Victor Shyba
afe7ed5b05 adjust size 2021-03-19 19:58:13 -03:00
Victor Shyba
d4bf004d74 use a thread pool to sync changes 2021-03-19 19:58:13 -03:00
Victor Shyba
e4d06a088b include the channel being filtered/blocked 2021-03-19 19:58:13 -03:00
Victor Shyba
0929088b12 missing refresh step 2021-03-19 19:58:13 -03:00
Victor Shyba
7b4838fc9b dont update more than 400 items a time 2021-03-19 19:58:13 -03:00
Victor Shyba
0cf9533248 narrow update by query 2021-03-19 19:58:13 -03:00
Victor Shyba
84ff0b8a9f general timeout 2021-03-19 19:58:13 -03:00
Victor Shyba
d467dcfeaf increase sync queue 2021-03-19 19:58:13 -03:00
Victor Shyba
8e68ba4751 fix join, refresh before update 2021-03-19 19:58:13 -03:00
Victor Shyba
0f2a85ba9f simplify sync 2021-03-19 19:58:13 -03:00