Commit graph

527 commits

Author SHA1 Message Date
Victor Shyba f9471f297e apply filter and block from ES script lang 2021-03-19 19:58:13 -03:00
Victor Shyba 146b693e4a exclude title and description 2021-03-19 19:58:13 -03:00
Victor Shyba e2441ea3e7 use prefix from ES docs 2021-03-19 19:58:13 -03:00
Victor Shyba 119e51912e fix partial id 2021-03-19 19:58:13 -03:00
Victor Shyba dd950f5b0d tag can have empty space 2021-03-19 19:58:13 -03:00
Victor Shyba 78a9bad1e1 no indexer_task 2021-03-19 19:58:13 -03:00
Victor Shyba 0c6eaf5484 fix resolve partial id 2021-03-19 19:58:13 -03:00
Victor Shyba 1010068ddb disable refresh interval. start with 3 shards 2021-03-19 19:58:13 -03:00
Victor Shyba ee7b37d3f3 also normalize the name supplied by user 2021-03-19 19:58:13 -03:00
Victor Shyba 143d82d242 normalized, not normalized_name 2021-03-19 19:58:13 -03:00
Victor Shyba 8b91b38855 update winners in one go 2021-03-19 19:58:13 -03:00
Victor Shyba 1098f0d2a3 use normalized name instead 2021-03-19 19:58:13 -03:00
Victor Shyba ab53cec022 fix is_controlling sync 2021-03-19 19:58:13 -03:00
Victor Shyba 90106f5f08 all test_claim_commands tests green 2021-03-19 19:58:13 -03:00
Victor Shyba 9924b7b438 reposts and tag inheritance 2021-03-19 19:58:13 -03:00
Victor Shyba aa37faab0a use porter analyzer with weights on full text search 2021-03-19 19:58:13 -03:00
Victor Shyba dc10f8ce72 ignore errors when deleting 2021-03-19 19:58:13 -03:00
Victor Shyba 996686c1da claim search and resolve translated to ES queries 2021-03-19 19:58:13 -03:00
Victor Shyba 488785d013 add indexer task 2021-03-19 19:58:13 -03:00
Victor Shyba 3abdc01230 index ES during sync 2021-03-19 19:58:13 -03:00
Victor Shyba 74df4fab83 change column to has_source and document both flags 2021-03-14 10:11:42 -04:00
Victor Shyba 1e5cd3d7a1 typo, fix tests 2021-03-14 10:11:42 -04:00
Victor Shyba a54e9b64aa add no_source claim_search filter 2021-03-14 10:11:42 -04:00
Lex Berezhny 4343073c00 clients can connect to wallet server even when they are not reachable by UDP 2021-03-12 13:29:55 -05:00
Victor Shyba ae337807f5 get merkles outside thread cooperatively 2021-03-10 13:05:17 -03:00
Victor Shyba 9ae30ac08e during subscribe, hash address only when its time 2021-03-10 12:51:58 -03:00
zeppi 183fb9f9ff provide --resolve tag for collection claim, separate from resolving its contents
bugfix and docs generation

review changes
2021-03-04 00:03:16 -05:00
Lex Berezhny 9815ddef1f fixes stalling client reconnect issue 2021-03-03 23:31:59 -05:00
Lex Berezhny 0cec80f676 fixes transaction signing bug when tx had no change outputs 2021-03-01 11:20:08 -05:00
Lex Berezhny b153e4bb9f added support to claim_search for filtering collections via --claim_type 2021-02-16 11:08:54 -05:00
Lex Berezhny 7486ee9537 do not tx._reset() in generate_channel_private_key 2021-02-08 09:56:03 -05:00
Lex Berezhny f80dd2b307 tx._reset() the transaction after internal changes 2021-02-08 09:56:03 -05:00
Lex Berezhny b208cf6d32 significant performance improvement when creating transactions and with txo spend specifically 2021-02-08 09:56:03 -05:00
Lex Berezhny ccb250b410 txo_list --resolve now works for signed supports by resolving the signing channel 2021-01-26 20:53:36 -05:00
Lex Berezhny eedcc2034d fixes #3166 2021-01-25 10:56:48 -05:00
Jack Robison 0f5f1aebed
accept connections from ledger clients in spv server 2021-01-21 21:37:39 -05:00
Jack Robison 1f9d0f4582
pylint 2021-01-21 16:30:54 -05:00
Jack Robison 20efdc70b3
use UDP ping for wallet server selection
-only connect to one spv server at a time
-remove session pool
2021-01-21 16:15:42 -05:00
Jack Robison f0d8fb8f1a
add UDP based ping protocol for spv servers 2021-01-21 16:08:33 -05:00
Jack Robison f7a380e9b7
start prometheus before block processing 2021-01-21 16:01:33 -05:00
Jack Robison e9c7cf6f63
logging 2021-01-21 14:51:59 -05:00
Jack Robison 68f1661452
add LRUCache with no prometheus metrics 2021-01-21 14:37:08 -05:00
Jack Robison 36fd1b91ae
cache server features 2021-01-21 14:37:08 -05:00
Jack Robison a4ec430ac0
improve subscription performance 2021-01-21 14:37:08 -05:00
Lex Berezhny bf0118c8ef added support for --not_channel_id to txo_list commands 2021-01-21 14:03:59 -05:00
Victor Shyba b882f1a010 log checkpoint finishing for gathering information on wal/shm file bug 2020-12-31 16:11:22 -03:00
Jack Robison 0758b85179
skip null args given to channel_ids in claim search 2020-12-29 21:41:07 -05:00
Jack Robison ab3d9bd080
block cache metrics 2020-12-29 21:40:39 -05:00
Jack Robison 9dd5159414
increase server cache sizes 2020-12-23 20:47:05 -05:00
Jack Robison 13e38d6fd8
use LRUCache instead of pylru in wallet server 2020-12-23 20:47:05 -05:00
Jack Robison 10dcb64715
lru cache metrics 2020-12-23 20:47:03 -05:00
Victor Shyba adb418aafc group cache hits in a single dict 2020-12-23 21:37:20 -03:00
Victor Shyba 270da80d64 return whole batches 2020-12-23 19:10:19 -03:00
Victor Shyba b2027cfd66 sync in batches of 10, clearing after 2020-12-23 18:30:52 -03:00
Jack Robison 7a7446c8bd
force resync blockchain.db 2020-12-22 20:08:26 -05:00
Jack Robison 8c71b744f3
fix request_synced_transactions edge cases 2020-12-22 15:19:08 -05:00
Jack Robison 4cbf4230e8
fix txi.txo_ref 2020-12-22 14:05:37 -05:00
Victor Shyba 6a610187e0 cache bypass 2020-12-22 13:01:30 -03:00
Victor Shyba eb2a4aebba unrestricted and reusing verified cache hits 2020-12-21 16:49:08 -03:00
Victor Shyba 21a2e67755 fix rebase 2020-12-20 21:39:36 -03:00
Victor Shyba 3b9e312615 fix verification 2020-12-20 21:25:26 -03:00
Victor Shyba 26dab04c9e checkpoint 2020-12-20 21:25:26 -03:00
Victor Shyba 00713c0d11 asyncgens 2020-12-20 21:25:26 -03:00
Victor Shyba 751b5f3027 refactor duplicate code 2020-12-20 21:25:26 -03:00
Victor Shyba e8261b000e wip, see jack 2020-12-20 21:23:58 -03:00
Victor Shyba 41ecb70297 join network can only happen after initial header sync returns 2020-12-20 21:15:11 -03:00
Jack Robison e3a4964787
catch any remaining index errors in fs_tx_hash 2020-12-16 01:28:30 -05:00
Jack Robison 9bf72910a4
fix off by one error in reorg 2020-12-16 01:27:03 -05:00
Jack Robison ee39e20e6d
logging 2020-12-16 01:26:19 -05:00
Jack Robison 0821ce44b5
remove unused heavy call 2020-12-15 19:22:12 -05:00
Jack Robison 674ce02e58
logging 2020-12-14 14:38:36 -05:00
Jack Robison 8dfa2767ec
new_touched 2020-12-14 13:52:26 -05:00
Jack Robison 20dad7f07f
only notify hashxs touched since last notification 2020-12-14 13:42:20 -05:00
Jack Robison 751cc4c44d
don't deserialize mempool in a thread 2020-12-13 13:48:50 -05:00
Jack Robison 2318e6d8e9
faster fs_transactions 2020-12-13 13:48:02 -05:00
Jack Robison 8da73ad3dd
improve hash_to_hex_str performance 2020-12-08 15:39:03 -05:00
Jack Robison b8c16d8ac5
fix mempool txs being re-requested with each address notification 2020-12-08 15:39:03 -05:00
Jack Robison eae0290978
fix mempool notification bloat 2020-12-07 13:16:37 -05:00
Jack Robison 3467d1fed0
fix tests 2020-12-03 22:16:28 -05:00
Jack Robison d02ff232e5
_single_batch 2020-12-03 20:50:25 -05:00
Jack Robison 2d1c6a5402
fix 2020-12-03 19:15:15 -05:00
Jack Robison eab3b65629
calculate notifications for all subscriptions instead of per session 2020-12-03 19:15:15 -05:00
Jack Robison 929617273d
resync blockchain.db 2020-12-03 15:56:35 -05:00
Jack Robison 2717bf7d49
increase default tx_cache_size setting 2020-12-03 15:56:35 -05:00
Jack Robison 9b4afe9816
store txids in memory, faster address subscription/history 2020-11-25 16:30:11 -05:00
Jack Robison af1d7813e9
ensure transactions are returned in the order they're requested 2020-11-24 11:52:07 -05:00
Jack Robison c02750edbd
fix tx_merkle 2020-11-23 16:48:24 -05:00
Jack Robison faeba9a7e4
faster blockchain.transaction.get_batch 2020-11-23 12:05:47 -05:00
Jack Robison 190d238a1f
faster read_headers 2020-11-23 12:03:11 -05:00
Jack Robison 923d817751
store headers in memory 2020-11-20 22:37:42 -05:00
Victor Shyba b8b9dcc2ee WAL_CHECKPOINT on close 2020-11-20 20:16:03 -03:00
Jack Robison 34f4f12eb9
pylint 2020-11-20 16:15:57 -05:00
Jack Robison fa63bf758d
delete single_call_context, use session pool 2020-11-20 15:52:11 -05:00
Jack Robison 2c7fd58e34
threaded compress headers 2020-11-20 13:09:54 -05:00
Jack Robison f2fd42b47a
use write_batch 2020-11-19 18:37:49 -05:00
Jack Robison 1b4ccad938
allow settings TRENDING_ALGORITHMS to nothing 2020-11-19 18:36:34 -05:00
Jack Robison a9de1ce8e0
fix fs_transactions 2020-11-19 12:40:46 -05:00
Jack Robison 632d8d02d2
remove dead code 2020-11-17 16:17:24 -05:00
Jack Robison 48aeb26e02
threaded_get_merkle
-remove dead code
2020-11-17 16:05:08 -05:00
Jack Robison 1694af8b5e
get transactions from leveldb instead of lbrycrd rpc 2020-11-17 15:30:48 -05:00
Jack Robison 83bcab9cd2
cleanup 2020-11-17 15:30:48 -05:00
Jack Robison bdc7f4b3f5
combine tx_count_db and hashes_db, add tx db 2020-11-17 15:30:48 -05:00
Victor Shyba 2619e162c1 recover invalid state described on #3026 2020-11-09 16:03:38 -05:00
Jack Robison bbed9b94c1
null language filtering for claim search 2020-11-05 21:23:49 -05:00
Brendon J. Brewer 1cdff47477
Rewrite of variable_decay.py for speed improvements 2020-11-05 21:15:55 -05:00
Jack Robison d83936a66a
fix uncaught error 2020-11-04 22:21:35 -05:00
Jack Robison 5517d2bf56
fix new_sdk_server arg for resolve 2020-11-03 16:23:31 -05:00
Jack Robison 925a458abe
tags 2020-11-03 10:39:08 -05:00
Jack Robison 76946c447f
use single_call_context for claim_search and resolve 2020-11-03 10:39:08 -05:00
Jack Robison ab067d1d3a fix unclean closing of blockchain.db 2020-10-26 16:23:28 -04:00
Alex Grintsvayg 0a0ac3b7c9 pass-through for new support_sum api 2020-10-16 12:03:01 -04:00
Lex Berezhny ec2f6c6b80 lint 2020-09-29 07:52:33 -04:00
Akinwale Ariwodola 838cc60161 conditional import 2020-09-29 07:52:33 -04:00
Akinwale Ariwodola 310c61a5cc check KIVY_BUILD env variable 2020-09-29 07:52:33 -04:00
Jack Robison 46dc15dd29
use language indexes 2020-09-28 14:45:51 -04:00
Jack Robison 8dc654b513
add language table and indexes 2020-09-28 14:45:51 -04:00
Jack Robison 7000ac3f3f
update indexed tags 2020-09-28 14:45:51 -04:00
Lex Berezhny 7b666efcf8 lbrycrd download path 2020-09-24 20:11:40 -04:00
Lex Berezhny ec24ebf2cf added --limit_claims_per_channel argument to claim_search to only return up to the specified number of claims per channel 2020-08-19 10:51:31 -04:00
Lex Berezhny 9f1ed6e8c3 release_all_outputs on startup 2020-08-04 15:03:58 -04:00
Lex Berezhny 9c5f940b00 claim search forwarding to new sdk 2020-08-04 12:11:02 -04:00
Lex Berezhny 455b4043b8 new resolve 2020-08-04 11:33:39 -04:00
Lex Berezhny 8996aafe0d always cast modified_on to int() 2020-07-21 19:14:57 -04:00
Lex Berezhny ba649d4b94 convert modified_on timestamp to int to avoid floating point conflicts when syncing 2020-07-20 14:24:53 -04:00
Lex Berezhny 2f81e9d374 pylint 2020-07-14 23:13:14 -04:00
Jack Robison 2d8703bb8d pylint 2020-07-14 23:13:14 -04:00
Jack Robison 76e60d9bc3 logging 2020-07-14 23:13:14 -04:00
Jack Robison 9d5370be5f fix 2020-07-14 23:13:14 -04:00
Jack Robison fc1a06bc45 fix 2020-07-14 23:13:14 -04:00
Jack Robison fce80374f4 batched sync 2020-07-14 23:13:14 -04:00
Jack Robison 420c9f10c2 remove _update_cache_item 2020-07-14 23:13:14 -04:00
Jack Robison 5a39681a2e log 2020-07-14 23:13:14 -04:00
Jack Robison 7a1b7db7c8 support multiple blocks with blockchain.transaction.get_batch 2020-07-14 23:13:14 -04:00
Jack Robison 03a643da52 use block cache 2020-07-14 23:13:14 -04:00
Jack Robison ac1a8b4daf
metric for time to send notifications 2020-07-02 18:57:36 -04:00
Jack Robison 28838c1759
notifications_in_flight_metric 2020-07-02 18:57:36 -04:00
Jack Robison 50ecb0dac9
fix notify 2020-07-02 18:57:36 -04:00
Jack Robison e22bc01cbd
re-add wakeup event, add address history metric 2020-07-02 18:57:36 -04:00
Jack Robison 6c28713a4c
read history in one loop 2020-07-02 18:57:36 -04:00
Jack Robison fc9023386c
non-blocking history lookup in notify 2020-07-02 18:57:36 -04:00
Jack Robison e6cae9bcc3
remove mempool wakeup event, lower refresh delay 2020-07-02 18:57:36 -04:00
Jack Robison a9eeca1302
mempool processing time metric 2020-07-02 18:57:36 -04:00
Jack Robison 8c695e42ca
fix sqlite coin chooser floor 2020-07-02 18:57:36 -04:00
Jack Robison 0aa7fd47d5
combine loops 2020-07-02 18:57:36 -04:00
Jack Robison 70596042d6
mempool_process_time_metric 2020-07-02 18:57:36 -04:00
Jack Robison caf616234b
flush databases during sync 2020-07-02 18:57:36 -04:00
Jack Robison 375187aa70
tx hashes db 2020-07-02 18:57:36 -04:00
Jack Robison 71eccdc0e3
hashes path 2020-07-02 18:57:36 -04:00
Jack Robison 639b1e48f5
blocks dir 2020-07-02 18:57:36 -04:00
Jack Robison 0bb4cb4472
tx count db 2020-07-02 18:57:36 -04:00
Jack Robison cc51543851
headers db 2020-07-02 18:57:36 -04:00
Jack Robison 98565eb67c
run read_raw_block in executor 2020-07-02 18:57:36 -04:00
Jack Robison fa60b9f9d3
logging 2020-07-02 17:35:38 -04:00
Jack Robison 872b89ee93
fix mempool conflicts following cancelled api calls that send transactions with the blocking flag 2020-06-05 20:06:42 -04:00
Jack Robison ae53062518
integration test for sqlite coin chooser 2020-06-05 20:06:42 -04:00
Jack Robison 17f76c9cb3
leave inputs for rejected tx reserved 2020-06-05 20:06:42 -04:00
Jack Robison 5de944146a
logging 2020-06-05 20:06:42 -04:00
Jack Robison a32a2ef04e
add sqlite coin chooser 2020-06-05 20:06:42 -04:00
Lex Berezhny 9c426373f2 lint 2020-06-05 15:49:18 -04:00
Lex Berezhny c03e30a01f added support for signed supports 2020-06-05 15:49:18 -04:00
Lex Berezhny 07f7a77ac0
Merge pull request #2971 from lbryio/faster-publish
Speed up publish response time by not blocking on updating the content claim db
2020-06-05 15:47:40 -04:00
Jack Robison 1beb13dd80
fix attribute error 2020-06-04 09:31:37 -04:00
Jack Robison ddae84abb3
fix pending_count 2020-06-04 09:31:37 -04:00
Jack Robison 9d44bbdb48
don't block the notification loop on sending the notifications 2020-06-04 09:25:41 -04:00
Jack Robison 8d93dd5adc
improve wallet server address history cache 2020-06-04 09:15:21 -04:00
Jack Robison 4bbd850898
fix uncaught ValueError in hashX_unsubscribe 2020-05-25 10:25:04 -04:00
Jack Robison 34eae6e608
fix wallet server prometheus bucket sizes 2020-05-25 10:24:31 -04:00
Jack Robison cae7792a1e
add transaction_cache_size to config 2020-05-25 10:16:18 -04:00
Lex Berezhny 590c892a6a re-set channel on txo 2020-05-18 12:27:22 -04:00
Lex Berezhny ff8a50c366 fixed bug with leaky information between outputs 2020-05-18 11:16:01 -04:00
Victor Shyba a0fea30a11
make wait check every second instead of once 2020-05-12 19:44:02 -04:00
Jack Robison c22482f907
channel private key generation in a thread pool 2020-05-11 14:54:31 -04:00
Jack Robison 750ff448ad
comments 2020-05-04 13:47:37 -04:00
Jack Robison e3abab6d4d
pylint 2020-05-04 12:15:48 -04:00
Jack Robison d3ffae72fb
buckets 2020-05-02 22:30:25 -04:00
Jack Robison 3469abaefd
write lock metrics 2020-05-02 21:23:17 -04:00
Jack Robison 797364ee5c
refactor prometheus metrics 2020-05-02 15:01:07 -04:00
Jack Robison 36c05fc4b9
move wallet server prometheus
-only run wallet server metrics for the wallet server
2020-05-01 12:54:35 -04:00
Jack Robison 79624febc0
prevent pileup of writes blocking reads 2020-05-01 12:48:41 -04:00
Jack Robison 0a9d4de126
include write lock in try/finally 2020-05-01 11:40:57 -04:00
Jack Robison ff7bed720a
don't close the connection upon a cancelled request 2020-04-29 12:32:31 -04:00
Victor Shyba 58f77b2a1c load/dump header file using executor 2020-04-27 10:34:49 -04:00
Victor Shyba 7170e69b22 test fixes + leave tx plot always on estimations 2020-04-27 10:34:49 -04:00
Victor Shyba 239ee2437c estimate only whats not downloaded 2020-04-27 10:34:49 -04:00
Victor Shyba ced368db31 hold headers file in memory during runtime 2020-04-27 10:34:49 -04:00
Tom 49458d1085 fix: reposts being returned for single tags 2020-04-22 23:16:30 -04:00
Jack Robison 21c112d059
lbrycrd url 2020-04-20 12:16:31 -04:00
Jack Robison 7ffdfd12f8
faster not tags 2020-04-17 10:47:01 -04:00
Jack Robison cb9a30f285
faster query 2020-04-17 10:47:01 -04:00
Lex Berezhny f5d7570102 fix issue with --exclude_internal_transfers where it was filtering out sent payments 2020-04-16 17:55:49 -04:00
Jack Robison d615f6761a
automatically batch large resolve requests 2020-04-06 12:03:33 -04:00
Jack Robison 5b29894048
add reset clients counter to prometheus 2020-04-06 12:03:33 -04:00
Victor Shyba b2f70c7120 return none for unconfirmed time estimation 2020-04-06 06:03:35 -03:00
Jack Robison 87cdf1e3a0
improve test_reorg_change_claim_height 2020-04-05 16:58:36 -04:00
Jack Robison 57fd47022e
test_reorg_change_claim_height 2020-04-03 13:39:44 -04:00
Jack Robison 5eafd3bf6b
feedback 2020-04-03 13:39:44 -04:00
Jack Robison 640b5b0ea9
delete_claims_above_height with thread lock 2020-04-03 13:39:44 -04:00
Jack Robison e4fb2f4680
test_reorg_dropping_claim 2020-04-03 13:39:44 -04:00
Jack Robison f7065c6f0c
add reorg count metric to prometheus 2020-04-03 13:39:44 -04:00
Jack Robison 64f7f837e7
delete claims above reorg height from the database 2020-04-03 13:39:44 -04:00