Commit graph

495 commits

Author SHA1 Message Date
Victor Shyba
0929088b12 missing refresh step 2021-03-19 19:58:13 -03:00
Victor Shyba
7b4838fc9b dont update more than 400 items a time 2021-03-19 19:58:13 -03:00
Victor Shyba
0cf9533248 narrow update by query 2021-03-19 19:58:13 -03:00
Victor Shyba
84ff0b8a9f general timeout 2021-03-19 19:58:13 -03:00
Victor Shyba
d467dcfeaf increase sync queue 2021-03-19 19:58:13 -03:00
Victor Shyba
8e68ba4751 fix join, refresh before update 2021-03-19 19:58:13 -03:00
Victor Shyba
0f2a85ba9f simplify sync 2021-03-19 19:58:13 -03:00
Victor Shyba
7674a0a91e backport fixes from testing server 2021-03-19 19:58:13 -03:00
Victor Shyba
5bc1a66572 32 slices and add censor type to fields 2021-03-19 19:58:13 -03:00
Victor Shyba
9b56067213 raise request timeout for content filtering 2021-03-19 19:58:13 -03:00
Victor Shyba
9a9df2fc3c apply filtering only to whats unfiltered 2021-03-19 19:58:13 -03:00
Victor Shyba
9989d8d1d4 refresh after delete 2021-03-19 19:58:13 -03:00
Victor Shyba
f9471f297e apply filter and block from ES script lang 2021-03-19 19:58:13 -03:00
Victor Shyba
146b693e4a exclude title and description 2021-03-19 19:58:13 -03:00
Victor Shyba
e2441ea3e7 use prefix from ES docs 2021-03-19 19:58:13 -03:00
Victor Shyba
119e51912e fix partial id 2021-03-19 19:58:13 -03:00
Victor Shyba
dd950f5b0d tag can have empty space 2021-03-19 19:58:13 -03:00
Victor Shyba
78a9bad1e1 no indexer_task 2021-03-19 19:58:13 -03:00
Victor Shyba
0c6eaf5484 fix resolve partial id 2021-03-19 19:58:13 -03:00
Victor Shyba
1010068ddb disable refresh interval. start with 3 shards 2021-03-19 19:58:13 -03:00
Victor Shyba
ee7b37d3f3 also normalize the name supplied by user 2021-03-19 19:58:13 -03:00
Victor Shyba
143d82d242 normalized, not normalized_name 2021-03-19 19:58:13 -03:00
Victor Shyba
8b91b38855 update winners in one go 2021-03-19 19:58:13 -03:00
Victor Shyba
1098f0d2a3 use normalized name instead 2021-03-19 19:58:13 -03:00
Victor Shyba
ab53cec022 fix is_controlling sync 2021-03-19 19:58:13 -03:00
Victor Shyba
90106f5f08 all test_claim_commands tests green 2021-03-19 19:58:13 -03:00
Victor Shyba
9924b7b438 reposts and tag inheritance 2021-03-19 19:58:13 -03:00
Victor Shyba
aa37faab0a use porter analyzer with weights on full text search 2021-03-19 19:58:13 -03:00
Victor Shyba
dc10f8ce72 ignore errors when deleting 2021-03-19 19:58:13 -03:00
Victor Shyba
996686c1da claim search and resolve translated to ES queries 2021-03-19 19:58:13 -03:00
Victor Shyba
488785d013 add indexer task 2021-03-19 19:58:13 -03:00
Victor Shyba
3abdc01230 index ES during sync 2021-03-19 19:58:13 -03:00
Victor Shyba
74df4fab83 change column to has_source and document both flags 2021-03-14 10:11:42 -04:00
Victor Shyba
1e5cd3d7a1 typo, fix tests 2021-03-14 10:11:42 -04:00
Victor Shyba
a54e9b64aa add no_source claim_search filter 2021-03-14 10:11:42 -04:00
Lex Berezhny
4343073c00 clients can connect to wallet server even when they are not reachable by UDP 2021-03-12 13:29:55 -05:00
Victor Shyba
ae337807f5 get merkles outside thread cooperatively 2021-03-10 13:05:17 -03:00
Victor Shyba
9ae30ac08e during subscribe, hash address only when its time 2021-03-10 12:51:58 -03:00
Lex Berezhny
b153e4bb9f added support to claim_search for filtering collections via --claim_type 2021-02-16 11:08:54 -05:00
Jack Robison
0f5f1aebed
accept connections from ledger clients in spv server 2021-01-21 21:37:39 -05:00
Jack Robison
f0d8fb8f1a
add UDP based ping protocol for spv servers 2021-01-21 16:08:33 -05:00
Jack Robison
f7a380e9b7
start prometheus before block processing 2021-01-21 16:01:33 -05:00
Jack Robison
e9c7cf6f63
logging 2021-01-21 14:51:59 -05:00
Jack Robison
68f1661452
add LRUCache with no prometheus metrics 2021-01-21 14:37:08 -05:00
Jack Robison
36fd1b91ae
cache server features 2021-01-21 14:37:08 -05:00
Jack Robison
a4ec430ac0
improve subscription performance 2021-01-21 14:37:08 -05:00
Jack Robison
0758b85179
skip null args given to channel_ids in claim search 2020-12-29 21:41:07 -05:00
Jack Robison
ab3d9bd080
block cache metrics 2020-12-29 21:40:39 -05:00
Jack Robison
9dd5159414
increase server cache sizes 2020-12-23 20:47:05 -05:00
Jack Robison
13e38d6fd8
use LRUCache instead of pylru in wallet server 2020-12-23 20:47:05 -05:00
Jack Robison
e3a4964787
catch any remaining index errors in fs_tx_hash 2020-12-16 01:28:30 -05:00
Jack Robison
9bf72910a4
fix off by one error in reorg 2020-12-16 01:27:03 -05:00
Jack Robison
ee39e20e6d
logging 2020-12-16 01:26:19 -05:00
Jack Robison
0821ce44b5
remove unused heavy call 2020-12-15 19:22:12 -05:00
Jack Robison
674ce02e58
logging 2020-12-14 14:38:36 -05:00
Jack Robison
8dfa2767ec
new_touched 2020-12-14 13:52:26 -05:00
Jack Robison
20dad7f07f
only notify hashxs touched since last notification 2020-12-14 13:42:20 -05:00
Jack Robison
751cc4c44d
don't deserialize mempool in a thread 2020-12-13 13:48:50 -05:00
Jack Robison
2318e6d8e9
faster fs_transactions 2020-12-13 13:48:02 -05:00
Jack Robison
8da73ad3dd
improve hash_to_hex_str performance 2020-12-08 15:39:03 -05:00
Jack Robison
b8c16d8ac5
fix mempool txs being re-requested with each address notification 2020-12-08 15:39:03 -05:00
Jack Robison
eae0290978
fix mempool notification bloat 2020-12-07 13:16:37 -05:00
Jack Robison
eab3b65629
calculate notifications for all subscriptions instead of per session 2020-12-03 19:15:15 -05:00
Jack Robison
9b4afe9816
store txids in memory, faster address subscription/history 2020-11-25 16:30:11 -05:00
Jack Robison
af1d7813e9
ensure transactions are returned in the order they're requested 2020-11-24 11:52:07 -05:00
Jack Robison
c02750edbd
fix tx_merkle 2020-11-23 16:48:24 -05:00
Jack Robison
faeba9a7e4
faster blockchain.transaction.get_batch 2020-11-23 12:05:47 -05:00
Jack Robison
190d238a1f
faster read_headers 2020-11-23 12:03:11 -05:00
Jack Robison
923d817751
store headers in memory 2020-11-20 22:37:42 -05:00
Jack Robison
fa63bf758d
delete single_call_context, use session pool 2020-11-20 15:52:11 -05:00
Jack Robison
2c7fd58e34
threaded compress headers 2020-11-20 13:09:54 -05:00
Jack Robison
f2fd42b47a
use write_batch 2020-11-19 18:37:49 -05:00
Jack Robison
1b4ccad938
allow settings TRENDING_ALGORITHMS to nothing 2020-11-19 18:36:34 -05:00
Jack Robison
a9de1ce8e0
fix fs_transactions 2020-11-19 12:40:46 -05:00
Jack Robison
632d8d02d2
remove dead code 2020-11-17 16:17:24 -05:00
Jack Robison
48aeb26e02
threaded_get_merkle
-remove dead code
2020-11-17 16:05:08 -05:00
Jack Robison
1694af8b5e
get transactions from leveldb instead of lbrycrd rpc 2020-11-17 15:30:48 -05:00
Jack Robison
83bcab9cd2
cleanup 2020-11-17 15:30:48 -05:00
Jack Robison
bdc7f4b3f5
combine tx_count_db and hashes_db, add tx db 2020-11-17 15:30:48 -05:00
Jack Robison
bbed9b94c1
null language filtering for claim search 2020-11-05 21:23:49 -05:00
Brendon J. Brewer
1cdff47477
Rewrite of variable_decay.py for speed improvements 2020-11-05 21:15:55 -05:00
Jack Robison
925a458abe
tags 2020-11-03 10:39:08 -05:00
Jack Robison
46dc15dd29
use language indexes 2020-09-28 14:45:51 -04:00
Jack Robison
8dc654b513
add language table and indexes 2020-09-28 14:45:51 -04:00
Jack Robison
7000ac3f3f
update indexed tags 2020-09-28 14:45:51 -04:00
Lex Berezhny
ec24ebf2cf added --limit_claims_per_channel argument to claim_search to only return up to the specified number of claims per channel 2020-08-19 10:51:31 -04:00
Jack Robison
7a1b7db7c8 support multiple blocks with blockchain.transaction.get_batch 2020-07-14 23:13:14 -04:00
Jack Robison
03a643da52 use block cache 2020-07-14 23:13:14 -04:00
Jack Robison
ac1a8b4daf
metric for time to send notifications 2020-07-02 18:57:36 -04:00
Jack Robison
28838c1759
notifications_in_flight_metric 2020-07-02 18:57:36 -04:00
Jack Robison
50ecb0dac9
fix notify 2020-07-02 18:57:36 -04:00
Jack Robison
e22bc01cbd
re-add wakeup event, add address history metric 2020-07-02 18:57:36 -04:00
Jack Robison
6c28713a4c
read history in one loop 2020-07-02 18:57:36 -04:00
Jack Robison
fc9023386c
non-blocking history lookup in notify 2020-07-02 18:57:36 -04:00
Jack Robison
e6cae9bcc3
remove mempool wakeup event, lower refresh delay 2020-07-02 18:57:36 -04:00
Jack Robison
a9eeca1302
mempool processing time metric 2020-07-02 18:57:36 -04:00
Jack Robison
0aa7fd47d5
combine loops 2020-07-02 18:57:36 -04:00
Jack Robison
70596042d6
mempool_process_time_metric 2020-07-02 18:57:36 -04:00
Jack Robison
caf616234b
flush databases during sync 2020-07-02 18:57:36 -04:00
Jack Robison
375187aa70
tx hashes db 2020-07-02 18:57:36 -04:00
Jack Robison
71eccdc0e3
hashes path 2020-07-02 18:57:36 -04:00
Jack Robison
639b1e48f5
blocks dir 2020-07-02 18:57:36 -04:00
Jack Robison
0bb4cb4472
tx count db 2020-07-02 18:57:36 -04:00
Jack Robison
cc51543851
headers db 2020-07-02 18:57:36 -04:00
Jack Robison
98565eb67c
run read_raw_block in executor 2020-07-02 18:57:36 -04:00
Jack Robison
9d44bbdb48
don't block the notification loop on sending the notifications 2020-06-04 09:25:41 -04:00
Jack Robison
8d93dd5adc
improve wallet server address history cache 2020-06-04 09:15:21 -04:00
Jack Robison
4bbd850898
fix uncaught ValueError in hashX_unsubscribe 2020-05-25 10:25:04 -04:00
Jack Robison
34eae6e608
fix wallet server prometheus bucket sizes 2020-05-25 10:24:31 -04:00
Jack Robison
797364ee5c
refactor prometheus metrics 2020-05-02 15:01:07 -04:00
Jack Robison
36c05fc4b9
move wallet server prometheus
-only run wallet server metrics for the wallet server
2020-05-01 12:54:35 -04:00
Tom
49458d1085 fix: reposts being returned for single tags 2020-04-22 23:16:30 -04:00
Jack Robison
7ffdfd12f8
faster not tags 2020-04-17 10:47:01 -04:00
Jack Robison
cb9a30f285
faster query 2020-04-17 10:47:01 -04:00
Jack Robison
5b29894048
add reset clients counter to prometheus 2020-04-06 12:03:33 -04:00
Jack Robison
87cdf1e3a0
improve test_reorg_change_claim_height 2020-04-05 16:58:36 -04:00
Jack Robison
640b5b0ea9
delete_claims_above_height with thread lock 2020-04-03 13:39:44 -04:00
Jack Robison
f7065c6f0c
add reorg count metric to prometheus 2020-04-03 13:39:44 -04:00
Jack Robison
64f7f837e7
delete claims above reorg height from the database 2020-04-03 13:39:44 -04:00
Jack Robison
44b2964a6a
add blockchain.transaction.get_batch 2020-03-19 19:20:30 -04:00
Jack Robison
8fddb57e0a
reduce block lru cache sizes 2020-03-19 19:20:30 -04:00
Jack Robison
c7a5a0cab0
add blockchain.transaction.info 2020-03-19 19:20:30 -04:00
Brendon J. Brewer
5f043b9a78 variable decay 2020-03-12 10:33:15 +13:00
Lex Berezhny
05228529b0 fix tests 2020-03-03 20:26:07 -05:00
Lex Berezhny
3cbeadfbc3 lbrynet status command returns features available of currenty connected wallet server, including thetrending algorithm used 2020-03-03 20:04:57 -05:00
Jack Robison
dc6194f862
perf counter 2020-03-01 15:17:21 -05:00
Jack Robison
de50214e1f
block lru cache 2020-03-01 15:17:21 -05:00
Jack Robison
5ec4a88c35
block hash lru cache 2020-03-01 15:17:05 -05:00
Jack Robison
9d5760d899
fix test 2020-03-01 14:41:14 -05:00
Jack Robison
7945e1ea3c
mempool threadpool 2020-03-01 12:40:13 -05:00
Jack Robison
d94c40e371
leveldb threadpool 2020-03-01 12:40:13 -05:00
Jack Robison
31f22122e8
block processor threadpool 2020-03-01 12:40:13 -05:00
Lex Berezhny
33679b56fd update claim_type_* indexes to be more peformant 2020-02-27 12:07:38 -05:00
Lex Berezhny
8c7908b200 reposts inherit the tags of the original claim 2020-02-22 12:23:11 -05:00
Lex Berezhny
d57900a069 claim_search --not_channel_ids excludes the channel itself and not just claims in the channel 2020-02-21 21:59:46 -05:00
Jack Robison
2f1d08e417
add MAX_RECEIVE wallet server setting 2020-02-20 14:11:16 -05:00
Victor Shyba
ae9ba14b59 use both donation and payment addresses separately 2020-02-18 18:10:58 -03:00
Victor Shyba
ad6c6fbe35 fixes from review and add analytics 2020-02-18 18:10:58 -03:00
Victor Shyba
d317a4042c change daily_fee to be lbc instead of dewies 2020-02-18 18:10:58 -03:00
Victor Shyba
c2d717aba5 old code from #2683 2020-02-18 18:10:58 -03:00
Lex Berezhny
0b2d9e15b4 added order bys for fee_amount 2020-02-17 17:24:48 -05:00
Lex Berezhny
d14e5e75e8 all three order bys for duration filter 2020-02-17 17:24:48 -05:00
Lex Berezhny
df11ef34d2 index cleanup 2020-02-17 17:24:48 -05:00
Lex Berezhny
4fc619262c replacing indexes on trending_global, trending_mixed to be trending_group, trending_mixed 2020-02-17 17:24:48 -05:00
Lex Berezhny
71cb03345b removed default order_by for claim_search 2020-02-17 16:26:29 -05:00
Jack Robison
9b46d03c91
fix race between setting items in and clearing the resolve/claim_search cache 2020-02-13 11:17:21 -05:00
Jack Robison
78eef25f1a
add reposted_claim_hash index
fixes https://github.com/lbryio/lbry-sdk/issues/2782
2020-02-12 17:39:57 -05:00
Lex Berezhny
605fc8ecd8 claim_search by --claim_type=repost support 2020-02-12 11:51:35 -05:00
Lex Berezhny
be4515ec9a use sets for __in constraint values 2020-02-12 10:39:25 -05:00
Brendon J. Brewer
3697d9e1f0 Tweaked some parameters, back to 134 2020-02-09 17:38:34 -05:00
Brendon J. Brewer
94524f7330 lint 2020-02-09 17:38:34 -05:00
Brendon J. Brewer
17d8a3e5d6 Use claim_hash not claim_id 2020-02-09 17:38:24 -05:00
Brendon J. Brewer
39996d7612 Only create file during install if logging is turned on 2020-02-09 17:37:55 -05:00
Lex Berezhny
9607d21828 censored searches/resolves include metadata of channel which did the censoring 2020-02-07 18:50:29 -05:00
Jack Robison
bf4cbe1204
add version to session count metric 2020-02-04 12:18:32 -05:00
Jack Robison
28eee4da25
add version metrics to prometheus 2020-02-04 10:52:08 -05:00
Jack Robison
933ccf6deb
add block updates to prometheus 2020-02-04 10:26:22 -05:00
Jack Robison
9ac41322e5
add cpu count to prometheus 2020-02-04 10:25:40 -05:00
Lex Berezhny
1e81d6f848 added database index for various channel_hash based queries 2020-02-03 20:28:39 -05:00
Jack Robison
dd1de530c3
catch TimeoutError when attempting to send notifications 2020-02-03 18:46:17 -05:00
Lex Berezhny
bfe711bd42 fixing tests 2020-02-01 13:29:55 -05:00
Lex Berezhny
15abf49211 censored resolve responses return appropriate error 2020-02-01 12:53:39 -05:00
Brendon J. Brewer
22bca7a16e Length zero could also just mean there aren't any claims 2020-01-31 07:34:43 +13:00
jessop
73c958222b add duration to claim search params 2020-01-29 15:46:47 -05:00
Lex Berezhny
790bbfcc99 integrate ar trending algorithm into configurable trending system 2020-01-28 18:11:32 -05:00
Brendon J. Brewer
994e70d43a ar trending algorithm 2020-01-28 18:11:32 -05:00
Jack Robison
11530c675d
initialize_request_handlers, remove local rpc handlers 2020-01-28 15:54:22 -05:00
Jack Robison
1c474352fe
add wallet server prometheus metrics
-sessions gauge
-requests counter
-response times histogram
-notification counter
-request errors counter
-interrupt counter
-operational error counter
-internal error counter
-reader executor times histogram
-pending queries gauge
-lbrycrd request counter
-client versions counter
2020-01-28 15:54:22 -05:00
Jack Robison
3b2d635390
Revert "Revert "configurable trending algorithms""
This reverts commit 55e08f96a6.
2020-01-28 15:53:57 -05:00
Jack Robison
55e08f96a6
Revert "configurable trending algorithms" 2020-01-28 15:03:09 -05:00
Lex Berezhny
01ed71b243 configurable trending 2020-01-28 11:58:27 -05:00
Lex Berezhny
595c058517 moved trending.py to trending/zscore.py 2020-01-28 11:58:27 -05:00
Alex Grintsvayg
e04b0a6995
add docker tag to prometheus version info 2020-01-27 14:16:18 -05:00
Alex Grintsvayg
0732ab6ad5
add min_version and method name to metrics 2020-01-23 13:13:20 -05:00
Alex Grintsvayg
7fd56e0add
build_type.py -> build_info.py 2020-01-23 13:13:19 -05:00
Lex Berezhny
8d8a5b36b6
Merge pull request #2737 from lbryio/fix_repost_resolve_bug
fix bug where repost claims were not being fully resolved
2020-01-22 23:32:55 -05:00
Lex Berezhny
a70cc7beb1 fix bug where repost claims were not being fully resolved 2020-01-22 22:59:05 -05:00
Alex Grintsvayg
d9f809864f
include version and build info in prometheus output 2020-01-22 19:13:58 -05:00
Lex Berezhny
bb23f509d7 rename BLOCKING_CHANNELS_IDS/FILTERING_CHANNELS_IDS to remove double S 2020-01-22 11:02:01 -05:00
Alex Grintsvayg
6c07141abd
Add prometheus metrics collection to client and server 2020-01-22 10:28:17 -05:00
Lex Berezhny
2cd7ea257c Added support to differentiate between filtering and blocking for content censoring 2020-01-22 08:39:14 -05:00
Alex Grintsvayg
1299c9162c
Revert "Add prometheus metrics collection to client and server"
This reverts commit 59a5bacb2e.
2020-01-21 10:30:49 -05:00
Alex Grintsvayg
8d28bb5a66
bump min protocol version 2020-01-20 12:53:10 -05:00
Lex Berezhny
9d79c52d20
add index for claim_type and release_time 2020-01-20 12:50:20 -05:00
Lex Berezhny
801f05f45e
fix json api generator 2020-01-20 12:50:16 -05:00
Lex Berezhny
345196aa3b
add pagination for claim_search 2020-01-20 12:49:42 -05:00
Lex Berezhny
86cedfe8b2
using multiprocessing.Manager to keep blocked content synced between readers 2020-01-20 12:49:41 -05:00
Alex Grintsvayg
59a5bacb2e
Add prometheus metrics collection to client and server 2020-01-20 12:43:49 -05:00
Jack Robison
b34f44c867
logging 2020-01-11 00:17:32 -05:00
Jack Robison
38b108752e
batched blockchain.address.subscribe 2020-01-09 23:40:04 -05:00
Victor Shyba
e73c081d36 remove bandwidth_limit env option 2020-01-07 19:57:34 -05:00
Victor Shyba
60194cbafd remove bw_limit on server 2020-01-07 19:57:34 -05:00
Lex Berezhny
78732513c8 set wallet server min/max 0.52.0-0.99.0 2020-01-06 12:17:09 -05:00
Lex Berezhny
fb1af9e3d2 update imports and more merging 2020-01-03 03:08:15 -05:00
Lex Berezhny
2968f74c6c moved lbry up one level 2020-01-01 15:57:56 -05:00