568 commits

Author	SHA1	Message	Date
Victor Shyba	ec89bcac8e	improve sync script for no-downtime maintenance	2021-03-19 19:58:13 -03:00
Victor Shyba	a916c1f4ad	check if db file exists before sync	2021-03-19 19:58:13 -03:00
Victor Shyba	a9a0ac92d7	ignore unset flag	2021-03-19 19:58:13 -03:00
Victor Shyba	da8a8bd1ef	filter+fts and tests for edge cases	2021-03-19 19:58:13 -03:00
Victor Shyba	67817005b5	check ES synced without a process and wait for ES	2021-03-19 19:58:13 -03:00
Jack Robison	24d11de5a7	torba-elastic-sync	2021-03-19 19:58:13 -03:00
Victor Shyba	e12fab90d1	docker compose update	2021-03-19 19:58:13 -03:00
Victor Shyba	0a194b5b01	claim_ids query	2021-03-19 19:58:13 -03:00
Victor Shyba	8d028adc53	be a writer by default	2021-03-19 19:58:13 -03:00
Victor Shyba	dfca15395e	claim id is also a keyword	2021-03-19 19:58:13 -03:00
Victor Shyba	e21f2362fe	apply reorg deletion as well	2021-03-19 19:58:13 -03:00
Victor Shyba	038a5f999f	cache encoded headers	2021-03-19 19:58:13 -03:00
Victor Shyba	5d3704c7ea	reader mode	2021-03-19 19:58:13 -03:00
Victor Shyba	87037c06c9	remove reader code	2021-03-19 19:58:13 -03:00
Victor Shyba	dd412c0f50	delete sqlite fts	2021-03-19 19:58:13 -03:00
Victor Shyba	bf44befff6	backport fixes from server	2021-03-19 19:58:13 -03:00
Victor Shyba	e61874bb6f	only repeat search if it has blocked items	2021-03-19 19:58:13 -03:00
Victor Shyba	ec9a3a4f7c	do not page filtered	2021-03-19 19:58:13 -03:00
Victor Shyba	e439a3a8dc	advanced resolve	2021-03-19 19:58:13 -03:00
Victor Shyba	19f70d7a11	create changelog trigger	2021-03-19 19:58:13 -03:00
Victor Shyba	afe7ed5b05	adjust size	2021-03-19 19:58:13 -03:00
Victor Shyba	d4bf004d74	use a thread pool to sync changes	2021-03-19 19:58:13 -03:00
Victor Shyba	e4d06a088b	include the channel being filtered/blocked	2021-03-19 19:58:13 -03:00
Victor Shyba	0929088b12	missing refresh step	2021-03-19 19:58:13 -03:00
Victor Shyba	7b4838fc9b	dont update more than 400 items a time	2021-03-19 19:58:13 -03:00
Victor Shyba	0cf9533248	narrow update by query	2021-03-19 19:58:13 -03:00
Victor Shyba	84ff0b8a9f	general timeout	2021-03-19 19:58:13 -03:00
Victor Shyba	d467dcfeaf	increase sync queue	2021-03-19 19:58:13 -03:00
Victor Shyba	8e68ba4751	fix join, refresh before update	2021-03-19 19:58:13 -03:00
Victor Shyba	0f2a85ba9f	simplify sync	2021-03-19 19:58:13 -03:00
Victor Shyba	7674a0a91e	backport fixes from testing server	2021-03-19 19:58:13 -03:00
Victor Shyba	5bc1a66572	32 slices and add censor type to fields	2021-03-19 19:58:13 -03:00
Victor Shyba	9b56067213	raise request timeout for content filtering	2021-03-19 19:58:13 -03:00
Victor Shyba	9a9df2fc3c	apply filtering only to whats unfiltered	2021-03-19 19:58:13 -03:00
Victor Shyba	9989d8d1d4	refresh after delete	2021-03-19 19:58:13 -03:00
Victor Shyba	f9471f297e	apply filter and block from ES script lang	2021-03-19 19:58:13 -03:00
Victor Shyba	146b693e4a	exclude title and description	2021-03-19 19:58:13 -03:00
Victor Shyba	e2441ea3e7	use prefix from ES docs	2021-03-19 19:58:13 -03:00
Victor Shyba	119e51912e	fix partial id	2021-03-19 19:58:13 -03:00
Victor Shyba	dd950f5b0d	tag can have empty space	2021-03-19 19:58:13 -03:00
Victor Shyba	78a9bad1e1	no indexer_task	2021-03-19 19:58:13 -03:00
Victor Shyba	0c6eaf5484	fix resolve partial id	2021-03-19 19:58:13 -03:00
Victor Shyba	1010068ddb	disable refresh interval. start with 3 shards	2021-03-19 19:58:13 -03:00
Victor Shyba	ee7b37d3f3	also normalize the name supplied by user	2021-03-19 19:58:13 -03:00
Victor Shyba	143d82d242	normalized, not normalized_name	2021-03-19 19:58:13 -03:00
Victor Shyba	8b91b38855	update winners in one go	2021-03-19 19:58:13 -03:00
Victor Shyba	1098f0d2a3	use normalized name instead	2021-03-19 19:58:13 -03:00
Victor Shyba	ab53cec022	fix is_controlling sync	2021-03-19 19:58:13 -03:00
Victor Shyba	90106f5f08	all test_claim_commands tests green	2021-03-19 19:58:13 -03:00
Victor Shyba	9924b7b438	reposts and tag inheritance	2021-03-19 19:58:13 -03:00
Victor Shyba	aa37faab0a	use porter analyzer with weights on full text search	2021-03-19 19:58:13 -03:00
Victor Shyba	dc10f8ce72	ignore errors when deleting	2021-03-19 19:58:13 -03:00
Victor Shyba	996686c1da	claim search and resolve translated to ES queries	2021-03-19 19:58:13 -03:00
Victor Shyba	488785d013	add indexer task	2021-03-19 19:58:13 -03:00
Victor Shyba	3abdc01230	index ES during sync	2021-03-19 19:58:13 -03:00
Victor Shyba	74df4fab83	change column to has_source and document both flags	2021-03-14 10:11:42 -04:00
Victor Shyba	1e5cd3d7a1	typo, fix tests	2021-03-14 10:11:42 -04:00
Victor Shyba	a54e9b64aa	add no_source claim_search filter	2021-03-14 10:11:42 -04:00
Lex Berezhny	4343073c00	clients can connect to wallet server even when they are not reachable by UDP	2021-03-12 13:29:55 -05:00
Victor Shyba	ae337807f5	get merkles outside thread cooperatively	2021-03-10 13:05:17 -03:00
Victor Shyba	9ae30ac08e	during subscribe, hash address only when its time	2021-03-10 12:51:58 -03:00
Lex Berezhny	b153e4bb9f	added support to claim_search for filtering collections via --claim_type	2021-02-16 11:08:54 -05:00
Jack Robison	0f5f1aebed	accept connections from ledger clients in spv server	2021-01-21 21:37:39 -05:00
Jack Robison	f0d8fb8f1a	add UDP based ping protocol for spv servers	2021-01-21 16:08:33 -05:00
Jack Robison	f7a380e9b7	start prometheus before block processing	2021-01-21 16:01:33 -05:00
Jack Robison	e9c7cf6f63	logging	2021-01-21 14:51:59 -05:00
Jack Robison	68f1661452	add LRUCache with no prometheus metrics	2021-01-21 14:37:08 -05:00
Jack Robison	36fd1b91ae	cache server features	2021-01-21 14:37:08 -05:00
Jack Robison	a4ec430ac0	improve subscription performance	2021-01-21 14:37:08 -05:00
Jack Robison	0758b85179	skip null args given to channel_ids in claim search	2020-12-29 21:41:07 -05:00
Jack Robison	ab3d9bd080	block cache metrics	2020-12-29 21:40:39 -05:00
Jack Robison	9dd5159414	increase server cache sizes	2020-12-23 20:47:05 -05:00
Jack Robison	13e38d6fd8	use LRUCache instead of pylru in wallet server	2020-12-23 20:47:05 -05:00
Jack Robison	e3a4964787	catch any remaining index errors in fs_tx_hash	2020-12-16 01:28:30 -05:00
Jack Robison	9bf72910a4	fix off by one error in reorg	2020-12-16 01:27:03 -05:00
Jack Robison	ee39e20e6d	logging	2020-12-16 01:26:19 -05:00
Jack Robison	0821ce44b5	remove unused heavy call	2020-12-15 19:22:12 -05:00
Jack Robison	674ce02e58	logging	2020-12-14 14:38:36 -05:00
Jack Robison	8dfa2767ec	new_touched	2020-12-14 13:52:26 -05:00
Jack Robison	20dad7f07f	only notify hashxs touched since last notification	2020-12-14 13:42:20 -05:00
Jack Robison	751cc4c44d	don't deserialize mempool in a thread	2020-12-13 13:48:50 -05:00
Jack Robison	2318e6d8e9	faster fs_transactions	2020-12-13 13:48:02 -05:00
Jack Robison	8da73ad3dd	improve hash_to_hex_str performance	2020-12-08 15:39:03 -05:00
Jack Robison	b8c16d8ac5	fix mempool txs being re-requested with each address notification	2020-12-08 15:39:03 -05:00
Jack Robison	eae0290978	fix mempool notification bloat	2020-12-07 13:16:37 -05:00
Jack Robison	eab3b65629	calculate notifications for all subscriptions instead of per session	2020-12-03 19:15:15 -05:00
Jack Robison	9b4afe9816	store txids in memory, faster address subscription/history	2020-11-25 16:30:11 -05:00
Jack Robison	af1d7813e9	ensure transactions are returned in the order they're requested	2020-11-24 11:52:07 -05:00
Jack Robison	c02750edbd	fix tx_merkle	2020-11-23 16:48:24 -05:00
Jack Robison	faeba9a7e4	faster blockchain.transaction.get_batch	2020-11-23 12:05:47 -05:00
Jack Robison	190d238a1f	faster read_headers	2020-11-23 12:03:11 -05:00
Jack Robison	923d817751	store headers in memory	2020-11-20 22:37:42 -05:00
Jack Robison	fa63bf758d	delete single_call_context, use session pool	2020-11-20 15:52:11 -05:00
Jack Robison	2c7fd58e34	threaded compress headers	2020-11-20 13:09:54 -05:00
Jack Robison	f2fd42b47a	use write_batch	2020-11-19 18:37:49 -05:00
Jack Robison	1b4ccad938	allow settings TRENDING_ALGORITHMS to nothing	2020-11-19 18:36:34 -05:00
Jack Robison	a9de1ce8e0	fix fs_transactions	2020-11-19 12:40:46 -05:00
Jack Robison	632d8d02d2	remove dead code	2020-11-17 16:17:24 -05:00
Jack Robison	48aeb26e02	threaded_get_merkle ... -remove dead code	2020-11-17 16:05:08 -05:00
Jack Robison	1694af8b5e	get transactions from leveldb instead of lbrycrd rpc	2020-11-17 15:30:48 -05:00
Jack Robison	83bcab9cd2	cleanup	2020-11-17 15:30:48 -05:00
Jack Robison	bdc7f4b3f5	combine tx_count_db and hashes_db, add tx db	2020-11-17 15:30:48 -05:00
Jack Robison	bbed9b94c1	null language filtering for claim search	2020-11-05 21:23:49 -05:00
Brendon J. Brewer	1cdff47477	Rewrite of variable_decay.py for speed improvements	2020-11-05 21:15:55 -05:00
Jack Robison	925a458abe	tags	2020-11-03 10:39:08 -05:00
Jack Robison	46dc15dd29	use language indexes	2020-09-28 14:45:51 -04:00
Jack Robison	8dc654b513	add language table and indexes	2020-09-28 14:45:51 -04:00
Jack Robison	7000ac3f3f	update indexed tags	2020-09-28 14:45:51 -04:00
Lex Berezhny	ec24ebf2cf	added --limit_claims_per_channel argument to claim_search to only return up to the specified number of claims per channel	2020-08-19 10:51:31 -04:00
Jack Robison	7a1b7db7c8	support multiple blocks with blockchain.transaction.get_batch	2020-07-14 23:13:14 -04:00
Jack Robison	03a643da52	use block cache	2020-07-14 23:13:14 -04:00
Jack Robison	ac1a8b4daf	metric for time to send notifications	2020-07-02 18:57:36 -04:00
Jack Robison	28838c1759	notifications_in_flight_metric	2020-07-02 18:57:36 -04:00
Jack Robison	50ecb0dac9	fix notify	2020-07-02 18:57:36 -04:00
Jack Robison	e22bc01cbd	re-add wakeup event, add address history metric	2020-07-02 18:57:36 -04:00
Jack Robison	6c28713a4c	read history in one loop	2020-07-02 18:57:36 -04:00
Jack Robison	fc9023386c	non-blocking history lookup in notify	2020-07-02 18:57:36 -04:00
Jack Robison	e6cae9bcc3	remove mempool wakeup event, lower refresh delay	2020-07-02 18:57:36 -04:00
Jack Robison	a9eeca1302	mempool processing time metric	2020-07-02 18:57:36 -04:00
Jack Robison	0aa7fd47d5	combine loops	2020-07-02 18:57:36 -04:00
Jack Robison	70596042d6	mempool_process_time_metric	2020-07-02 18:57:36 -04:00
Jack Robison	caf616234b	flush databases during sync	2020-07-02 18:57:36 -04:00
Jack Robison	375187aa70	tx hashes db	2020-07-02 18:57:36 -04:00
Jack Robison	71eccdc0e3	hashes path	2020-07-02 18:57:36 -04:00
Jack Robison	639b1e48f5	blocks dir	2020-07-02 18:57:36 -04:00
Jack Robison	0bb4cb4472	tx count db	2020-07-02 18:57:36 -04:00
Jack Robison	cc51543851	headers db	2020-07-02 18:57:36 -04:00
Jack Robison	98565eb67c	run read_raw_block in executor	2020-07-02 18:57:36 -04:00
Jack Robison	9d44bbdb48	don't block the notification loop on sending the notifications	2020-06-04 09:25:41 -04:00
Jack Robison	8d93dd5adc	improve wallet server address history cache	2020-06-04 09:15:21 -04:00
Jack Robison	4bbd850898	fix uncaught ValueError in hashX_unsubscribe	2020-05-25 10:25:04 -04:00
Jack Robison	34eae6e608	fix wallet server prometheus bucket sizes	2020-05-25 10:24:31 -04:00
Jack Robison	797364ee5c	refactor prometheus metrics	2020-05-02 15:01:07 -04:00
Jack Robison	36c05fc4b9	move wallet server prometheus ... -only run wallet server metrics for the wallet server	2020-05-01 12:54:35 -04:00
Tom	49458d1085	fix: reposts being returned for single tags	2020-04-22 23:16:30 -04:00
Jack Robison	7ffdfd12f8	faster not tags	2020-04-17 10:47:01 -04:00
Jack Robison	cb9a30f285	faster query	2020-04-17 10:47:01 -04:00
Jack Robison	5b29894048	add reset clients counter to prometheus	2020-04-06 12:03:33 -04:00
Jack Robison	87cdf1e3a0	improve test_reorg_change_claim_height	2020-04-05 16:58:36 -04:00
Jack Robison	640b5b0ea9	delete_claims_above_height with thread lock	2020-04-03 13:39:44 -04:00
Jack Robison	f7065c6f0c	add reorg count metric to prometheus	2020-04-03 13:39:44 -04:00
Jack Robison	64f7f837e7	delete claims above reorg height from the database	2020-04-03 13:39:44 -04:00
Jack Robison	44b2964a6a	add blockchain.transaction.get_batch	2020-03-19 19:20:30 -04:00
Jack Robison	8fddb57e0a	reduce block lru cache sizes	2020-03-19 19:20:30 -04:00
Jack Robison	c7a5a0cab0	add blockchain.transaction.info	2020-03-19 19:20:30 -04:00
Brendon J. Brewer	5f043b9a78	variable decay	2020-03-12 10:33:15 +13:00
Lex Berezhny	05228529b0	fix tests	2020-03-03 20:26:07 -05:00
Lex Berezhny	3cbeadfbc3	lbrynet status command returns features available of currenty connected wallet server, including thetrending algorithm used	2020-03-03 20:04:57 -05:00
Jack Robison	dc6194f862	perf counter	2020-03-01 15:17:21 -05:00
Jack Robison	de50214e1f	block lru cache	2020-03-01 15:17:21 -05:00
Jack Robison	5ec4a88c35	block hash lru cache	2020-03-01 15:17:05 -05:00
Jack Robison	9d5760d899	fix test	2020-03-01 14:41:14 -05:00
Jack Robison	7945e1ea3c	mempool threadpool	2020-03-01 12:40:13 -05:00
Jack Robison	d94c40e371	leveldb threadpool	2020-03-01 12:40:13 -05:00
Jack Robison	31f22122e8	block processor threadpool	2020-03-01 12:40:13 -05:00
Lex Berezhny	33679b56fd	update claim_type_* indexes to be more peformant	2020-02-27 12:07:38 -05:00
Lex Berezhny	8c7908b200	reposts inherit the tags of the original claim	2020-02-22 12:23:11 -05:00
Lex Berezhny	d57900a069	claim_search --not_channel_ids excludes the channel itself and not just claims in the channel	2020-02-21 21:59:46 -05:00
Jack Robison	2f1d08e417	add MAX_RECEIVE wallet server setting	2020-02-20 14:11:16 -05:00
Victor Shyba	ae9ba14b59	use both donation and payment addresses separately	2020-02-18 18:10:58 -03:00
Victor Shyba	ad6c6fbe35	fixes from review and add analytics	2020-02-18 18:10:58 -03:00
Victor Shyba	d317a4042c	change daily_fee to be lbc instead of dewies	2020-02-18 18:10:58 -03:00
Victor Shyba	c2d717aba5	old code from #2683	2020-02-18 18:10:58 -03:00
Lex Berezhny	0b2d9e15b4	added order bys for fee_amount	2020-02-17 17:24:48 -05:00
Lex Berezhny	d14e5e75e8	all three order bys for duration filter	2020-02-17 17:24:48 -05:00
Lex Berezhny	df11ef34d2	index cleanup	2020-02-17 17:24:48 -05:00
Lex Berezhny	4fc619262c	replacing indexes on trending_global, trending_mixed to be trending_group, trending_mixed	2020-02-17 17:24:48 -05:00
Lex Berezhny	71cb03345b	removed default order_by for claim_search	2020-02-17 16:26:29 -05:00
Jack Robison	9b46d03c91	fix race between setting items in and clearing the resolve/claim_search cache	2020-02-13 11:17:21 -05:00
Jack Robison	78eef25f1a	add reposted_claim_hash index ... fixes https://github.com/lbryio/lbry-sdk/issues/2782	2020-02-12 17:39:57 -05:00
Lex Berezhny	605fc8ecd8	claim_search by --claim_type=repost support	2020-02-12 11:51:35 -05:00
Lex Berezhny	be4515ec9a	use sets for __in constraint values	2020-02-12 10:39:25 -05:00
Brendon J. Brewer	3697d9e1f0	Tweaked some parameters, back to 134	2020-02-09 17:38:34 -05:00
Brendon J. Brewer	94524f7330	lint	2020-02-09 17:38:34 -05:00
Brendon J. Brewer	17d8a3e5d6	Use claim_hash not claim_id	2020-02-09 17:38:24 -05:00
Brendon J. Brewer	39996d7612	Only create file during install if logging is turned on	2020-02-09 17:37:55 -05:00
Lex Berezhny	9607d21828	censored searches/resolves include metadata of channel which did the censoring	2020-02-07 18:50:29 -05:00
Jack Robison	bf4cbe1204	add version to session count metric	2020-02-04 12:18:32 -05:00
Jack Robison	28eee4da25	add version metrics to prometheus	2020-02-04 10:52:08 -05:00
Jack Robison	933ccf6deb	add block updates to prometheus	2020-02-04 10:26:22 -05:00
Jack Robison	9ac41322e5	add cpu count to prometheus	2020-02-04 10:25:40 -05:00
Lex Berezhny	1e81d6f848	added database index for various channel_hash based queries	2020-02-03 20:28:39 -05:00
Jack Robison	dd1de530c3	catch TimeoutError when attempting to send notifications	2020-02-03 18:46:17 -05:00
Lex Berezhny	bfe711bd42	fixing tests	2020-02-01 13:29:55 -05:00
Lex Berezhny	15abf49211	censored resolve responses return appropriate error	2020-02-01 12:53:39 -05:00
Brendon J. Brewer	22bca7a16e	Length zero could also just mean there aren't any claims	2020-01-31 07:34:43 +13:00
jessop	73c958222b	add duration to claim search params	2020-01-29 15:46:47 -05:00
Lex Berezhny	790bbfcc99	integrate ar trending algorithm into configurable trending system	2020-01-28 18:11:32 -05:00
Brendon J. Brewer	994e70d43a	ar trending algorithm	2020-01-28 18:11:32 -05:00
Jack Robison	11530c675d	initialize_request_handlers, remove local rpc handlers	2020-01-28 15:54:22 -05:00
Jack Robison	1c474352fe	add wallet server prometheus metrics ... -sessions gauge -requests counter -response times histogram -notification counter -request errors counter -interrupt counter -operational error counter -internal error counter -reader executor times histogram -pending queries gauge -lbrycrd request counter -client versions counter	2020-01-28 15:54:22 -05:00
Jack Robison	3b2d635390	Revert "Revert "configurable trending algorithms"" ... This reverts commit 55e08f96a6.	2020-01-28 15:53:57 -05:00
Jack Robison	55e08f96a6	Revert "configurable trending algorithms"	2020-01-28 15:03:09 -05:00
Lex Berezhny	01ed71b243	configurable trending	2020-01-28 11:58:27 -05:00
Lex Berezhny	595c058517	moved trending.py to trending/zscore.py	2020-01-28 11:58:27 -05:00
Alex Grintsvayg	e04b0a6995	add docker tag to prometheus version info	2020-01-27 14:16:18 -05:00
Alex Grintsvayg	0732ab6ad5	add min_version and method name to metrics	2020-01-23 13:13:20 -05:00
Alex Grintsvayg	7fd56e0add	build_type.py -> build_info.py	2020-01-23 13:13:19 -05:00
Lex Berezhny	8d8a5b36b6	Merge pull request #2737 from lbryio/fix_repost_resolve_bug ... fix bug where repost claims were not being fully resolved	2020-01-22 23:32:55 -05:00
Lex Berezhny	a70cc7beb1	fix bug where repost claims were not being fully resolved	2020-01-22 22:59:05 -05:00
Alex Grintsvayg	d9f809864f	include version and build info in prometheus output	2020-01-22 19:13:58 -05:00
Lex Berezhny	bb23f509d7	rename BLOCKING_CHANNELS_IDS/FILTERING_CHANNELS_IDS to remove double S	2020-01-22 11:02:01 -05:00
Alex Grintsvayg	6c07141abd	Add prometheus metrics collection to client and server	2020-01-22 10:28:17 -05:00
Lex Berezhny	2cd7ea257c	Added support to differentiate between filtering and blocking for content censoring	2020-01-22 08:39:14 -05:00
Alex Grintsvayg	1299c9162c	Revert "Add prometheus metrics collection to client and server" ... This reverts commit 59a5bacb2e.	2020-01-21 10:30:49 -05:00
Alex Grintsvayg	8d28bb5a66	bump min protocol version	2020-01-20 12:53:10 -05:00
Lex Berezhny	9d79c52d20	add index for claim_type and release_time	2020-01-20 12:50:20 -05:00
Lex Berezhny	801f05f45e	fix json api generator	2020-01-20 12:50:16 -05:00
Lex Berezhny	345196aa3b	add pagination for claim_search	2020-01-20 12:49:42 -05:00
Lex Berezhny	86cedfe8b2	using multiprocessing.Manager to keep blocked content synced between readers	2020-01-20 12:49:41 -05:00
Alex Grintsvayg	59a5bacb2e	Add prometheus metrics collection to client and server	2020-01-20 12:43:49 -05:00
Jack Robison	b34f44c867	logging	2020-01-11 00:17:32 -05:00
Jack Robison	38b108752e	batched blockchain.address.subscribe	2020-01-09 23:40:04 -05:00
Victor Shyba	e73c081d36	remove bandwidth_limit env option	2020-01-07 19:57:34 -05:00
Victor Shyba	60194cbafd	remove bw_limit on server	2020-01-07 19:57:34 -05:00
Lex Berezhny	78732513c8	set wallet server min/max 0.52.0-0.99.0	2020-01-06 12:17:09 -05:00
Lex Berezhny	fb1af9e3d2	update imports and more merging	2020-01-03 03:08:15 -05:00
Lex Berezhny	2968f74c6c	moved lbry up one level	2020-01-01 15:57:56 -05:00