866 commits

Author	SHA1	Message	Date
Jack Robison	165f3bb270	refactor trending	2021-10-05 16:44:49 -04:00
Jack Robison	0ba75153f3	trending fixes	2021-10-05 16:44:49 -04:00
Jack Robison	db2789990f	make app backward compatible with trending_score ... -update trending decay function to zero out low trending score values faster	2021-10-05 16:44:49 -04:00
Jack Robison	acaf299bcb	log time to update and decay trending in elasticsearch	2021-10-05 16:44:49 -04:00
Jack Robison	1940301824	skip integrity errors for trending spikes	2021-10-05 16:44:49 -04:00
Jack Robison	34576e880d	update trending in elasticsearch ... -add TrendingPrefixSpike to leveldb -expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings	2021-10-05 16:44:49 -04:00
Brendon J. Brewer	65c0668d40	constants	2021-10-05 16:44:49 -04:00
Brendon J. Brewer	53bd2bcbfe	Put trending score into ES	2021-10-05 16:44:49 -04:00
Brendon J. Brewer	388724fccb	Mark claims as touched	2021-10-05 16:44:49 -04:00
Jack Robison	231eabb013	fix non normalized canonical urls	2021-10-05 16:44:49 -04:00
Jack Robison	54903fc2ea	handle unicode error for unnormalized names	2021-10-05 16:44:49 -04:00
Jack Robison	3a1baf0700	prefix db	2021-10-05 16:44:49 -04:00
Brendon J. Brewer	0c0e36b6f8	trending	2021-10-05 16:44:49 -04:00
Jack Robison	234c03db09	fix claims not having non-normalized names	2021-10-05 16:44:49 -04:00
Jack Robison	28aa7da349	merge conflicts	2021-10-05 16:44:49 -04:00
Jack Robison	c51e344b87	fix missing fields in reposts	2021-10-05 16:44:49 -04:00
Jack Robison	54461dfa75	fix merge conflicts and simplify extract_doc	2021-10-05 16:44:49 -04:00
Jack Robison	2d48e93f74	fix bulk es sync	2021-10-05 16:44:49 -04:00
Jack Robison	af22646322	fix tests	2021-10-05 16:44:49 -04:00
Jack Robison	722b42a93e	fix tests	2021-10-05 16:44:49 -04:00
Jack Robison	8f9e7f77a7	handle invalid claim update	2021-10-05 16:44:49 -04:00
Jack Robison	09bb1ba494	fix keeping claim_hash_to_txo and txo_to_claim in sync	2021-10-05 16:44:49 -04:00
Victor Shyba	d4137428ff	implement blocking and filtering	2021-10-05 16:44:49 -04:00
Jack Robison	b4d6c4f5b7	fix _get_pending_claim_name	2021-10-05 16:44:49 -04:00
Jack Robison	ffbe59ece5	fix applying expiration fork	2021-10-05 16:44:49 -04:00
Jack Robison	fab9c90ccb	update iterators to use pack_partial_key	2021-10-05 16:44:49 -04:00
Jack Robison	fb1a774bc4	delete lbry/wallet/server/storage.py ... -expose leveldb lru cache size as `CACHE_MB` hub param	2021-10-05 16:44:49 -04:00
Jack Robison	98bc7d1e0e	remove dead code	2021-10-05 16:44:49 -04:00
Jack Robison	f7622f24b2	non blocking mempool loop	2021-10-05 16:44:49 -04:00
Jack Robison	f0a195a6d4	faster es sync	2021-10-05 16:44:49 -04:00
Jack Robison	180ba27d84	run advance_block in threadpool	2021-10-05 16:44:49 -04:00
Jack Robison	f944671f86	use claim_to_txo cache	2021-10-05 16:44:49 -04:00
Jack Robison	def2903f7d	faster _cached_get_active_amount for claims ... -remove dead code	2021-10-05 16:44:49 -04:00
Jack Robison	0273a4e839	fix claim search by fee for claims without fees	2021-10-05 16:44:49 -04:00
Jack Robison	f8d2f02c5d	clear claim_to_txo cache before reading	2021-10-05 16:44:49 -04:00
Jack Robison	25147d8897	handle claims that dont exist in ES sync	2021-10-05 16:44:49 -04:00
Jack Robison	0fb6f05fba	in memory claim_to_txo and txo_to_claim dictionaries	2021-10-05 16:44:49 -04:00
Jack Robison	4e4e899356	fix spend_utxo	2021-10-05 16:44:49 -04:00
Jack Robison	5a01dbf269	split flush from advance_block	2021-10-05 16:44:49 -04:00
Jack Robison	30b923b283	rename extend_ops	2021-10-05 16:44:49 -04:00
Jack Robison	73ba381d20	faster spend_utxo	2021-10-05 16:44:49 -04:00
Jack Robison	1a5912877e	faster get_future_activated	2021-10-05 16:44:49 -04:00
Jack Robison	813e506b68	threadpool	2021-10-05 16:44:49 -04:00
Jack Robison	077ca987f7	cleanup	2021-10-05 16:44:49 -04:00
Jack Robison	c632a7a6a5	fix getting block hash during reorg	2021-10-05 16:44:49 -04:00
Jack Robison	e33e767510	fix test	2021-10-05 16:44:49 -04:00
Jack Robison	ac82617aa9	fix spends in address histories	2021-10-05 16:44:49 -04:00
Jack Robison	a35dfd1fd1	faster es sync	2021-10-05 16:44:49 -04:00
Jack Robison	c28aae9913	fix expiring channels	2021-10-05 16:44:49 -04:00
Jack Robison	c26a99e65c	fix abandoning signed claims in the same tx as their channel ... -fix canonical/short url in es	2021-10-05 16:44:49 -04:00
Jack Robison	ca57dcfc2f	handle failure to generate a short id	2021-10-05 16:44:49 -04:00
Jack Robison	df5662dd69	fix resolve by short id	2021-10-05 16:44:49 -04:00
Jack Robison	1ac7831f3c	move MemPool into BlockProcessor	2021-10-05 16:44:49 -04:00
Jack Robison	292d272a94	combine MemPool and Notifications classes	2021-10-05 16:44:49 -04:00
Jack Robison	a6ee8dc66e	fix touched hashXs notifications	2021-10-05 16:44:49 -04:00
Jack Robison	496f89f184	reorg claims in the search index	2021-10-05 16:44:49 -04:00
Jack Robison	7a56eff1ac	small fixes	2021-10-05 16:44:49 -04:00
Jack Robison	07e182aa16	rename	2021-10-05 16:44:49 -04:00
Jack Robison	7de06aa1e0	delete stale code	2021-10-05 16:44:49 -04:00
Jack Robison	3955b64405	simplify advance and reorg	2021-10-05 16:44:49 -04:00
Jack Robison	2bb55d681d	update limited_history	2021-10-05 16:44:49 -04:00
Jack Robison	f94e6ac527	update lookup_utxos	2021-10-05 16:44:49 -04:00
Jack Robison	b344f17b86	update RevertableOpStack	2021-10-05 16:44:49 -04:00
Jack Robison	677b8cb633	add remaining db prefixes	2021-10-05 16:44:49 -04:00
Jack Robison	a1ddd762e0	cleanup	2021-10-05 16:44:49 -04:00
Jack Robison	68474e4057	skip es sync during initial hub sync, halt the hub upon finishing initial sync	2021-10-05 16:44:49 -04:00
Jack Robison	a84b9ee396	fix es sync	2021-10-05 16:44:49 -04:00
Jack Robison	b9c2ee745a	fix non localhost elasticsearch	2021-10-05 16:44:49 -04:00
Jack Robison	c68f9f6f16	fix signed claim invalidation corner cases	2021-10-05 16:44:49 -04:00
Jack Robison	229cb85a6a	extra deletes ... -the channel_to_claim/claim_to_channel entries already get deleted when the claim txo is spent	2021-10-05 16:44:49 -04:00
Jack Robison	e5c22fa665	fix has_no_source for reposts	2021-10-05 16:44:49 -04:00
Jack Robison	8bcfff05d7	update channel_to_claim and claim_to_channel at the same time	2021-10-05 16:44:49 -04:00
Jack Robison	6416ee8151	typing and fix error string	2021-10-05 16:44:49 -04:00
Jack Robison	f8eceb48e6	update staged txo_to_claim after invalidating channel sig ... -fixes abandon of claim with invalidated signature and an update in same block	2021-10-05 16:44:49 -04:00
Jack Robison	310c483bfa	missing channel_to_claim delete	2021-10-05 16:44:49 -04:00
Jack Robison	a8f20361aa	fix RepostKey	2021-10-05 16:44:49 -04:00
Jack Robison	290be69d99	typing	2021-10-05 16:44:49 -04:00
Jack Robison	3b96bd7ea0	fix	2021-10-05 16:44:49 -04:00
Jack Robison	dc2f22f5fa	cleanup	2021-10-05 16:44:49 -04:00
Jack Robison	821be29f41	rename effective_amount prefix	2021-10-05 16:44:49 -04:00
Jack Robison	52ff1a12ff	fix undeleted claim_to_channel record	2021-10-05 16:44:49 -04:00
Jack Robison	814699ef11	cleanup	2021-10-05 16:44:49 -04:00
Jack Robison	0c30838b25	fix mismatch in claim_to_txo<->txo_to_claim	2021-10-05 16:44:49 -04:00
Jack Robison	cf66c2a1ee	rename things ... -fix effective amount integrity error	2021-10-05 16:44:49 -04:00
Jack Robison	2ee419ffca	fix	2021-10-05 16:44:49 -04:00
Jack Robison	bfb9d696d7	pretty print	2021-10-05 16:44:49 -04:00
Jack Robison	bb2a34dd6b	fix duplicate activate	2021-10-05 16:44:49 -04:00
Jack Robison	ed652c0c56	fix updating resolve by effective amount after abandoning support	2021-10-05 16:44:49 -04:00
Jack Robison	1dc961d6eb	use RevertableOpStack in _get_takeover_ops	2021-10-05 16:44:49 -04:00
Jack Robison	d119fcfc98	remove debug prints	2021-10-05 16:44:49 -04:00
Jack Robison	4d3573724a	add RevertableOpStack to verify consistency of ops as they're staged	2021-10-05 16:44:49 -04:00
Jack Robison	8b37a66075	fix fee amount overflow in es	2021-10-05 16:44:49 -04:00
Jack Robison	ba4f32075a	faster claim producer ... -make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits	2021-10-05 16:44:49 -04:00
Jack Robison	218be22576	imports	2021-10-05 16:44:49 -04:00
Jack Robison	7688293716	close db in sync script	2021-10-05 16:44:49 -04:00
Jack Robison	458f8533c4	try default block size	2021-10-05 16:44:49 -04:00
Jack Robison	34502752fc	update elastic sync	2021-10-05 16:44:49 -04:00
Jack Robison	d6758fd823	invalidate channel signatures upon channel abandon	2021-10-05 16:44:49 -04:00
Jack Robison	65700e790e	_prepare_claim_for_sync generators	2021-10-05 16:44:49 -04:00
Jack Robison	7c34e4bb96	logging	2021-10-05 16:44:49 -04:00
Jack Robison	d0d6e3563b	use default sync=False during write_batch	2021-10-05 16:44:49 -04:00
Jack Robison	a2619f8c78	genesis_bytes attribute	2021-10-05 16:44:49 -04:00
Jack Robison	42d07fd2f0	fix	2021-10-05 16:44:49 -04:00
Jack Robison	8bea10960f	disable es (revert)	2021-10-05 16:44:49 -04:00
Jack Robison	9cbb19c304	_cached_get_active_amount	2021-10-05 16:44:49 -04:00
Jack Robison	1b94dfd712	fix removing unactivated support	2021-10-05 16:44:49 -04:00
Jack Robison	9f3604d739	debug	2021-10-05 16:44:49 -04:00
Jack Robison	4a1b2be269	leveldb tuning	2021-10-05 16:44:49 -04:00
Jack Robison	962dc1b55b	debug	2021-10-05 16:44:49 -04:00
Jack Robison	07c86502f6	refactor ClaimToTXO prefix	2021-10-05 16:44:49 -04:00
Jack Robison	adb188e5d0	filter abandoned claims from those considered for early activation	2021-10-05 16:44:49 -04:00
Jack Robison	ce031dc6b8	only do early takeover on a larger amount (fix case where they're equal)	2021-10-05 16:44:49 -04:00
Jack Robison	18b5f03247	filter supported claim hashes for claims that dont exist from early takeover/activations	2021-10-05 16:44:49 -04:00
Jack Robison	8a555ecf1c	remove extra open functions	2021-10-05 16:44:49 -04:00
Jack Robison	1b325b9acd	fix flush id	2021-10-05 16:44:49 -04:00
Jack Robison	1bdaddb319	fix clearing pending_support caches upon abandon	2021-10-05 16:44:49 -04:00
Jack Robison	7896e177ef	fix putting spent unactivated supports in removed_active_support	2021-10-05 16:44:49 -04:00
Jack Robison	ce8e659008	fix syncing claim to es where channel is in the same block	2021-10-05 16:44:49 -04:00
Jack Robison	27be5deeb2	ignore activation for headless supports	2021-10-05 16:44:49 -04:00
Jack Robison	515f270c3a	faster get_future_activated	2021-10-05 16:44:49 -04:00
Jack Robison	ffff3bd334	debugging	2021-10-05 16:44:49 -04:00
Jack Robison	f493f13b25	prints	2021-10-05 16:44:49 -04:00
Jack Robison	e605c14b13	flush count	2021-10-05 16:44:49 -04:00
Jack Robison	338488f16d	tests	2021-10-05 16:44:49 -04:00
Jack Robison	2abc67c3e8	reposts	2021-10-05 16:44:49 -04:00
Jack Robison	eb1ba143ec	fix updating the ES search index ... -update search index to use ResolveResult tuples	2021-10-05 16:44:49 -04:00
Jack Robison	6f5bca0f67	bid ordered resolve, feed ES claim data from block processor	2021-10-05 16:44:49 -04:00
Jack Robison	407cd8dd4b	fix duplicate update op for early activating claim	2021-10-05 16:44:49 -04:00
Jack Robison	62a4f0fc04	fix early takeovers by not-yet activated claims	2021-10-05 16:44:49 -04:00
Jack Robison	3eb9d23108	require previous_winning arg for get_takeover_name_ops	2021-10-05 16:44:49 -04:00
Jack Robison	410d4aeb21	fix takeover edge case ... if a claim with a higher value than that of a claim taking over a name exists but isn't yet activated, activate it early and have it take over the name	2021-10-05 16:44:49 -04:00
Jack Robison	0a28d216fd	comments	2021-10-05 16:44:49 -04:00
Jack Robison	b69faf6920	bid ordered resolve (WIP)	2021-10-05 16:44:49 -04:00
Jack Robison	e77f9981df	DBError	2021-10-05 16:44:49 -04:00
Jack Robison	d27c2cc1e9	remove unused COIN file	2021-10-05 16:44:49 -04:00
Jack Robison	586b19675e	claim takeovers	2021-10-05 16:44:49 -04:00
Jack Robison	f2907536b4	move get_expiration_height and claimtrie constants to Coin class	2021-10-05 16:44:49 -04:00
Jack Robison	9a11ac06bf	claim activations and takeovers (WIP)	2021-10-05 16:44:49 -04:00
Jack Robison	aa3b18f848	advance_blocks -> advance_block	2021-10-05 16:44:49 -04:00
Jack Robison	103bdc151f	dead code	2021-10-05 16:44:49 -04:00
Jack Robison	6d4c1cd879	LBRYBlockProcessor -> BlockProcessor ... - temporarily disable claim_search	2021-10-05 16:44:49 -04:00
Jack Robison	cacbe30871	rebase	2021-10-05 16:44:49 -04:00
Jack Robison	04bb7b4919	add wrapper for getnamesintrie ... -used for verifying db state against lbrycrd	2021-10-05 16:44:49 -04:00
Jack Robison	b7df277a5c	db state struct ... -remove dead code	2021-10-05 16:44:49 -04:00
Jack Robison	c681041b48	claim expiration	2021-10-05 16:44:49 -04:00
Jack Robison	923834c784	get_claim_by_claim_id	2021-10-05 16:44:49 -04:00
Jack Robison	588edf98be	claims db ... -move all leveldb prefixes to DB_PREFIXES enum -add serializable RevertableOp interface for key/value puts and deletes -resolve urls from leveldb	2021-10-05 16:44:49 -04:00
Jack Robison	28c603ad5f	transaction_num_mapping	2021-10-05 16:44:49 -04:00
Jack Robison	6988a47e02	disable sqlite in block processor	2021-10-05 16:44:49 -04:00
Jack Robison	2c8ceb1217	named tuples	2021-10-05 16:44:49 -04:00
Jack Robison	ccac4ffa24	consolidate flush_backup	2021-10-05 16:44:49 -04:00
Jack Robison	4258cef9bd	remove lbry.wallet.server.history	2021-10-05 16:44:49 -04:00
Jack Robison	62cc6dfe76	consolidate leveldb block advance/reorg ... -move methods from History to LevelDB	2021-10-05 16:44:49 -04:00
Jack Robison	9f224a971b	atomic flush_dbs	2021-10-05 16:44:49 -04:00
Jack Robison	cf5dba9157	combine leveldb databases	2021-10-05 16:44:49 -04:00
Jack Robison	23035b9aa0	Merkle staticmethods	2021-10-05 16:44:49 -04:00
Victor Shyba	c2dcc4c898	avoid [''] on peers list	2021-09-13 15:57:21 -03:00
Victor Shyba	5061a35e66	remove ignored output from hub node	2021-09-10 18:57:20 -03:00
Victor Shyba	cd9a1e8c9e	default to legacy search for this release	2021-09-10 18:57:20 -03:00
Victor Shyba	b64aa51c0c	fix stream_types being an integer	2021-09-10 18:57:20 -03:00
Victor Shyba	8206441834	run CI for old and new setups	2021-09-10 18:57:20 -03:00
Victor Shyba	57dffaa2ce	update hub to beta release	2021-09-10 18:57:20 -03:00
Victor Shyba	9e81dd2360	refactor arguments fixup	2021-09-10 18:57:20 -03:00
Victor Shyba	1c31ec66f2	simplify operator handling	2021-09-10 18:57:20 -03:00
Victor Shyba	270192486a	translate grpc errors to RPCError	2021-09-10 18:57:20 -03:00
Victor Shyba	a799503c97	update fields from hub	2021-09-10 18:57:20 -03:00
Victor Shyba	9685928087	there is no first_search	2021-09-10 18:57:20 -03:00
Victor Shyba	0e4b2fad99	specify index name	2021-09-10 18:57:20 -03:00
Jeffrey Picard	046147eb1d	updates for fields	2021-09-10 18:57:20 -03:00
Jeffrey Picard	4544a074d9	Move the go hub settings from network to ledger config and hook reset ... correctly.	2021-09-10 18:57:20 -03:00
Jeffrey Picard	9b78501392	Set default server to the networks default and use go hub by default	2021-09-10 18:57:20 -03:00
Jeffrey Picard	a4955a2b79	remove uneeded prints	2021-09-10 18:57:20 -03:00
Jeffrey Picard	15a56ca25e	tons of small changes squashed together	2021-09-10 18:57:20 -03:00
Jeffrey Picard	9f65799a3d	uncomment tests, add remove_duplicates param ... Cleanup prints and commented out code remove print don't do list claims cleanup	2021-09-10 18:57:20 -03:00
Jeffrey Picard	a97fc6dba8	cleanup and reorgnazing some stuff ... Fixing tests relabel failing tests properly run all the tests for the hub cleanup HubNode	2021-09-10 18:57:20 -03:00
Jeffrey Picard	c124e88d12	grpc client for python	2021-09-10 18:57:20 -03:00
Jeffrey Picard	17f3870296	Add tests for hub ... Have the basic starting /stopping / querying. Still don't have the hub jsonrpc stuff working right and from the looks of it I need to clearify some of the logic in the claim search function itself because it's not returning the correct number of claims anyways. get the integration working with grpcurl Got tests working, still need to port the rest of them ported all of the claim search tests still a few failing due to not having inflation working, and there's something weird with limit_claims_per_channel that needs to be fixed.	2021-09-10 18:57:20 -03:00
Cristian Vicas	45bf6c3bf3	Drop comment_* apis. ... Refactored dangling functions. Added unit test.	2021-09-02 11:38:29 +03:00
Lex Berezhny	dc1d9e1c84	upgrade pylint and fix lint errors	2021-08-20 22:36:35 -04:00
Jack Robison	2445c00c7e	raise WalletNotLoadedError in get_wallet_or_error instead of ValueError	2021-08-17 11:30:58 -04:00
Victor Shyba	21c71bfac1	update sync utility	2021-08-09 18:33:47 -03:00
Victor Shyba	6baaed3581	refactor query with new fields	2021-08-09 18:33:47 -03:00
Victor Shyba	152dbfd5d1	reflect fee_currency, fee_amount and duration on repost searches	2021-08-09 18:33:47 -03:00
Victor Shyba	a56d14086b	reflect media_type on repost searches	2021-08-09 18:33:47 -03:00
Victor Shyba	aee87693f8	reflect stream_type on repost searches	2021-08-09 18:33:47 -03:00
Victor Shyba	e222b6ad9c	log that a invalid query happened	2021-08-09 15:07:44 -03:00
Victor Shyba	19b17374e8	throttle instead of disconnecting	2021-08-09 15:07:44 -03:00
Victor Shyba	43989122bb	add error type and message to error readme and update code	2021-08-09 15:07:44 -03:00
Victor Shyba	72712d6047	raise and disconnect if too many parameters are used on search	2021-08-09 15:07:44 -03:00
Victor Shyba	c591792de9	has_source is a special case	2021-07-22 16:25:55 -03:00
Victor Shyba	3108543ae5	3 missing fields	2021-07-22 16:25:55 -03:00
Victor Shyba	1eb221c743	translate reposted, signature_valid and normalized	2021-07-22 16:25:55 -03:00
Alex Grin	bebf6bc2e7	Update constants.py	2021-07-22 16:25:55 -03:00
Alex Grin	9e91cc2138	Update constants.py	2021-07-22 16:25:55 -03:00
Victor Shyba	5bd411ca27	filtering hash->id	2021-07-22 16:25:55 -03:00
Victor Shyba	a533cda6f0	ES: all _hash to _id	2021-07-22 16:25:55 -03:00
Jack Robison	9a8f9f0a94	fix stuck notification due to mempool/notification race	2021-07-20 15:14:10 -04:00
Victor Shyba	0ccafd5b53	make get_or_create_usable_address respect the generator lock	2021-07-19 14:09:52 -04:00
Victor Shyba	3de0982a4a	limit request error logging to 16k	2021-07-07 18:39:38 -03:00
Victor Shyba	c2184fb3bf	run migration on history db open	2021-07-07 18:39:38 -03:00