Commit graph

2114 commits

Author SHA1 Message Date
Jack Robison a98ea1e66a update sync script to handle ES falling behind leveldb on shutdown 2021-10-20 23:41:11 -04:00
Jack Robison 3dec697816 logging 2021-10-20 23:41:11 -04:00
Jack Robison b05d071a1c update Env to accept parameters from cli args 2021-10-20 23:41:11 -04:00
Jack Robison a27d3b9689 set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2021-10-20 23:41:11 -04:00
Jack Robison 1facc0cd01 remove unused hub env settings 2021-10-20 23:41:11 -04:00
FemtosecondLaser 6b8d4a444b Modified ensure_directory_exists() to check if the directory is writable by the process. 2021-10-20 15:26:16 +01:00
Jack Robison 6bef09a3b1 update lbry-hub-elastic-sync to support resyncing recent blocks 2021-10-19 15:53:20 -04:00
Jack Robison e35319e5a2 add CACHE_ALL_CLAIM_TXOS hub setting 2021-10-19 15:53:20 -04:00
Jack Robison 0e548b3812 remove dead code 2021-10-19 15:53:20 -04:00
Jack Robison bfac02ccab add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2021-10-19 15:53:20 -04:00
Jack Robison 7ea1a2b361 sleeps 2021-10-19 15:53:20 -04:00
Jack Robison 99df418f1d improve resolve caching 2021-10-19 15:53:20 -04:00
Jack Robison 6416d8ce9c threadpools for block processor and es sync reader 2021-10-19 15:53:20 -04:00
Jack Robison 22b43a2b01 doc strings 2021-10-19 15:53:20 -04:00
Jack Robison 05e5d24c5e improve claims_producer performance 2021-10-19 15:53:20 -04:00
Jack Robison eabcc30367 resolve lru cache 2021-10-19 15:53:20 -04:00
Jack Robison f5e0ef5223 add block_txs index 2021-10-19 15:53:20 -04:00
Jack Robison f46d9330b0 smaller caches 2021-10-19 15:53:20 -04:00
Jack Robison b62a0b4607 Update daemon.py
docstring
2021-10-15 09:40:15 -04:00
Cristian Vicas 1f044321fb Updated documentation for RPC calls: status, blob_list. 2021-10-15 09:40:15 -04:00
belikor 9509acc490
file_manager: raise new InvalidStreamURLError if the URL is invalid
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
2021-10-15 08:59:37 -04:00
Jack Robison d3516f299e
clear es attributes during initial sync 2021-10-08 16:34:48 -04:00
Jack Robison 79630767c2
fix setting references on txos in extra_txos 2021-10-08 16:34:15 -04:00
Jack Robison 084a76d075
fix reposted channel being missing from resolve result
-improve names of the resolve related methods in `LevelDB`
2021-10-07 15:09:13 -04:00
Jack Robison 43432a9e48
fix compactify script 2021-10-07 00:37:55 -04:00
Jack Robison d64a5bc12f
fix test 2021-10-06 23:53:17 -04:00
Jack Robison ccf03fc07b
only save undo info for blocks within reorg limit 2021-10-06 12:07:42 -04:00
Jack Robison a7c45da10c
fix channel count 2021-10-06 00:02:16 -04:00
Jack Robison 8167af9b4a
sort touched or deleted claim hashes 2021-10-05 16:44:49 -04:00
Jack Robison 4cf76123e5
block processor db refactoring
-access db through HubDB class, don't use plyvel.DB directly
-add channel count and support amount prefixes
2021-10-05 16:44:49 -04:00
Jack Robison 01ee4b23e6
fix and add test for abandoning a controlling in the same block a new claim is made 2021-10-05 16:44:49 -04:00
Jack Robison 09db868a28
fix ES index name so it stays the same within a test case 2021-10-05 16:44:49 -04:00
Jack Robison 33e8ef75ff
fix bug with early takeover by an update 2021-10-05 16:44:49 -04:00
Jack Robison 86f21da28b
fix activating non existent claim 2021-10-05 16:44:49 -04:00
Jack Robison 89cd6a9aa4
add tests for takeovers from amount changes in updates before/on/after activation 2021-10-05 16:44:49 -04:00
Jack Robison 18e1256037
batch address history notifications 2021-10-05 16:44:49 -04:00
Jack Robison 02cf478d91
improve leveldb caching 2021-10-05 16:44:49 -04:00
Jack Robison 6ec70192fe
refactor reload_blocking_filtering_streams 2021-10-05 16:44:49 -04:00
Jack Robison 8c75098a9a
fix filtering error upon abandon 2021-10-05 16:44:49 -04:00
Jack Robison 72500f6948
faster read_claim_txos 2021-10-05 16:44:49 -04:00
Jack Robison 37ec9ab464
remove unused executor 2021-10-05 16:44:49 -04:00
Victor Shyba 82fe2a4c8d
fix blocking and filtering 2021-10-05 16:44:49 -04:00
Jack Robison 91a07cfaee
fix logging number of notified sessions 2021-10-05 16:44:49 -04:00
Jack Robison 709f5e9a65
fix update that initiates takeover not being delayed 2021-10-05 16:44:49 -04:00
Jack Robison b2f9ef21cc
use hub binary from https://github.com/lbryio/hub/pull/13 2021-10-05 16:44:49 -04:00
Jack Robison be6b72edcd
handle invalid release time 2021-10-05 16:44:49 -04:00
Jack Robison ece2d1e78a
name and normalized -> claim_name and normalized_name
-update generated pb files
2021-10-05 16:44:49 -04:00
Jack Robison 1ee1a5f2a1
fix es sync.py 2021-10-05 16:44:49 -04:00
Jack Robison a567326853
fix all_claims_producer 2021-10-05 16:44:49 -04:00
Jack Robison 6231861dd6
merge conflicts 2021-10-05 16:44:49 -04:00
Jack Robison 1ff7b77ee0
claim search fixes 2021-10-05 16:44:49 -04:00
Jack Robison 9365708bb2
fix release_time and creation_timestamp 2021-10-05 16:44:49 -04:00
Jack Robison d23a0a8589
delete unused code 2021-10-05 16:44:49 -04:00
Jack Robison 58ad1f3876
non blocking claim producer 2021-10-05 16:44:49 -04:00
Jack Robison 2138e7ea33
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 32f8c9e59f
renormalization 2021-10-05 16:44:49 -04:00
Jack Robison 3a16edd8a6
fix trending overflow 2021-10-05 16:44:49 -04:00
Jack Robison 165f3bb270
refactor trending 2021-10-05 16:44:49 -04:00
Jack Robison 0ba75153f3
trending fixes 2021-10-05 16:44:49 -04:00
Jack Robison db2789990f
make app backward compatible with trending_score
-update trending decay function to zero out low trending score values faster
2021-10-05 16:44:49 -04:00
Jack Robison acaf299bcb
log time to update and decay trending in elasticsearch 2021-10-05 16:44:49 -04:00
Jack Robison 1940301824
skip integrity errors for trending spikes 2021-10-05 16:44:49 -04:00
Jack Robison 34576e880d
update trending in elasticsearch
-add TrendingPrefixSpike to leveldb
-expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings
2021-10-05 16:44:49 -04:00
Brendon J. Brewer 65c0668d40
constants 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 53bd2bcbfe
Put trending score into ES 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 388724fccb
Mark claims as touched 2021-10-05 16:44:49 -04:00
Jack Robison 231eabb013
fix non normalized canonical urls 2021-10-05 16:44:49 -04:00
Jack Robison 54903fc2ea
handle unicode error for unnormalized names 2021-10-05 16:44:49 -04:00
Jack Robison 3a1baf0700
prefix db 2021-10-05 16:44:49 -04:00
Brendon J. Brewer 0c0e36b6f8
trending 2021-10-05 16:44:49 -04:00
Jack Robison 234c03db09
fix claims not having non-normalized names 2021-10-05 16:44:49 -04:00
Jack Robison 28aa7da349
merge conflicts 2021-10-05 16:44:49 -04:00
Jack Robison c51e344b87
fix missing fields in reposts 2021-10-05 16:44:49 -04:00
Jack Robison 54461dfa75
fix merge conflicts and simplify extract_doc 2021-10-05 16:44:49 -04:00
Jack Robison 2d48e93f74
fix bulk es sync 2021-10-05 16:44:49 -04:00
Jack Robison af22646322
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 722b42a93e
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison 8f9e7f77a7
handle invalid claim update 2021-10-05 16:44:49 -04:00
Jack Robison 09bb1ba494
fix keeping claim_hash_to_txo and txo_to_claim in sync 2021-10-05 16:44:49 -04:00
Victor Shyba d4137428ff
implement blocking and filtering 2021-10-05 16:44:49 -04:00
Jack Robison b4d6c4f5b7
fix _get_pending_claim_name 2021-10-05 16:44:49 -04:00
Jack Robison ffbe59ece5
fix applying expiration fork 2021-10-05 16:44:49 -04:00
Jack Robison fab9c90ccb
update iterators to use pack_partial_key 2021-10-05 16:44:49 -04:00
Jack Robison fb1a774bc4
delete lbry/wallet/server/storage.py
-expose leveldb lru cache size as `CACHE_MB` hub param
2021-10-05 16:44:49 -04:00
Jack Robison 98bc7d1e0e
remove dead code 2021-10-05 16:44:49 -04:00
Jack Robison f7622f24b2
non blocking mempool loop 2021-10-05 16:44:49 -04:00
Jack Robison f0a195a6d4
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison 180ba27d84
run advance_block in threadpool 2021-10-05 16:44:49 -04:00
Jack Robison f944671f86
use claim_to_txo cache 2021-10-05 16:44:49 -04:00
Jack Robison def2903f7d
faster _cached_get_active_amount for claims
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison 0273a4e839
fix claim search by fee for claims without fees 2021-10-05 16:44:49 -04:00
Jack Robison f8d2f02c5d
clear claim_to_txo cache before reading 2021-10-05 16:44:49 -04:00
Jack Robison 25147d8897
handle claims that dont exist in ES sync 2021-10-05 16:44:49 -04:00
Jack Robison 0fb6f05fba
in memory claim_to_txo and txo_to_claim dictionaries 2021-10-05 16:44:49 -04:00
Jack Robison 4e4e899356
fix spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison 5a01dbf269
split flush from advance_block 2021-10-05 16:44:49 -04:00
Jack Robison 30b923b283
rename extend_ops 2021-10-05 16:44:49 -04:00
Jack Robison 73ba381d20
faster spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison 1a5912877e
faster get_future_activated 2021-10-05 16:44:49 -04:00
Jack Robison 813e506b68
threadpool 2021-10-05 16:44:49 -04:00
Jack Robison 077ca987f7
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison c632a7a6a5
fix getting block hash during reorg 2021-10-05 16:44:49 -04:00
Jack Robison e33e767510
fix test 2021-10-05 16:44:49 -04:00
Jack Robison ac82617aa9
fix spends in address histories 2021-10-05 16:44:49 -04:00
Jack Robison a35dfd1fd1
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison c28aae9913
fix expiring channels 2021-10-05 16:44:49 -04:00
Jack Robison c26a99e65c
fix abandoning signed claims in the same tx as their channel
-fix canonical/short url in es
2021-10-05 16:44:49 -04:00
Jack Robison ca57dcfc2f
handle failure to generate a short id 2021-10-05 16:44:49 -04:00
Jack Robison df5662dd69
fix resolve by short id 2021-10-05 16:44:49 -04:00
Jack Robison 1ac7831f3c
move MemPool into BlockProcessor 2021-10-05 16:44:49 -04:00
Jack Robison 292d272a94
combine MemPool and Notifications classes 2021-10-05 16:44:49 -04:00
Jack Robison a6ee8dc66e
fix touched hashXs notifications 2021-10-05 16:44:49 -04:00
Jack Robison 496f89f184
reorg claims in the search index 2021-10-05 16:44:49 -04:00
Jack Robison 7a56eff1ac
small fixes 2021-10-05 16:44:49 -04:00
Jack Robison 07e182aa16
rename 2021-10-05 16:44:49 -04:00
Jack Robison 7de06aa1e0
delete stale code 2021-10-05 16:44:49 -04:00
Jack Robison 3955b64405
simplify advance and reorg 2021-10-05 16:44:49 -04:00
Jack Robison 2bb55d681d
update limited_history 2021-10-05 16:44:49 -04:00
Jack Robison f94e6ac527
update lookup_utxos 2021-10-05 16:44:49 -04:00
Jack Robison b344f17b86
update RevertableOpStack 2021-10-05 16:44:49 -04:00
Jack Robison 677b8cb633
add remaining db prefixes 2021-10-05 16:44:49 -04:00
Jack Robison a1ddd762e0
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 68474e4057
skip es sync during initial hub sync, halt the hub upon finishing initial sync 2021-10-05 16:44:49 -04:00
Jack Robison a84b9ee396
fix es sync 2021-10-05 16:44:49 -04:00
Jack Robison b9c2ee745a
fix non localhost elasticsearch 2021-10-05 16:44:49 -04:00
Jack Robison 615e489d8d
fix stream_update --clear_channel flag 2021-10-05 16:44:49 -04:00
Jack Robison c68f9f6f16
fix signed claim invalidation corner cases 2021-10-05 16:44:49 -04:00
Jack Robison 229cb85a6a
extra deletes
-the channel_to_claim/claim_to_channel entries already get deleted when the claim txo is spent
2021-10-05 16:44:49 -04:00
Jack Robison e5c22fa665
fix has_no_source for reposts 2021-10-05 16:44:49 -04:00
Jack Robison 8bcfff05d7
update channel_to_claim and claim_to_channel at the same time 2021-10-05 16:44:49 -04:00
Jack Robison 6416ee8151
typing and fix error string 2021-10-05 16:44:49 -04:00
Jack Robison f8eceb48e6
update staged txo_to_claim after invalidating channel sig
-fixes abandon of claim with invalidated signature and an update in same block
2021-10-05 16:44:49 -04:00
Jack Robison 310c483bfa
missing channel_to_claim delete 2021-10-05 16:44:49 -04:00
Jack Robison a8f20361aa
fix RepostKey 2021-10-05 16:44:49 -04:00
Jack Robison 290be69d99
typing 2021-10-05 16:44:49 -04:00
Jack Robison 3b96bd7ea0
fix 2021-10-05 16:44:49 -04:00
Jack Robison dc2f22f5fa
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 821be29f41
rename effective_amount prefix 2021-10-05 16:44:49 -04:00
Jack Robison 52ff1a12ff
fix undeleted claim_to_channel record 2021-10-05 16:44:49 -04:00
Jack Robison 814699ef11
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison 0c30838b25
fix mismatch in claim_to_txo<->txo_to_claim 2021-10-05 16:44:49 -04:00
Jack Robison cf66c2a1ee
rename things
-fix effective amount integrity error
2021-10-05 16:44:49 -04:00
Jack Robison 2ee419ffca
fix 2021-10-05 16:44:49 -04:00
Jack Robison bfb9d696d7
pretty print 2021-10-05 16:44:49 -04:00
Jack Robison bb2a34dd6b
fix duplicate activate 2021-10-05 16:44:49 -04:00
Jack Robison ed652c0c56
fix updating resolve by effective amount after abandoning support 2021-10-05 16:44:49 -04:00
Jack Robison 1dc961d6eb
use RevertableOpStack in _get_takeover_ops 2021-10-05 16:44:49 -04:00
Jack Robison d119fcfc98
remove debug prints 2021-10-05 16:44:49 -04:00
Jack Robison 4d3573724a
add RevertableOpStack to verify consistency of ops as they're staged 2021-10-05 16:44:49 -04:00
Jack Robison 8b37a66075
fix fee amount overflow in es 2021-10-05 16:44:49 -04:00
Jack Robison ba4f32075a
faster claim producer
-make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits
2021-10-05 16:44:49 -04:00
Jack Robison 218be22576
imports 2021-10-05 16:44:49 -04:00
Jack Robison 7688293716
close db in sync script 2021-10-05 16:44:49 -04:00
Jack Robison 458f8533c4
try default block size 2021-10-05 16:44:49 -04:00
Jack Robison 34502752fc
update elastic sync 2021-10-05 16:44:49 -04:00
Jack Robison d6758fd823
invalidate channel signatures upon channel abandon 2021-10-05 16:44:49 -04:00
Jack Robison 65700e790e
_prepare_claim_for_sync generators 2021-10-05 16:44:49 -04:00
Jack Robison 7c34e4bb96
logging 2021-10-05 16:44:49 -04:00
Jack Robison d0d6e3563b
use default sync=False during write_batch 2021-10-05 16:44:49 -04:00
Jack Robison a2619f8c78
genesis_bytes attribute 2021-10-05 16:44:49 -04:00
Jack Robison 42d07fd2f0
fix 2021-10-05 16:44:49 -04:00
Jack Robison 8bea10960f
disable es (revert) 2021-10-05 16:44:49 -04:00
Jack Robison 9cbb19c304
_cached_get_active_amount 2021-10-05 16:44:49 -04:00
Jack Robison 1b94dfd712
fix removing unactivated support 2021-10-05 16:44:49 -04:00
Jack Robison 9f3604d739
debug 2021-10-05 16:44:49 -04:00
Jack Robison 4a1b2be269
leveldb tuning 2021-10-05 16:44:49 -04:00
Jack Robison 962dc1b55b
debug 2021-10-05 16:44:49 -04:00
Jack Robison 07c86502f6
refactor ClaimToTXO prefix 2021-10-05 16:44:49 -04:00
Jack Robison adb188e5d0
filter abandoned claims from those considered for early activation 2021-10-05 16:44:49 -04:00
Jack Robison ce031dc6b8
only do early takeover on a larger amount (fix case where they're equal) 2021-10-05 16:44:49 -04:00
Jack Robison 18b5f03247
filter supported claim hashes for claims that dont exist from early takeover/activations 2021-10-05 16:44:49 -04:00
Jack Robison 8a555ecf1c
remove extra open functions 2021-10-05 16:44:49 -04:00
Jack Robison 1b325b9acd
fix flush id 2021-10-05 16:44:49 -04:00
Jack Robison 1bdaddb319
fix clearing pending_support caches upon abandon 2021-10-05 16:44:49 -04:00
Jack Robison 7896e177ef
fix putting spent unactivated supports in removed_active_support 2021-10-05 16:44:49 -04:00
Jack Robison ce8e659008
fix syncing claim to es where channel is in the same block 2021-10-05 16:44:49 -04:00
Jack Robison 27be5deeb2
ignore activation for headless supports 2021-10-05 16:44:49 -04:00
Jack Robison 515f270c3a
faster get_future_activated 2021-10-05 16:44:49 -04:00
Jack Robison ffff3bd334
debugging 2021-10-05 16:44:49 -04:00
Jack Robison f493f13b25
prints 2021-10-05 16:44:49 -04:00
Jack Robison e605c14b13
flush count 2021-10-05 16:44:49 -04:00
Jack Robison 338488f16d
tests 2021-10-05 16:44:49 -04:00
Jack Robison 2abc67c3e8
reposts 2021-10-05 16:44:49 -04:00
Jack Robison eb1ba143ec
fix updating the ES search index
-update search index to use ResolveResult tuples
2021-10-05 16:44:49 -04:00
Jack Robison 6f5bca0f67
bid ordered resolve, feed ES claim data from block processor 2021-10-05 16:44:49 -04:00
Jack Robison 407cd8dd4b
fix duplicate update op for early activating claim 2021-10-05 16:44:49 -04:00
Jack Robison 62a4f0fc04
fix early takeovers by not-yet activated claims 2021-10-05 16:44:49 -04:00
Jack Robison 3eb9d23108
require previous_winning arg for get_takeover_name_ops 2021-10-05 16:44:49 -04:00
Jack Robison 410d4aeb21
fix takeover edge case
if a claim with a higher value than that of a claim taking over a name exists but isn't yet activated, activate it early and have it take over the name
2021-10-05 16:44:49 -04:00
Jack Robison 0a28d216fd
comments 2021-10-05 16:44:49 -04:00
Jack Robison b69faf6920
bid ordered resolve (WIP) 2021-10-05 16:44:49 -04:00
Jack Robison e77f9981df
DBError 2021-10-05 16:44:49 -04:00
Jack Robison d27c2cc1e9
remove unused COIN file 2021-10-05 16:44:49 -04:00
Jack Robison 586b19675e
claim takeovers 2021-10-05 16:44:49 -04:00
Jack Robison f2907536b4
move get_expiration_height and claimtrie constants to Coin class 2021-10-05 16:44:49 -04:00
Jack Robison 9a11ac06bf
claim activations and takeovers (WIP) 2021-10-05 16:44:49 -04:00
Jack Robison aa3b18f848
advance_blocks -> advance_block 2021-10-05 16:44:49 -04:00
Jack Robison 103bdc151f
dead code 2021-10-05 16:44:49 -04:00
Jack Robison 6d4c1cd879
LBRYBlockProcessor -> BlockProcessor
- temporarily disable claim_search
2021-10-05 16:44:49 -04:00
Jack Robison cacbe30871
rebase 2021-10-05 16:44:49 -04:00
Jack Robison bfeeacb230
tests 2021-10-05 16:44:49 -04:00
Jack Robison 04bb7b4919
add wrapper for getnamesintrie
-used for verifying db state against lbrycrd
2021-10-05 16:44:49 -04:00
Jack Robison b7df277a5c
db state struct
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison c681041b48
claim expiration 2021-10-05 16:44:49 -04:00
Jack Robison 923834c784
get_claim_by_claim_id 2021-10-05 16:44:49 -04:00
Jack Robison 588edf98be
claims db
-move all leveldb prefixes to DB_PREFIXES enum
-add serializable RevertableOp interface for key/value puts and deletes
-resolve urls from leveldb
2021-10-05 16:44:49 -04:00
Jack Robison 28c603ad5f
transaction_num_mapping 2021-10-05 16:44:49 -04:00
Jack Robison 6988a47e02
disable sqlite in block processor 2021-10-05 16:44:49 -04:00
Jack Robison 2c8ceb1217
named tuples 2021-10-05 16:44:49 -04:00
Jack Robison ccac4ffa24
consolidate flush_backup 2021-10-05 16:44:49 -04:00
Jack Robison 4258cef9bd
remove lbry.wallet.server.history 2021-10-05 16:44:49 -04:00
Jack Robison 62cc6dfe76
consolidate leveldb block advance/reorg
-move methods from History to LevelDB
2021-10-05 16:44:49 -04:00
Jack Robison 9f224a971b
atomic flush_dbs 2021-10-05 16:44:49 -04:00
Jack Robison cf5dba9157
combine leveldb databases 2021-10-05 16:44:49 -04:00
Jack Robison 23035b9aa0
Merkle staticmethods 2021-10-05 16:44:49 -04:00
Lex Berezhny 84908ec8ec v0.105.0 2021-10-05 11:29:39 -04:00
Victor Shyba dade49743b fix file reflect and add test 2021-10-04 19:26:05 -03:00
Lex Berezhny dfa6701c43 disk space metrics 2021-10-03 19:33:18 -04:00
Victor Shyba 763ca69a73 dht: use bytes hex/fromhex instead of binascii 2021-09-30 13:26:33 -03:00
Victor Shyba 6bf3b152bf add grin to dht known list 2021-09-30 13:26:33 -03:00
Victor Shyba aa19f85996 add madiator to known dht nodes 2021-09-30 13:26:33 -03:00
Victor Shyba 71a19191f8 add dht seed node script 2021-09-30 13:26:33 -03:00
Victor Shyba 38a0f20a33 fix conflict with imported function 2021-09-30 13:24:17 -03:00
Victor Shyba c35192108c errors for empyt and misssing file on publish 2021-09-30 13:24:17 -03:00
Victor Shyba 245b564f13 generalize stream empty to argument empty 2021-09-30 13:24:17 -03:00
Victor Shyba 0d8d1ea4f3 empty stream name error for user input 2021-09-30 13:24:17 -03:00
Victor Shyba 27a427a363 error for missing channel private key 2021-09-30 13:24:17 -03:00
Victor Shyba 2ff028a694 error for already purchased claims 2021-09-30 13:24:17 -03:00
belikor bbbaf59591 daemon: fix documentation in the file_list docstring
This is necessary to produce the `docs/api.json`
(through `scripts/generate_json_api.py`)
with correct information, and to be able to parse this file later on
by other tools.
2021-09-23 21:00:31 -05:00
Lex Berezhny 169419896f v0.104.0 2021-09-22 18:39:01 -04:00
Lex Berezhny fc4407ef7e revert release 2021-09-22 18:11:41 -04:00
Lex Berezhny 03735a125f v0.104.0 2021-09-22 14:02:52 -04:00
Lex Berezhny 9b9794b5e0 default is_mine to true during migration 2021-09-20 09:23:42 -04:00
Lex Berezhny cfe6c82a31 tests 2021-09-19 21:38:09 -04:00
Lex Berezhny 3e30228d95 lint 2021-09-15 10:49:03 -04:00
Lex Berezhny 7264b53e5f during disk clean your own sd blob is now kept and file status of deleted files is set to stopped 2021-09-15 10:37:08 -04:00
Lex Berezhny 60836d8523 db migration and other fixes 2021-09-15 09:10:06 -04:00
Lex Berezhny ef89c2e47a use databse to track blob disk space use and preserve own blobs 2021-09-15 09:10:06 -04:00
Lex Berezhny 2d9e3e1847 v0.103.0 2021-09-14 23:25:32 -04:00
Victor Shyba c2dcc4c898 avoid [''] on peers list 2021-09-13 15:57:21 -03:00
Victor Shyba 358ef4536f add ConflictingInputValueError for claim_id+claim_ids 2021-09-10 18:57:20 -03:00
Victor Shyba 5061a35e66 remove ignored output from hub node 2021-09-10 18:57:20 -03:00
Victor Shyba cd9a1e8c9e default to legacy search for this release 2021-09-10 18:57:20 -03:00
Victor Shyba 40d26cb868 fix error msg to match Go msg 2021-09-10 18:57:20 -03:00
Victor Shyba b64aa51c0c fix stream_types being an integer 2021-09-10 18:57:20 -03:00
Victor Shyba 8206441834 run CI for old and new setups 2021-09-10 18:57:20 -03:00
Victor Shyba d713783736 ignore default values 2021-09-10 18:57:20 -03:00
Victor Shyba 57dffaa2ce update hub to beta release 2021-09-10 18:57:20 -03:00
Victor Shyba 9e81dd2360 refactor arguments fixup 2021-09-10 18:57:20 -03:00
Victor Shyba e2798969d7 claim_id is an invertible field, not a repeated 2021-09-10 18:57:20 -03:00
Victor Shyba 1c31ec66f2 simplify operator handling 2021-09-10 18:57:20 -03:00
Victor Shyba 241f9fc7b0 not_claim_id/not_claim_ids is not a search parameter 2021-09-10 18:57:20 -03:00
Victor Shyba 270192486a translate grpc errors to RPCError 2021-09-10 18:57:20 -03:00
Victor Shyba a799503c97 update fields from hub 2021-09-10 18:57:20 -03:00
Victor Shyba 9685928087 there is no first_search 2021-09-10 18:57:20 -03:00
Victor Shyba 0e4b2fad99 specify index name 2021-09-10 18:57:20 -03:00
Victor Shyba 3c4571a4e0 remove fallback 2021-09-10 18:57:20 -03:00
Jeffrey Picard 046147eb1d updates for fields 2021-09-10 18:57:20 -03:00
Jeffrey Picard 7834520e54 update code to be consistent with field renames 2021-09-10 18:57:20 -03:00
Jeffrey Picard 8e5b4d4b6f hardcode port 2021-09-10 18:57:20 -03:00
Jeffrey Picard 4544a074d9 Move the go hub settings from network to ledger config and hook reset
correctly.
2021-09-10 18:57:20 -03:00
Jeffrey Picard 9b78501392 Set default server to the networks default and use go hub by default 2021-09-10 18:57:20 -03:00
Jeffrey Picard a4955a2b79 remove uneeded prints 2021-09-10 18:57:20 -03:00
Jeffrey Picard 92ae1a565b updates protobuf 2021-09-10 18:57:20 -03:00
Jeffrey Picard 15a56ca25e tons of small changes squashed together 2021-09-10 18:57:20 -03:00
Jeffrey Picard 9dcaa829ea update protobufs 2021-09-10 18:57:20 -03:00
Jeffrey Picard 9f65799a3d uncomment tests, add remove_duplicates param
Cleanup prints and commented out code

remove print

don't do list claims

cleanup
2021-09-10 18:57:20 -03:00
Jeffrey Picard 886587848b protobuf changes
more protobuf changes (fix imports)
2021-09-10 18:57:20 -03:00
Jeffrey Picard a97fc6dba8 cleanup and reorgnazing some stuff
Fixing tests

relabel failing tests properly

run all the tests for the hub

cleanup HubNode
2021-09-10 18:57:20 -03:00
Jeffrey Picard c124e88d12 grpc client for python 2021-09-10 18:57:20 -03:00
Jeffrey Picard 17f3870296 Add tests for hub
Have the basic starting /stopping / querying. Still don't have the hub
jsonrpc stuff working right and from the looks of it I need to clearify
some of the logic in the claim search function itself because it's not
returning the correct number of claims anyways.

get the integration working with grpcurl

Got tests working, still need to port the rest of them

ported all of the claim search tests

still a few failing due to not having inflation working, and there's something weird
with limit_claims_per_channel that needs to be fixed.
2021-09-10 18:57:20 -03:00
Cristian Vicas 45bf6c3bf3 Drop comment_* apis.
Refactored dangling functions.
Added unit test.
2021-09-02 11:38:29 +03:00
Cristian Vicas fef0cc764d Drop comment_* apis
Removed the comment API
Removed tests for the comment API
Removed the documentation section
Removed the comment server configuration
2021-09-02 08:51:00 +03:00
belikor d26c06dbf3 jsonrpc_support_sum: remove the + signs from the docstring
These symbols came from 0a0ac3b7c9 and were probably added
accidentally to the beginning of the line by copying and pasting
some diffs.
2021-08-25 13:28:02 -05:00
Lex Berezhny dc1d9e1c84 upgrade pylint and fix lint errors 2021-08-20 22:36:35 -04:00
Lex Berezhny d5bae3a8c6 manually set save_files=True in unit tests 2021-08-19 09:31:17 -04:00
Lex Berezhny 87094fc83f changed default setting save_files to be false 2021-08-17 15:47:18 -04:00
Lex Berezhny 7582c221d1 v0.102.0 2021-08-17 14:16:17 -04:00
Jack Robison eccedada40
add TODOs for errors raised that aren't defined in lbry.error 2021-08-17 12:31:03 -04:00
Jack Robison 25d54accf8
return api errors from wallet_add and wallet_create 2021-08-17 12:30:17 -04:00
Jack Robison d07685f0e9
only log tracebacks for api errors not defined in lbry.error 2021-08-17 11:30:58 -04:00
Jack Robison 2445c00c7e
raise WalletNotLoadedError in get_wallet_or_error instead of ValueError 2021-08-17 11:30:58 -04:00
Lex Berezhny 4614c7d4c2 clean blobs after waiting interval instead of immediately on startup 2021-08-17 09:52:44 -04:00
Lex Berezhny 3433c9e708 return number of files deleted 2021-08-16 17:03:40 -04:00
Lex Berezhny 2cd5d75a2e return true/false if clean was performed 2021-08-16 17:02:13 -04:00
Lex Berezhny 4edab7bb7f fix sorting by DirEntry error 2021-08-16 14:41:16 -04:00
Lex Berezhny 51d21d8c86 working disk cleanup 2021-08-16 14:15:12 -04:00
Lex Berezhny b4c3307cdf fixed tests 2021-08-13 10:32:46 -04:00
Lex Berezhny 4e8d10cb44 disk space manager and status API 2021-08-13 10:32:46 -04:00
Victor Shyba 21c71bfac1 update sync utility 2021-08-09 18:33:47 -03:00
Victor Shyba 6baaed3581 refactor query with new fields 2021-08-09 18:33:47 -03:00
Victor Shyba 152dbfd5d1 reflect fee_currency, fee_amount and duration on repost searches 2021-08-09 18:33:47 -03:00
Victor Shyba a56d14086b reflect media_type on repost searches 2021-08-09 18:33:47 -03:00
Victor Shyba aee87693f8 reflect stream_type on repost searches 2021-08-09 18:33:47 -03:00
Alex Grin 976b4affd9
Merge pull request #3383 from lbryio/dht_log 2021-08-09 17:10:01 -04:00
Victor Shyba e222b6ad9c log that a invalid query happened 2021-08-09 15:07:44 -03:00
Victor Shyba 19b17374e8 throttle instead of disconnecting 2021-08-09 15:07:44 -03:00
Victor Shyba 43989122bb add error type and message to error readme and update code 2021-08-09 15:07:44 -03:00
Victor Shyba 72712d6047 raise and disconnect if too many parameters are used on search 2021-08-09 15:07:44 -03:00
Victor Shyba 0b52d2cc15 log invalid port as a warning instead of an exception 2021-08-03 15:29:52 -03:00
Lex Berezhny 279a365cb1 v0.101.1 2021-07-27 11:12:06 -04:00
Lex Berezhny 2c9e00da56 revert version 2021-07-27 11:10:16 -04:00
Lex Berezhny b7d58bcdbc v0.101.1 2021-07-26 17:01:25 -04:00
Lex Berezhny 13a856b843 revert version 2021-07-26 17:00:32 -04:00
Lex Berezhny 60cf6c6b97 v0.101.1 2021-07-26 16:02:28 -04:00
Victor Shyba 1849c02cb6 fix typo in kwargs key 2021-07-26 16:02:48 -03:00
Lex Berezhny 1ec74a89e2
Merge pull request #3367 from belikor/fix-search-claim-id
fix error when using `--claim_id` with `lbrynet claim search`
2021-07-23 10:08:57 -04:00
Victor Shyba c591792de9 has_source is a special case 2021-07-22 16:25:55 -03:00
Victor Shyba 3108543ae5 3 missing fields 2021-07-22 16:25:55 -03:00
Victor Shyba 1eb221c743 translate reposted, signature_valid and normalized 2021-07-22 16:25:55 -03:00
Alex Grin bebf6bc2e7 Update constants.py 2021-07-22 16:25:55 -03:00
Alex Grin 9e91cc2138 Update constants.py 2021-07-22 16:25:55 -03:00
Victor Shyba c5b939cfb7 fix tests 2021-07-22 16:25:55 -03:00
Victor Shyba 5bd411ca27 filtering hash->id 2021-07-22 16:25:55 -03:00
Victor Shyba a533cda6f0 ES: all _hash to _id 2021-07-22 16:25:55 -03:00
Lex Berezhny fe4b07b8ae v0.101.0 2021-07-21 12:35:16 -04:00
Lex Berezhny f9f2ccd904 revert version 2021-07-21 12:28:41 -04:00
Lex Berezhny 8d9073cd31 v0.101.0 2021-07-20 22:52:44 -04:00
Jack Robison 9a8f9f0a94
fix stuck notification due to mempool/notification race 2021-07-20 15:14:10 -04:00
belikor 6421cecafb daemon: fix --claim_id with lbrynet claim search
For some reason, when using `claim_search`
with `--claim_id`, the arguments dictionary will also
contain `claim_ids` with an empty list, even if we didn't specify it.
```
lbrynet claim search --claim_id=8945573bcfcb7f8276187dfbb93545eac4ebf71a
```

Using both `claim_id` and `claim_ids` will raise a `ValueError`
exception so the daemon won't return a valid result
even if the claim ID is in fact valid.

So if `claim_id` exists, we need to discard `claim_ids`
if it is empty, before proceeding with the rest of the code.

On the other hand, if `claim_ids` is used, and `claim_id` is absent,
there will be no problem as `claim_id` won't be added to the dictionary.
```
lbrynet claim search --claim_ids=8945573bcfcb7f8276187dfbb93545eac4ebf71a
```
2021-07-19 22:24:43 -05:00
Alex Grin 74116cc550
Merge branch 'master' into propagate_external_ip_change 2021-07-19 14:12:19 -04:00
Victor Shyba 0ccafd5b53 make get_or_create_usable_address respect the generator lock 2021-07-19 14:09:52 -04:00
belikor b7791d2845 exchange_rate_manager: raise exception if 'error' is in json_response
If the error is not handled, the running daemon will continuously
print the following error message:
```
Traceback (most recent call last):
  File "lbry/extras/daemon/exchange_rate_manager.py", line 77, in get_rate
  File "lbry/extras/daemon/exchange_rate_manager.py", line 189, in get_rate_from_response
KeyError: 0
```

This started happening when the UPBit exchange decided to delist
the LBC coin.

Normally `json_response` should be a dictionary, not a list,
so `json_response[0]` causes an error.

By checking for the `'error'` key, we can raise the proper exception.

Once this is done, the message will be a warning, not a traceback.
```
WARNING  lbry.extras.daemon.exchange_rate_manager:92:
Failed to get exchange rate from UPbit: result not found
```
2021-07-19 13:41:49 -04:00
Victor Shyba 6254f53716 propagate external ip changes from upnp component to dht node protocol 2021-07-08 03:46:05 -03:00
Victor Shyba 3de0982a4a limit request error logging to 16k 2021-07-07 18:39:38 -03:00
Victor Shyba c2184fb3bf run migration on history db open 2021-07-07 18:39:38 -03:00
Victor Shyba 919c09fcb0 add migration 2021-07-07 18:39:38 -03:00
Victor Shyba 1d9dbd40ec increase flush counter to 32 bits 2021-07-07 18:39:38 -03:00
Victor Shyba 309e957a85 add concurrent_hub_requests conf 2021-06-24 21:21:19 -03:00
Victor Shyba d7007e402e move request semaphore to session and apply to all requests 2021-06-24 21:02:41 -03:00
Victor Shyba 91323a21cf add hub_timeout and propagate it to network code 2021-06-24 21:02:41 -03:00
Lex Berezhny fea893d76c v0.100.0 2021-06-22 13:33:03 -04:00
Lex Berezhny 761bc6ba4c revert release and fix test 2021-06-22 13:32:41 -04:00
Lex Berezhny 75172feb4e v0.100.0 2021-06-22 12:53:23 -04:00
Lex Berezhny 3285fb1608 revert release 2021-06-22 12:52:48 -04:00
Lex Berezhny 03a4c6910d v0.100.0 2021-06-22 12:51:36 -04:00
Lex Berezhny 485b958599 revert release 2021-06-22 12:50:11 -04:00
Lex Berezhny da47ba2f67 v0.100.0 2021-06-22 11:11:02 -04:00
Lex Berezhny c39195488a bug fix 2021-06-22 11:07:58 -04:00
Lex Berezhny 0946c72b88 lint 2021-06-22 11:07:58 -04:00
Lex Berezhny 7d49b046d4 added support to config for determining if value is set and implemented hub selection logic 2021-06-22 11:07:58 -04:00
Lex Berezhny 5f0426c840 country jurisdiction added to hub UDP protocol 2021-06-22 11:07:58 -04:00
Lex Berezhny 73e239cc5f client side hub discovery pub/sub and hub metadata stored, removed old peers implementation 2021-06-22 11:07:58 -04:00
Lex Berezhny ad670f721a working client peer hub 2021-06-22 11:07:58 -04:00
Lex Berezhny 028a4a70cf wallet server federation, client portion 2021-06-22 11:07:58 -04:00
Lex Berezhny e457b2f0d6 fix trending to use built-in sqlite instead of apsw 2021-06-15 18:22:42 -04:00
Lex Berezhny c9cf7fd4d4 drop apsw in wallet.server.db.elasticsearch.sync 2021-06-15 18:22:42 -04:00
Lex Berezhny 25e16c3565 dropping apsw 2021-06-15 18:22:42 -04:00
Victor Shyba fb438dc108 remove the unregister call 2021-06-04 12:47:16 -03:00
Victor Shyba 4e6b4f179b add items() to LRUCache 2021-06-04 12:20:44 -03:00