LBRYCommunity/lbry-sdk
1
0
Fork 1
Code Issues Pull requests Projects Releases 282 Packages Wiki Activity
8899 commits 110 branches 641 tags 31 MiB
Commit graph

1950 commits

Author SHA1 Message Date
Victor Shyba
a19060c08d log unexpected errors, rename task/loop 2021-11-09 14:27:06 -05:00
Victor Shyba
fa2ad88cc4 clear cache on test assertions 2021-11-09 14:27:06 -05:00
Victor Shyba
63cbcd0956 make sure the downloader always stops gracefully 2021-11-09 14:27:06 -05:00
Victor Shyba
d6d0ebf8f4 cache space stats from running components so status is instant 2021-11-09 14:27:06 -05:00
Victor Shyba
0d810d92ca add index for blob table so size summaries are faster 2021-11-09 14:27:06 -05:00
Victor Shyba
1ff914a6f4 download from stored announcements and dont reannounce 2021-11-09 14:27:06 -05:00
Victor Shyba
5959b1be72 improve disk space manager status, include more info and unify space queries 2021-11-09 14:27:06 -05:00
Victor Shyba
d12a214c05 normal_blobs->stream_blobs, proactive->background 2021-11-09 14:27:06 -05:00
Victor Shyba
3a83052f2e fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full 2021-11-09 14:27:06 -05:00
Victor Shyba
510b44ca92 move more logic out of the downloader component 2021-11-09 14:27:06 -05:00
Victor Shyba
15edb6756d extract background downloader to its own class 2021-11-09 14:27:06 -05:00
Victor Shyba
fbfd02b08b add analytics event for network disk space 2021-11-09 14:27:06 -05:00
Victor Shyba
b39c26fc86 announce orphan blobs manually, as that was done when save stream 2021-11-09 14:27:06 -05:00
Victor Shyba
95b2c8d175 cleanup background downloader blobs from conf 2021-11-09 14:27:06 -05:00
Victor Shyba
d52748b09f separated network seeding space metrics 2021-11-09 14:27:06 -05:00
Victor Shyba
34d18a3a9a don't save streams for network blobs and bypass disk space manager 2021-11-09 14:27:06 -05:00
Victor Shyba
3b27d6a9b5 add conf for network seeding space limit 2021-11-09 14:27:06 -05:00
Victor Shyba
703c391f99 schedule the download task instead 2021-11-09 14:27:06 -05:00
Victor Shyba
13667df374 download from DHT 2021-11-09 14:27:06 -05:00
Victor Shyba
8800d6985f drop channel support, prepare to hook into DHT 2021-11-09 14:27:06 -05:00
Victor Shyba
364b8f2605 handle case where something that isn't a sd blob gets hit 2021-11-09 14:27:06 -05:00
Victor Shyba
67b9ea9deb no api yet 2021-11-09 14:27:06 -05:00
Victor Shyba
b78f2336a7 download only blobs 2021-11-09 14:27:06 -05:00
Victor Shyba
23a5ce3df7 fix exception arguments 2021-11-09 14:27:06 -05:00
Victor Shyba
8f88e28e50 test add/remove/list subscriptions 2021-11-09 14:27:06 -05:00
Victor Shyba
9cf6139557 fix and test main api 2021-11-09 14:27:06 -05:00
Victor Shyba
d556065a8b download all blobs and check that on tests 2021-11-09 14:27:06 -05:00
Victor Shyba
951716f7dc create downloader component and initial tests 2021-11-09 14:27:06 -05:00
Victor Shyba
c42b76dcb8 dont lose results on duplicates, just warn 2021-11-08 10:50:47 -05:00
Victor Shyba
a73582d9ae remove tried_for_this_blob so banned peers are retried for same blob 2021-11-08 10:50:47 -05:00
Cristian Vicas
42c4fc7557 Bug [#2070] where blob_get RPC timed out. 
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.

Bug [#2070] where blob_get RPC timed out.

Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
 2021-11-08 10:49:48 -05:00
Jack Robison
ddbbb6f1dd
use mempool cache in transaction_get_batch 2021-10-27 20:19:08 -04:00
Lex Berezhny
ff21a92330
Merge pull request #3457 from FemtosecondLaser/feature/3270-check-default-download-dir-writable 
Modified ensure_directory_exists() to check if the directory is writable by the process.
 2021-10-27 11:00:13 -04:00
Jack Robison
1e391d211b
fix attempting to update trending on abandoned claims 2021-10-23 18:39:04 -04:00
Jack Robison
1a74d6604d skip loading tx/claim caches in the elastic sync script when not needed 2021-10-22 15:10:35 -04:00
Jack Robison
48505c2968 update trending with help from @eggplantbren 2021-10-21 00:17:12 -04:00
Jack Robison
a98ea1e66a update sync script to handle ES falling behind leveldb on shutdown 2021-10-20 23:41:11 -04:00
Jack Robison
3dec697816 logging 2021-10-20 23:41:11 -04:00
Jack Robison
b05d071a1c update Env to accept parameters from cli args 2021-10-20 23:41:11 -04:00
Jack Robison
a27d3b9689 set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2021-10-20 23:41:11 -04:00
Jack Robison
1facc0cd01 remove unused hub env settings 2021-10-20 23:41:11 -04:00
FemtosecondLaser
6b8d4a444b Modified ensure_directory_exists() to check if the directory is writable by the process. 2021-10-20 15:26:16 +01:00
Jack Robison
6bef09a3b1 update lbry-hub-elastic-sync to support resyncing recent blocks 2021-10-19 15:53:20 -04:00
Jack Robison
e35319e5a2 add CACHE_ALL_CLAIM_TXOS hub setting 2021-10-19 15:53:20 -04:00
Jack Robison
0e548b3812 remove dead code 2021-10-19 15:53:20 -04:00
Jack Robison
bfac02ccab add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2021-10-19 15:53:20 -04:00
Jack Robison
7ea1a2b361 sleeps 2021-10-19 15:53:20 -04:00
Jack Robison
99df418f1d improve resolve caching 2021-10-19 15:53:20 -04:00
Jack Robison
6416d8ce9c threadpools for block processor and es sync reader 2021-10-19 15:53:20 -04:00
Jack Robison
22b43a2b01 doc strings 2021-10-19 15:53:20 -04:00
Jack Robison
05e5d24c5e improve claims_producer performance 2021-10-19 15:53:20 -04:00
Jack Robison
eabcc30367 resolve lru cache 2021-10-19 15:53:20 -04:00
Jack Robison
f5e0ef5223 add block_txs index 2021-10-19 15:53:20 -04:00
Jack Robison
f46d9330b0 smaller caches 2021-10-19 15:53:20 -04:00
Jack Robison
b62a0b4607 Update daemon.py 
docstring
 2021-10-15 09:40:15 -04:00
Cristian Vicas
1f044321fb Updated documentation for RPC calls: status, blob_list. 2021-10-15 09:40:15 -04:00
belikor
9509acc490
file_manager: raise new InvalidStreamURLError if the URL is invalid 
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
 2021-10-15 08:59:37 -04:00
Jack Robison
d3516f299e
clear es attributes during initial sync 2021-10-08 16:34:48 -04:00
Jack Robison
79630767c2
fix setting references on txos in extra_txos 2021-10-08 16:34:15 -04:00
Jack Robison
084a76d075
fix reposted channel being missing from resolve result 
-improve names of the resolve related methods in `LevelDB`
 2021-10-07 15:09:13 -04:00
Jack Robison
43432a9e48
fix compactify script 2021-10-07 00:37:55 -04:00
Jack Robison
d64a5bc12f
fix test 2021-10-06 23:53:17 -04:00
Jack Robison
ccf03fc07b
only save undo info for blocks within reorg limit 2021-10-06 12:07:42 -04:00
Jack Robison
a7c45da10c
fix channel count 2021-10-06 00:02:16 -04:00
Jack Robison
8167af9b4a
sort touched or deleted claim hashes 2021-10-05 16:44:49 -04:00
Jack Robison
4cf76123e5
block processor db refactoring 
-access db through HubDB class, don't use plyvel.DB directly
-add channel count and support amount prefixes
 2021-10-05 16:44:49 -04:00
Jack Robison
01ee4b23e6
fix and add test for abandoning a controlling in the same block a new claim is made 2021-10-05 16:44:49 -04:00
Jack Robison
09db868a28
fix ES index name so it stays the same within a test case 2021-10-05 16:44:49 -04:00
Jack Robison
33e8ef75ff
fix bug with early takeover by an update 2021-10-05 16:44:49 -04:00
Jack Robison
86f21da28b
fix activating non existent claim 2021-10-05 16:44:49 -04:00
Jack Robison
89cd6a9aa4
add tests for takeovers from amount changes in updates before/on/after activation 2021-10-05 16:44:49 -04:00
Jack Robison
18e1256037
batch address history notifications 2021-10-05 16:44:49 -04:00
Jack Robison
02cf478d91
improve leveldb caching 2021-10-05 16:44:49 -04:00
Jack Robison
6ec70192fe
refactor reload_blocking_filtering_streams 2021-10-05 16:44:49 -04:00
Jack Robison
8c75098a9a
fix filtering error upon abandon 2021-10-05 16:44:49 -04:00
Jack Robison
72500f6948
faster read_claim_txos 2021-10-05 16:44:49 -04:00
Jack Robison
37ec9ab464
remove unused executor 2021-10-05 16:44:49 -04:00
Victor Shyba
82fe2a4c8d
fix blocking and filtering 2021-10-05 16:44:49 -04:00
Jack Robison
91a07cfaee
fix logging number of notified sessions 2021-10-05 16:44:49 -04:00
Jack Robison
709f5e9a65
fix update that initiates takeover not being delayed 2021-10-05 16:44:49 -04:00
Jack Robison
b2f9ef21cc
use hub binary from https://github.com/lbryio/hub/pull/13 2021-10-05 16:44:49 -04:00
Jack Robison
be6b72edcd
handle invalid release time 2021-10-05 16:44:49 -04:00
Jack Robison
ece2d1e78a
name and normalized -> claim_name and normalized_name 
-update generated pb files
 2021-10-05 16:44:49 -04:00
Jack Robison
1ee1a5f2a1
fix es sync.py 2021-10-05 16:44:49 -04:00
Jack Robison
a567326853
fix all_claims_producer 2021-10-05 16:44:49 -04:00
Jack Robison
6231861dd6
merge conflicts 2021-10-05 16:44:49 -04:00
Jack Robison
1ff7b77ee0
claim search fixes 2021-10-05 16:44:49 -04:00
Jack Robison
9365708bb2
fix release_time and creation_timestamp 2021-10-05 16:44:49 -04:00
Jack Robison
d23a0a8589
delete unused code 2021-10-05 16:44:49 -04:00
Jack Robison
58ad1f3876
non blocking claim producer 2021-10-05 16:44:49 -04:00
Jack Robison
2138e7ea33
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison
32f8c9e59f
renormalization 2021-10-05 16:44:49 -04:00
Jack Robison
3a16edd8a6
fix trending overflow 2021-10-05 16:44:49 -04:00
Jack Robison
165f3bb270
refactor trending 2021-10-05 16:44:49 -04:00
Jack Robison
0ba75153f3
trending fixes 2021-10-05 16:44:49 -04:00
Jack Robison
db2789990f
make app backward compatible with trending_score 
-update trending decay function to zero out low trending score values faster
 2021-10-05 16:44:49 -04:00
Jack Robison
acaf299bcb
log time to update and decay trending in elasticsearch 2021-10-05 16:44:49 -04:00
Jack Robison
1940301824
skip integrity errors for trending spikes 2021-10-05 16:44:49 -04:00
Jack Robison
34576e880d
update trending in elasticsearch 
-add TrendingPrefixSpike to leveldb
-expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings
 2021-10-05 16:44:49 -04:00
Brendon J. Brewer
65c0668d40
constants 2021-10-05 16:44:49 -04:00
Brendon J. Brewer
53bd2bcbfe
Put trending score into ES 2021-10-05 16:44:49 -04:00
Brendon J. Brewer
388724fccb
Mark claims as touched 2021-10-05 16:44:49 -04:00
Jack Robison
231eabb013
fix non normalized canonical urls 2021-10-05 16:44:49 -04:00
Jack Robison
54903fc2ea
handle unicode error for unnormalized names 2021-10-05 16:44:49 -04:00
Jack Robison
3a1baf0700
prefix db 2021-10-05 16:44:49 -04:00
Brendon J. Brewer
0c0e36b6f8
trending 2021-10-05 16:44:49 -04:00
Jack Robison
234c03db09
fix claims not having non-normalized names 2021-10-05 16:44:49 -04:00
Jack Robison
28aa7da349
merge conflicts 2021-10-05 16:44:49 -04:00
Jack Robison
c51e344b87
fix missing fields in reposts 2021-10-05 16:44:49 -04:00
Jack Robison
54461dfa75
fix merge conflicts and simplify extract_doc 2021-10-05 16:44:49 -04:00
Jack Robison
2d48e93f74
fix bulk es sync 2021-10-05 16:44:49 -04:00
Jack Robison
af22646322
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison
722b42a93e
fix tests 2021-10-05 16:44:49 -04:00
Jack Robison
8f9e7f77a7
handle invalid claim update 2021-10-05 16:44:49 -04:00
Jack Robison
09bb1ba494
fix keeping claim_hash_to_txo and txo_to_claim in sync 2021-10-05 16:44:49 -04:00
Victor Shyba
d4137428ff
implement blocking and filtering 2021-10-05 16:44:49 -04:00
Jack Robison
b4d6c4f5b7
fix _get_pending_claim_name 2021-10-05 16:44:49 -04:00
Jack Robison
ffbe59ece5
fix applying expiration fork 2021-10-05 16:44:49 -04:00
Jack Robison
fab9c90ccb
update iterators to use pack_partial_key 2021-10-05 16:44:49 -04:00
Jack Robison
fb1a774bc4
delete lbry/wallet/server/storage.py 
-expose leveldb lru cache size as `CACHE_MB` hub param
 2021-10-05 16:44:49 -04:00
Jack Robison
98bc7d1e0e
remove dead code 2021-10-05 16:44:49 -04:00
Jack Robison
f7622f24b2
non blocking mempool loop 2021-10-05 16:44:49 -04:00
Jack Robison
f0a195a6d4
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison
180ba27d84
run advance_block in threadpool 2021-10-05 16:44:49 -04:00
Jack Robison
f944671f86
use claim_to_txo cache 2021-10-05 16:44:49 -04:00
Jack Robison
def2903f7d
faster _cached_get_active_amount for claims 
-remove dead code
 2021-10-05 16:44:49 -04:00
Jack Robison
0273a4e839
fix claim search by fee for claims without fees 2021-10-05 16:44:49 -04:00
Jack Robison
f8d2f02c5d
clear claim_to_txo cache before reading 2021-10-05 16:44:49 -04:00
Jack Robison
25147d8897
handle claims that dont exist in ES sync 2021-10-05 16:44:49 -04:00
Jack Robison
0fb6f05fba
in memory claim_to_txo and txo_to_claim dictionaries 2021-10-05 16:44:49 -04:00
Jack Robison
4e4e899356
fix spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison
5a01dbf269
split flush from advance_block 2021-10-05 16:44:49 -04:00
Jack Robison
30b923b283
rename extend_ops 2021-10-05 16:44:49 -04:00
Jack Robison
73ba381d20
faster spend_utxo 2021-10-05 16:44:49 -04:00
Jack Robison
1a5912877e
faster get_future_activated 2021-10-05 16:44:49 -04:00
Jack Robison
813e506b68
threadpool 2021-10-05 16:44:49 -04:00
Jack Robison
077ca987f7
cleanup 2021-10-05 16:44:49 -04:00
Jack Robison
c632a7a6a5
fix getting block hash during reorg 2021-10-05 16:44:49 -04:00
Jack Robison
e33e767510
fix test 2021-10-05 16:44:49 -04:00
Jack Robison
ac82617aa9
fix spends in address histories 2021-10-05 16:44:49 -04:00
Jack Robison
a35dfd1fd1
faster es sync 2021-10-05 16:44:49 -04:00
Jack Robison
c28aae9913
fix expiring channels 2021-10-05 16:44:49 -04:00
Jack Robison
c26a99e65c
fix abandoning signed claims in the same tx as their channel 
-fix canonical/short url in es
 2021-10-05 16:44:49 -04:00
Jack Robison
ca57dcfc2f
handle failure to generate a short id 2021-10-05 16:44:49 -04:00
Jack Robison
df5662dd69
fix resolve by short id 2021-10-05 16:44:49 -04:00
Jack Robison
1ac7831f3c
move MemPool into BlockProcessor 2021-10-05 16:44:49 -04:00
Jack Robison
292d272a94
combine MemPool and Notifications classes 2021-10-05 16:44:49 -04:00
Jack Robison
a6ee8dc66e
fix touched hashXs notifications 2021-10-05 16:44:49 -04:00
Jack Robison
496f89f184
reorg claims in the search index 2021-10-05 16:44:49 -04:00
Jack Robison
7a56eff1ac
small fixes 2021-10-05 16:44:49 -04:00