Jack Robison
7531401623
keep touched_or_deleted records
2021-11-21 13:52:03 -05:00
FemtosecondLaser
e6c1dc251e
changed addTimeout to add_timeout for lint compliance
2021-11-20 00:47:46 +00:00
FemtosecondLaser
dca7977051
added timeout of async operations to integration test setup/teardown
2021-11-20 00:22:25 +00:00
Brendon J. Brewer
3f6fe995b8
Rename trending
2021-11-16 10:59:10 -05:00
Jack Robison
1e00fb369d
fix missing es notification for support amount changing
2021-11-15 00:58:18 -05:00
Jack Robison
54b522383a
improve tests
2021-11-15 00:58:18 -05:00
Jack Robison
3fe1582432
fix duplicate trending notification to ES
2021-11-15 00:58:18 -05:00
Jack Robison
85eddd2100
fix effective amount for resolve/ES being off while claims/supports are unactivated
2021-11-15 00:58:18 -05:00
Jack Robison
0ca98678f7
update default tcp/blob port to be the same as the default udp/dht port (4444)
2021-11-10 13:02:28 -03:00
Victor Shyba
a19060c08d
log unexpected errors, rename task/loop
2021-11-09 14:27:06 -05:00
Victor Shyba
fa2ad88cc4
clear cache on test assertions
2021-11-09 14:27:06 -05:00
Victor Shyba
63cbcd0956
make sure the downloader always stops gracefully
2021-11-09 14:27:06 -05:00
Victor Shyba
d6d0ebf8f4
cache space stats from running components so status is instant
2021-11-09 14:27:06 -05:00
Victor Shyba
0d810d92ca
add index for blob table so size summaries are faster
2021-11-09 14:27:06 -05:00
Victor Shyba
1ff914a6f4
download from stored announcements and dont reannounce
2021-11-09 14:27:06 -05:00
Victor Shyba
5959b1be72
improve disk space manager status, include more info and unify space queries
2021-11-09 14:27:06 -05:00
Victor Shyba
d12a214c05
normal_blobs->stream_blobs, proactive->background
2021-11-09 14:27:06 -05:00
Victor Shyba
3a83052f2e
fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full
2021-11-09 14:27:06 -05:00
Victor Shyba
510b44ca92
move more logic out of the downloader component
2021-11-09 14:27:06 -05:00
Victor Shyba
15edb6756d
extract background downloader to its own class
2021-11-09 14:27:06 -05:00
Victor Shyba
fbfd02b08b
add analytics event for network disk space
2021-11-09 14:27:06 -05:00
Victor Shyba
b39c26fc86
announce orphan blobs manually, as that was done when save stream
2021-11-09 14:27:06 -05:00
Victor Shyba
95b2c8d175
cleanup background downloader blobs from conf
2021-11-09 14:27:06 -05:00
Victor Shyba
d52748b09f
separated network seeding space metrics
2021-11-09 14:27:06 -05:00
Victor Shyba
34d18a3a9a
don't save streams for network blobs and bypass disk space manager
2021-11-09 14:27:06 -05:00
Victor Shyba
3b27d6a9b5
add conf for network seeding space limit
2021-11-09 14:27:06 -05:00
Victor Shyba
703c391f99
schedule the download task instead
2021-11-09 14:27:06 -05:00
Victor Shyba
13667df374
download from DHT
2021-11-09 14:27:06 -05:00
Victor Shyba
8800d6985f
drop channel support, prepare to hook into DHT
2021-11-09 14:27:06 -05:00
Victor Shyba
364b8f2605
handle case where something that isn't a sd blob gets hit
2021-11-09 14:27:06 -05:00
Victor Shyba
67b9ea9deb
no api yet
2021-11-09 14:27:06 -05:00
Victor Shyba
b78f2336a7
download only blobs
2021-11-09 14:27:06 -05:00
Victor Shyba
23a5ce3df7
fix exception arguments
2021-11-09 14:27:06 -05:00
Victor Shyba
8f88e28e50
test add/remove/list subscriptions
2021-11-09 14:27:06 -05:00
Victor Shyba
9cf6139557
fix and test main api
2021-11-09 14:27:06 -05:00
Victor Shyba
d556065a8b
download all blobs and check that on tests
2021-11-09 14:27:06 -05:00
Victor Shyba
951716f7dc
create downloader component and initial tests
2021-11-09 14:27:06 -05:00
Victor Shyba
c42b76dcb8
dont lose results on duplicates, just warn
2021-11-08 10:50:47 -05:00
Victor Shyba
a73582d9ae
remove tried_for_this_blob so banned peers are retried for same blob
2021-11-08 10:50:47 -05:00
Cristian Vicas
42c4fc7557
Bug [ #2070 ] where blob_get RPC timed out.
...
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
Bug [#2070 ] where blob_get RPC timed out.
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
2021-11-08 10:49:48 -05:00
Jack Robison
ddbbb6f1dd
use mempool cache in transaction_get_batch
2021-10-27 20:19:08 -04:00
Lex Berezhny
ff21a92330
Merge pull request #3457 from FemtosecondLaser/feature/3270-check-default-download-dir-writable
...
Modified ensure_directory_exists() to check if the directory is writable by the process.
2021-10-27 11:00:13 -04:00
Jack Robison
1e391d211b
fix attempting to update trending on abandoned claims
2021-10-23 18:39:04 -04:00
Jack Robison
1a74d6604d
skip loading tx/claim caches in the elastic sync script when not needed
2021-10-22 15:10:35 -04:00
Jack Robison
48505c2968
update trending with help from @eggplantbren
2021-10-21 00:17:12 -04:00
Jack Robison
a98ea1e66a
update sync script to handle ES falling behind leveldb on shutdown
2021-10-20 23:41:11 -04:00
Jack Robison
3dec697816
logging
2021-10-20 23:41:11 -04:00
Jack Robison
b05d071a1c
update Env to accept parameters from cli args
2021-10-20 23:41:11 -04:00
Jack Robison
a27d3b9689
set default CACHE_MB
to 1024mb and the default QUERY_TIMEOUT_MS
to 10s
2021-10-20 23:41:11 -04:00
Jack Robison
1facc0cd01
remove unused hub env settings
2021-10-20 23:41:11 -04:00
FemtosecondLaser
6b8d4a444b
Modified ensure_directory_exists() to check if the directory is writable by the process.
2021-10-20 15:26:16 +01:00
Jack Robison
6bef09a3b1
update lbry-hub-elastic-sync to support resyncing recent blocks
2021-10-19 15:53:20 -04:00
Jack Robison
e35319e5a2
add CACHE_ALL_CLAIM_TXOS
hub setting
2021-10-19 15:53:20 -04:00
Jack Robison
0e548b3812
remove dead code
2021-10-19 15:53:20 -04:00
Jack Robison
bfac02ccab
add CACHE_ALL_TX_HASHES
setting to optionally use more memory to save i/o
2021-10-19 15:53:20 -04:00
Jack Robison
7ea1a2b361
sleeps
2021-10-19 15:53:20 -04:00
Jack Robison
99df418f1d
improve resolve caching
2021-10-19 15:53:20 -04:00
Jack Robison
6416d8ce9c
threadpools for block processor and es sync reader
2021-10-19 15:53:20 -04:00
Jack Robison
22b43a2b01
doc strings
2021-10-19 15:53:20 -04:00
Jack Robison
05e5d24c5e
improve claims_producer performance
2021-10-19 15:53:20 -04:00
Jack Robison
eabcc30367
resolve lru cache
2021-10-19 15:53:20 -04:00
Jack Robison
f5e0ef5223
add block_txs index
2021-10-19 15:53:20 -04:00
Jack Robison
f46d9330b0
smaller caches
2021-10-19 15:53:20 -04:00
Jack Robison
b62a0b4607
Update daemon.py
...
docstring
2021-10-15 09:40:15 -04:00
Cristian Vicas
1f044321fb
Updated documentation for RPC calls: status, blob_list.
2021-10-15 09:40:15 -04:00
belikor
9509acc490
file_manager: raise new InvalidStreamURLError
if the URL is invalid
...
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.
For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```
```
Traceback (most recent call last):
File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
if not URL.parse(uri).has_stream:
File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```
Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```
This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
2021-10-15 08:59:37 -04:00
Jack Robison
d3516f299e
clear es attributes during initial sync
2021-10-08 16:34:48 -04:00
Jack Robison
79630767c2
fix setting references on txos in extra_txos
2021-10-08 16:34:15 -04:00
Jack Robison
084a76d075
fix reposted channel being missing from resolve result
...
-improve names of the resolve related methods in `LevelDB`
2021-10-07 15:09:13 -04:00
Jack Robison
43432a9e48
fix compactify script
2021-10-07 00:37:55 -04:00
Jack Robison
d64a5bc12f
fix test
2021-10-06 23:53:17 -04:00
Jack Robison
ccf03fc07b
only save undo info for blocks within reorg limit
2021-10-06 12:07:42 -04:00
Jack Robison
a7c45da10c
fix channel count
2021-10-06 00:02:16 -04:00
Jack Robison
8167af9b4a
sort touched or deleted claim hashes
2021-10-05 16:44:49 -04:00
Jack Robison
4cf76123e5
block processor db refactoring
...
-access db through HubDB class, don't use plyvel.DB directly
-add channel count and support amount prefixes
2021-10-05 16:44:49 -04:00
Jack Robison
01ee4b23e6
fix and add test for abandoning a controlling in the same block a new claim is made
2021-10-05 16:44:49 -04:00
Jack Robison
09db868a28
fix ES index name so it stays the same within a test case
2021-10-05 16:44:49 -04:00
Jack Robison
33e8ef75ff
fix bug with early takeover by an update
2021-10-05 16:44:49 -04:00
Jack Robison
86f21da28b
fix activating non existent claim
2021-10-05 16:44:49 -04:00
Jack Robison
89cd6a9aa4
add tests for takeovers from amount changes in updates before/on/after activation
2021-10-05 16:44:49 -04:00
Jack Robison
18e1256037
batch address history notifications
2021-10-05 16:44:49 -04:00
Jack Robison
02cf478d91
improve leveldb caching
2021-10-05 16:44:49 -04:00
Jack Robison
6ec70192fe
refactor reload_blocking_filtering_streams
2021-10-05 16:44:49 -04:00
Jack Robison
8c75098a9a
fix filtering error upon abandon
2021-10-05 16:44:49 -04:00
Jack Robison
72500f6948
faster read_claim_txos
2021-10-05 16:44:49 -04:00
Jack Robison
37ec9ab464
remove unused executor
2021-10-05 16:44:49 -04:00
Victor Shyba
82fe2a4c8d
fix blocking and filtering
2021-10-05 16:44:49 -04:00
Jack Robison
91a07cfaee
fix logging number of notified sessions
2021-10-05 16:44:49 -04:00
Jack Robison
709f5e9a65
fix update that initiates takeover not being delayed
2021-10-05 16:44:49 -04:00
Jack Robison
b2f9ef21cc
use hub binary from https://github.com/lbryio/hub/pull/13
2021-10-05 16:44:49 -04:00
Jack Robison
be6b72edcd
handle invalid release time
2021-10-05 16:44:49 -04:00
Jack Robison
ece2d1e78a
name
and normalized
-> claim_name
and normalized_name
...
-update generated pb files
2021-10-05 16:44:49 -04:00
Jack Robison
1ee1a5f2a1
fix es sync.py
2021-10-05 16:44:49 -04:00
Jack Robison
a567326853
fix all_claims_producer
2021-10-05 16:44:49 -04:00
Jack Robison
6231861dd6
merge conflicts
2021-10-05 16:44:49 -04:00
Jack Robison
1ff7b77ee0
claim search fixes
2021-10-05 16:44:49 -04:00
Jack Robison
9365708bb2
fix release_time and creation_timestamp
2021-10-05 16:44:49 -04:00
Jack Robison
d23a0a8589
delete unused code
2021-10-05 16:44:49 -04:00
Jack Robison
58ad1f3876
non blocking claim producer
2021-10-05 16:44:49 -04:00
Jack Robison
2138e7ea33
fix tests
2021-10-05 16:44:49 -04:00
Jack Robison
32f8c9e59f
renormalization
2021-10-05 16:44:49 -04:00
Jack Robison
3a16edd8a6
fix trending overflow
2021-10-05 16:44:49 -04:00
Jack Robison
165f3bb270
refactor trending
2021-10-05 16:44:49 -04:00
Jack Robison
0ba75153f3
trending fixes
2021-10-05 16:44:49 -04:00
Jack Robison
db2789990f
make app backward compatible with trending_score
...
-update trending decay function to zero out low trending score values faster
2021-10-05 16:44:49 -04:00
Jack Robison
acaf299bcb
log time to update and decay trending in elasticsearch
2021-10-05 16:44:49 -04:00
Jack Robison
1940301824
skip integrity errors for trending spikes
2021-10-05 16:44:49 -04:00
Jack Robison
34576e880d
update trending in elasticsearch
...
-add TrendingPrefixSpike to leveldb
-expose `TRENDING_HALF_LIFE`, `TRENDING_WHALE_HALF_LIFE` and `TRENDING_WHALE_THRESHOLD` hub settings
2021-10-05 16:44:49 -04:00
Brendon J. Brewer
65c0668d40
constants
2021-10-05 16:44:49 -04:00
Brendon J. Brewer
53bd2bcbfe
Put trending score into ES
2021-10-05 16:44:49 -04:00
Brendon J. Brewer
388724fccb
Mark claims as touched
2021-10-05 16:44:49 -04:00
Jack Robison
231eabb013
fix non normalized canonical urls
2021-10-05 16:44:49 -04:00
Jack Robison
54903fc2ea
handle unicode error for unnormalized names
2021-10-05 16:44:49 -04:00
Jack Robison
3a1baf0700
prefix db
2021-10-05 16:44:49 -04:00
Brendon J. Brewer
0c0e36b6f8
trending
2021-10-05 16:44:49 -04:00
Jack Robison
234c03db09
fix claims not having non-normalized names
2021-10-05 16:44:49 -04:00
Jack Robison
28aa7da349
merge conflicts
2021-10-05 16:44:49 -04:00
Jack Robison
c51e344b87
fix missing fields in reposts
2021-10-05 16:44:49 -04:00
Jack Robison
54461dfa75
fix merge conflicts and simplify extract_doc
2021-10-05 16:44:49 -04:00
Jack Robison
2d48e93f74
fix bulk es sync
2021-10-05 16:44:49 -04:00
Jack Robison
af22646322
fix tests
2021-10-05 16:44:49 -04:00
Jack Robison
722b42a93e
fix tests
2021-10-05 16:44:49 -04:00
Jack Robison
8f9e7f77a7
handle invalid claim update
2021-10-05 16:44:49 -04:00
Jack Robison
09bb1ba494
fix keeping claim_hash_to_txo and txo_to_claim in sync
2021-10-05 16:44:49 -04:00
Victor Shyba
d4137428ff
implement blocking and filtering
2021-10-05 16:44:49 -04:00
Jack Robison
b4d6c4f5b7
fix _get_pending_claim_name
2021-10-05 16:44:49 -04:00
Jack Robison
ffbe59ece5
fix applying expiration fork
2021-10-05 16:44:49 -04:00
Jack Robison
fab9c90ccb
update iterators to use pack_partial_key
2021-10-05 16:44:49 -04:00
Jack Robison
fb1a774bc4
delete lbry/wallet/server/storage.py
...
-expose leveldb lru cache size as `CACHE_MB` hub param
2021-10-05 16:44:49 -04:00
Jack Robison
98bc7d1e0e
remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison
f7622f24b2
non blocking mempool loop
2021-10-05 16:44:49 -04:00
Jack Robison
f0a195a6d4
faster es sync
2021-10-05 16:44:49 -04:00
Jack Robison
180ba27d84
run advance_block in threadpool
2021-10-05 16:44:49 -04:00
Jack Robison
f944671f86
use claim_to_txo cache
2021-10-05 16:44:49 -04:00
Jack Robison
def2903f7d
faster _cached_get_active_amount for claims
...
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison
0273a4e839
fix claim search by fee for claims without fees
2021-10-05 16:44:49 -04:00
Jack Robison
f8d2f02c5d
clear claim_to_txo cache before reading
2021-10-05 16:44:49 -04:00
Jack Robison
25147d8897
handle claims that dont exist in ES sync
2021-10-05 16:44:49 -04:00
Jack Robison
0fb6f05fba
in memory claim_to_txo and txo_to_claim dictionaries
2021-10-05 16:44:49 -04:00
Jack Robison
4e4e899356
fix spend_utxo
2021-10-05 16:44:49 -04:00
Jack Robison
5a01dbf269
split flush from advance_block
2021-10-05 16:44:49 -04:00
Jack Robison
30b923b283
rename extend_ops
2021-10-05 16:44:49 -04:00
Jack Robison
73ba381d20
faster spend_utxo
2021-10-05 16:44:49 -04:00
Jack Robison
1a5912877e
faster get_future_activated
2021-10-05 16:44:49 -04:00
Jack Robison
813e506b68
threadpool
2021-10-05 16:44:49 -04:00
Jack Robison
077ca987f7
cleanup
2021-10-05 16:44:49 -04:00
Jack Robison
c632a7a6a5
fix getting block hash during reorg
2021-10-05 16:44:49 -04:00
Jack Robison
e33e767510
fix test
2021-10-05 16:44:49 -04:00
Jack Robison
ac82617aa9
fix spends in address histories
2021-10-05 16:44:49 -04:00
Jack Robison
a35dfd1fd1
faster es sync
2021-10-05 16:44:49 -04:00
Jack Robison
c28aae9913
fix expiring channels
2021-10-05 16:44:49 -04:00
Jack Robison
c26a99e65c
fix abandoning signed claims in the same tx as their channel
...
-fix canonical/short url in es
2021-10-05 16:44:49 -04:00
Jack Robison
ca57dcfc2f
handle failure to generate a short id
2021-10-05 16:44:49 -04:00
Jack Robison
df5662dd69
fix resolve by short id
2021-10-05 16:44:49 -04:00
Jack Robison
1ac7831f3c
move MemPool into BlockProcessor
2021-10-05 16:44:49 -04:00
Jack Robison
292d272a94
combine MemPool and Notifications classes
2021-10-05 16:44:49 -04:00
Jack Robison
a6ee8dc66e
fix touched hashXs notifications
2021-10-05 16:44:49 -04:00
Jack Robison
496f89f184
reorg claims in the search index
2021-10-05 16:44:49 -04:00
Jack Robison
7a56eff1ac
small fixes
2021-10-05 16:44:49 -04:00
Jack Robison
07e182aa16
rename
2021-10-05 16:44:49 -04:00
Jack Robison
7de06aa1e0
delete stale code
2021-10-05 16:44:49 -04:00
Jack Robison
3955b64405
simplify advance and reorg
2021-10-05 16:44:49 -04:00
Jack Robison
2bb55d681d
update limited_history
2021-10-05 16:44:49 -04:00
Jack Robison
f94e6ac527
update lookup_utxos
2021-10-05 16:44:49 -04:00
Jack Robison
b344f17b86
update RevertableOpStack
2021-10-05 16:44:49 -04:00
Jack Robison
677b8cb633
add remaining db prefixes
2021-10-05 16:44:49 -04:00
Jack Robison
a1ddd762e0
cleanup
2021-10-05 16:44:49 -04:00
Jack Robison
68474e4057
skip es sync during initial hub sync, halt the hub upon finishing initial sync
2021-10-05 16:44:49 -04:00
Jack Robison
a84b9ee396
fix es sync
2021-10-05 16:44:49 -04:00
Jack Robison
b9c2ee745a
fix non localhost elasticsearch
2021-10-05 16:44:49 -04:00
Jack Robison
615e489d8d
fix stream_update
--clear_channel flag
2021-10-05 16:44:49 -04:00
Jack Robison
c68f9f6f16
fix signed claim invalidation corner cases
2021-10-05 16:44:49 -04:00
Jack Robison
229cb85a6a
extra deletes
...
-the channel_to_claim/claim_to_channel entries already get deleted when the claim txo is spent
2021-10-05 16:44:49 -04:00
Jack Robison
e5c22fa665
fix has_no_source for reposts
2021-10-05 16:44:49 -04:00
Jack Robison
8bcfff05d7
update channel_to_claim and claim_to_channel at the same time
2021-10-05 16:44:49 -04:00
Jack Robison
6416ee8151
typing and fix error string
2021-10-05 16:44:49 -04:00
Jack Robison
f8eceb48e6
update staged txo_to_claim after invalidating channel sig
...
-fixes abandon of claim with invalidated signature and an update in same block
2021-10-05 16:44:49 -04:00
Jack Robison
310c483bfa
missing channel_to_claim delete
2021-10-05 16:44:49 -04:00
Jack Robison
a8f20361aa
fix RepostKey
2021-10-05 16:44:49 -04:00
Jack Robison
290be69d99
typing
2021-10-05 16:44:49 -04:00
Jack Robison
3b96bd7ea0
fix
2021-10-05 16:44:49 -04:00
Jack Robison
dc2f22f5fa
cleanup
2021-10-05 16:44:49 -04:00
Jack Robison
821be29f41
rename effective_amount prefix
2021-10-05 16:44:49 -04:00
Jack Robison
52ff1a12ff
fix undeleted claim_to_channel record
2021-10-05 16:44:49 -04:00
Jack Robison
814699ef11
cleanup
2021-10-05 16:44:49 -04:00
Jack Robison
0c30838b25
fix mismatch in claim_to_txo<->txo_to_claim
2021-10-05 16:44:49 -04:00
Jack Robison
cf66c2a1ee
rename things
...
-fix effective amount integrity error
2021-10-05 16:44:49 -04:00
Jack Robison
2ee419ffca
fix
2021-10-05 16:44:49 -04:00
Jack Robison
bfb9d696d7
pretty print
2021-10-05 16:44:49 -04:00
Jack Robison
bb2a34dd6b
fix duplicate activate
2021-10-05 16:44:49 -04:00
Jack Robison
ed652c0c56
fix updating resolve by effective amount after abandoning support
2021-10-05 16:44:49 -04:00
Jack Robison
1dc961d6eb
use RevertableOpStack in _get_takeover_ops
2021-10-05 16:44:49 -04:00
Jack Robison
d119fcfc98
remove debug prints
2021-10-05 16:44:49 -04:00
Jack Robison
4d3573724a
add RevertableOpStack to verify consistency of ops as they're staged
2021-10-05 16:44:49 -04:00
Jack Robison
8b37a66075
fix fee amount overflow in es
2021-10-05 16:44:49 -04:00
Jack Robison
ba4f32075a
faster claim producer
...
-make batches of claim txos from the iterator, and sort by tx hash before fetching to maximize cache and read ahead hits
2021-10-05 16:44:49 -04:00
Jack Robison
218be22576
imports
2021-10-05 16:44:49 -04:00
Jack Robison
7688293716
close db in sync script
2021-10-05 16:44:49 -04:00
Jack Robison
458f8533c4
try default block size
2021-10-05 16:44:49 -04:00
Jack Robison
34502752fc
update elastic sync
2021-10-05 16:44:49 -04:00