Commit graph

1841 commits

Author SHA1 Message Date
Victor Shyba
02994164be fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full 2022-03-15 15:34:09 -03:00
Victor Shyba
c3b89a896b move more logic out of the downloader component 2022-03-15 15:34:09 -03:00
Victor Shyba
b0bf6eff16 extract background downloader to its own class 2022-03-15 15:34:09 -03:00
Victor Shyba
47c4cb800d add analytics event for network disk space 2022-03-15 15:34:09 -03:00
Victor Shyba
c5f6a4ca1c announce orphan blobs manually, as that was done when save stream 2022-03-15 15:34:09 -03:00
Victor Shyba
813108b9d8 cleanup background downloader blobs from conf 2022-03-15 15:34:09 -03:00
Victor Shyba
95703b3af5 separated network seeding space metrics 2022-03-15 15:34:09 -03:00
Victor Shyba
2ceec7c3d4 don't save streams for network blobs and bypass disk space manager 2022-03-15 15:34:09 -03:00
Victor Shyba
3985f1578e add conf for network seeding space limit 2022-03-15 15:34:09 -03:00
Victor Shyba
355ccb1e32 schedule the download task instead 2022-03-15 15:34:09 -03:00
Victor Shyba
8d6a7101f6 download from DHT 2022-03-15 15:34:09 -03:00
Victor Shyba
abbd8473bb drop channel support, prepare to hook into DHT 2022-03-15 15:34:09 -03:00
Victor Shyba
02b2103d94 handle case where something that isn't a sd blob gets hit 2022-03-15 15:34:09 -03:00
Victor Shyba
85474ae381 no api yet 2022-03-15 15:34:09 -03:00
Victor Shyba
9d3c401abb download only blobs 2022-03-15 15:34:09 -03:00
Victor Shyba
602cd5bd2e fix exception arguments 2022-03-15 15:34:09 -03:00
Victor Shyba
4b88b191e7 test add/remove/list subscriptions 2022-03-15 15:34:09 -03:00
Victor Shyba
e449cd01ba fix and test main api 2022-03-15 15:34:08 -03:00
Victor Shyba
b1db42acf4 download all blobs and check that on tests 2022-03-15 15:34:08 -03:00
Victor Shyba
fcbe8cf00b create downloader component and initial tests 2022-03-15 15:34:08 -03:00
Victor Shyba
7c1c04674c dont lose results on duplicates, just warn 2022-03-15 15:34:08 -03:00
Victor Shyba
694aeab75c remove tried_for_this_blob so banned peers are retried for same blob 2022-03-15 15:34:08 -03:00
Cristian Vicas
90aaf64b77 Bug [#2070] where blob_get RPC timed out.
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.

Bug [#2070] where blob_get RPC timed out.

Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
2022-03-15 15:34:08 -03:00
Jack Robison
6eeabb1a1a use mempool cache in transaction_get_batch 2022-03-15 15:34:08 -03:00
FemtosecondLaser
bed2e253b2 Modified ensure_directory_exists() to check if the directory is writable by the process. 2022-03-15 15:34:08 -03:00
Jack Robison
9029b4b5c3 fix attempting to update trending on abandoned claims 2022-03-15 15:34:08 -03:00
Jack Robison
b3705073a3 skip loading tx/claim caches in the elastic sync script when not needed 2022-03-15 15:34:08 -03:00
Jack Robison
755e8ce101 update trending with help from @eggplantbren 2022-03-15 15:34:08 -03:00
Jack Robison
fd2ab47a16 update sync script to handle ES falling behind leveldb on shutdown 2022-03-15 15:34:08 -03:00
Jack Robison
9aafb7a743 logging 2022-03-15 15:34:08 -03:00
Jack Robison
3cd2227c29 update Env to accept parameters from cli args 2022-03-15 15:34:08 -03:00
Jack Robison
27cc83c03b set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2022-03-15 15:34:08 -03:00
Jack Robison
cb6db3f3d8 remove unused hub env settings 2022-03-15 15:34:08 -03:00
Jack Robison
9020e39a83 update lbry-hub-elastic-sync to support resyncing recent blocks 2022-03-15 15:34:08 -03:00
Jack Robison
64509ca95d add CACHE_ALL_CLAIM_TXOS hub setting 2022-03-15 15:34:08 -03:00
Jack Robison
dafd62104b remove dead code 2022-03-15 15:34:08 -03:00
Jack Robison
e09588e433 add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2022-03-15 15:34:08 -03:00
Jack Robison
50e00192e8 sleeps 2022-03-15 15:34:08 -03:00
Jack Robison
e6d470f110 improve resolve caching 2022-03-15 15:34:08 -03:00
Jack Robison
eeaf9a72e2 threadpools for block processor and es sync reader 2022-03-15 15:34:08 -03:00
Jack Robison
a4ad1bb0a9 doc strings 2022-03-15 15:34:08 -03:00
Jack Robison
e0086682b9 improve claims_producer performance 2022-03-15 15:34:08 -03:00
Jack Robison
15ac2ade59 resolve lru cache 2022-03-15 15:34:08 -03:00
Jack Robison
09e0d5c55e add block_txs index 2022-03-15 15:34:08 -03:00
Jack Robison
dfe855c0c9 smaller caches 2022-03-15 15:34:08 -03:00
Jack Robison
a53b454d86 Update daemon.py
docstring
2022-03-15 15:34:08 -03:00
Cristian Vicas
01557b599a Updated documentation for RPC calls: status, blob_list. 2022-03-15 15:34:08 -03:00
belikor
65323b4169 file_manager: raise new InvalidStreamURLError if the URL is invalid
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
2022-03-15 15:34:08 -03:00
Jack Robison
cdef8b4852 clear es attributes during initial sync 2022-03-15 15:34:08 -03:00
Jack Robison
8d72142390 fix setting references on txos in extra_txos 2022-03-15 15:34:08 -03:00