LBRYCommunity/lbry-sdk
1
0
Fork 1
Code Issues Pull requests Packages Projects Releases 282 Wiki Activity
9005 commits 110 branches 641 tags 31 MiB
Commit graph

2026 commits

Author SHA1 Message Date
Victor Shyba 03b59ac6fc dont set head blob to announce on save 2022-02-20 22:33:57 -03:00
Victor Shyba 43ac3336d7 break tie by length 2022-02-20 22:24:04 -03:00
Victor Shyba d12c78db74 fix and test case for blob_clean after disabling network storage 2022-02-20 22:24:04 -03:00
Victor Shyba bb60c385d5 put back all the peers, get rid of re_add 2022-02-08 21:41:52 -03:00
Alex Grintsvayg 7c7a0d4bdf
let stream_update work on non-stream claims 2022-02-08 09:28:17 -05:00
Jeffrey Picard e0ea6383e2
Update __init__.py 
Update go hub binary to fix es sync test.
 2022-02-04 12:17:19 -05:00
Lex Berezhny cba9c16a06 fix 2022-02-04 12:07:41 -05:00
Lex Berezhny dd68fb077b prevent creation of change which is below the dust threshold of 1000 dewies 2022-02-04 12:07:41 -05:00
Victor Shyba c0f512ace7 bump DHT peer manager cache to 16384 2022-02-02 16:54:42 -03:00
Victor Shyba c9d637b4da add gauge for queue size 2022-02-02 11:56:42 -03:00
Victor Shyba ae3e8fadf5 count announcements and how many peers we were able to announce to 2022-02-02 11:56:42 -03:00
Eugene Dubinin 9b463a8cab adds tests for guess_media_type 
removes unnecessary comments
 2022-01-29 20:49:42 +02:00
Eugene Dubinin babc54a240 adjusts code style 2022-01-29 15:25:17 +02:00
Eugene Dubinin 5836a93b21 fixes KeyError on missing synonyms 2022-01-29 15:25:17 +02:00
Eugene Dubinin 557348e345 detect media_type from the file contents 2022-01-29 15:25:17 +02:00
Lex Berezhny 3a496902f8 wallet locking/unlocking no longer breaks deterministic channel keys 2022-01-24 09:45:08 -05:00
Lex Berezhny 302461b446 updated based on code review 2022-01-17 11:08:28 -05:00
Lex Berezhny ac201c718e drop dependency on cryptography library in wallet module 2022-01-17 10:43:59 -05:00
Victor Shyba 0618053bd4 remove request_flight metric 2022-01-12 12:41:04 -03:00
Victor Shyba fd9dcbf9a8 add granular metric for stored blob prefix, for network announcements calculation 2022-01-12 12:39:23 -03:00
Victor Shyba beb8583436 change colliding bits metric to gauge 2022-01-12 12:39:23 -03:00
Victor Shyba b44e2c0b38 count bit collisions between 8 and 16 2022-01-12 12:39:23 -03:00
Victor Shyba 06e94640b5 add counter for peers with colliding bytes 2022-01-12 12:39:23 -03:00
Victor Shyba ff36bdc802 add requests in flight and error 2022-01-12 12:39:23 -03:00
Victor Shyba 46f576de46 add request received 2022-01-12 12:39:23 -03:00
Victor Shyba 7b09c34fce add request_sent and request_time metric on dht 2022-01-12 12:39:23 -03:00
Victor Shyba a22f50aa84 add storing_peers and peer_manager_keys 2022-01-12 12:39:23 -03:00
Victor Shyba 2d9130b4e0 prometheus: move blobs_stored and peers to SDK. add buckets_in_routing_table 2022-01-12 12:39:23 -03:00
Victor Shyba 5cb4c06d0c add prefix_neighbors_count to routing table debug api 2022-01-12 12:39:23 -03:00
Lex Berezhny aaa11c02bf added integration test 2022-01-10 08:46:10 -05:00
vertbyqb d2ebbf5db6 jsonrpc_channel_sign - Convert hexdata to a string before signing 
Fixes #3533
 2022-01-10 08:46:10 -05:00
Victor Shyba a8523996a9 extract cache values, increase peer cache to 2048 2022-01-07 12:58:52 -03:00
Victor Shyba f586de2bbe DHT bugfix: failures tracking should be bound to 2048 LRU cache size 2022-01-07 12:46:00 -03:00
Victor Shyba 7df02303b2 fix missing docopt argument 2022-01-05 17:10:31 -03:00
Victor Shyba f89c75e642 bump hub version to latest supporting sd_hash search 2022-01-05 17:10:31 -03:00
Victor Shyba d2c1961101 update hub protobuf including sd_hash field 2022-01-05 17:10:31 -03:00
Victor Shyba 2a4c5a48bf increase indexed sd_hash prefix to 4 chars 2022-01-05 17:10:31 -03:00
Victor Shyba 5f5f39a4aa enable and test prefix search for sd hash 2022-01-05 17:10:31 -03:00
Victor Shyba df54cc04af sync and search sd_hash 2022-01-05 17:10:31 -03:00
Victor Shyba 19fa274227 add sd hash to API 2022-01-05 17:10:31 -03:00
Lex Berezhny 4822792ee2 create nondetermnistic channel in test to replicate old test behavior 2021-12-22 18:29:46 -05:00
Lex Berezhny 23c10faff5 lint 2021-12-22 18:29:46 -05:00
Lex Berezhny 1eaa195363 reduced crypto dependency in wallet to coincurve 2021-12-22 18:29:46 -05:00
Lex Berezhny fb57cfa5d8 moved imports for lint 2021-12-22 18:29:46 -05:00
Lex Berezhny d815a6f02c use ecdsa for signing/veryfing instead of coincurve due to compatibility issues 2021-12-22 18:29:46 -05:00
Lex Berezhny 8216f4a873 work in progress 2021-12-22 18:29:46 -05:00
Lex Berezhny e4cc4521d9 channel key generation no longer arbitrarily bounded 2021-12-22 18:29:46 -05:00
Lex Berezhny 6bd9b3744d progress, channel keys generate deterministically now 2021-12-22 18:29:46 -05:00
Lex Berezhny f741b00768 progress on deterministic channel keys 2021-12-22 18:29:46 -05:00
Lex Berezhny e5268f43e7 changes default coin selection strategy from standard to prefer_confirmed 2021-12-21 10:22:09 -05:00
Victor Shyba 54d6fb9da4 do not limit DHT results by K, respect max_results 2021-12-09 14:34:55 -03:00
Victor Shyba 3d5c9cc1c2 clarify DHT debug logging on key and operation 2021-12-09 14:32:30 -03:00
Jeffrey Picard d66f46e07b Switch RangeField back to ints 2021-12-03 18:12:38 -05:00
Jeffrey Picard 3436965b33 Debugging 2021-12-03 17:22:52 -05:00
Jeffrey Picard 1b322dc404
Update protobufs, go hub shim, and claim test. 2021-12-03 13:03:00 -05:00
Jack Robison 58341f4ff1
remove unused ES fields 2021-12-03 13:03:00 -05:00
Jack Robison 0d3ca80008
support lists of constraints for all range fields 2021-12-03 13:03:00 -05:00
Lex Berezhny 63437712cd
Merge pull request #3490 from ghost/integration_test_setup_cleanup_timeouts 
added timeout of async operations to integration test setup/teardown
 2021-12-02 19:52:44 -05:00
Jack Robison 26d0e87f46 v0.106.0 2021-12-02 17:17:00 -05:00
Jack Robison 7bb293e5d6 update claim_search doc 
backward compatibility for `trending_mixed`, `trending_local`, `trending_global`, and `trending_group` args to `claim_search`
 2021-12-02 14:51:52 -05:00
Lex Berezhny e4777f9314
Merge branch 'master' into integration_test_setup_cleanup_timeouts 2021-12-01 22:08:18 -05:00
Jack Robison 1aa66c6038
update header checkpoints 2021-12-01 18:46:24 -05:00
Victor Shyba e7458edb72 test case for stream_type search on claims missing source + fix 2021-12-01 18:42:47 -05:00
Lex Berezhny 9e43060d41 fee per name env var 2021-12-01 10:22:34 -05:00
FemtosecondLaser d69486fb6e returned conditional check in add_timeout() as it was making test_node.py tests unhappy 2021-11-30 01:01:35 +00:00
FemtosecondLaser d4ebfdbc3c removed conditional check in add_timeout() 2021-11-29 22:56:50 +00:00
FemtosecondLaser e00c3db71a
Merge branch 'master' into integration_test_setup_cleanup_timeouts 2021-11-29 21:50:05 +00:00
Jack Robison 7531401623
keep touched_or_deleted records 2021-11-21 13:52:03 -05:00
FemtosecondLaser e6c1dc251e changed addTimeout to add_timeout for lint compliance 2021-11-20 00:47:46 +00:00
FemtosecondLaser dca7977051 added timeout of async operations to integration test setup/teardown 2021-11-20 00:22:25 +00:00
Brendon J. Brewer 3f6fe995b8 Rename trending 2021-11-16 10:59:10 -05:00
Jack Robison 1e00fb369d fix missing es notification for support amount changing 2021-11-15 00:58:18 -05:00
Jack Robison 54b522383a improve tests 2021-11-15 00:58:18 -05:00
Jack Robison 3fe1582432 fix duplicate trending notification to ES 2021-11-15 00:58:18 -05:00
Jack Robison 85eddd2100 fix effective amount for resolve/ES being off while claims/supports are unactivated 2021-11-15 00:58:18 -05:00
Jack Robison 0ca98678f7 update default tcp/blob port to be the same as the default udp/dht port (4444) 2021-11-10 13:02:28 -03:00
Victor Shyba a19060c08d log unexpected errors, rename task/loop 2021-11-09 14:27:06 -05:00
Victor Shyba fa2ad88cc4 clear cache on test assertions 2021-11-09 14:27:06 -05:00
Victor Shyba 63cbcd0956 make sure the downloader always stops gracefully 2021-11-09 14:27:06 -05:00
Victor Shyba d6d0ebf8f4 cache space stats from running components so status is instant 2021-11-09 14:27:06 -05:00
Victor Shyba 0d810d92ca add index for blob table so size summaries are faster 2021-11-09 14:27:06 -05:00
Victor Shyba 1ff914a6f4 download from stored announcements and dont reannounce 2021-11-09 14:27:06 -05:00
Victor Shyba 5959b1be72 improve disk space manager status, include more info and unify space queries 2021-11-09 14:27:06 -05:00
Victor Shyba d12a214c05 normal_blobs->stream_blobs, proactive->background 2021-11-09 14:27:06 -05:00
Victor Shyba 3a83052f2e fix free space calculation, test it and give a margin of 10mb before starting so it doesnt insist when full 2021-11-09 14:27:06 -05:00
Victor Shyba 510b44ca92 move more logic out of the downloader component 2021-11-09 14:27:06 -05:00
Victor Shyba 15edb6756d extract background downloader to its own class 2021-11-09 14:27:06 -05:00
Victor Shyba fbfd02b08b add analytics event for network disk space 2021-11-09 14:27:06 -05:00
Victor Shyba b39c26fc86 announce orphan blobs manually, as that was done when save stream 2021-11-09 14:27:06 -05:00
Victor Shyba 95b2c8d175 cleanup background downloader blobs from conf 2021-11-09 14:27:06 -05:00
Victor Shyba d52748b09f separated network seeding space metrics 2021-11-09 14:27:06 -05:00
Victor Shyba 34d18a3a9a don't save streams for network blobs and bypass disk space manager 2021-11-09 14:27:06 -05:00
Victor Shyba 3b27d6a9b5 add conf for network seeding space limit 2021-11-09 14:27:06 -05:00
Victor Shyba 703c391f99 schedule the download task instead 2021-11-09 14:27:06 -05:00
Victor Shyba 13667df374 download from DHT 2021-11-09 14:27:06 -05:00
Victor Shyba 8800d6985f drop channel support, prepare to hook into DHT 2021-11-09 14:27:06 -05:00
Victor Shyba 364b8f2605 handle case where something that isn't a sd blob gets hit 2021-11-09 14:27:06 -05:00
Victor Shyba 67b9ea9deb no api yet 2021-11-09 14:27:06 -05:00
Victor Shyba b78f2336a7 download only blobs 2021-11-09 14:27:06 -05:00
Victor Shyba 23a5ce3df7 fix exception arguments 2021-11-09 14:27:06 -05:00
Victor Shyba 8f88e28e50 test add/remove/list subscriptions 2021-11-09 14:27:06 -05:00
Victor Shyba 9cf6139557 fix and test main api 2021-11-09 14:27:06 -05:00
Victor Shyba d556065a8b download all blobs and check that on tests 2021-11-09 14:27:06 -05:00
Victor Shyba 951716f7dc create downloader component and initial tests 2021-11-09 14:27:06 -05:00
Victor Shyba c42b76dcb8 dont lose results on duplicates, just warn 2021-11-08 10:50:47 -05:00
Victor Shyba a73582d9ae remove tried_for_this_blob so banned peers are retried for same blob 2021-11-08 10:50:47 -05:00
Cristian Vicas 42c4fc7557 Bug [#2070] where blob_get RPC timed out. 
Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.

Bug [#2070] where blob_get RPC timed out.

Both stream.downloader and blob_exchange.downloader paths are adding the fixed_peers list to the DHT node.
Tested jsonrpc_blob_get daemon call.
 2021-11-08 10:49:48 -05:00
Jack Robison ddbbb6f1dd
use mempool cache in transaction_get_batch 2021-10-27 20:19:08 -04:00
Lex Berezhny ff21a92330
Merge pull request #3457 from FemtosecondLaser/feature/3270-check-default-download-dir-writable 
Modified ensure_directory_exists() to check if the directory is writable by the process.
 2021-10-27 11:00:13 -04:00
Jack Robison 1e391d211b
fix attempting to update trending on abandoned claims 2021-10-23 18:39:04 -04:00
Jack Robison 1a74d6604d skip loading tx/claim caches in the elastic sync script when not needed 2021-10-22 15:10:35 -04:00
Jack Robison 48505c2968 update trending with help from @eggplantbren 2021-10-21 00:17:12 -04:00
Jack Robison a98ea1e66a update sync script to handle ES falling behind leveldb on shutdown 2021-10-20 23:41:11 -04:00
Jack Robison 3dec697816 logging 2021-10-20 23:41:11 -04:00
Jack Robison b05d071a1c update Env to accept parameters from cli args 2021-10-20 23:41:11 -04:00
Jack Robison a27d3b9689 set default CACHE_MB to 1024mb and the default QUERY_TIMEOUT_MS to 10s 2021-10-20 23:41:11 -04:00
Jack Robison 1facc0cd01 remove unused hub env settings 2021-10-20 23:41:11 -04:00
FemtosecondLaser 6b8d4a444b Modified ensure_directory_exists() to check if the directory is writable by the process. 2021-10-20 15:26:16 +01:00
Jack Robison 6bef09a3b1 update lbry-hub-elastic-sync to support resyncing recent blocks 2021-10-19 15:53:20 -04:00
Jack Robison e35319e5a2 add CACHE_ALL_CLAIM_TXOS hub setting 2021-10-19 15:53:20 -04:00
Jack Robison 0e548b3812 remove dead code 2021-10-19 15:53:20 -04:00
Jack Robison bfac02ccab add CACHE_ALL_TX_HASHES setting to optionally use more memory to save i/o 2021-10-19 15:53:20 -04:00
Jack Robison 7ea1a2b361 sleeps 2021-10-19 15:53:20 -04:00
Jack Robison 99df418f1d improve resolve caching 2021-10-19 15:53:20 -04:00
Jack Robison 6416d8ce9c threadpools for block processor and es sync reader 2021-10-19 15:53:20 -04:00
Jack Robison 22b43a2b01 doc strings 2021-10-19 15:53:20 -04:00
Jack Robison 05e5d24c5e improve claims_producer performance 2021-10-19 15:53:20 -04:00
Jack Robison eabcc30367 resolve lru cache 2021-10-19 15:53:20 -04:00
Jack Robison f5e0ef5223 add block_txs index 2021-10-19 15:53:20 -04:00
Jack Robison f46d9330b0 smaller caches 2021-10-19 15:53:20 -04:00
Jack Robison b62a0b4607 Update daemon.py 
docstring
 2021-10-15 09:40:15 -04:00
Cristian Vicas 1f044321fb Updated documentation for RPC calls: status, blob_list. 2021-10-15 09:40:15 -04:00
belikor 9509acc490
file_manager: raise new InvalidStreamURLError if the URL is invalid 
When using `lbrynet get URL`, if the URL is not a valid URL
the function `url.URL.parse` will raise a `ValueError` exception
which will produce a whole backtrace.

For example, this is the case if we provide a channel name
with a forward slash but without a stream name.
```
lbrynet get @Non-existing/
```

```
Traceback (most recent call last):
  File "/opt/git/lbry-sdk/lbry/file/file_manager.py", line 84, in download_from_uri
    if not URL.parse(uri).has_stream:
  File "/opt/git/lbry-sdk/lbry/schema/url.py", line 114, in parse
    raise ValueError('Invalid LBRY URL')
ValueError: Invalid LBRY URL
WARNING  lbry.extras.daemon.daemon:1110: Error downloading Non-existing/: Invalid LBRY URL
```

Now we raise a new `InvalidStreamURLError` which can be trapped in the upper functions
that use `url.URL.parse` such as `FileManager.download_from_uri`.
If we do this the traceback won't be shown.
```
WARNING  lbry.file.file_manager:252:
Failed to download Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
WARNING  lbry.extras.daemon.daemon:1110:
Error downloading Non-existing/: Invalid LBRY stream URL: '@Non-existing/'
```

This handles the case when trying to download only "channel" parts
without the claim part.
```
lbrynet get @Non-existing
lbrynet get @Non-existing/
lbrynet get Non-existing/
```
 2021-10-15 08:59:37 -04:00
Jack Robison d3516f299e
clear es attributes during initial sync 2021-10-08 16:34:48 -04:00
Jack Robison 79630767c2
fix setting references on txos in extra_txos 2021-10-08 16:34:15 -04:00
Jack Robison 084a76d075
fix reposted channel being missing from resolve result 
-improve names of the resolve related methods in `LevelDB`
 2021-10-07 15:09:13 -04:00
Jack Robison 43432a9e48
fix compactify script 2021-10-07 00:37:55 -04:00
Jack Robison d64a5bc12f
fix test 2021-10-06 23:53:17 -04:00
Jack Robison ccf03fc07b
only save undo info for blocks within reorg limit 2021-10-06 12:07:42 -04:00
Jack Robison a7c45da10c
fix channel count 2021-10-06 00:02:16 -04:00
Jack Robison 8167af9b4a
sort touched or deleted claim hashes 2021-10-05 16:44:49 -04:00
Jack Robison 4cf76123e5
block processor db refactoring 
-access db through HubDB class, don't use plyvel.DB directly
-add channel count and support amount prefixes
 2021-10-05 16:44:49 -04:00
Jack Robison 01ee4b23e6
fix and add test for abandoning a controlling in the same block a new claim is made 2021-10-05 16:44:49 -04:00
Jack Robison 09db868a28
fix ES index name so it stays the same within a test case 2021-10-05 16:44:49 -04:00
Jack Robison 33e8ef75ff
fix bug with early takeover by an update 2021-10-05 16:44:49 -04:00
Jack Robison 86f21da28b
fix activating non existent claim 2021-10-05 16:44:49 -04:00
Jack Robison 89cd6a9aa4
add tests for takeovers from amount changes in updates before/on/after activation 2021-10-05 16:44:49 -04:00
Jack Robison 18e1256037
batch address history notifications 2021-10-05 16:44:49 -04:00
Jack Robison 02cf478d91
improve leveldb caching 2021-10-05 16:44:49 -04:00
Jack Robison 6ec70192fe
refactor reload_blocking_filtering_streams 2021-10-05 16:44:49 -04:00