Victor Shyba
8c3552dd34
port reflector
2018-08-24 11:40:43 -04:00
Lex Berezhny
971252d5d1
undo simplejson commit
2018-08-24 11:37:53 -04:00
hackrush
c01716a6c0
Using simplejson and fixed some encodes and decodes
2018-08-24 11:37:53 -04:00
Lex Berezhny
43bef9447c
progress on publish command: py3 porting and integration tests
2018-08-24 11:37:53 -04:00
Jack Robison
4c97d2af2e
fix request_needed_blobs for partially reflected streams
2018-06-15 10:10:49 -04:00
Jack Robison
aa4240de85
track successful reflector uploads in sqlite to minimize how many streams are attempted by auto re-reflect
...
-increase the default `auto_re_reflect_interval` to a day
2018-05-08 14:41:56 -04:00
Jack Robison
9a9b2f47ce
pass sd_hash to reflector client factory instead of looking it up
2018-03-29 09:46:34 -04:00
Jack Robison
3085e28490
don't try to resend failed blobs to reflector
...
-fixes infinite loop where client keeps trying to send failing blobs, which may be failing because they are invalid
-return list of reflected blob hashes from BlobReflectorClient
2018-03-22 16:53:16 -04:00
Jack Robison
679c2f403f
fix incorrectly raised DownloadCanceledError for already verified blobs
...
-make InvalidDataError message clearer
-fix BlobReflectorClient result
2018-03-22 13:23:12 -04:00
Jack Robison
b28bdbd752
rename existing reupload.reflect_stream --> reupload.reflect_file, add a reupload.reflect_stream function
2018-03-19 13:55:11 -04:00
Jack Robison
81de5fbbf4
verify_will_announce_head_and_sd_blobs
2018-03-08 16:46:39 -05:00
Jack Robison
9a8cac20e7
update reflector client and server to use SQLiteStorage
2018-02-13 10:27:51 -05:00
Jack Robison
64e04f8a68
refactor start_lbry_file
2018-01-04 21:28:09 -05:00
Jack Robison
bf46f5616b
Merge branch 'add_lbry_file_manager_to_reflector_2'
2017-12-29 12:39:50 -05:00
Jack Robison
a6f0c5fb4c
pylint fixes
...
fix exception formatting and a circular import
2017-12-15 20:25:20 -05:00
Alex Grintsvayg
ebed4f010b
fixes #1029 - stupid error messages
2017-12-13 13:56:44 -05:00
Jack Robison
329d74f6be
don't include file names when logging stream related info
2017-12-06 15:09:35 -05:00
Kay Kurokawa
63c6d3580d
do not yield on add_lbry_file() when obtaining sd blob as reflector server
2017-12-05 13:18:16 -05:00
Kay Kurokawa
ab667015c4
add lbry_file_manager to reflector server and add files to it
2017-12-04 13:08:33 -05:00
Kay Kurokawa
70709b4ca4
remove use of name from reflector client, use file name instead
2017-11-28 11:53:15 -05:00
Jack Robison
d1511cba54
fix reading sd blob
...
TODO: add __enter__ and __exit__ methods to HashBlobReader to let it be
used as a contextmanager
2017-10-05 17:58:18 -04:00
Kay Kurokawa
0a919c3eab
fix inconsistency in how blob_hashes_to_send is used, preventing proper reflecting
2017-10-03 11:22:07 -04:00
Kay Kurokawa
6268e43509
make sure to close blob files when connection is lost on reflector client
2017-10-02 16:30:58 -04:00
Kay Kurokawa
a4ea49cf65
read handle was not being closed after file transfer
2017-09-27 17:30:20 -04:00
Kay Kurokawa
aab43c8d6c
close the reader directly instead of calling close_read_handle()
2017-09-27 17:30:20 -04:00
Jack Robison
4b8700268f
discover stream info for uploads via reflector v1
2017-09-22 14:46:23 -04:00
Kay Kurokawa
1a8e11ead3
work on reflector head blob announce only
2017-09-22 14:39:28 -04:00
Jack Robison
6cbe86d057
rename is_validated() to get_is_verified() to distinguish from verified property
2017-09-20 14:02:34 -04:00
Kay Kurokawa
e92321a9c1
have BlobFile.open_for_writing() return the writer instead of write and close functions
2017-09-20 13:59:23 -04:00
Jack Robison
b98cd24e10
don't reflect empty last blob
2017-09-20 13:59:20 -04:00
Jack Robison
c971cce717
file_reflect
2017-08-04 16:08:03 -04:00
Jack Robison
d5f2481af5
handle ip addresses given as reflector servers
2017-08-04 16:08:02 -04:00
Kay Kurokawa
98f90a4ad6
fixing minor debugging bug
2017-03-08 15:32:00 -05:00
Jack Robison
9744287370
change attribute "uri" to "name"
2017-03-07 18:29:44 -05:00
jobevers
9e2cda0cdc
remove more upload_allowed arguments
2017-02-15 06:48:58 -06:00
jobevers
4fb5e932e6
fix indentation
2017-02-15 06:48:58 -06:00
Job Evers-Meltzer
2a45c91175
fix get_blob calls with upload_allowed arguments
2017-02-15 06:48:58 -06:00
Job Evers-Meltzer
a38e5c855e
Remove upload_allowed attribute
...
This is unused and largely pulluting the argument list
of too many functions.
2017-02-15 06:48:50 -06:00
Jack Robison
7720724ec0
reflect lbry_files in looping call in EncryptedFileManager
...
-remove ManagedEncryptedFileDownloader._reupload
-clean up reflector functions in Daemon, move to reflector.reupload
-check ConnectionLost in reflector client
-close sd blob file handle when it wont be sent (otherwise read handle stays open)
-log reflector sd info
-give reflector client factory a lbry file
2017-02-13 15:13:37 -05:00
Alex Grintsvayg
c7106dfc97
reduce loudness of reflector message
2017-02-13 12:12:32 -05:00
Jack Robison
777419b4a9
log traceback on failed blob upload
...
-move status message on connectionDone to client, indicate if blobs
were sent or not (and how many blobs reflector still needs, if any)
-only try uploading failed blob once after first failure, to prevent
indefinite retries
2017-02-08 16:18:34 -05:00
Jack Robison
f2ddc9bd98
fix recursion depth bug upon failed blob
2017-02-08 14:28:30 -05:00
Jack Robison
d053db8dfd
add uri to stream reflector for better logging
2017-02-08 14:28:16 -05:00
Jack Robison
a58a81a841
stopProducing in reflector client file_sender when uploading is done
...
this fixes exceptions.ValueError: I/O operation on closed file in
twisted.protocols.basic.ResumeProducing
2017-02-08 14:09:03 -05:00
Jack Robison
f614a13db4
update documentation
2017-02-06 15:56:00 -05:00
Jack Robison
6fae07d29e
reflector v2
...
-add {‘sd_blob_hash’: …, ‘sd_blob_size ‘: …} query type with
{‘send_sd_blob’: True/False, ‘needed_blobs’: []} response
this allows the reflector client to know how much of a stream reflector
already has covered, as to minimize the number of subsequent requests
and prevent streams from being partially reflected
-remove empty {} request
2017-02-06 15:55:46 -05:00
Jack Robison
74a872d763
fix pylint errors
2017-01-25 13:10:38 -05:00
Jack Robison
5fa2dfeca7
try reflecting all the blobs in a stream
...
previously only the sd blob was reflected, if the server indicated it
needed the blob then the rest of the stream would follow. this allowed
for many streams to be partially reflected, where for whatever reason
the connection was broken before the full upload was completed. this
meant that on a subsequent run, the client would falsely believe
reflector had the whole stream when it actually only had some portion
of it.
this solution isn’t ideal, I’m most of the way done with a better one,
but this can be deployed now.
2017-01-25 13:03:48 -05:00
Job Evers-Meltzer
fa8190356a
replace log_support.failure calls
2016-12-15 18:58:15 -06:00
Job Evers-Meltzer
c30ea04959
Remove commented code
...
This is the result of running eradicate (https://github.com/myint/eradicate )
on the code and double-checking the changes.
2016-12-15 18:58:06 -06:00
