Commit graph

77 commits

Author SHA1 Message Date
Jack Robison
9a8cac20e7
update reflector client and server to use SQLiteStorage 2018-02-13 10:27:51 -05:00
Jack Robison
64e04f8a68
refactor start_lbry_file 2018-01-04 21:28:09 -05:00
Jack Robison
bf46f5616b
Merge branch 'add_lbry_file_manager_to_reflector_2' 2017-12-29 12:39:50 -05:00
Jack Robison
a6f0c5fb4c
pylint fixes
fix exception formatting and a circular import
2017-12-15 20:25:20 -05:00
Alex Grintsvayg
ebed4f010b fixes - stupid error messages 2017-12-13 13:56:44 -05:00
Jack Robison
329d74f6be
don't include file names when logging stream related info 2017-12-06 15:09:35 -05:00
Kay Kurokawa
63c6d3580d do not yield on add_lbry_file() when obtaining sd blob as reflector server 2017-12-05 13:18:16 -05:00
Kay Kurokawa
ab667015c4
add lbry_file_manager to reflector server and add files to it 2017-12-04 13:08:33 -05:00
Kay Kurokawa
70709b4ca4
remove use of name from reflector client, use file name instead 2017-11-28 11:53:15 -05:00
Jack Robison
d1511cba54 fix reading sd blob
TODO: add __enter__ and __exit__ methods to HashBlobReader to let it be
used as a contextmanager
2017-10-05 17:58:18 -04:00
Kay Kurokawa
0a919c3eab fix inconsistency in how blob_hashes_to_send is used, preventing proper reflecting 2017-10-03 11:22:07 -04:00
Kay Kurokawa
6268e43509 make sure to close blob files when connection is lost on reflector client 2017-10-02 16:30:58 -04:00
Kay Kurokawa
a4ea49cf65 read handle was not being closed after file transfer 2017-09-27 17:30:20 -04:00
Kay Kurokawa
aab43c8d6c close the reader directly instead of calling close_read_handle() 2017-09-27 17:30:20 -04:00
Jack Robison
4b8700268f
discover stream info for uploads via reflector v1 2017-09-22 14:46:23 -04:00
Kay Kurokawa
1a8e11ead3
work on reflector head blob announce only 2017-09-22 14:39:28 -04:00
Jack Robison
6cbe86d057
rename is_validated() to get_is_verified() to distinguish from verified property 2017-09-20 14:02:34 -04:00
Kay Kurokawa
e92321a9c1
have BlobFile.open_for_writing() return the writer instead of write and close functions 2017-09-20 13:59:23 -04:00
Jack Robison
b98cd24e10
don't reflect empty last blob 2017-09-20 13:59:20 -04:00
Jack Robison
c971cce717
file_reflect 2017-08-04 16:08:03 -04:00
Jack Robison
d5f2481af5
handle ip addresses given as reflector servers 2017-08-04 16:08:02 -04:00
Kay Kurokawa
98f90a4ad6 fixing minor debugging bug 2017-03-08 15:32:00 -05:00
Jack Robison
9744287370 change attribute "uri" to "name" 2017-03-07 18:29:44 -05:00
jobevers
9e2cda0cdc remove more upload_allowed arguments 2017-02-15 06:48:58 -06:00
jobevers
4fb5e932e6 fix indentation 2017-02-15 06:48:58 -06:00
Job Evers-Meltzer
2a45c91175 fix get_blob calls with upload_allowed arguments 2017-02-15 06:48:58 -06:00
Job Evers-Meltzer
a38e5c855e Remove upload_allowed attribute
This is unused and largely pulluting the argument list
of too many functions.
2017-02-15 06:48:50 -06:00
Jack Robison
7720724ec0 reflect lbry_files in looping call in EncryptedFileManager
-remove ManagedEncryptedFileDownloader._reupload

-clean up reflector functions in Daemon, move to reflector.reupload

-check ConnectionLost in reflector client

-close sd blob file handle when it wont be sent (otherwise read handle stays open)

-log reflector sd info

-give reflector client factory a lbry file
2017-02-13 15:13:37 -05:00
Alex Grintsvayg
c7106dfc97 reduce loudness of reflector message 2017-02-13 12:12:32 -05:00
Jack Robison
777419b4a9 log traceback on failed blob upload
-move status message on connectionDone to client, indicate if blobs
were sent or not (and how many blobs reflector still needs, if any)

-only try uploading failed blob once after first failure, to prevent
indefinite retries
2017-02-08 16:18:34 -05:00
Jack Robison
f2ddc9bd98 fix recursion depth bug upon failed blob 2017-02-08 14:28:30 -05:00
Jack Robison
d053db8dfd add uri to stream reflector for better logging 2017-02-08 14:28:16 -05:00
Jack Robison
a58a81a841 stopProducing in reflector client file_sender when uploading is done
this fixes exceptions.ValueError: I/O operation on closed file in
twisted.protocols.basic.ResumeProducing
2017-02-08 14:09:03 -05:00
Jack Robison
f614a13db4 update documentation 2017-02-06 15:56:00 -05:00
Jack Robison
6fae07d29e reflector v2
-add {‘sd_blob_hash’: …, ‘sd_blob_size ‘: …} query type with
{‘send_sd_blob’: True/False, ‘needed_blobs’: []} response

this allows the reflector client to know how much of a stream reflector
already has covered, as to minimize the number of subsequent requests
and prevent streams from being partially reflected

-remove empty {} request
2017-02-06 15:55:46 -05:00
Jack Robison
74a872d763 fix pylint errors 2017-01-25 13:10:38 -05:00
Jack Robison
5fa2dfeca7 try reflecting all the blobs in a stream
previously only the sd blob was reflected, if the server indicated it
needed the blob then the rest of the stream would follow. this allowed
for many streams to be partially reflected, where for whatever reason
the connection was broken before the full upload was completed. this
meant that on a subsequent run, the client would falsely believe
reflector had the whole stream when it actually only had some portion
of it.

this solution isn’t ideal, I’m most of the way done with a better one,
but this can be deployed now.
2017-01-25 13:03:48 -05:00
Job Evers-Meltzer
fa8190356a replace log_support.failure calls 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer
c30ea04959 Remove commented code
This is the result of running eradicate (https://github.com/myint/eradicate)
on the code and double-checking the changes.
2016-12-15 18:58:06 -06:00
Job Evers-Meltzer
b64fa51567 fixup long lines 2016-12-07 09:38:34 -05:00
Job Evers-Meltzer
0941618048 move reflector blob client to its own module 2016-11-14 19:33:36 -06:00
Job Evers-Meltzer
85dee27c92 whitespace 2016-11-14 19:33:36 -06:00
Job Evers-Meltzer
61a354fa60 add bad-whitespace check to pylint 2016-11-09 11:27:07 -06:00
Jack
9de7354ea0 drop connection on failed read blob 2016-11-01 16:32:34 -04:00
Jack
cfbdda411a whoops 2016-11-01 13:23:02 -04:00
Jack
96c4837cb8 clean up errback 2016-11-01 13:19:13 -04:00
Jack
4cffc32328 catch availability check error 2016-11-01 13:05:19 -04:00
Jack
110a9b1e0f whoops 2016-10-28 15:56:22 -04:00
Jack
035a1cf758 feedback from job 2016-10-28 15:48:51 -04:00
Jack
369cd516c0 move availability check and fix from ManagedEncryptedFileDownloader to ReflectorAvailabilityHelper 2016-10-26 16:17:01 -04:00