initial commit

.gitignore

@@ -0,0 +1 @@
+*.pyc

INSTALL

@@ -0,0 +1,34 @@
+Prerequisites
+-------------
+
+To use the LBRYWallet, which enables spending and accepting LBRYcrds in exchange for data, the
+LBRYcrd application (insert link to LBRYcrd website here) must be installed and running. If
+this is not desired, the testing client can be used to simulate trading points, which is
+built into LBRYnet.
+
+on Ubuntu:
+
+sudo apt-get install libgmp3-dev build-essential python-dev python-pip
+
+Getting the source
+------------------
+
+Don't you already have it?
+
+Setting up the environment
+--------------------------
+
+It's recommended that you use a virtualenv
+
+sudo apt-get install python-virtualenv
+cd <source base directory>
+virtualenv .
+source bin/activate
+
+(to deactivate the virtualenv, enter 'deactivate')
+
+python setup.py install
+
+this will install all of the libraries and a few applications
+
+For running the file sharing application, see RUNNING

LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2015, LBRY, Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README

@@ -0,0 +1,57 @@
+LBRYnet
+=======
+
+LBRYnet is a fully decentralized network for distributing data. It consists of peers uploading
+and downloading data from other peers, possibly in exchange for payments, and a distributed hash
+table, used by peers to discover other peers.
+
+Overview
+--------
+
+On LBRYnet, data is broken into chunks, and each chunk is specified by its sha384 hash sum. This
+guarantees that peers can verify the correctness of each chunk without having to know anything
+about its contents, and can confidently re-transmit the chunk to other peers. Peers wishing to
+transmit chunks to other peers announce to the distributed hash table that they are associated
+with the sha384 hash sum in question. When a peer wants to download that chunk from the network,
+it asks the distributed hash table which peers are associated with that sha384 hash sum. The
+distributed hash table can also be used more generally. It simply stores IP addresses and
+ports which are associated with 384-bit numbers, and can be used by any type of application to
+help peers find each other. For example, an application for which clients don't know all of the
+necessary chunks may use some identifier, chosen by the application, to find clients which do
+know all of the necessary chunks.
+
+Running
+-------
+
+LBRYnet comes with an file sharing application, called 'lbrynet-console', which breaks
+files into chunks, encrypts them with a symmetric key, computes their sha384 hash sum, generates
+a special file called a 'stream descriptor' containing the hash sums and some other file metadata,
+and makes the chunks available for download by other peers. A peer wishing to download the file
+must first obtain the 'stream descriptor' and then may open it with his 'lbrynet-console' client,
+download all of the chunks by locating peers with the chunks via the DHT, and then combine the
+chunks into the original file, according to the metadata included in the 'stream descriptor'.
+
+To install and use this client, see INSTALL and RUNNING
+
+Installation
+------------
+
+See INSTALL
+
+Developers
+----------
+
+Documentation: doc.lbry.io
+Source code: trac.lbry.io/browser
+
+To contribute to the development of LBRYnet or lbrynet-console, contact jimmy@lbry.io
+
+Support
+-------
+
+Send all support requests to jimmy@lbry.io
+
+License
+-------
+
+See LICENSE

RUNNING

@@ -0,0 +1,52 @@
+To install LBRYnet and lbrynet-console, see INSTALL
+
+lbrynet-console is a console application which makes use of the LBRYnet to share files.
+
+In particular, lbrynet-console splits files into encrypted chunks of data compatible with
+LBRYnet, groups all metadata into a 'stream descriptor file' which can be sent directly to
+others wishing to obtain the file, or can be itself turned into a chunk compatible with
+LBRYnet and downloaded via LBRYnet by anyone knowing its sha384 hashsum. lbrynet-console
+also acts as a client whichreads a stream descriptor file, downloads the chunks of data
+specified by the hash sums found in the stream descriptor file, decrypts them according to
+metadata found in the stream, and reconstructs the original file. lbrynet-console features
+a server so that clients can connect to it and download the chunks and other data gotten
+from files created locally and files that have been downloaded from LBRYnet.
+
+lbrynet-console also has a plugin system. There are two plugins: a live stream proof of
+concept which is currently far behind the development of the rest of the application and
+therefore will not run, and a plugin which attempts to determine which chunks on the
+network should be downloaded in order for the application to turn a profit. It will run,
+but its usefulness is extremely limited.
+
+Passing '--help' to lbrynet-console will cause it to print out a quick help message
+describing other command line options to the application.
+
+Once the application has been started, the user is presented with a numbered list of
+actions which looks something like this:
+
+...
+[2] Toggle whether an LBRY File is running
+[3] Create an LBRY File from file
+[4] Publish a stream descriptor file to the DHT for an LBRY File
+...
+
+To perform an action, type the desired number and then hit enter. For example, if you wish
+to create an LBRY file from a file as described in the beginning of this document, type 3 and
+hit enter.
+
+If the application needs more input in order to for the action to be taken, the application
+will continue to print prompts for input until it has received what it needs.
+
+For example, when creating an LBRY file from a file, the application needs to know which file
+it's supposed to use to create the LBRY file, so the user will be prompted for it:
+
+File name:
+
+The user should input the desired file name and hit enter, at which point the application
+will go about splitting the file and making it available on the network.
+
+Some actions will produce sub-menus of actions, which work the same way.
+
+A more detailed user guide is available at doc.lbry.io
+
+Any issues may be reported to jimmy@lbry.io

ez_setup.py

@@ -0,0 +1,332 @@
+#!/usr/bin/env python
+"""Bootstrap setuptools installation
+
+To use setuptools in your package's setup.py, include this
+file in the same directory and add this to the top of your setup.py::
+
+    from ez_setup import use_setuptools
+    use_setuptools()
+
+To require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, simply supply
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import os
+import shutil
+import sys
+import tempfile
+import zipfile
+import optparse
+import subprocess
+import platform
+import textwrap
+import contextlib
+
+from distutils import log
+
+try:
+    from urllib.request import urlopen
+except ImportError:
+    from urllib2 import urlopen
+
+try:
+    from site import USER_SITE
+except ImportError:
+    USER_SITE = None
+
+DEFAULT_VERSION = "4.0.1"
+DEFAULT_URL = "https://pypi.python.org/packages/source/s/setuptools/"
+
+def _python_cmd(*args):
+    """
+    Return True if the command succeeded.
+    """
+    args = (sys.executable,) + args
+    return subprocess.call(args) == 0
+
+
+def _install(archive_filename, install_args=()):
+    with archive_context(archive_filename):
+        # installing
+        log.warn('Installing Setuptools')
+        if not _python_cmd('setup.py', 'install', *install_args):
+            log.warn('Something went wrong during the installation.')
+            log.warn('See the error message above.')
+            # exitcode will be 2
+            return 2
+
+
+def _build_egg(egg, archive_filename, to_dir):
+    with archive_context(archive_filename):
+        # building an egg
+        log.warn('Building a Setuptools egg in %s', to_dir)
+        _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir)
+    # returning the result
+    log.warn(egg)
+    if not os.path.exists(egg):
+        raise IOError('Could not build the egg.')
+
+
+class ContextualZipFile(zipfile.ZipFile):
+    """
+    Supplement ZipFile class to support context manager for Python 2.6
+    """
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, traceback):
+        self.close()
+
+    def __new__(cls, *args, **kwargs):
+        """
+        Construct a ZipFile or ContextualZipFile as appropriate
+        """
+        if hasattr(zipfile.ZipFile, '__exit__'):
+            return zipfile.ZipFile(*args, **kwargs)
+        return super(ContextualZipFile, cls).__new__(cls)
+
+
+@contextlib.contextmanager
+def archive_context(filename):
+    # extracting the archive
+    tmpdir = tempfile.mkdtemp()
+    log.warn('Extracting in %s', tmpdir)
+    old_wd = os.getcwd()
+    try:
+        os.chdir(tmpdir)
+        with ContextualZipFile(filename) as archive:
+            archive.extractall()
+
+        # going in the directory
+        subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
+        os.chdir(subdir)
+        log.warn('Now working in %s', subdir)
+        yield
+
+    finally:
+        os.chdir(old_wd)
+        shutil.rmtree(tmpdir)
+
+
+def _do_download(version, download_base, to_dir, download_delay):
+    egg = os.path.join(to_dir, 'setuptools-%s-py%d.%d.egg'
+                       % (version, sys.version_info[0], sys.version_info[1]))
+    if not os.path.exists(egg):
+        archive = download_setuptools(version, download_base,
+                                      to_dir, download_delay)
+        _build_egg(egg, archive, to_dir)
+    sys.path.insert(0, egg)
+
+    # Remove previously-imported pkg_resources if present (see
+    # https://bitbucket.org/pypa/setuptools/pull-request/7/ for details).
+    if 'pkg_resources' in sys.modules:
+        del sys.modules['pkg_resources']
+
+    import setuptools
+    setuptools.bootstrap_install_from = egg
+
+
+def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
+        to_dir=os.curdir, download_delay=15):
+    to_dir = os.path.abspath(to_dir)
+    rep_modules = 'pkg_resources', 'setuptools'
+    imported = set(sys.modules).intersection(rep_modules)
+    try:
+        import pkg_resources
+    except ImportError:
+        return _do_download(version, download_base, to_dir, download_delay)
+    try:
+        pkg_resources.require("setuptools>=" + version)
+        return
+    except pkg_resources.DistributionNotFound:
+        return _do_download(version, download_base, to_dir, download_delay)
+    except pkg_resources.VersionConflict as VC_err:
+        if imported:
+            msg = textwrap.dedent("""
+                The required version of setuptools (>={version}) is not available,
+                and can't be installed while this script is running. Please
+                install a more recent version first, using
+                'easy_install -U setuptools'.
+
+                (Currently using {VC_err.args[0]!r})
+                """).format(VC_err=VC_err, version=version)
+            sys.stderr.write(msg)
+            sys.exit(2)
+
+        # otherwise, reload ok
+        del pkg_resources, sys.modules['pkg_resources']
+        return _do_download(version, download_base, to_dir, download_delay)
+
+def _clean_check(cmd, target):
+    """
+    Run the command to download target. If the command fails, clean up before
+    re-raising the error.
+    """
+    try:
+        subprocess.check_call(cmd)
+    except subprocess.CalledProcessError:
+        if os.access(target, os.F_OK):
+            os.unlink(target)
+        raise
+
+def download_file_powershell(url, target):
+    """
+    Download the file at url to target using Powershell (which will validate
+    trust). Raise an exception if the command cannot complete.
+    """
+    target = os.path.abspath(target)
+    ps_cmd = (
+        "[System.Net.WebRequest]::DefaultWebProxy.Credentials = "
+        "[System.Net.CredentialCache]::DefaultCredentials; "
+        "(new-object System.Net.WebClient).DownloadFile(%(url)r, %(target)r)"
+        % vars()
+    )
+    cmd = [
+        'powershell',
+        '-Command',
+        ps_cmd,
+    ]
+    _clean_check(cmd, target)
+
+def has_powershell():
+    if platform.system() != 'Windows':
+        return False
+    cmd = ['powershell', '-Command', 'echo test']
+    with open(os.path.devnull, 'wb') as devnull:
+        try:
+            subprocess.check_call(cmd, stdout=devnull, stderr=devnull)
+        except Exception:
+            return False
+    return True
+
+download_file_powershell.viable = has_powershell
+
+def download_file_curl(url, target):
+    cmd = ['curl', url, '--silent', '--output', target]
+    _clean_check(cmd, target)
+
+def has_curl():
+    cmd = ['curl', '--version']
+    with open(os.path.devnull, 'wb') as devnull:
+        try:
+            subprocess.check_call(cmd, stdout=devnull, stderr=devnull)
+        except Exception:
+            return False
+    return True
+
+download_file_curl.viable = has_curl
+
+def download_file_wget(url, target):
+    cmd = ['wget', url, '--quiet', '--output-document', target]
+    _clean_check(cmd, target)
+
+def has_wget():
+    cmd = ['wget', '--version']
+    with open(os.path.devnull, 'wb') as devnull:
+        try:
+            subprocess.check_call(cmd, stdout=devnull, stderr=devnull)
+        except Exception:
+            return False
+    return True
+
+download_file_wget.viable = has_wget
+
+def download_file_insecure(url, target):
+    """
+    Use Python to download the file, even though it cannot authenticate the
+    connection.
+    """
+    src = urlopen(url)
+    try:
+        # Read all the data in one block.
+        data = src.read()
+    finally:
+        src.close()
+
+    # Write all the data in one block to avoid creating a partial file.
+    with open(target, "wb") as dst:
+        dst.write(data)
+
+download_file_insecure.viable = lambda: True
+
+def get_best_downloader():
+    downloaders = (
+        download_file_powershell,
+        download_file_curl,
+        download_file_wget,
+        download_file_insecure,
+    )
+    viable_downloaders = (dl for dl in downloaders if dl.viable())
+    return next(viable_downloaders, None)
+
+def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
+        to_dir=os.curdir, delay=15, downloader_factory=get_best_downloader):
+    """
+    Download setuptools from a specified location and return its filename
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end
+    with a '/'). `to_dir` is the directory where the egg will be downloaded.
+    `delay` is the number of seconds to pause before an actual download
+    attempt.
+
+    ``downloader_factory`` should be a function taking no arguments and
+    returning a function for downloading a URL to a target.
+    """
+    # making sure we use the absolute path
+    to_dir = os.path.abspath(to_dir)
+    zip_name = "setuptools-%s.zip" % version
+    url = download_base + zip_name
+    saveto = os.path.join(to_dir, zip_name)
+    if not os.path.exists(saveto):  # Avoid repeated downloads
+        log.warn("Downloading %s", url)
+        downloader = downloader_factory()
+        downloader(url, saveto)
+    return os.path.realpath(saveto)
+
+def _build_install_args(options):
+    """
+    Build the arguments to 'python setup.py install' on the setuptools package
+    """
+    return ['--user'] if options.user_install else []
+
+def _parse_args():
+    """
+    Parse the command line for options
+    """
+    parser = optparse.OptionParser()
+    parser.add_option(
+        '--user', dest='user_install', action='store_true', default=False,
+        help='install in user site package (requires Python 2.6 or later)')
+    parser.add_option(
+        '--download-base', dest='download_base', metavar="URL",
+        default=DEFAULT_URL,
+        help='alternative URL from where to download the setuptools package')
+    parser.add_option(
+        '--insecure', dest='downloader_factory', action='store_const',
+        const=lambda: download_file_insecure, default=get_best_downloader,
+        help='Use internal, non-validating downloader'
+    )
+    parser.add_option(
+        '--version', help="Specify which version to download",
+        default=DEFAULT_VERSION,
+    )
+    options, args = parser.parse_args()
+    # positional arguments are ignored
+    return options
+
+def main():
+    """Install or upgrade setuptools and EasyInstall"""
+    options = _parse_args()
+    archive = download_setuptools(
+        version=options.version,
+        download_base=options.download_base,
+        downloader_factory=options.downloader_factory,
+    )
+    return _install(archive, _build_install_args(options))
+
+if __name__ == '__main__':
+    sys.exit(main())

lbrynet/conf.py

@@ -0,0 +1,25 @@
+"""
+Some network wide and also application specific parameters
+"""
+
+
+import os
+
+
+MAX_HANDSHAKE_SIZE = 2**16
+MAX_REQUEST_SIZE = 2**16
+MAX_BLOB_REQUEST_SIZE = 2**16
+MAX_RESPONSE_INFO_SIZE = 2**16
+MAX_BLOB_INFOS_TO_REQUEST = 20
+BLOBFILES_DIR = ".blobfiles"
+BLOB_SIZE = 2**21
+MIN_BLOB_DATA_PAYMENT_RATE = .5  # points/megabyte
+MIN_BLOB_INFO_PAYMENT_RATE = 2.0  # points/1000 infos
+MIN_VALUABLE_BLOB_INFO_PAYMENT_RATE = 5.0  # points/1000 infos
+MIN_VALUABLE_BLOB_HASH_PAYMENT_RATE = 5.0  # points/1000 infos
+MAX_CONNECTIONS_PER_STREAM = 5
+
+POINTTRADER_SERVER = 'http://ec2-54-187-192-68.us-west-2.compute.amazonaws.com:2424'
+#POINTTRADER_SERVER = 'http://127.0.0.1:2424'
+
+CRYPTSD_FILE_EXTENSION = ".cryptsd"

lbrynet/core/BlobInfo.py

@@ -0,0 +1,18 @@
+class BlobInfo(object):
+    """
+    This structure is used to represent the metadata of a blob.
+
+    @ivar blob_hash: The sha384 hashsum of the blob's data.
+    @type blob_hash: string, hex-encoded
+
+    @ivar blob_num: For streams, the position of the blob in the stream.
+    @type blob_num: integer
+
+    @ivar length: The length of the blob in bytes.
+    @type length: integer
+    """
+
+    def __init__(self, blob_hash, blob_num, length):
+        self.blob_hash = blob_hash
+        self.blob_num = blob_num
+        self.length = length

lbrynet/core/BlobManager.py

@@ -0,0 +1,438 @@
+import logging
+import os
+import leveldb
+import time
+import json
+from twisted.internet import threads, defer, reactor, task
+from twisted.python.failure import Failure
+from lbrynet.core.HashBlob import BlobFile, TempBlob, BlobFileCreator, TempBlobCreator
+from lbrynet.core.server.DHTHashAnnouncer import DHTHashSupplier
+from lbrynet.core.utils import is_valid_blobhash
+from lbrynet.core.cryptoutils import get_lbry_hash_obj
+
+
+class BlobManager(DHTHashSupplier):
+    """This class is subclassed by classes which keep track of which blobs are available
+       and which give access to new/existing blobs"""
+    def __init__(self, hash_announcer):
+        DHTHashSupplier.__init__(self, hash_announcer)
+
+    def setup(self):
+        pass
+
+    def get_blob(self, blob_hash, upload_allowed, length):
+        pass
+
+    def get_blob_creator(self):
+        pass
+
+    def _make_new_blob(self, blob_hash, upload_allowed, length):
+        pass
+
+    def blob_completed(self, blob, next_announce_time=None):
+        pass
+
+    def completed_blobs(self, blobs_to_check):
+        pass
+
+    def hashes_to_announce(self):
+        pass
+
+    def creator_finished(self, blob_creator):
+        pass
+
+    def delete_blob(self, blob_hash):
+        pass
+
+    def get_blob_length(self, blob_hash):
+        pass
+
+    def check_consistency(self):
+        pass
+
+    def blob_requested(self, blob_hash):
+        pass
+
+    def blob_downloaded(self, blob_hash):
+        pass
+
+    def blob_searched_on(self, blob_hash):
+        pass
+
+    def blob_paid_for(self, blob_hash, amount):
+        pass
+
+
+class DiskBlobManager(BlobManager):
+    """This class stores blobs on the hard disk"""
+    def __init__(self, hash_announcer, blob_dir, db_dir):
+        BlobManager.__init__(self, hash_announcer)
+        self.blob_dir = blob_dir
+        self.db_dir = db_dir
+        self.db = None
+        self.blob_type = BlobFile
+        self.blob_creator_type = BlobFileCreator
+        self.blobs = {}
+        self.blob_hashes_to_delete = {}  # {blob_hash: being_deleted (True/False)}
+        self._next_manage_call = None
+
+    def setup(self):
+        d = threads.deferToThread(self._open_db)
+        d.addCallback(lambda _: self._manage())
+        return d
+
+    def stop(self):
+        if self._next_manage_call is not None and self._next_manage_call.active():
+            self._next_manage_call.cancel()
+            self._next_manage_call = None
+        self.db = None
+        return defer.succeed(True)
+
+    def get_blob(self, blob_hash, upload_allowed, length=None):
+        """Return a blob identified by blob_hash, which may be a new blob or a blob that is already on the hard disk"""
+        # TODO: if blob.upload_allowed and upload_allowed is False, change upload_allowed in blob and on disk
+        if blob_hash in self.blobs:
+            return defer.succeed(self.blobs[blob_hash])
+        return self._make_new_blob(blob_hash, upload_allowed, length)
+
+    def get_blob_creator(self):
+        return self.blob_creator_type(self, self.blob_dir)
+
+    def _make_new_blob(self, blob_hash, upload_allowed, length=None):
+        blob = self.blob_type(self.blob_dir, blob_hash, upload_allowed, length)
+        self.blobs[blob_hash] = blob
+        d = threads.deferToThread(self._completed_blobs, [blob_hash])
+
+        def check_completed(completed_blobs):
+
+            def set_length(length):
+                blob.length = length
+
+            if len(completed_blobs) == 1 and completed_blobs[0] == blob_hash:
+                blob.verified = True
+                inner_d = threads.deferToThread(self._get_blob_length, blob_hash)
+                inner_d.addCallback(set_length)
+                inner_d.addCallback(lambda _: blob)
+            else:
+                inner_d = defer.succeed(blob)
+            return inner_d
+
+        d.addCallback(check_completed)
+        return d
+
+    def blob_completed(self, blob, next_announce_time=None):
+        if next_announce_time is None:
+            next_announce_time = time.time()
+        return threads.deferToThread(self._add_completed_blob, blob.blob_hash, blob.length,
+                                     time.time(), next_announce_time)
+
+    def completed_blobs(self, blobs_to_check):
+        return threads.deferToThread(self._completed_blobs, blobs_to_check)
+
+    def hashes_to_announce(self):
+        next_announce_time = time.time() + self.hash_reannounce_time
+        return threads.deferToThread(self._get_blobs_to_announce, next_announce_time)
+
+    def creator_finished(self, blob_creator):
+        logging.debug("blob_creator.blob_hash: %s", blob_creator.blob_hash)
+        assert blob_creator.blob_hash is not None
+        assert blob_creator.blob_hash not in self.blobs
+        assert blob_creator.length is not None
+        new_blob = self.blob_type(self.blob_dir, blob_creator.blob_hash, True, blob_creator.length)
+        new_blob.verified = True
+        self.blobs[blob_creator.blob_hash] = new_blob
+        if self.hash_announcer is not None:
+            self.hash_announcer.immediate_announce([blob_creator.blob_hash])
+            next_announce_time = time.time() + self.hash_reannounce_time
+            d = self.blob_completed(new_blob, next_announce_time)
+        else:
+            d = self.blob_completed(new_blob)
+        return d
+
+    def delete_blobs(self, blob_hashes):
+        for blob_hash in blob_hashes:
+            if not blob_hash in self.blob_hashes_to_delete:
+                self.blob_hashes_to_delete[blob_hash] = False
+
+    def update_all_last_verified_dates(self, timestamp):
+        return threads.deferToThread(self._update_all_last_verified_dates, timestamp)
+
+    def immediate_announce_all_blobs(self):
+        d = threads.deferToThread(self._get_all_verified_blob_hashes)
+        d.addCallback(self.hash_announcer.immediate_announce)
+        return d
+
+    def get_blob_length(self, blob_hash):
+        return threads.deferToThread(self._get_blob_length, blob_hash)
+
+    def check_consistency(self):
+        return threads.deferToThread(self._check_consistency)
+
+    def _manage(self):
+        from twisted.internet import reactor
+
+        d = self._delete_blobs_marked_for_deletion()
+
+        def set_next_manage_call():
+            self._next_manage_call = reactor.callLater(1, self._manage)
+
+        d.addCallback(lambda _: set_next_manage_call())
+
+    def _delete_blobs_marked_for_deletion(self):
+
+        def remove_from_list(b_h):
+            del self.blob_hashes_to_delete[b_h]
+            return b_h
+
+        def set_not_deleting(err, b_h):
+            logging.warning("Failed to delete blob %s. Reason: %s", str(b_h), err.getErrorMessage())
+            self.blob_hashes_to_delete[b_h] = False
+            return err
+
+        def delete_from_db(result):
+            b_hs = [r[1] for r in result if r[0] is True]
+            if b_hs:
+                d = threads.deferToThread(self._delete_blobs_from_db, b_hs)
+            else:
+                d = defer.succeed(True)
+
+            def log_error(err):
+                logging.warning("Failed to delete completed blobs from the db: %s", err.getErrorMessage())
+
+            d.addErrback(log_error)
+            return d
+
+        def delete(blob, b_h):
+            d = blob.delete()
+            d.addCallbacks(lambda _: remove_from_list(b_h), set_not_deleting, errbackArgs=(b_h,))
+            return d
+
+        ds = []
+        for blob_hash, being_deleted in self.blob_hashes_to_delete.items():
+            if being_deleted is False:
+                self.blob_hashes_to_delete[blob_hash] = True
+                d = self.get_blob(blob_hash, True)
+                d.addCallbacks(delete, set_not_deleting, callbackArgs=(blob_hash,), errbackArgs=(blob_hash,))
+                ds.append(d)
+        dl = defer.DeferredList(ds, consumeErrors=True)
+        dl.addCallback(delete_from_db)
+        return defer.DeferredList(ds)
+
+    ######### database calls #########
+
+    def _open_db(self):
+        self.db = leveldb.LevelDB(os.path.join(self.db_dir, "blobs.db"))
+
+    def _add_completed_blob(self, blob_hash, length, timestamp, next_announce_time=None):
+        logging.debug("Adding a completed blob. blob_hash=%s, length=%s", blob_hash, str(length))
+        if next_announce_time is None:
+            next_announce_time = timestamp
+        self.db.Put(blob_hash, json.dumps((length, timestamp, next_announce_time)), sync=True)
+
+    def _completed_blobs(self, blobs_to_check):
+        blobs = []
+        for b in blobs_to_check:
+            if is_valid_blobhash(b):
+                try:
+                    length, verified_time, next_announce_time = json.loads(self.db.Get(b))
+                except KeyError:
+                    continue
+                file_path = os.path.join(self.blob_dir, b)
+                if os.path.isfile(file_path):
+                    if verified_time > os.path.getctime(file_path):
+                        blobs.append(b)
+        return blobs
+
+    def _get_blob_length(self, blob):
+        length, verified_time, next_announce_time = json.loads(self.db.Get(blob))
+        return length
+
+    def _update_blob_verified_timestamp(self, blob, timestamp):
+        length, old_verified_time, next_announce_time = json.loads(self.db.Get(blob))
+        self.db.Put(blob, json.dumps((length, timestamp, next_announce_time)), sync=True)
+
+    def _get_blobs_to_announce(self, next_announce_time):
+        # TODO: See if the following would be better for handling announce times:
+        # TODO:    Have a separate db for them, and read the whole thing into memory
+        # TODO:    on startup, and then write changes to db when they happen
+        blobs = []
+        batch = leveldb.WriteBatch()
+        current_time = time.time()
+        for blob_hash, blob_info in self.db.RangeIter():
+            length, verified_time, announce_time = json.loads(blob_info)
+            if announce_time < current_time:
+                batch.Put(blob_hash, json.dumps((length, verified_time, next_announce_time)))
+                blobs.append(blob_hash)
+        self.db.Write(batch, sync=True)
+        return blobs
+
+    def _update_all_last_verified_dates(self, timestamp):
+        batch = leveldb.WriteBatch()
+        for blob_hash, blob_info in self.db.RangeIter():
+            length, verified_time, announce_time = json.loads(blob_info)
+            batch.Put(blob_hash, json.dumps((length, timestamp, announce_time)))
+        self.db.Write(batch, sync=True)
+
+    def _delete_blobs_from_db(self, blob_hashes):
+        batch = leveldb.WriteBatch()
+        for blob_hash in blob_hashes:
+            batch.Delete(blob_hash)
+        self.db.Write(batch, sync=True)
+
+    def _check_consistency(self):
+        batch = leveldb.WriteBatch()
+        current_time = time.time()
+        for blob_hash, blob_info in self.db.RangeIter():
+            length, verified_time, announce_time = json.loads(blob_info)
+            file_path = os.path.join(self.blob_dir, blob_hash)
+            if os.path.isfile(file_path):
+                if verified_time < os.path.getctime(file_path):
+                    h = get_lbry_hash_obj()
+                    len_so_far = 0
+                    f = open(file_path)
+                    while True:
+                        data = f.read(2**12)
+                        if not data:
+                            break
+                        h.update(data)
+                        len_so_far += len(data)
+                    if len_so_far == length and h.hexdigest() == blob_hash:
+                        batch.Put(blob_hash, json.dumps((length, current_time, announce_time)))
+        self.db.Write(batch, sync=True)
+
+    def _get_all_verified_blob_hashes(self):
+        blob_hashes = []
+        for blob_hash, blob_info in self.db.RangeIter():
+            length, verified_time, announce_time = json.loads(blob_info)
+            file_path = os.path.join(self.blob_dir, blob_hash)
+            if os.path.isfile(file_path):
+                if verified_time > os.path.getctime(file_path):
+                    blob_hashes.append(blob_hash)
+        return blob_hashes
+
+
+class TempBlobManager(BlobManager):
+    """This class stores blobs in memory"""
+    def __init__(self, hash_announcer):
+        BlobManager.__init__(self, hash_announcer)
+        self.blob_type = TempBlob
+        self.blob_creator_type = TempBlobCreator
+        self.blobs = {}
+        self.blob_next_announces = {}
+        self.blob_hashes_to_delete = {}  # {blob_hash: being_deleted (True/False)}
+        self._next_manage_call = None
+
+    def setup(self):
+        self._manage()
+        return defer.succeed(True)
+
+    def stop(self):
+        if self._next_manage_call is not None and self._next_manage_call.active():
+            self._next_manage_call.cancel()
+            self._next_manage_call = None
+
+    def get_blob(self, blob_hash, upload_allowed, length=None):
+        if blob_hash in self.blobs:
+            return defer.succeed(self.blobs[blob_hash])
+        return self._make_new_blob(blob_hash, upload_allowed, length)
+
+    def get_blob_creator(self):
+        return self.blob_creator_type(self)
+
+    def _make_new_blob(self, blob_hash, upload_allowed, length=None):
+        blob = self.blob_type(blob_hash, upload_allowed, length)
+        self.blobs[blob_hash] = blob
+        return defer.succeed(blob)
+
+    def blob_completed(self, blob, next_announce_time=None):
+        if next_announce_time is None:
+            next_announce_time = time.time()
+        self.blob_next_announces[blob.blob_hash] = next_announce_time
+        return defer.succeed(True)
+
+    def completed_blobs(self, blobs_to_check):
+        blobs = [b.blob_hash for b in self.blobs.itervalues() if b.blob_hash in blobs_to_check and b.is_validated()]
+        return defer.succeed(blobs)
+
+    def hashes_to_announce(self):
+        now = time.time()
+        blobs = [blob_hash for blob_hash, announce_time in self.blob_next_announces.iteritems() if announce_time < now]
+        next_announce_time = now + self.hash_reannounce_time
+        for b in blobs:
+            self.blob_next_announces[b] = next_announce_time
+        return defer.succeed(blobs)
+
+    def creator_finished(self, blob_creator):
+        assert blob_creator.blob_hash is not None
+        assert blob_creator.blob_hash not in self.blobs
+        assert blob_creator.length is not None
+        new_blob = self.blob_type(blob_creator.blob_hash, True, blob_creator.length)
+        new_blob.verified = True
+        new_blob.data_buffer = blob_creator.data_buffer
+        new_blob.length = blob_creator.length
+        self.blobs[blob_creator.blob_hash] = new_blob
+        if self.hash_announcer is not None:
+            self.hash_announcer.immediate_announce([blob_creator.blob_hash])
+            next_announce_time = time.time() + self.hash_reannounce_time
+            d = self.blob_completed(new_blob, next_announce_time)
+        else:
+            d = self.blob_completed(new_blob)
+        d.addCallback(lambda _: new_blob)
+        return d
+
+    def delete_blobs(self, blob_hashes):
+        for blob_hash in blob_hashes:
+            if not blob_hash in self.blob_hashes_to_delete:
+                self.blob_hashes_to_delete[blob_hash] = False
+
+    def get_blob_length(self, blob_hash):
+        if blob_hash in self.blobs:
+            if self.blobs[blob_hash].length is not None:
+                return defer.succeed(self.blobs[blob_hash].length)
+        return defer.fail(ValueError("No such blob hash is known"))
+
+    def immediate_announce_all_blobs(self):
+        return self.hash_announcer.immediate_announce(self.blobs.iterkeys())
+
+    def _manage(self):
+        from twisted.internet import reactor
+
+        d = self._delete_blobs_marked_for_deletion()
+
+        def set_next_manage_call():
+            logging.info("Setting the next manage call in %s", str(self))
+            self._next_manage_call = reactor.callLater(1, self._manage)
+
+        d.addCallback(lambda _: set_next_manage_call())
+
+    def _delete_blobs_marked_for_deletion(self):
+
+        def remove_from_list(b_h):
+            del self.blob_hashes_to_delete[b_h]
+            logging.info("Deleted blob %s", blob_hash)
+            return b_h
+
+        def set_not_deleting(err, b_h):
+            logging.warning("Failed to delete blob %s. Reason: %s", str(b_h), err.getErrorMessage())
+            self.blob_hashes_to_delete[b_h] = False
+            return b_h
+
+        ds = []
+        for blob_hash, being_deleted in self.blob_hashes_to_delete.items():
+            if being_deleted is False:
+                if blob_hash in self.blobs:
+                    self.blob_hashes_to_delete[blob_hash] = True
+                    logging.info("Found a blob marked for deletion: %s", blob_hash)
+                    blob = self.blobs[blob_hash]
+                    d = blob.delete()
+
+                    d.addCallbacks(lambda _: remove_from_list(blob_hash), set_not_deleting,
+                                   errbackArgs=(blob_hash,))
+
+                    ds.append(d)
+                else:
+                    remove_from_list(blob_hash)
+                    d = defer.fail(Failure(ValueError("No such blob known")))
+                    logging.warning("Blob %s cannot be deleted because it is unknown")
+                    ds.append(d)
+        return defer.DeferredList(ds)

lbrynet/core/DownloadOption.py

@@ -0,0 +1,6 @@
+class DownloadOption(object):
+    def __init__(self, option_types, long_description, short_description, default):
+        self.option_types = option_types
+        self.long_description = long_description
+        self.short_description = short_description
+        self.default = default

lbrynet/core/Error.py

@@ -0,0 +1,48 @@
+class PriceDisagreementError(Exception):
+    pass
+
+
+class DuplicateStreamHashError(Exception):
+    pass
+
+
+class DownloadCanceledError(Exception):
+    pass
+
+
+class RequestCanceledError(Exception):
+    pass
+
+
+class InsufficientFundsError(Exception):
+    pass
+
+
+class ConnectionClosedBeforeResponseError(Exception):
+    pass
+
+
+class UnknownNameError(Exception):
+    def __init__(self, name):
+        self.name = name
+
+
+class InvalidStreamInfoError(Exception):
+    def __init__(self, name):
+        self.name = name
+
+
+class MisbehavingPeerError(Exception):
+    pass
+
+
+class InvalidDataError(MisbehavingPeerError):
+    pass
+
+
+class NoResponseError(MisbehavingPeerError):
+    pass
+
+
+class InvalidResponseError(MisbehavingPeerError):
+    pass

lbrynet/core/HashAnnouncer.py

@@ -0,0 +1,15 @@
+class DummyHashAnnouncer(object):
+    def __init__(self, *args):
+        pass
+
+    def run_manage_loop(self):
+        pass
+
+    def stop(self):
+        pass
+
+    def add_supplier(self, *args):
+        pass
+
+    def immediate_announce(self, *args):
+        pass

lbrynet/core/HashBlob.py

@@ -0,0 +1,391 @@
+from StringIO import StringIO
+import logging
+import os
+import tempfile
+import threading
+import shutil
+from twisted.internet import interfaces, defer, threads
+from twisted.protocols.basic import FileSender
+from twisted.python.failure import Failure
+from zope.interface import implements
+from lbrynet.conf import BLOB_SIZE
+from lbrynet.core.Error import DownloadCanceledError, InvalidDataError
+from lbrynet.core.cryptoutils import get_lbry_hash_obj
+
+
+class HashBlobReader(object):
+    implements(interfaces.IConsumer)
+
+    def __init__(self, write_func):
+        self.write_func = write_func
+
+    def registerProducer(self, producer, streaming):
+
+        from twisted.internet import reactor
+
+        self.producer = producer
+        self.streaming = streaming
+        if self.streaming is False:
+            reactor.callLater(0, self.producer.resumeProducing)
+
+    def unregisterProducer(self):
+        pass
+
+    def write(self, data):
+
+        from twisted.internet import reactor
+
+        self.write_func(data)
+        if self.streaming is False:
+            reactor.callLater(0, self.producer.resumeProducing)
+
+
+class HashBlobWriter(object):
+    def __init__(self, write_handle, length_getter, finished_cb):
+        self.write_handle = write_handle
+        self.length_getter = length_getter
+        self.finished_cb = finished_cb
+        self.hashsum = get_lbry_hash_obj()
+        self.len_so_far = 0
+
+    def write(self, data):
+        self.hashsum.update(data)
+        self.len_so_far += len(data)
+        if self.len_so_far > self.length_getter():
+            self.finished_cb(self, Failure(InvalidDataError("Length so far is greater than the expected length."
+                                                            " %s to %s" % (str(self.len_so_far),
+                                                                           str(self.length_getter())))))
+        else:
+            self.write_handle.write(data)
+            if self.len_so_far == self.length_getter():
+                self.finished_cb(self)
+
+    def cancel(self, reason=None):
+        if reason is None:
+            reason = Failure(DownloadCanceledError())
+        self.finished_cb(self, reason)
+
+
+class HashBlob(object):
+    """A chunk of data available on the network which is specified by a hashsum"""
+
+    def __init__(self, blob_hash, upload_allowed, length=None):
+        self.blob_hash = blob_hash
+        self.length = length
+        self.writers = {}  # {Peer: writer, finished_deferred}
+        self.finished_deferred = None
+        self.verified = False
+        self.upload_allowed = upload_allowed
+        self.readers = 0
+
+    def set_length(self, length):
+        if self.length is not None and length == self.length:
+            return True
+        if self.length is None and 0 <= length <= BLOB_SIZE:
+            self.length = length
+            return True
+        logging.warning("Got an invalid length. Previous length: %s, Invalid length: %s", str(self.length), str(length))
+        return False
+
+    def get_length(self):
+        return self.length
+
+    def is_validated(self):
+        if self.verified:
+            return True
+        else:
+            return False
+
+    def is_downloading(self):
+        if self.writers:
+            return True
+        return False
+
+    def read(self, write_func):
+
+        def close_self(*args):
+            self.close_read_handle(file_handle)
+            return args[0]
+
+        file_sender = FileSender()
+        reader = HashBlobReader(write_func)
+        file_handle = self.open_for_reading()
+        if file_handle is not None:
+            d = file_sender.beginFileTransfer(file_handle, reader)
+            d.addCallback(close_self)
+        else:
+            d = defer.fail(ValueError("Could not read the blob"))
+        return d
+
+    def writer_finished(self, writer, err=None):
+
+        def fire_finished_deferred():
+            self.verified = True
+            for p, (w, finished_deferred) in self.writers.items():
+                if w == writer:
+                    finished_deferred.callback(self)
+                    del self.writers[p]
+                    return True
+            logging.warning("Somehow, the writer that was accepted as being valid was already removed. writer: %s",
+                            str(writer))
+            return False
+
+        def errback_finished_deferred(err):
+            for p, (w, finished_deferred) in self.writers.items():
+                if w == writer:
+                    finished_deferred.errback(err)
+                    del self.writers[p]
+
+        def cancel_other_downloads():
+            for p, (w, finished_deferred) in self.writers.items():
+                w.cancel()
+
+        if err is None:
+            if writer.len_so_far == self.length and writer.hashsum.hexdigest() == self.blob_hash:
+                if self.verified is False:
+                    d = self._save_verified_blob(writer)
+                    d.addCallbacks(lambda _: fire_finished_deferred(), errback_finished_deferred)
+                    d.addCallback(lambda _: cancel_other_downloads())
+                else:
+                    errback_finished_deferred(Failure(DownloadCanceledError()))
+                    d = defer.succeed(True)
+            else:
+                err_string = "length vs expected: {0}, {1}, hash vs expected: {2}, {3}"
+                err_string = err_string.format(self.length, writer.len_so_far, self.blob_hash,
+                                               writer.hashsum.hexdigest())
+                errback_finished_deferred(Failure(InvalidDataError(err_string)))
+                d = defer.succeed(True)
+        else:
+            errback_finished_deferred(err)
+            d = defer.succeed(True)
+
+        d.addBoth(lambda _: self._close_writer(writer))
+        return d
+
+    def open_for_writing(self, peer):
+        pass
+
+    def open_for_reading(self):
+        pass
+
+    def delete(self):
+        pass
+
+    def close_read_handle(self, file_handle):
+        pass
+
+    def _close_writer(self, writer):
+        pass
+
+    def _save_verified_blob(self, writer):
+        pass
+
+    def __str__(self):
+        return self.blob_hash[:16]
+
+    def __repr__(self):
+        return str(self)
+
+
+class BlobFile(HashBlob):
+    """A HashBlob which will be saved to the hard disk of the downloader"""
+
+    def __init__(self, blob_dir, *args):
+        HashBlob.__init__(self, *args)
+        self.blob_dir = blob_dir
+        self.file_path = os.path.join(blob_dir, self.blob_hash)
+        self.setting_verified_blob_lock = threading.Lock()
+        self.moved_verified_blob = False
+
+    def open_for_writing(self, peer):
+        if not peer in self.writers:
+            logging.debug("Opening %s to be written by %s", str(self), str(peer))
+            write_file = tempfile.NamedTemporaryFile(delete=False, dir=self.blob_dir)
+            finished_deferred = defer.Deferred()
+            writer = HashBlobWriter(write_file, self.get_length, self.writer_finished)
+
+            self.writers[peer] = (writer, finished_deferred)
+            return finished_deferred, writer.write, writer.cancel
+        logging.warning("Tried to download the same file twice simultaneously from the same peer")
+        return None, None, None
+
+    def open_for_reading(self):
+        if self.verified is True:
+            file_handle = None
+            try:
+                file_handle = open(self.file_path, 'rb')
+                self.readers += 1
+                return file_handle
+            except IOError:
+                self.close_read_handle(file_handle)
+        return None
+
+    def delete(self):
+        if not self.writers and not self.readers:
+            self.verified = False
+            self.moved_verified_blob = False
+
+            def delete_from_file_system():
+                if os.path.isfile(self.file_path):
+                    os.remove(self.file_path)
+
+            d = threads.deferToThread(delete_from_file_system)
+
+            def log_error(err):
+                logging.warning("An error occurred deleting %s: %s", str(self.file_path), err.getErrorMessage())
+                return err
+
+            d.addErrback(log_error)
+            return d
+        else:
+            return defer.fail(Failure(ValueError("File is currently being read or written and cannot be deleted")))
+
+    def close_read_handle(self, file_handle):
+        if file_handle is not None:
+            file_handle.close()
+            self.readers -= 1
+
+    def _close_writer(self, writer):
+        if writer.write_handle is not None:
+            logging.debug("Closing %s", str(self))
+            name = writer.write_handle.name
+            writer.write_handle.close()
+            threads.deferToThread(os.remove, name)
+            writer.write_handle = None
+
+    def _save_verified_blob(self, writer):
+
+        def move_file():
+            with self.setting_verified_blob_lock:
+                if self.moved_verified_blob is False:
+                    temp_file_name = writer.write_handle.name
+                    writer.write_handle.close()
+                    shutil.move(temp_file_name, self.file_path)
+                    writer.write_handle = None
+                    self.moved_verified_blob = True
+                    return True
+                else:
+                    raise DownloadCanceledError()
+
+        return threads.deferToThread(move_file)
+
+
+class TempBlob(HashBlob):
+    """A HashBlob which will only exist in memory"""
+    def __init__(self, *args):
+        HashBlob.__init__(self, *args)
+        self.data_buffer = ""
+
+    def open_for_writing(self, peer):
+        if not peer in self.writers:
+            temp_buffer = StringIO()
+            finished_deferred = defer.Deferred()
+            writer = HashBlobWriter(temp_buffer, self.get_length, self.writer_finished)
+
+            self.writers[peer] = (writer, finished_deferred)
+            return finished_deferred, writer.write, writer.cancel
+        return None, None, None
+
+    def open_for_reading(self):
+        if self.verified is True:
+            return StringIO(self.data_buffer)
+        return None
+
+    def delete(self):
+        if not self.writers and not self.readers:
+            self.verified = False
+            self.data_buffer = ''
+            return defer.succeed(True)
+        else:
+            return defer.fail(Failure(ValueError("Blob is currently being read or written and cannot be deleted")))
+
+    def close_read_handle(self, file_handle):
+        file_handle.close()
+
+    def _close_writer(self, writer):
+        if writer.write_handle is not None:
+            writer.write_handle.close()
+            writer.write_handle = None
+
+    def _save_verified_blob(self, writer):
+        if not self.data_buffer:
+            self.data_buffer = writer.write_handle.getvalue()
+            writer.write_handle.close()
+            writer.write_handle = None
+            return defer.succeed(True)
+        else:
+            return defer.fail(Failure(DownloadCanceledError()))
+
+
+class HashBlobCreator(object):
+    def __init__(self, blob_manager):
+        self.blob_manager = blob_manager
+        self.hashsum = get_lbry_hash_obj()
+        self.len_so_far = 0
+        self.blob_hash = None
+        self.length = None
+
+    def open(self):
+        pass
+
+    def close(self):
+        self.length = self.len_so_far
+        if self.length == 0:
+            self.blob_hash = None
+        else:
+            self.blob_hash = self.hashsum.hexdigest()
+        d = self._close()
+
+        if self.blob_hash is not None:
+            d.addCallback(lambda _: self.blob_manager.creator_finished(self))
+            d.addCallback(lambda _: self.blob_hash)
+        else:
+            d.addCallback(lambda _: None)
+        return d
+
+    def write(self, data):
+        self.hashsum.update(data)
+        self.len_so_far += len(data)
+        self._write(data)
+
+    def _close(self):
+        pass
+
+    def _write(self, data):
+        pass
+
+
+class BlobFileCreator(HashBlobCreator):
+    def __init__(self, blob_manager, blob_dir):
+        HashBlobCreator.__init__(self, blob_manager)
+        self.blob_dir = blob_dir
+        self.out_file = tempfile.NamedTemporaryFile(delete=False, dir=self.blob_dir)
+
+    def _close(self):
+        temp_file_name = self.out_file.name
+        self.out_file.close()
+
+        def change_file_name():
+            shutil.move(temp_file_name, os.path.join(self.blob_dir, self.blob_hash))
+            return True
+
+        if self.blob_hash is not None:
+            d = threads.deferToThread(change_file_name)
+        else:
+            d = defer.succeed(True)
+        return d
+
+    def _write(self, data):
+        self.out_file.write(data)
+
+
+class TempBlobCreator(HashBlobCreator):
+    def __init__(self, blob_manager):
+        HashBlobCreator.__init__(self, blob_manager)
+        self.data_buffer = ''
+
+    def _close(self):
+        return defer.succeed(True)
+
+    def _write(self, data):
+        self.data_buffer += data

lbrynet/core/LBRYcrdWallet.py

@@ -0,0 +1,468 @@
+from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, ILBRYWallet
+from lbrynet.core.client.ClientRequest import ClientRequest
+from lbrynet.core.Error import UnknownNameError, InvalidStreamInfoError, RequestCanceledError
+from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException
+from twisted.internet import threads, reactor, defer, task
+from twisted.python.failure import Failure
+from collections import defaultdict, deque
+from zope.interface import implements
+from decimal import Decimal
+import datetime
+import logging
+import json
+import subprocess
+import socket
+import time
+import os
+
+
+class ReservedPoints(object):
+    def __init__(self, identifier, amount):
+        self.identifier = identifier
+        self.amount = amount
+
+
+class LBRYcrdWallet(object):
+    """This class implements the LBRYWallet interface for the LBRYcrd payment system"""
+    implements(ILBRYWallet)
+
+    def __init__(self, rpc_user, rpc_pass, rpc_url, rpc_port, start_lbrycrdd=False,
+                 wallet_dir=None, wallet_conf=None):
+        self.rpc_conn_string = "http://%s:%s@%s:%s" % (rpc_user, rpc_pass, rpc_url, str(rpc_port))
+        self.next_manage_call = None
+        self.wallet_balance = Decimal(0.0)
+        self.total_reserved_points = Decimal(0.0)
+        self.peer_addresses = {}  # {Peer: string}
+        self.queued_payments = defaultdict(Decimal)  # {address(string): amount(Decimal)}
+        self.expected_balances = defaultdict(Decimal)  # {address(string): amount(Decimal)}
+        self.current_address_given_to_peer = {}  # {Peer: address(string)}
+        self.expected_balance_at_time = deque()  # (Peer, address(string), amount(Decimal), time(datetime), count(int),
+                                                 # incremental_amount(float))
+        self.max_expected_payment_time = datetime.timedelta(minutes=3)
+        self.stopped = True
+        self.start_lbrycrdd = start_lbrycrdd
+        self.started_lbrycrdd = False
+        self.wallet_dir = wallet_dir
+        self.wallet_conf = wallet_conf
+        self.lbrycrdd = None
+        self.manage_running = False
+
+    def start(self):
+
+        def make_connection():
+            if self.start_lbrycrdd is True:
+                self._start_daemon()
+            logging.info("Trying to connect to %s", self.rpc_conn_string)
+            self.rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+            logging.info("Connected!")
+
+        def start_manage():
+            self.stopped = False
+            self.manage()
+            return True
+
+        d = threads.deferToThread(make_connection)
+        d.addCallback(lambda _: start_manage())
+        return d
+
+    def stop(self):
+        self.stopped = True
+        # If self.next_manage_call is None, then manage is currently running or else
+        # start has not been called, so set stopped and do nothing else.
+        if self.next_manage_call is not None:
+            self.next_manage_call.cancel()
+            self.next_manage_call = None
+
+        d = self.manage()
+        if self.start_lbrycrdd is True:
+            d.addBoth(lambda _: self._stop_daemon())
+        return d
+
+    def manage(self):
+        logging.info("Doing manage")
+        self.next_manage_call = None
+        have_set_manage_running = [False]
+
+        def check_if_manage_running():
+
+            d = defer.Deferred()
+
+            def fire_if_not_running():
+                if self.manage_running is False:
+                    self.manage_running = True
+                    have_set_manage_running[0] = True
+                    d.callback(True)
+                else:
+                    task.deferLater(reactor, 1, fire_if_not_running)
+
+            fire_if_not_running()
+            return d
+
+        d = check_if_manage_running()
+
+        d.addCallback(lambda _: self._check_expected_balances())
+
+        d.addCallback(lambda _: self._send_payments())
+
+        d.addCallback(lambda _: threads.deferToThread(self._get_wallet_balance))
+
+        def set_wallet_balance(balance):
+            self.wallet_balance = balance
+
+        d.addCallback(set_wallet_balance)
+
+        def set_next_manage_call():
+            if not self.stopped:
+                self.next_manage_call = reactor.callLater(60, self.manage)
+
+        d.addCallback(lambda _: set_next_manage_call())
+
+        def log_error(err):
+            logging.error("Something went wrong during manage. Error message: %s", err.getErrorMessage())
+            return err
+
+        d.addErrback(log_error)
+
+        def set_manage_not_running(arg):
+            if have_set_manage_running[0] is True:
+                self.manage_running = False
+            return arg
+
+        d.addBoth(set_manage_not_running)
+        return d
+
+    def get_info_exchanger(self):
+        return LBRYcrdAddressRequester(self)
+
+    def get_wallet_info_query_handler_factory(self):
+        return LBRYcrdAddressQueryHandlerFactory(self)
+
+    def get_balance(self):
+        d = threads.deferToThread(self._get_wallet_balance)
+        return d
+
+    def reserve_points(self, peer, amount):
+        """
+        Ensure a certain amount of points are available to be sent as payment, before the service is rendered
+
+        @param peer: The peer to which the payment will ultimately be sent
+
+        @param amount: The amount of points to reserve
+
+        @return: A ReservedPoints object which is given to send_points once the service has been rendered
+        """
+        rounded_amount = Decimal(str(round(amount, 8)))
+        #if peer in self.peer_addresses:
+        if self.wallet_balance >= self.total_reserved_points + rounded_amount:
+            self.total_reserved_points += rounded_amount
+            return ReservedPoints(peer, rounded_amount)
+        return None
+
+    def cancel_point_reservation(self, reserved_points):
+        """
+        Return all of the points that were reserved previously for some ReservedPoints object
+
+        @param reserved_points: ReservedPoints previously returned by reserve_points
+
+        @return: None
+        """
+        self.total_reserved_points -= reserved_points.amount
+
+    def send_points(self, reserved_points, amount):
+        """
+        Schedule a payment to be sent to a peer
+
+        @param reserved_points: ReservedPoints object previously returned by reserve_points
+
+        @param amount: amount of points to actually send, must be less than or equal to the
+            amount reserved in reserved_points
+
+        @return: Deferred which fires when the payment has been scheduled
+        """
+        rounded_amount = Decimal(str(round(amount, 8)))
+        peer = reserved_points.identifier
+        assert(rounded_amount <= reserved_points.amount)
+        assert(peer in self.peer_addresses)
+        self.queued_payments[self.peer_addresses[peer]] += rounded_amount
+        # make any unused points available
+        self.total_reserved_points -= (reserved_points.amount - rounded_amount)
+        logging.info("ordering that %s points be sent to %s", str(rounded_amount),
+                     str(self.peer_addresses[peer]))
+        peer.update_stats('points_sent', amount)
+        return defer.succeed(True)
+
+    def add_expected_payment(self, peer, amount):
+        """Increase the number of points expected to be paid by a peer"""
+        rounded_amount = Decimal(str(round(amount, 8)))
+        assert(peer in self.current_address_given_to_peer)
+        address = self.current_address_given_to_peer[peer]
+        logging.info("expecting a payment at address %s in the amount of %s", str(address), str(rounded_amount))
+        self.expected_balances[address] += rounded_amount
+        expected_balance = self.expected_balances[address]
+        expected_time = datetime.datetime.now() + self.max_expected_payment_time
+        self.expected_balance_at_time.append((peer, address, expected_balance, expected_time, 0, amount))
+        peer.update_stats('expected_points', amount)
+
+    def update_peer_address(self, peer, address):
+        self.peer_addresses[peer] = address
+
+    def get_new_address_for_peer(self, peer):
+        def set_address_for_peer(address):
+            self.current_address_given_to_peer[peer] = address
+            return address
+        d = threads.deferToThread(self._get_new_address)
+        d.addCallback(set_address_for_peer)
+        return d
+
+    def get_stream_info_for_name(self, name):
+
+        def get_stream_info_from_value(result):
+            r_dict = {}
+            if 'value' in result:
+                value = result['value']
+                try:
+                    value_dict = json.loads(value)
+                except ValueError:
+                    return Failure(InvalidStreamInfoError(name))
+                if 'stream_hash' in value_dict:
+                    r_dict['stream_hash'] = value_dict['stream_hash']
+                if 'name' in value_dict:
+                    r_dict['name'] = value_dict['name']
+                if 'description' in value_dict:
+                    r_dict['description'] = value_dict['description']
+                return r_dict
+            return Failure(UnknownNameError(name))
+
+        d = threads.deferToThread(self._get_value_for_name, name)
+        d.addCallback(get_stream_info_from_value)
+        return d
+
+    def claim_name(self, stream_hash, name, amount):
+        value = json.dumps({"stream_hash": stream_hash})
+        d = threads.deferToThread(self._claim_name, name, value, amount)
+        return d
+
+    def get_available_balance(self):
+        return float(self.wallet_balance - self.total_reserved_points)
+
+    def get_new_address(self):
+        return threads.deferToThread(self._get_new_address)
+
+    def _start_daemon(self):
+
+        if os.name == "nt":
+            si = subprocess.STARTUPINFO
+            si.dwFlags = subprocess.STARTF_USESHOWWINDOW
+            si.wShowWindow = subprocess.SW_HIDE
+            self.lbrycrdd = subprocess.Popen(["lbrycrdd.exe", "-datadir=%s" % self.wallet_dir,
+                                              "-conf=%s" % self.wallet_conf], startupinfo=si)
+        else:
+            self.lbrycrdd = subprocess.Popen(["./lbrycrdd", "-datadir=%s" % self.wallet_dir,
+                                              "-conf=%s" % self.wallet_conf])
+        self.started_lbrycrdd = True
+
+        tries = 0
+        while tries < 5:
+            try:
+                rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+                rpc_conn.getinfo()
+                break
+            except (socket.error, JSONRPCException):
+                tries += 1
+                logging.warning("Failed to connect to lbrycrdd.")
+                if tries < 5:
+                    time.sleep(2 ** tries)
+                    logging.warning("Trying again in %d seconds", 2 ** tries)
+                else:
+                    logging.warning("Giving up.")
+        else:
+            self.lbrycrdd.terminate()
+            raise ValueError("Couldn't open lbrycrdd")
+
+    def _stop_daemon(self):
+        if self.lbrycrdd is not None and self.started_lbrycrdd is True:
+            d = threads.deferToThread(self._rpc_stop)
+            return d
+        return defer.succeed(True)
+
+    def _check_expected_balances(self):
+        now = datetime.datetime.now()
+        balances_to_check = []
+        try:
+            while self.expected_balance_at_time[0][3] < now:
+                balances_to_check.append(self.expected_balance_at_time.popleft())
+        except IndexError:
+            pass
+        ds = []
+        for balance_to_check in balances_to_check:
+            d = threads.deferToThread(self._check_expected_balance, balance_to_check)
+            ds.append(d)
+        dl = defer.DeferredList(ds)
+
+        def handle_checks(results):
+            from future_builtins import zip
+            for balance, (success, result) in zip(balances_to_check, results):
+                peer = balance[0]
+                if success is True:
+                    if result is False:
+                        if balance[4] <= 1:  # first or second strike, give them another chance
+                            new_expected_balance = (balance[0],
+                                                    balance[1],
+                                                    balance[2],
+                                                    datetime.datetime.now() + self.max_expected_payment_time,
+                                                    balance[4] + 1,
+                                                    balance[5])
+                            self.expected_balance_at_time.append(new_expected_balance)
+                            peer.update_score(-5.0)
+                        else:
+                            peer.update_score(-50.0)
+                    else:
+                        if balance[4] == 0:
+                            peer.update_score(balance[5])
+                        peer.update_stats('points_received', balance[5])
+                else:
+                    logging.warning("Something went wrong checking a balance. Peer: %s, account: %s,"
+                                    "expected balance: %s, expected time: %s, count: %s, error: %s",
+                                    str(balance[0]), str(balance[1]), str(balance[2]), str(balance[3]),
+                                    str(balance[4]), str(result.getErrorMessage()))
+
+        dl.addCallback(handle_checks)
+        return dl
+
+    def _check_expected_balance(self, expected_balance):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        logging.info("Checking balance of address %s", str(expected_balance[1]))
+        balance = rpc_conn.getreceivedbyaddress(expected_balance[1])
+        logging.debug("received balance: %s", str(balance))
+        logging.debug("expected balance: %s", str(expected_balance[2]))
+        return balance >= expected_balance[2]
+
+    def _send_payments(self):
+        logging.info("Trying to send payments, if there are any to be sent")
+
+        def do_send(payments):
+            rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+            rpc_conn.sendmany("", payments)
+
+        payments_to_send = {}
+        for address, points in self.queued_payments.items():
+            logging.info("Should be sending %s points to %s", str(points), str(address))
+            payments_to_send[address] = float(points)
+            self.total_reserved_points -= points
+            self.wallet_balance -= points
+            del self.queued_payments[address]
+        if payments_to_send:
+            logging.info("Creating a transaction with outputs %s", str(payments_to_send))
+            return threads.deferToThread(do_send, payments_to_send)
+        logging.info("There were no payments to send")
+        return defer.succeed(True)
+
+    def _get_wallet_balance(self):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        return rpc_conn.getbalance("")
+
+    def _get_new_address(self):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        return rpc_conn.getnewaddress()
+
+    def _get_value_for_name(self, name):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        return rpc_conn.getvalueforname(name)
+
+    def _claim_name(self, name, value, amount):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        return str(rpc_conn.claimname(name, value, amount))
+
+    def _rpc_stop(self):
+        rpc_conn = AuthServiceProxy(self.rpc_conn_string)
+        rpc_conn.stop()
+        self.lbrycrdd.wait()
+
+
+class LBRYcrdAddressRequester(object):
+    implements([IRequestCreator])
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+        self._protocols = []
+
+    ######### IRequestCreator #########
+
+    def send_next_request(self, peer, protocol):
+
+        if not protocol in self._protocols:
+            r = ClientRequest({'lbrycrd_address': True}, 'lbrycrd_address')
+            d = protocol.add_request(r)
+            d.addCallback(self._handle_address_response, peer, r, protocol)
+            d.addErrback(self._request_failed, peer)
+            self._protocols.append(protocol)
+            return defer.succeed(True)
+        else:
+            return defer.succeed(False)
+
+    ######### internal calls #########
+
+    def _handle_address_response(self, response_dict, peer, request, protocol):
+        assert request.response_identifier in response_dict, \
+            "Expected %s in dict but did not get it" % request.response_identifier
+        assert protocol in self._protocols, "Responding protocol is not in our list of protocols"
+        address = response_dict[request.response_identifier]
+        self.wallet.update_peer_address(peer, address)
+
+    def _request_failed(self, err, peer):
+        if not err.check(RequestCanceledError):
+            logging.warning("A peer failed to send a valid public key response. Error: %s, peer: %s",
+                            err.getErrorMessage(), str(peer))
+            #return err
+
+
+class LBRYcrdAddressQueryHandlerFactory(object):
+    implements(IQueryHandlerFactory)
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+
+    ######### IQueryHandlerFactory #########
+
+    def build_query_handler(self):
+        q_h = LBRYcrdAddressQueryHandler(self.wallet)
+        return q_h
+
+    def get_primary_query_identifier(self):
+        return 'lbrycrd_address'
+
+    def get_description(self):
+        return "LBRYcrd Address - an address for receiving payments via LBRYcrd"
+
+
+class LBRYcrdAddressQueryHandler(object):
+    implements(IQueryHandler)
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+        self.query_identifiers = ['lbrycrd_address']
+        self.address = None
+        self.peer = None
+
+    ######### IQueryHandler #########
+
+    def register_with_request_handler(self, request_handler, peer):
+        self.peer = peer
+        request_handler.register_query_handler(self, self.query_identifiers)
+
+    def handle_queries(self, queries):
+
+        def create_response(address):
+            self.address = address
+            fields = {'lbrycrd_address': address}
+            return fields
+
+        if self.query_identifiers[0] in queries:
+            d = self.wallet.get_new_address_for_peer(self.peer)
+            d.addCallback(create_response)
+            return d
+        if self.address is None:
+            logging.warning("Expected a request for an address, but did not receive one")
+            return defer.fail(Failure(ValueError("Expected but did not receive an address request")))
+        else:
+            return defer.succeed({})

lbrynet/core/PTCWallet.py

@@ -0,0 +1,315 @@
+from collections import defaultdict
+import logging
+import leveldb
+import os
+import time
+from Crypto.Hash import SHA512
+from Crypto.PublicKey import RSA
+from lbrynet.core.client.ClientRequest import ClientRequest
+from lbrynet.core.Error import RequestCanceledError
+from lbrynet.interfaces import IRequestCreator, IQueryHandlerFactory, IQueryHandler, ILBRYWallet
+from lbrynet.pointtraderclient import pointtraderclient
+from twisted.internet import defer, threads
+from zope.interface import implements
+from twisted.python.failure import Failure
+from lbrynet.core.LBRYcrdWallet import ReservedPoints
+
+
+class PTCWallet(object):
+    """This class sends payments to peers and also ensures that expected payments are received.
+       This class is only intended to be used for testing."""
+    implements(ILBRYWallet)
+
+    def __init__(self, db_dir):
+        self.db_dir = db_dir
+        self.db = None
+        self.private_key = None
+        self.encoded_public_key = None
+        self.peer_pub_keys = {}
+        self.queued_payments = defaultdict(int)
+        self.expected_payments = defaultdict(list)
+        self.received_payments = defaultdict(list)
+        self.next_manage_call = None
+        self.payment_check_window = 3 * 60  # 3 minutes
+        self.new_payments_expected_time = time.time() - self.payment_check_window
+        self.known_transactions = []
+        self.total_reserved_points = 0.0
+        self.wallet_balance = 0.0
+
+    def manage(self):
+        """Send payments, ensure expected payments are received"""
+
+        from twisted.internet import reactor
+
+        if time.time() < self.new_payments_expected_time + self.payment_check_window:
+            d1 = self._get_new_payments()
+        else:
+            d1 = defer.succeed(None)
+        d1.addCallback(lambda _: self._check_good_standing())
+        d2 = self._send_queued_points()
+        self.next_manage_call = reactor.callLater(60, self.manage)
+        dl = defer.DeferredList([d1, d2])
+        dl.addCallback(lambda _: self.get_balance())
+
+        def set_balance(balance):
+            self.wallet_balance = balance
+
+        dl.addCallback(set_balance)
+        return dl
+
+    def stop(self):
+        if self.next_manage_call is not None:
+            self.next_manage_call.cancel()
+            self.next_manage_call = None
+        d = self.manage()
+        self.next_manage_call.cancel()
+        self.next_manage_call = None
+        self.db = None
+        return d
+
+    def start(self):
+
+        def save_key(success, private_key):
+            if success is True:
+                threads.deferToThread(self.save_private_key,  private_key.exportKey())
+                return True
+            return False
+
+        def register_private_key(private_key):
+            self.private_key = private_key
+            self.encoded_public_key = self.private_key.publickey().exportKey()
+            d_r = pointtraderclient.register_new_account(private_key)
+            d_r.addCallback(save_key, private_key)
+            return d_r
+
+        def ensure_private_key_exists(encoded_private_key):
+            if encoded_private_key is not None:
+                self.private_key = RSA.importKey(encoded_private_key)
+                self.encoded_public_key = self.private_key.publickey().exportKey()
+                return True
+            else:
+                create_d = threads.deferToThread(RSA.generate, 4096)
+                create_d.addCallback(register_private_key)
+                return create_d
+
+        def start_manage():
+            self.manage()
+            return True
+        d = threads.deferToThread(self._open_db)
+        d.addCallback(lambda _: threads.deferToThread(self.get_wallet_private_key))
+        d.addCallback(ensure_private_key_exists)
+        d.addCallback(lambda _: start_manage())
+        return d
+
+    def get_info_exchanger(self):
+        return PointTraderKeyExchanger(self)
+
+    def get_wallet_info_query_handler_factory(self):
+        return PointTraderKeyQueryHandlerFactory(self)
+
+    def reserve_points(self, peer, amount):
+        """
+        Ensure a certain amount of points are available to be sent as payment, before the service is rendered
+
+        @param peer: The peer to which the payment will ultimately be sent
+
+        @param amount: The amount of points to reserve
+
+        @return: A ReservedPoints object which is given to send_points once the service has been rendered
+        """
+        if self.wallet_balance >= self.total_reserved_points + amount:
+            self.total_reserved_points += amount
+            return ReservedPoints(peer, amount)
+        return None
+
+    def cancel_point_reservation(self, reserved_points):
+        """
+        Return all of the points that were reserved previously for some ReservedPoints object
+
+        @param reserved_points: ReservedPoints previously returned by reserve_points
+
+        @return: None
+        """
+        self.total_reserved_points -= reserved_points.amount
+
+    def send_points(self, reserved_points, amount):
+        """
+        Schedule a payment to be sent to a peer
+
+        @param reserved_points: ReservedPoints object previously returned by reserve_points
+
+        @param amount: amount of points to actually send, must be less than or equal to the
+            amount reserved in reserved_points
+
+        @return: Deferred which fires when the payment has been scheduled
+        """
+        self.queued_payments[reserved_points.identifier] += amount
+        # make any unused points available
+        self.total_reserved_points -= reserved_points.amount - amount
+        reserved_points.identifier.update_stats('points_sent', amount)
+        d = defer.succeed(True)
+        return d
+
+    def _send_queued_points(self):
+        ds = []
+        for peer, points in self.queued_payments.items():
+            if peer in self.peer_pub_keys:
+                d = pointtraderclient.send_points(self.private_key, self.peer_pub_keys[peer], points)
+                self.wallet_balance -= points
+                self.total_reserved_points -= points
+                ds.append(d)
+                del self.queued_payments[peer]
+            else:
+                logging.warning("Don't have a payment address for peer %s. Can't send %s points.",
+                                str(peer), str(points))
+        return defer.DeferredList(ds)
+
+    def get_balance(self):
+        """Return the balance of this wallet"""
+        d = pointtraderclient.get_balance(self.private_key)
+        return d
+
+    def add_expected_payment(self, peer, amount):
+        """Increase the number of points expected to be paid by a peer"""
+        self.expected_payments[peer].append((amount, time.time()))
+        self.new_payments_expected_time = time.time()
+        peer.update_stats('expected_points', amount)
+
+    def set_public_key_for_peer(self, peer, pub_key):
+        self.peer_pub_keys[peer] = pub_key
+
+    def _get_new_payments(self):
+
+        def add_new_transactions(transactions):
+            for transaction in transactions:
+                if transaction[1] == self.encoded_public_key:
+                    t_hash = SHA512.new()
+                    t_hash.update(transaction[0])
+                    t_hash.update(transaction[1])
+                    t_hash.update(str(transaction[2]))
+                    t_hash.update(transaction[3])
+                    if t_hash.hexdigest() not in self.known_transactions:
+                        self.known_transactions.append(t_hash.hexdigest())
+                        self._add_received_payment(transaction[0], transaction[2])
+
+        d = pointtraderclient.get_recent_transactions(self.private_key)
+        d.addCallback(add_new_transactions)
+        return d
+
+    def _add_received_payment(self, encoded_other_public_key, amount):
+        self.received_payments[encoded_other_public_key].append((amount, time.time()))
+
+    def _check_good_standing(self):
+        for peer, expected_payments in self.expected_payments.iteritems():
+            expected_cutoff = time.time() - 90
+            min_expected_balance = sum([a[0] for a in expected_payments if a[1] < expected_cutoff])
+            received_balance = 0
+            if self.peer_pub_keys[peer] in self.received_payments:
+                received_balance = sum([a[0] for a in self.received_payments[self.peer_pub_keys[peer]]])
+            if min_expected_balance > received_balance:
+                logging.warning("Account in bad standing: %s (pub_key: %s), expected amount = %s, received_amount = %s",
+                                str(peer), self.peer_pub_keys[peer], str(min_expected_balance), str(received_balance))
+
+    def _open_db(self):
+        self.db = leveldb.LevelDB(os.path.join(self.db_dir, "ptcwallet.db"))
+
+    def save_private_key(self, private_key):
+        self.db.Put("private_key", private_key)
+
+    def get_wallet_private_key(self):
+        try:
+            return self.db.Get("private_key")
+        except KeyError:
+            return None
+
+
+class PointTraderKeyExchanger(object):
+    implements([IRequestCreator])
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+        self._protocols = []
+
+    ######### IRequestCreator #########
+
+    def send_next_request(self, peer, protocol):
+        if not protocol in self._protocols:
+            r = ClientRequest({'public_key': self.wallet.encoded_public_key},
+                              'public_key')
+            d = protocol.add_request(r)
+            d.addCallback(self._handle_exchange_response, peer, r, protocol)
+            d.addErrback(self._request_failed, peer)
+            self._protocols.append(protocol)
+            return defer.succeed(True)
+        else:
+            return defer.succeed(False)
+
+    ######### internal calls #########
+
+    def _handle_exchange_response(self, response_dict, peer, request, protocol):
+        assert request.response_identifier in response_dict, \
+            "Expected %s in dict but did not get it" % request.response_identifier
+        assert protocol in self._protocols, "Responding protocol is not in our list of protocols"
+        peer_pub_key = response_dict[request.response_identifier]
+        self.wallet.set_public_key_for_peer(peer, peer_pub_key)
+        return True
+
+    def _request_failed(self, err, peer):
+        if not err.check(RequestCanceledError):
+            logging.warning("A peer failed to send a valid public key response. Error: %s, peer: %s",
+                            err.getErrorMessage(), str(peer))
+            #return err
+
+
+class PointTraderKeyQueryHandlerFactory(object):
+    implements(IQueryHandlerFactory)
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+
+    ######### IQueryHandlerFactory #########
+
+    def build_query_handler(self):
+        q_h = PointTraderKeyQueryHandler(self.wallet)
+        return q_h
+
+    def get_primary_query_identifier(self):
+        return 'public_key'
+
+    def get_description(self):
+        return "Point Trader Address - an address for receiving payments on the point trader testing network"
+
+
+class PointTraderKeyQueryHandler(object):
+    implements(IQueryHandler)
+
+    def __init__(self, wallet):
+        self.wallet = wallet
+        self.query_identifiers = ['public_key']
+        self.public_key = None
+        self.peer = None
+
+    ######### IQueryHandler #########
+
+    def register_with_request_handler(self, request_handler, peer):
+        self.peer = peer
+        request_handler.register_query_handler(self, self.query_identifiers)
+
+    def handle_queries(self, queries):
+        if self.query_identifiers[0] in queries:
+            new_encoded_pub_key = queries[self.query_identifiers[0]]
+            try:
+                RSA.importKey(new_encoded_pub_key)
+            except (ValueError, TypeError, IndexError):
+                logging.warning("Client sent an invalid public key.")
+                return defer.fail(Failure(ValueError("Client sent an invalid public key")))
+            self.public_key = new_encoded_pub_key
+            self.wallet.set_public_key_for_peer(self.peer, self.public_key)
+            logging.debug("Received the client's public key: %s", str(self.public_key))
+            fields = {'public_key': self.wallet.encoded_public_key}
+            return defer.succeed(fields)
+        if self.public_key is None:
+            logging.warning("Expected a public key, but did not receive one")
+            return defer.fail(Failure(ValueError("Expected but did not receive a public key")))
+        else:
+            return defer.succeed({})

lbrynet/core/PaymentRateManager.py

@@ -0,0 +1,29 @@
+class BasePaymentRateManager(object):
+    def __init__(self, rate):
+        self.min_blob_data_payment_rate = rate
+
+
+class PaymentRateManager(object):
+    def __init__(self, base, rate=None):
+        """
+        @param base: a BasePaymentRateManager
+
+        @param rate: the min blob data payment rate
+        """
+        self.base = base
+        self.min_blob_data_payment_rate = rate
+        self.points_paid = 0.0
+
+    def get_rate_blob_data(self, peer):
+        return self.get_effective_min_blob_data_payment_rate()
+
+    def accept_rate_blob_data(self, peer, payment_rate):
+        return payment_rate >= self.get_effective_min_blob_data_payment_rate()
+
+    def get_effective_min_blob_data_payment_rate(self):
+        if self.min_blob_data_payment_rate is None:
+            return self.base.min_blob_data_payment_rate
+        return self.min_blob_data_payment_rate
+
+    def record_points_paid(self, amount):
+        self.points_paid += amount

lbrynet/core/Peer.py

@@ -0,0 +1,36 @@
+from collections import defaultdict
+import datetime
+
+
+class Peer(object):
+    def __init__(self, host, port):
+        self.host = host
+        self.port = port
+        self.attempt_connection_at = None
+        self.down_count = 0
+        self.score = 0
+        self.stats = defaultdict(float)  # {string stat_type, float count}
+
+    def is_available(self):
+        if (self.attempt_connection_at is None or
+            datetime.datetime.today() > self.attempt_connection_at):
+            return True
+        return False
+
+    def report_up(self):
+        self.down_count = 0
+        self.attempt_connection_at = None
+
+    def report_down(self):
+        self.down_count += 1
+        timeout_time = datetime.timedelta(seconds=60 * self.down_count)
+        self.attempt_connection_at = datetime.datetime.today() + timeout_time
+
+    def update_score(self, score_change):
+        self.score += score_change
+
+    def update_stats(self, stat_type, count):
+        self.stats[stat_type] += count
+
+    def __str__(self):
+        return self.host + ":" + str(self.port)

lbrynet/core/PeerFinder.py

@@ -0,0 +1,19 @@
+from twisted.internet import defer
+
+
+class DummyPeerFinder(object):
+    """This class finds peers which have announced to the DHT that they have certain blobs"""
+    def __init__(self):
+        pass
+
+    def run_manage_loop(self):
+        pass
+
+    def stop(self):
+        pass
+
+    def find_peers_for_blob(self, blob_hash):
+        return defer.succeed([])
+
+    def get_most_popular_hashes(self, num_to_return):
+        return []

lbrynet/core/PeerManager.py

@@ -0,0 +1,14 @@
+from lbrynet.core.Peer import Peer
+
+
+class PeerManager(object):
+    def __init__(self):
+        self.peers = []
+
+    def get_peer(self, host, port):
+        for peer in self.peers:
+            if peer.host == host and peer.port == port:
+                return peer
+        peer = Peer(host, port)
+        self.peers.append(peer)
+        return peer

lbrynet/core/RateLimiter.py

@@ -0,0 +1,206 @@
+from zope.interface import implements
+from lbrynet.interfaces import IRateLimiter
+
+
+class DummyRateLimiter(object):
+    def __init__(self):
+        self.dl_bytes_this_second = 0
+        self.ul_bytes_this_second = 0
+        self.total_dl_bytes = 0
+        self.total_ul_bytes = 0
+        self.target_dl = 0
+        self.target_ul = 0
+        self.ul_delay = 0.00
+        self.dl_delay = 0.00
+        self.next_tick = None
+
+    def tick(self):
+
+        from twisted.internet import reactor
+
+        self.dl_bytes_this_second = 0
+        self.ul_bytes_this_second = 0
+        self.next_tick = reactor.callLater(1.0, self.tick)
+
+    def stop(self):
+        if self.next_tick is not None:
+            self.next_tick.cancel()
+            self.next_tick = None
+
+    def set_dl_limit(self, limit):
+        pass
+
+    def set_ul_limit(self, limit):
+        pass
+
+    def ul_wait_time(self):
+        return self.ul_delay
+
+    def dl_wait_time(self):
+        return self.dl_delay
+
+    def report_dl_bytes(self, num_bytes):
+        self.dl_bytes_this_second += num_bytes
+        self.total_dl_bytes += num_bytes
+
+    def report_ul_bytes(self, num_bytes):
+        self.ul_bytes_this_second += num_bytes
+        self.total_ul_bytes += num_bytes
+
+
+class RateLimiter(object):
+    """This class ensures that upload and download rates don't exceed specified maximums"""
+
+    implements(IRateLimiter)
+
+    #called by main application
+
+    def __init__(self, max_dl_bytes=None, max_ul_bytes=None):
+        self.max_dl_bytes = max_dl_bytes
+        self.max_ul_bytes = max_ul_bytes
+        self.dl_bytes_this_second = 0
+        self.ul_bytes_this_second = 0
+        self.total_dl_bytes = 0
+        self.total_ul_bytes = 0
+        self.next_tick = None
+        self.next_unthrottle_dl = None
+        self.next_unthrottle_ul = None
+
+        self.next_dl_check = None
+        self.next_ul_check = None
+
+        self.dl_check_interval = 1.0
+        self.ul_check_interval = 1.0
+
+        self.dl_throttled = False
+        self.ul_throttled = False
+
+        self.protocols = []
+
+    def tick(self):
+
+        from twisted.internet import reactor
+
+        # happens once per second
+        if self.next_dl_check is not None:
+            self.next_dl_check.cancel()
+            self.next_dl_check = None
+        if self.next_ul_check is not None:
+            self.next_ul_check.cancel()
+            self.next_ul_check = None
+        if self.max_dl_bytes is not None:
+            if self.dl_bytes_this_second == 0:
+                self.dl_check_interval = 1.0
+            else:
+                self.dl_check_interval = min(1.0, self.dl_check_interval *
+                                             self.max_dl_bytes / self.dl_bytes_this_second)
+            self.next_dl_check = reactor.callLater(self.dl_check_interval, self.check_dl)
+        if self.max_ul_bytes is not None:
+            if self.ul_bytes_this_second == 0:
+                self.ul_check_interval = 1.0
+            else:
+                self.ul_check_interval = min(1.0, self.ul_check_interval *
+                                             self.max_ul_bytes / self.ul_bytes_this_second)
+            self.next_ul_check = reactor.callLater(self.ul_check_interval, self.check_ul)
+        self.dl_bytes_this_second = 0
+        self.ul_bytes_this_second = 0
+        self.unthrottle_dl()
+        self.unthrottle_ul()
+        self.next_tick = reactor.callLater(1.0, self.tick)
+
+    def stop(self):
+        if self.next_tick is not None:
+            self.next_tick.cancel()
+            self.next_tick = None
+        if self.next_dl_check is not None:
+            self.next_dl_check.cancel()
+            self.next_dl_check = None
+        if self.next_ul_check is not None:
+            self.next_ul_check.cancel()
+            self.next_ul_check = None
+
+    def set_dl_limit(self, limit):
+        self.max_dl_bytes = limit
+
+    def set_ul_limit(self, limit):
+        self.max_ul_bytes = limit
+
+    #throttling
+
+    def check_dl(self):
+
+        from twisted.internet import reactor
+
+        self.next_dl_check = None
+
+        if self.dl_bytes_this_second > self.max_dl_bytes:
+            self.throttle_dl()
+        else:
+            self.next_dl_check = reactor.callLater(self.dl_check_interval, self.check_dl)
+            self.dl_check_interval = min(self.dl_check_interval * 2, 1.0)
+
+    def check_ul(self):
+
+        from twisted.internet import reactor
+
+        self.next_ul_check = None
+
+        if self.ul_bytes_this_second > self.max_ul_bytes:
+            self.throttle_ul()
+        else:
+            self.next_ul_check = reactor.callLater(self.ul_check_interval, self.check_ul)
+            self.ul_check_interval = min(self.ul_check_interval * 2, 1.0)
+
+    def throttle_dl(self):
+        if self.dl_throttled is False:
+            for protocol in self.protocols:
+                protocol.throttle_download()
+            self.dl_throttled = True
+
+    def throttle_ul(self):
+        if self.ul_throttled is False:
+            for protocol in self.protocols:
+                protocol.throttle_upload()
+            self.ul_throttled = True
+
+    def unthrottle_dl(self):
+        if self.dl_throttled is True:
+            for protocol in self.protocols:
+                protocol.unthrottle_download()
+            self.dl_throttled = False
+
+    def unthrottle_ul(self):
+        if self.ul_throttled is True:
+            for protocol in self.protocols:
+                protocol.unthrottle_upload()
+            self.ul_throttled = False
+
+    #deprecated
+
+    def ul_wait_time(self):
+        return 0
+
+    def dl_wait_time(self):
+        return 0
+
+    #called by protocols
+
+    def report_dl_bytes(self, num_bytes):
+        self.dl_bytes_this_second += num_bytes
+        self.total_dl_bytes += num_bytes
+
+    def report_ul_bytes(self, num_bytes):
+        self.ul_bytes_this_second += num_bytes
+        self.total_ul_bytes += num_bytes
+
+    def register_protocol(self, protocol):
+        if protocol not in self.protocols:
+            self.protocols.append(protocol)
+            if self.dl_throttled is True:
+                protocol.throttle_download()
+            if self.ul_throttled is True:
+                protocol.throttle_upload()
+
+    def unregister_protocol(self, protocol):
+        if protocol in self.protocols:
+            self.protocols.remove(protocol)

lbrynet/core/Session.py

@@ -0,0 +1,245 @@
+import logging
+import miniupnpc
+from lbrynet.core.PTCWallet import PTCWallet
+from lbrynet.core.BlobManager import DiskBlobManager, TempBlobManager
+from lbrynet.dht import node
+from lbrynet.core.PeerManager import PeerManager
+from lbrynet.core.RateLimiter import RateLimiter
+from lbrynet.core.client.DHTPeerFinder import DHTPeerFinder
+from lbrynet.core.HashAnnouncer import DummyHashAnnouncer
+from lbrynet.core.server.DHTHashAnnouncer import DHTHashAnnouncer
+from lbrynet.core.utils import generate_id
+from lbrynet.core.PaymentRateManager import BasePaymentRateManager
+from twisted.internet import threads, defer
+
+
+class LBRYSession(object):
+    """This class manages all important services common to any application that uses the network:
+    the hash announcer, which informs other peers that this peer is associated with some hash. Usually,
+    this means this peer has a blob identified by the hash in question, but it can be used for other
+    purposes.
+    the peer finder, which finds peers that are associated with some hash.
+    the blob manager, which keeps track of which blobs have been downloaded and provides access to them,
+    the rate limiter, which attempts to ensure download and upload rates stay below a set maximum,
+    and upnp, which opens holes in compatible firewalls so that remote peers can connect to this peer."""
+    def __init__(self, blob_data_payment_rate, db_dir=None, lbryid=None, peer_manager=None, dht_node_port=None,
+                 known_dht_nodes=None, peer_finder=None, hash_announcer=None,
+                 blob_dir=None, blob_manager=None, peer_port=None, use_upnp=True,
+                 rate_limiter=None, wallet=None):
+        """
+        @param blob_data_payment_rate: The default payment rate for blob data
+
+        @param db_dir: The directory in which levelDB files should be stored
+
+        @param lbryid: The unique ID of this node
+
+        @param peer_manager: An object which keeps track of all known peers. If None, a PeerManager will be created
+
+        @param dht_node_port: The port on which the dht node should listen for incoming connections
+
+        @param known_dht_nodes: A list of nodes which the dht node should use to bootstrap into the dht
+
+        @param peer_finder: An object which is used to look up peers that are associated with some hash. If None,
+            a DHTPeerFinder will be used, which looks for peers in the distributed hash table.
+
+        @param hash_announcer: An object which announces to other peers that this peer is associated with some hash.
+            If None, and peer_port is not None, a DHTHashAnnouncer will be used. If None and
+            peer_port is None, a DummyHashAnnouncer will be used, which will not actually announce
+            anything.
+
+        @param blob_dir: The directory in which blobs will be stored. If None and blob_manager is None, blobs will
+            be stored in memory only.
+
+        @param blob_manager: An object which keeps track of downloaded blobs and provides access to them. If None,
+            and blob_dir is not None, a DiskBlobManager will be used, with the given blob_dir.
+            If None and blob_dir is None, a TempBlobManager will be used, which stores blobs in
+            memory only.
+
+        @param peer_port: The port on which other peers should connect to this peer
+
+        @param use_upnp: Whether or not to try to open a hole in the firewall so that outside peers can connect to
+            this peer's peer_port and dht_node_port
+
+        @param rate_limiter: An object which keeps track of the amount of data transferred to and from this peer,
+            and can limit that rate if desired
+
+        @param wallet: An object which will be used to keep track of expected payments and which will pay peers.
+            If None, a wallet which uses the Point Trader system will be used, which is meant for testing
+            only
+
+        @return:
+        """
+        self.db_dir = db_dir
+
+        self.lbryid = lbryid
+
+        self.peer_manager = peer_manager
+
+        self.dht_node_port = dht_node_port
+        self.known_dht_nodes = known_dht_nodes
+        if self.known_dht_nodes is None:
+            self.known_dht_nodes = []
+        self.peer_finder = peer_finder
+        self.hash_announcer = hash_announcer
+
+        self.blob_dir = blob_dir
+        self.blob_manager = blob_manager
+
+        self.peer_port = peer_port
+
+        self.use_upnp = use_upnp
+
+        self.rate_limiter = rate_limiter
+
+        self.external_ip = '127.0.0.1'
+        self.upnp_handler = None
+        self.upnp_redirects_set = False
+
+        self.wallet = wallet
+
+        self.dht_node = None
+
+        self.base_payment_rate_manager = BasePaymentRateManager(blob_data_payment_rate)
+
+    def setup(self):
+        """Create the blob directory and database if necessary, start all desired services"""
+
+        logging.debug("Setting up the lbry session")
+
+        if self.lbryid is None:
+            self.lbryid = generate_id()
+
+        if self.wallet is None:
+            self.wallet = PTCWallet(self.db_dir)
+
+        if self.peer_manager is None:
+            self.peer_manager = PeerManager()
+
+        if self.use_upnp is True:
+            d = self._try_upnp()
+        else:
+            d = defer.succeed(True)
+
+        if self.peer_finder is None:
+            d.addCallback(lambda _: self._setup_dht())
+        else:
+            if self.hash_announcer is None and self.peer_port is not None:
+                logging.warning("The server has no way to advertise its available blobs.")
+                self.hash_announcer = DummyHashAnnouncer()
+
+        d.addCallback(lambda _: self._setup_other_components())
+        return d
+
+    def shut_down(self):
+        """Stop all services"""
+        ds = []
+        if self.dht_node is not None:
+            ds.append(defer.maybeDeferred(self.dht_node.stop))
+        ds.append(defer.maybeDeferred(self.rate_limiter.stop))
+        ds.append(defer.maybeDeferred(self.peer_finder.stop))
+        ds.append(defer.maybeDeferred(self.hash_announcer.stop))
+        ds.append(defer.maybeDeferred(self.wallet.stop))
+        ds.append(defer.maybeDeferred(self.blob_manager.stop))
+        if self.upnp_redirects_set is True:
+            ds.append(defer.maybeDeferred(self._unset_upnp))
+        return defer.DeferredList(ds)
+
+    def _try_upnp(self):
+
+        logging.debug("In _try_upnp")
+
+        def threaded_try_upnp():
+            if self.use_upnp is False:
+                logging.debug("Not using upnp")
+                return False
+            u = miniupnpc.UPnP()
+            num_devices_found = u.discover()
+            if num_devices_found > 0:
+                self.upnp_handler = u
+                u.selectigd()
+                external_ip = u.externalipaddress()
+                if external_ip != '0.0.0.0':
+                    self.external_ip = external_ip
+                if self.peer_port is not None:
+                    u.addportmapping(self.peer_port, 'TCP', u.lanaddr, self.peer_port, 'LBRY peer port', '')
+                if self.dht_node_port is not None:
+                    u.addportmapping(self.dht_node_port, 'UDP', u.lanaddr, self.dht_node_port, 'LBRY DHT port', '')
+                self.upnp_redirects_set = True
+                return True
+            return False
+
+        def upnp_failed(err):
+            logging.warning("UPnP failed. Reason: %s", err.getErrorMessage())
+            return False
+
+        d = threads.deferToThread(threaded_try_upnp)
+        d.addErrback(upnp_failed)
+        return d
+
+    def _setup_dht(self):
+
+        from twisted.internet import reactor
+
+        logging.debug("Starting the dht")
+
+        def match_port(h, p):
+            return h, p
+
+        def join_resolved_addresses(result):
+            addresses = []
+            for success, value in result:
+                if success is True:
+                    addresses.append(value)
+            return addresses
+
+        def start_dht(addresses):
+            self.dht_node.joinNetwork(addresses)
+            self.peer_finder.run_manage_loop()
+            self.hash_announcer.run_manage_loop()
+            return True
+
+        ds = []
+        for host, port in self.known_dht_nodes:
+            d = reactor.resolve(host)
+            d.addCallback(match_port, port)
+            ds.append(d)
+
+        self.dht_node = node.Node(udpPort=self.dht_node_port, lbryid=self.lbryid,
+                                  externalIP=self.external_ip)
+        self.peer_finder = DHTPeerFinder(self.dht_node, self.peer_manager)
+        if self.hash_announcer is None:
+            self.hash_announcer = DHTHashAnnouncer(self.dht_node, self.peer_port)
+
+        dl = defer.DeferredList(ds)
+        dl.addCallback(join_resolved_addresses)
+        dl.addCallback(start_dht)
+        return dl
+
+    def _setup_other_components(self):
+        logging.debug("Setting up the rest of the components")
+
+        if self.rate_limiter is None:
+            self.rate_limiter = RateLimiter()
+
+        if self.blob_manager is None:
+            if self.blob_dir is None:
+                self.blob_manager = TempBlobManager(self.hash_announcer)
+            else:
+                self.blob_manager = DiskBlobManager(self.hash_announcer, self.blob_dir, self.db_dir)
+
+        self.rate_limiter.tick()
+        d1 = self.blob_manager.setup()
+        d2 = self.wallet.start()
+        return defer.DeferredList([d1, d2], fireOnOneErrback=True)
+
+    def _unset_upnp(self):
+
+        def threaded_unset_upnp():
+            u = self.upnp_handler
+            if self.peer_port is not None:
+                u.deleteportmapping(self.peer_port, 'TCP')
+            if self.dht_node_port is not None:
+                u.deleteportmapping(self.dht_node_port, 'UDP')
+            self.upnp_redirects_set = False
+
+        return threads.deferToThread(threaded_unset_upnp)

lbrynet/core/StreamCreator.py

@@ -0,0 +1,73 @@
+import logging
+from twisted.internet import interfaces, defer
+from zope.interface import implements
+
+
+class StreamCreator(object):
+    """Classes which derive from this class create a 'stream', which can be any
+        collection of associated blobs and associated metadata. These classes
+        use the IConsumer interface to get data from an IProducer and transform
+        the data into a 'stream'"""
+
+    implements(interfaces.IConsumer)
+
+    def __init__(self, name):
+        """
+        @param name: the name of the stream
+        """
+        self.name = name
+        self.stopped = True
+        self.producer = None
+        self.streaming = None
+        self.blob_count = -1
+        self.current_blob = None
+        self.finished_deferreds = []
+
+    def _blob_finished(self, blob_info):
+        pass
+
+    def registerProducer(self, producer, streaming):
+
+        from twisted.internet import reactor
+
+        self.producer = producer
+        self.streaming = streaming
+        self.stopped = False
+        if streaming is False:
+            reactor.callLater(0, self.producer.resumeProducing)
+
+    def unregisterProducer(self):
+        self.stopped = True
+        self.producer = None
+
+    def stop(self):
+        """Stop creating the stream. Create the terminating zero-length blob."""
+        logging.debug("stop has been called for StreamCreator")
+        self.stopped = True
+        if self.current_blob is not None:
+            current_blob = self.current_blob
+            d = current_blob.close()
+            d.addCallback(self._blob_finished)
+            self.finished_deferreds.append(d)
+            self.current_blob = None
+        self._finalize()
+        dl = defer.DeferredList(self.finished_deferreds)
+        dl.addCallback(lambda _: self._finished())
+        return dl
+
+    def _finalize(self):
+        pass
+
+    def _finished(self):
+        pass
+
+    def write(self, data):
+
+        from twisted.internet import reactor
+
+        self._write(data)
+        if self.stopped is False and self.streaming is False:
+            reactor.callLater(0, self.producer.resumeProducing)
+
+    def _write(self, data):
+        pass

lbrynet/core/StreamDescriptor.py

@@ -0,0 +1,195 @@
+from collections import defaultdict
+import json
+import logging
+from twisted.internet import threads
+from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
+
+
+class StreamDescriptorReader(object):
+    """Classes which derive from this class read a stream descriptor file return
+       a dictionary containing the fields in the file"""
+    def __init__(self):
+        pass
+
+    def _get_raw_data(self):
+        """This method must be overridden by subclasses. It should return a deferred
+           which fires with the raw data in the stream descriptor"""
+        pass
+
+    def get_info(self):
+        """Return the fields contained in the file"""
+        d = self._get_raw_data()
+        d.addCallback(json.loads)
+        return d
+
+
+class PlainStreamDescriptorReader(StreamDescriptorReader):
+    """Read a stream descriptor file which is not a blob but a regular file"""
+    def __init__(self, stream_descriptor_filename):
+        StreamDescriptorReader.__init__(self)
+        self.stream_descriptor_filename = stream_descriptor_filename
+
+    def _get_raw_data(self):
+
+        def get_data():
+            with open(self.stream_descriptor_filename) as file_handle:
+                raw_data = file_handle.read()
+                return raw_data
+
+        return threads.deferToThread(get_data)
+
+
+class BlobStreamDescriptorReader(StreamDescriptorReader):
+    """Read a stream descriptor file which is a blob"""
+    def __init__(self, blob):
+        StreamDescriptorReader.__init__(self)
+        self.blob = blob
+
+    def _get_raw_data(self):
+
+        def get_data():
+            f = self.blob.open_for_reading()
+            if f is not None:
+                raw_data = f.read()
+                self.blob.close_read_handle(f)
+                return raw_data
+            else:
+                raise ValueError("Could not open the blob for reading")
+
+        return threads.deferToThread(get_data)
+
+
+class StreamDescriptorWriter(object):
+    """Classes which derive from this class write fields from a dictionary
+       of fields to a stream descriptor"""
+    def __init__(self):
+        pass
+
+    def create_descriptor(self, sd_info):
+        return self._write_stream_descriptor(json.dumps(sd_info))
+
+    def _write_stream_descriptor(self, raw_data):
+        """This method must be overridden by subclasses to write raw data to the stream descriptor"""
+        pass
+
+
+class PlainStreamDescriptorWriter(StreamDescriptorWriter):
+    def __init__(self, sd_file_name):
+        StreamDescriptorWriter.__init__(self)
+        self.sd_file_name = sd_file_name
+
+    def _write_stream_descriptor(self, raw_data):
+
+        def write_file():
+            logging.debug("Writing the sd file to disk")
+            with open(self.sd_file_name, 'w') as sd_file:
+                sd_file.write(raw_data)
+            return self.sd_file_name
+
+        return threads.deferToThread(write_file)
+
+
+class BlobStreamDescriptorWriter(StreamDescriptorWriter):
+    def __init__(self, blob_manager):
+        StreamDescriptorWriter.__init__(self)
+
+        self.blob_manager = blob_manager
+
+    def _write_stream_descriptor(self, raw_data):
+        logging.debug("Creating the new blob for the stream descriptor")
+        blob_creator = self.blob_manager.get_blob_creator()
+        blob_creator.write(raw_data)
+        logging.debug("Wrote the data to the new blob")
+        return blob_creator.close()
+
+
+class StreamDescriptorIdentifier(object):
+    """Tries to determine the type of stream described by the stream descriptor using the
+       'stream_type' field. Keeps a list of StreamDescriptorValidators and StreamDownloaderFactorys
+       and returns the appropriate ones based on the type of the stream descriptor given
+    """
+    def __init__(self):
+        self._sd_info_validators = {}  # {stream_type: IStreamDescriptorValidator}
+        self._stream_downloader_factories = defaultdict(list)  # {stream_type: [IStreamDownloaderFactory]}
+
+    def add_stream_info_validator(self, stream_type, sd_info_validator):
+        """
+        This is how the StreamDescriptorIdentifier learns about new types of stream descriptors.
+
+        There can only be one StreamDescriptorValidator for each type of stream.
+
+        @param stream_type: A string representing the type of stream descriptor. This must be unique to
+            this stream descriptor.
+
+        @param sd_info_validator: A class implementing the IStreamDescriptorValidator interface. This class's
+            constructor will be passed the raw metadata in the stream descriptor file and its 'validate' method
+            will then be called. If the validation step fails, an exception will be thrown, preventing the stream
+            descriptor from being further processed.
+
+        @return: None
+        """
+        self._sd_info_validators[stream_type] = sd_info_validator
+
+    def add_stream_downloader_factory(self, stream_type, factory):
+        """
+        Register a stream downloader factory with the StreamDescriptorIdentifier.
+
+        This is how the StreamDescriptorIdentifier determines what factories may be used to process different stream
+        descriptor files. There must be at least one factory for each type of stream added via
+        "add_stream_info_validator".
+
+        @param stream_type: A string representing the type of stream descriptor which the factory knows how to process.
+
+        @param factory: An object implementing the IStreamDownloaderFactory interface.
+
+        @return: None
+        """
+        self._stream_downloader_factories[stream_type].append(factory)
+
+    def get_info_and_factories_for_sd_file(self, sd_path):
+
+        sd_reader = PlainStreamDescriptorReader(sd_path)
+        d = sd_reader.get_info()
+        d.addCallback(self._return_info_and_factories)
+        return d
+
+    def get_info_and_factories_for_sd_blob(self, sd_blob):
+        sd_reader = BlobStreamDescriptorReader(sd_blob)
+        d = sd_reader.get_info()
+        d.addCallback(self._return_info_and_factories)
+        return d
+
+    def _get_factories(self, stream_type):
+        assert stream_type in self._stream_downloader_factories, "Unrecognized stream type: " + str(stream_type)
+        return self._stream_downloader_factories[stream_type]
+
+    def _get_validator(self, stream_type):
+        assert stream_type in self._sd_info_validators, "Unrecognized stream type: " + str(stream_type)
+        return self._sd_info_validators[stream_type]
+
+    def _return_info_and_factories(self, sd_info):
+        assert 'stream_type' in sd_info, 'Invalid stream descriptor. No stream_type parameter.'
+        stream_type = sd_info['stream_type']
+        factories = self._get_factories(stream_type)
+        validator = self._get_validator(stream_type)(sd_info)
+        d = validator.validate()
+
+        d.addCallback(lambda _: (validator, factories))
+        return d
+
+
+def download_sd_blob(session, blob_hash, payment_rate_manager):
+    """
+    Downloads a single blob from the network
+
+    @param session:
+
+    @param blob_hash:
+
+    @param payment_rate_manager:
+
+    @return: An object of type HashBlob
+    """
+    downloader = StandaloneBlobDownloader(blob_hash, session.blob_manager, session.peer_finder,
+                                          session.rate_limiter, payment_rate_manager, session.wallet)
+    return downloader.download()

lbrynet/core/__init__.py

@@ -0,0 +1,7 @@
+"""
+Classes and functions which can be used by any application wishing to make use of the LBRY network.
+
+This includes classes for connecting to other peers and downloading blobs from them, listening for
+connections from peers and responding to their requests, managing locally stored blobs, sending
+and receiving payments, and locating peers in the DHT.
+"""

lbrynet/core/client/BlobRequester.py

@@ -0,0 +1,307 @@
+from collections import defaultdict
+import logging
+from twisted.internet import defer
+from twisted.python.failure import Failure
+from zope.interface import implements
+from lbrynet.core.Error import PriceDisagreementError, DownloadCanceledError, InsufficientFundsError
+from lbrynet.core.Error import InvalidResponseError, RequestCanceledError, NoResponseError
+from lbrynet.core.client.ClientRequest import ClientRequest, ClientBlobRequest
+from lbrynet.interfaces import IRequestCreator
+
+
+class BlobRequester(object):
+    implements(IRequestCreator)
+
+    def __init__(self, blob_manager, peer_finder, payment_rate_manager, wallet, download_manager):
+        self.blob_manager = blob_manager
+        self.peer_finder = peer_finder
+        self.payment_rate_manager = payment_rate_manager
+        self.wallet = wallet
+        self.download_manager = download_manager
+        self._peers = defaultdict(int)  # {Peer: score}
+        self._available_blobs = defaultdict(list)  # {Peer: [blob_hash]}
+        self._unavailable_blobs = defaultdict(list)  # {Peer: [blob_hash]}}
+        self._protocol_prices = {}  # {ClientProtocol: price}
+        self._price_disagreements = []  # [Peer]
+        self._incompatible_peers = []
+
+    ######## IRequestCreator #########
+
+    def send_next_request(self, peer, protocol):
+        sent_request = False
+        if self._blobs_to_download() and self._should_send_request_to(peer):
+            a_r = self._get_availability_request(peer)
+            d_r = self._get_download_request(peer)
+            p_r = None
+
+            if a_r is not None or d_r is not None:
+                p_r = self._get_price_request(peer, protocol)
+
+            if a_r is not None:
+                d1 = protocol.add_request(a_r)
+                d1.addCallback(self._handle_availability, peer, a_r)
+                d1.addErrback(self._request_failed, "availability request", peer)
+                sent_request = True
+            if d_r is not None:
+                reserved_points = self._reserve_points(peer, protocol, d_r.max_pay_units)
+                if reserved_points is not None:
+                    # Note: The following three callbacks will be called when the blob has been
+                    # fully downloaded or canceled
+                    d_r.finished_deferred.addCallbacks(self._download_succeeded, self._download_failed,
+                                                       callbackArgs=(peer, d_r.blob),
+                                                       errbackArgs=(peer,))
+                    d_r.finished_deferred.addBoth(self._pay_or_cancel_payment, protocol, reserved_points, d_r.blob)
+                    d_r.finished_deferred.addErrback(self._handle_download_error, peer, d_r.blob)
+
+                    d2 = protocol.add_blob_request(d_r)
+                    # Note: The following two callbacks will be called as soon as the peer sends its
+                    # response, which will be before the blob has finished downloading, but may be
+                    # after the blob has been canceled. For example,
+                    # 1) client sends request to Peer A
+                    # 2) the blob is finished downloading from peer B, and therefore this one is canceled
+                    # 3) client receives response from Peer A
+                    # Therefore, these callbacks shouldn't rely on there being a blob about to be
+                    # downloaded.
+                    d2.addCallback(self._handle_incoming_blob, peer, d_r)
+                    d2.addErrback(self._request_failed, "download request", peer)
+
+                    sent_request = True
+                else:
+                    d_r.cancel(InsufficientFundsError())
+                    return defer.fail(InsufficientFundsError())
+            if sent_request is True:
+                if p_r is not None:
+                    d3 = protocol.add_request(p_r)
+                    d3.addCallback(self._handle_price_response, peer, p_r, protocol)
+                    d3.addErrback(self._request_failed, "price request", peer)
+        return defer.succeed(sent_request)
+
+    def get_new_peers(self):
+        d = self._get_hash_for_peer_search()
+        d.addCallback(self._find_peers_for_hash)
+        return d
+
+    ######### internal calls #########
+
+    def _download_succeeded(self, arg, peer, blob):
+        logging.info("Blob %s has been successfully downloaded from %s", str(blob), str(peer))
+        self._update_local_score(peer, 5.0)
+        peer.update_stats('blobs_downloaded', 1)
+        peer.update_score(5.0)
+        self.blob_manager.blob_completed(blob)
+        return arg
+
+    def _download_failed(self, reason, peer):
+        if not reason.check(DownloadCanceledError, PriceDisagreementError):
+            self._update_local_score(peer, -10.0)
+        return reason
+
+    def _pay_or_cancel_payment(self, arg, protocol, reserved_points, blob):
+        if blob.length != 0 and (not isinstance(arg, Failure) or arg.check(DownloadCanceledError)):
+            self._pay_peer(protocol, blob.length, reserved_points)
+        else:
+            self._cancel_points(reserved_points)
+        return arg
+
+    def _handle_download_error(self, err, peer, blob_to_download):
+        if not err.check(DownloadCanceledError, PriceDisagreementError, RequestCanceledError):
+            logging.warning("An error occurred while downloading %s from %s. Error: %s",
+                            blob_to_download.blob_hash, str(peer), err.getTraceback())
+        if err.check(PriceDisagreementError):
+            # Don't kill the whole connection just because a price couldn't be agreed upon.
+            # Other information might be desired by other request creators at a better rate.
+            return True
+        return err
+
+    def _get_hash_for_peer_search(self):
+        r = None
+        blobs_to_download = self._blobs_to_download()
+        if blobs_to_download:
+            blobs_without_sources = self._blobs_without_sources()
+            if not blobs_without_sources:
+                blob_hash = blobs_to_download[0].blob_hash
+            else:
+                blob_hash = blobs_without_sources[0].blob_hash
+            r = blob_hash
+        logging.debug("Blob requester peer search response: %s", str(r))
+        return defer.succeed(r)
+
+    def _find_peers_for_hash(self, h):
+        if h is None:
+            return None
+        else:
+            d = self.peer_finder.find_peers_for_blob(h)
+
+            def choose_best_peers(peers):
+                bad_peers = self._get_bad_peers()
+                return [p for p in peers if not p in bad_peers]
+
+            d.addCallback(choose_best_peers)
+            return d
+
+    def _should_send_request_to(self, peer):
+        if self._peers[peer] < -5.0:
+            return False
+        if peer in self._price_disagreements:
+            return False
+        if peer in self._incompatible_peers:
+            return False
+        return True
+
+    def _get_bad_peers(self):
+        return [p for p in self._peers.iterkeys() if not self._should_send_request_to(p)]
+
+    def _hash_available(self, blob_hash):
+        for peer in self._available_blobs:
+            if blob_hash in self._available_blobs[peer]:
+                return True
+        return False
+
+    def _hash_available_on(self, blob_hash, peer):
+        if blob_hash in self._available_blobs[peer]:
+            return True
+        return False
+
+    def _blobs_to_download(self):
+        needed_blobs = self.download_manager.needed_blobs()
+        return sorted(needed_blobs, key=lambda b: b.is_downloading())
+
+    def _blobs_without_sources(self):
+        return [b for b in self.download_manager.needed_blobs() if not self._hash_available(b.blob_hash)]
+
+    def _get_availability_request(self, peer):
+        all_needed = [b.blob_hash for b in self._blobs_to_download() if not b.blob_hash in self._available_blobs[peer]]
+        # sort them so that the peer will be asked first for blobs it hasn't said it doesn't have
+        to_request = sorted(all_needed, key=lambda b: b in self._unavailable_blobs[peer])[:20]
+        if to_request:
+            r_dict = {'requested_blobs': to_request}
+            response_identifier = 'available_blobs'
+            request = ClientRequest(r_dict, response_identifier)
+            return request
+        return None
+
+    def _get_download_request(self, peer):
+        request = None
+        to_download = [b for b in self._blobs_to_download() if self._hash_available_on(b.blob_hash, peer)]
+        while to_download and request is None:
+            blob_to_download = to_download[0]
+            to_download = to_download[1:]
+            if not blob_to_download.is_validated():
+                d, write_func, cancel_func = blob_to_download.open_for_writing(peer)
+
+                def counting_write_func(data):
+                    peer.update_stats('blob_bytes_downloaded', len(data))
+                    return write_func(data)
+
+                if d is not None:
+
+                    request_dict = {'requested_blob': blob_to_download.blob_hash}
+                    response_identifier = 'incoming_blob'
+
+                    request = ClientBlobRequest(request_dict, response_identifier, counting_write_func, d,
+                                                cancel_func, blob_to_download)
+
+                    logging.info("Requesting blob %s from %s", str(blob_to_download), str(peer))
+        return request
+
+    def _price_settled(self, protocol):
+        if protocol in self._protocol_prices:
+            return True
+        return False
+
+    def _get_price_request(self, peer, protocol):
+        request = None
+        if not protocol in self._protocol_prices:
+            self._protocol_prices[protocol] = self.payment_rate_manager.get_rate_blob_data(peer)
+            request_dict = {'blob_data_payment_rate': self._protocol_prices[protocol]}
+            request = ClientRequest(request_dict, 'blob_data_payment_rate')
+        return request
+
+    def _update_local_score(self, peer, amount):
+            self._peers[peer] += amount
+
+    def _reserve_points(self, peer, protocol, max_bytes):
+        assert protocol in self._protocol_prices
+        points_to_reserve = 1.0 * max_bytes * self._protocol_prices[protocol] / 2**20
+        return self.wallet.reserve_points(peer, points_to_reserve)
+
+    def _pay_peer(self, protocol, num_bytes, reserved_points):
+        assert num_bytes != 0
+        assert protocol in self._protocol_prices
+        point_amount = 1.0 * num_bytes * self._protocol_prices[protocol] / 2**20
+        self.wallet.send_points(reserved_points, point_amount)
+        self.payment_rate_manager.record_points_paid(point_amount)
+
+    def _cancel_points(self, reserved_points):
+        self.wallet.cancel_point_reservation(reserved_points)
+
+    def _handle_availability(self, response_dict, peer, request):
+        if not request.response_identifier in response_dict:
+            raise InvalidResponseError("response identifier not in response")
+        logging.debug("Received a response to the availability request")
+        blob_hashes = response_dict[request.response_identifier]
+        for blob_hash in blob_hashes:
+            if blob_hash in request.request_dict['requested_blobs']:
+                logging.debug("The server has indicated it has the following blob available: %s", blob_hash)
+                self._available_blobs[peer].append(blob_hash)
+                if blob_hash in self._unavailable_blobs[peer]:
+                    self._unavailable_blobs[peer].remove(blob_hash)
+                request.request_dict['requested_blobs'].remove(blob_hash)
+        for blob_hash in request.request_dict['requested_blobs']:
+            self._unavailable_blobs[peer].append(blob_hash)
+        return True
+
+    def _handle_incoming_blob(self, response_dict, peer, request):
+        if not request.response_identifier in response_dict:
+            return InvalidResponseError("response identifier not in response")
+        if not type(response_dict[request.response_identifier]) == dict:
+            return InvalidResponseError("response not a dict. got %s" %
+                                        (type(response_dict[request.response_identifier]),))
+        response = response_dict[request.response_identifier]
+        if 'error' in response:
+            # This means we're not getting our blob for some reason
+            if response['error'] == "RATE_UNSET":
+                # Stop the download with an error that won't penalize the peer
+                request.cancel(PriceDisagreementError())
+            else:
+                # The peer has done something bad so we should get out of here
+                return InvalidResponseError("Got an unknown error from the peer: %s" %
+                                            (response['error'],))
+        else:
+            if not 'blob_hash' in response:
+                return InvalidResponseError("Missing the required field 'blob_hash'")
+            if not response['blob_hash'] == request.request_dict['requested_blob']:
+                return InvalidResponseError("Incoming blob does not match expected. Incoming: %s. Expected: %s" %
+                                            (response['blob_hash'], request.request_dict['requested_blob']))
+            if not 'length' in response:
+                return InvalidResponseError("Missing the required field 'length'")
+            if not request.blob.set_length(response['length']):
+                return InvalidResponseError("Could not set the length of the blob")
+        return True
+
+    def _handle_price_response(self, response_dict, peer, request, protocol):
+        if not request.response_identifier in response_dict:
+            return InvalidResponseError("response identifier not in response")
+        assert protocol in self._protocol_prices
+        response = response_dict[request.response_identifier]
+        if response == "RATE_ACCEPTED":
+            return True
+        else:
+            del self._protocol_prices[protocol]
+            self._price_disagreements.append(peer)
+        return True
+
+    def _request_failed(self, reason, request_type, peer):
+        if reason.check(RequestCanceledError):
+            return
+        if reason.check(NoResponseError):
+            self._incompatible_peers.append(peer)
+            return
+        logging.warning("Blob requester: a request of type '%s' failed. Reason: %s, Error type: %s",
+                        str(request_type), reason.getErrorMessage(), reason.type)
+        self._update_local_score(peer, -10.0)
+        if isinstance(reason, InvalidResponseError):
+            peer.update_score(-10.0)
+        else:
+            peer.update_score(-2.0)
+        return reason

lbrynet/core/client/ClientProtocol.py

@@ -0,0 +1,235 @@
+import json
+import logging
+from twisted.internet import error, defer, reactor
+from twisted.internet.protocol import Protocol, ClientFactory
+from twisted.python import failure
+from lbrynet.conf import MAX_RESPONSE_INFO_SIZE as MAX_RESPONSE_SIZE
+from lbrynet.core.Error import ConnectionClosedBeforeResponseError, NoResponseError
+from lbrynet.core.Error import DownloadCanceledError, MisbehavingPeerError
+from lbrynet.core.Error import RequestCanceledError
+from lbrynet.interfaces import IRequestSender, IRateLimited
+from zope.interface import implements
+
+
+class ClientProtocol(Protocol):
+    implements(IRequestSender, IRateLimited)
+
+    ######### Protocol #########
+
+    def connectionMade(self):
+        self._connection_manager = self.factory.connection_manager
+        self._rate_limiter = self.factory.rate_limiter
+        self.peer = self.factory.peer
+        self._response_deferreds = {}
+        self._response_buff = ''
+        self._downloading_blob = False
+        self._blob_download_request = None
+        self._next_request = {}
+        self.connection_closed = False
+        self.connection_closing = False
+
+        self.peer.report_up()
+
+        self._ask_for_request()
+
+    def dataReceived(self, data):
+        self._rate_limiter.report_dl_bytes(len(data))
+        if self._downloading_blob is True:
+            self._blob_download_request.write(data)
+        else:
+            self._response_buff += data
+            if len(self._response_buff) > MAX_RESPONSE_SIZE:
+                logging.warning("Response is too large. Size %s", len(self._response_buff))
+                self.transport.loseConnection()
+            response, extra_data = self._get_valid_response(self._response_buff)
+            if response is not None:
+                self._response_buff = ''
+                self._handle_response(response)
+                if self._downloading_blob is True and len(extra_data) != 0:
+                    self._blob_download_request.write(extra_data)
+
+    def connectionLost(self, reason):
+        self.connection_closed = True
+        if reason.check(error.ConnectionDone):
+            err = failure.Failure(ConnectionClosedBeforeResponseError())
+        else:
+            err = reason
+        #if self._response_deferreds:
+        #    logging.warning("Lost connection with active response deferreds. %s", str(self._response_deferreds))
+        for key, d in self._response_deferreds.items():
+            del self._response_deferreds[key]
+            d.errback(err)
+        if self._blob_download_request is not None:
+            self._blob_download_request.cancel(err)
+        self._connection_manager.protocol_disconnected(self.peer, self)
+
+    ######### IRequestSender #########
+
+    def add_request(self, request):
+        if request.response_identifier in self._response_deferreds:
+            return defer.fail(failure.Failure(ValueError("There is already a request for that response active")))
+        self._next_request.update(request.request_dict)
+        d = defer.Deferred()
+        logging.debug("Adding a request. Request: %s", str(request))
+        self._response_deferreds[request.response_identifier] = d
+        return d
+
+    def add_blob_request(self, blob_request):
+        if self._blob_download_request is None:
+            d = self.add_request(blob_request)
+            self._blob_download_request = blob_request
+            blob_request.finished_deferred.addCallbacks(self._downloading_finished,
+                                                        self._downloading_failed)
+            blob_request.finished_deferred.addErrback(self._handle_response_error)
+            return d
+        else:
+            return defer.fail(failure.Failure(ValueError("There is already a blob download request active")))
+
+    def cancel_requests(self):
+        self.connection_closing = True
+        ds = []
+        err = failure.Failure(RequestCanceledError())
+        for key, d in self._response_deferreds.items():
+            del self._response_deferreds[key]
+            d.errback(err)
+            ds.append(d)
+        if self._blob_download_request is not None:
+            self._blob_download_request.cancel(err)
+            ds.append(self._blob_download_request.finished_deferred)
+            self._blob_download_request = None
+        return defer.DeferredList(ds)
+
+    ######### Internal request handling #########
+
+    def _handle_request_error(self, err):
+        logging.error("An unexpected error occurred creating or sending a request to %s. Error message: %s",
+                      str(self.peer), err.getTraceback())
+        self.transport.loseConnection()
+
+    def _ask_for_request(self):
+
+        if self.connection_closed is True or self.connection_closing is True:
+            return
+
+        def send_request_or_close(do_request):
+            if do_request is True:
+                request_msg, self._next_request = self._next_request, {}
+                self._send_request_message(request_msg)
+            else:
+                # The connection manager has indicated that this connection should be terminated
+                logging.info("Closing the connection to %s due to having no further requests to send", str(self.peer))
+                self.transport.loseConnection()
+
+        d = self._connection_manager.get_next_request(self.peer, self)
+        d.addCallback(send_request_or_close)
+        d.addErrback(self._handle_request_error)
+
+    def _send_request_message(self, request_msg):
+        # TODO: compare this message to the last one. If they're the same,
+        # TODO: incrementally delay this message.
+        m = json.dumps(request_msg)
+        self.transport.write(m)
+
+    def _get_valid_response(self, response_msg):
+        extra_data = None
+        response = None
+        curr_pos = 0
+        while 1:
+            next_close_paren = response_msg.find('}', curr_pos)
+            if next_close_paren != -1:
+                curr_pos = next_close_paren + 1
+                try:
+                    response = json.loads(response_msg[:curr_pos])
+                except ValueError:
+                    pass
+                else:
+                    extra_data = response_msg[curr_pos:]
+                    break
+            else:
+                break
+        return response, extra_data
+
+    def _handle_response_error(self, err):
+        # If an error gets to this point, log it and kill the connection.
+        if not err.check(MisbehavingPeerError, ConnectionClosedBeforeResponseError, DownloadCanceledError,
+                         RequestCanceledError):
+            logging.error("The connection to %s is closing due to an unexpected error: %s", str(self.peer),
+                          err.getErrorMessage())
+        if not err.check(RequestCanceledError):
+            self.transport.loseConnection()
+
+    def _handle_response(self, response):
+        ds = []
+        logging.debug("Handling a response. Current expected responses: %s", str(self._response_deferreds))
+        for key, val in response.items():
+            if key in self._response_deferreds:
+                d = self._response_deferreds[key]
+                del self._response_deferreds[key]
+                d.callback({key: val})
+                ds.append(d)
+        for k, d in self._response_deferreds.items():
+            del self._response_deferreds[k]
+            d.errback(failure.Failure(NoResponseError()))
+            ds.append(d)
+
+        if self._blob_download_request is not None:
+            self._downloading_blob = True
+            d = self._blob_download_request.finished_deferred
+            d.addErrback(self._handle_response_error)
+            ds.append(d)
+
+        dl = defer.DeferredList(ds)
+
+        dl.addCallback(lambda _: self._ask_for_request())
+
+    def _downloading_finished(self, arg):
+        logging.debug("The blob has finished downloading")
+        self._blob_download_request = None
+        self._downloading_blob = False
+        return arg
+
+    def _downloading_failed(self, err):
+        if err.check(DownloadCanceledError):
+            # TODO: (wish-list) it seems silly to close the connection over this, and it shouldn't
+            # TODO: always be this way. it's done this way now because the client has no other way
+            # TODO: of telling the server it wants the download to stop. It would be great if the
+            # TODO: protocol had such a mechanism.
+            logging.info("Closing the connection to %s because the download of blob %s was canceled",
+                         str(self.peer), str(self._blob_download_request.blob))
+            #self.transport.loseConnection()
+            #return True
+        return err
+
+    ######### IRateLimited #########
+
+    def throttle_upload(self):
+        pass
+
+    def unthrottle_upload(self):
+        pass
+
+    def throttle_download(self):
+        self.transport.pauseProducing()
+
+    def unthrottle_download(self):
+        self.transport.resumeProducing()
+
+
+class ClientProtocolFactory(ClientFactory):
+    protocol = ClientProtocol
+
+    def __init__(self, peer, rate_limiter, connection_manager):
+        self.peer = peer
+        self.rate_limiter = rate_limiter
+        self.connection_manager = connection_manager
+        self.p = None
+
+    def clientConnectionFailed(self, connector, reason):
+        self.peer.report_down()
+        self.connection_manager.protocol_disconnected(self.peer, connector)
+
+    def buildProtocol(self, addr):
+        p = self.protocol()
+        p.factory = self
+        self.p = p
+        return p

lbrynet/core/client/ClientRequest.py

@@ -0,0 +1,27 @@
+from lbrynet.conf import BLOB_SIZE
+
+
+class ClientRequest(object):
+    def __init__(self, request_dict, response_identifier=None):
+        self.request_dict = request_dict
+        self.response_identifier = response_identifier
+
+
+class ClientPaidRequest(ClientRequest):
+    def __init__(self, request_dict, response_identifier, max_pay_units):
+        ClientRequest.__init__(self, request_dict, response_identifier)
+        self.max_pay_units = max_pay_units
+
+
+class ClientBlobRequest(ClientPaidRequest):
+    def __init__(self, request_dict, response_identifier, write_func, finished_deferred,
+                 cancel_func, blob):
+        if blob.length is None:
+            max_pay_units = BLOB_SIZE
+        else:
+            max_pay_units = blob.length
+        ClientPaidRequest.__init__(self, request_dict, response_identifier, max_pay_units)
+        self.write = write_func
+        self.finished_deferred = finished_deferred
+        self.cancel = cancel_func
+        self.blob = blob

lbrynet/core/client/ConnectionManager.py

@@ -0,0 +1,177 @@
+import logging
+from twisted.internet import defer
+from zope.interface import implements
+from lbrynet import interfaces
+from lbrynet.conf import MAX_CONNECTIONS_PER_STREAM
+from lbrynet.core.client.ClientProtocol import ClientProtocolFactory
+from lbrynet.core.Error import InsufficientFundsError
+
+
+class ConnectionManager(object):
+    implements(interfaces.IConnectionManager)
+
+    def __init__(self, downloader, rate_limiter, primary_request_creators, secondary_request_creators):
+        self.downloader = downloader
+        self.rate_limiter = rate_limiter
+        self.primary_request_creators = primary_request_creators
+        self.secondary_request_creators = secondary_request_creators
+        self.peer_connections = {}  # {Peer: {'connection': connection,
+                                    #         'request_creators': [IRequestCreator if using this connection]}}
+        self.connections_closing = {}  # {Peer: deferred (fired when the connection is closed)}
+        self.next_manage_call = None
+
+    def start(self):
+        from twisted.internet import reactor
+
+        if self.next_manage_call is not None and self.next_manage_call.active() is True:
+            self.next_manage_call.cancel()
+        self.next_manage_call = reactor.callLater(0, self._manage)
+        return defer.succeed(True)
+
+    def stop(self):
+        if self.next_manage_call is not None and self.next_manage_call.active() is True:
+            self.next_manage_call.cancel()
+        self.next_manage_call = None
+        closing_deferreds = []
+        for peer in self.peer_connections.keys():
+
+            def close_connection(p):
+                logging.info("Abruptly closing a connection to %s due to downloading being paused",
+                             str(p))
+
+                if self.peer_connections[p]['factory'].p is not None:
+                    d = self.peer_connections[p]['factory'].p.cancel_requests()
+                else:
+                    d = defer.succeed(True)
+
+                def disconnect_peer():
+                    self.peer_connections[p]['connection'].disconnect()
+                    if p in self.peer_connections:
+                        del self.peer_connections[p]
+                    d = defer.Deferred()
+                    self.connections_closing[p] = d
+                    return d
+
+                d.addBoth(lambda _: disconnect_peer())
+                return d
+
+            closing_deferreds.append(close_connection(peer))
+        return defer.DeferredList(closing_deferreds)
+
+    def get_next_request(self, peer, protocol):
+
+        logging.debug("Trying to get the next request for peer %s", str(peer))
+
+        if not peer in self.peer_connections:
+            logging.debug("The peer has already been told to shut down.")
+            return defer.succeed(False)
+
+        def handle_error(err):
+            if err.check(InsufficientFundsError):
+                self.downloader.insufficient_funds()
+                return False
+            else:
+                return err
+
+        def check_if_request_sent(request_sent, request_creator):
+            if request_sent is False:
+                if request_creator in self.peer_connections[peer]['request_creators']:
+                    self.peer_connections[peer]['request_creators'].remove(request_creator)
+            else:
+                if not request_creator in self.peer_connections[peer]['request_creators']:
+                    self.peer_connections[peer]['request_creators'].append(request_creator)
+            return request_sent
+
+        def check_requests(requests):
+            have_request = True in [r[1] for r in requests if r[0] is True]
+            return have_request
+
+        def get_secondary_requests_if_necessary(have_request):
+            if have_request is True:
+                ds = []
+                for s_r_c in self.secondary_request_creators:
+                    d = s_r_c.send_next_request(peer, protocol)
+                    ds.append(d)
+                dl = defer.DeferredList(ds)
+            else:
+                dl = defer.succeed(None)
+            dl.addCallback(lambda _: have_request)
+            return dl
+
+        ds = []
+
+        for p_r_c in self.primary_request_creators:
+            d = p_r_c.send_next_request(peer, protocol)
+            d.addErrback(handle_error)
+            d.addCallback(check_if_request_sent, p_r_c)
+            ds.append(d)
+
+        dl = defer.DeferredList(ds, fireOnOneErrback=True)
+        dl.addCallback(check_requests)
+        dl.addCallback(get_secondary_requests_if_necessary)
+        return dl
+
+    def protocol_disconnected(self, peer, protocol):
+        if peer in self.peer_connections:
+            del self.peer_connections[peer]
+        if peer in self.connections_closing:
+            d = self.connections_closing[peer]
+            del self.connections_closing[peer]
+            d.callback(True)
+
+    def _rank_request_creator_connections(self):
+        """
+        @return: an ordered list of our request creators, ranked according to which has the least number of
+            connections open that it likes
+        """
+        def count_peers(request_creator):
+            return len([p for p in self.peer_connections.itervalues() if request_creator in p['request_creators']])
+
+        return sorted(self.primary_request_creators, key=count_peers)
+
+    def _connect_to_peer(self, peer):
+
+        from twisted.internet import reactor
+
+        if peer is not None:
+            logging.debug("Trying to connect to %s", str(peer))
+            factory = ClientProtocolFactory(peer, self.rate_limiter, self)
+            connection = reactor.connectTCP(peer.host, peer.port, factory)
+            self.peer_connections[peer] = {'connection': connection,
+                                           'request_creators': self.primary_request_creators[:],
+                                           'factory': factory}
+
+    def _manage(self):
+
+        from twisted.internet import reactor
+
+        def get_new_peers(request_creators):
+            logging.debug("Trying to get a new peer to connect to")
+            if len(request_creators) > 0:
+                logging.debug("Got a creator to check: %s", str(request_creators[0]))
+                d = request_creators[0].get_new_peers()
+                d.addCallback(lambda h: h if h is not None else get_new_peers(request_creators[1:]))
+                return d
+            else:
+                return defer.succeed(None)
+
+        def pick_best_peer(peers):
+            # TODO: Eventually rank them based on past performance/reputation. For now
+            # TODO: just pick the first to which we don't have an open connection
+            logging.debug("Got a list of peers to choose from: %s", str(peers))
+            if peers is None:
+                return None
+            for peer in peers:
+                if not peer in self.peer_connections:
+                    logging.debug("Got a good peer. Returning peer %s", str(peer))
+                    return peer
+            logging.debug("Couldn't find a good peer to connect to")
+            return None
+
+        if len(self.peer_connections) < MAX_CONNECTIONS_PER_STREAM:
+            ordered_request_creators = self._rank_request_creator_connections()
+            d = get_new_peers(ordered_request_creators)
+            d.addCallback(pick_best_peer)
+            d.addCallback(self._connect_to_peer)
+
+        self.next_manage_call = reactor.callLater(1, self._manage)

lbrynet/core/client/DHTPeerFinder.py

@@ -0,0 +1,47 @@
+import binascii
+from zope.interface import implements
+from lbrynet.interfaces import IPeerFinder
+
+
+class DHTPeerFinder(object):
+    """This class finds peers which have announced to the DHT that they have certain blobs"""
+    implements(IPeerFinder)
+
+    def __init__(self, dht_node, peer_manager):
+        self.dht_node = dht_node
+        self.peer_manager = peer_manager
+        self.peers = []
+        self.next_manage_call = None
+
+    def run_manage_loop(self):
+
+        from twisted.internet import reactor
+
+        self._manage_peers()
+        self.next_manage_call = reactor.callLater(60, self.run_manage_loop)
+
+    def stop(self):
+        if self.next_manage_call is not None and self.next_manage_call.active():
+            self.next_manage_call.cancel()
+            self.next_manage_call = None
+
+    def _manage_peers(self):
+        pass
+
+    def find_peers_for_blob(self, blob_hash):
+        bin_hash = binascii.unhexlify(blob_hash)
+
+        def filter_peers(peer_list):
+            good_peers = []
+            for host, port in peer_list:
+                peer = self.peer_manager.get_peer(host, port)
+                if peer.is_available() is True:
+                    good_peers.append(peer)
+            return good_peers
+
+        d = self.dht_node.getPeersForBlob(bin_hash)
+        d.addCallback(filter_peers)
+        return d
+
+    def get_most_popular_hashes(self, num_to_return):
+        return self.dht_node.get_most_popular_hashes(num_to_return)

lbrynet/core/client/DownloadManager.py

@@ -0,0 +1,115 @@
+import logging
+from twisted.internet import defer
+from twisted.python import failure
+from zope.interface import implements
+from lbrynet import interfaces
+
+
+class DownloadManager(object):
+    implements(interfaces.IDownloadManager)
+
+    def __init__(self, blob_manager, upload_allowed):
+        self.blob_manager = blob_manager
+        self.upload_allowed = upload_allowed
+        self.blob_requester = None
+        self.blob_info_finder = None
+        self.progress_manager = None
+        self.blob_handler = None
+        self.connection_manager = None
+
+        self.blobs = {}
+        self.blob_infos = {}
+
+    ######### IDownloadManager #########
+
+    def start_downloading(self):
+        d = self.blob_info_finder.get_initial_blobs()
+        logging.debug("Requested the initial blobs from the info finder")
+        d.addCallback(self.add_blobs_to_download)
+        d.addCallback(lambda _: self.resume_downloading())
+        return d
+
+    def resume_downloading(self):
+
+        def check_start(result, manager):
+            if isinstance(result, failure.Failure):
+                logging.error("Failed to start the %s: %s", manager, result.getErrorMessage())
+                return False
+            return True
+
+        d1 = self.progress_manager.start()
+        d1.addBoth(check_start, "progress manager")
+        d2 = self.connection_manager.start()
+        d2.addBoth(check_start, "connection manager")
+        dl = defer.DeferredList([d1, d2])
+        dl.addCallback(lambda xs: False not in xs)
+        return dl
+
+    def stop_downloading(self):
+
+        def check_stop(result, manager):
+            if isinstance(result, failure.Failure):
+                logging.error("Failed to stop the %s: %s", manager. result.getErrorMessage())
+                return False
+            return True
+
+        d1 = self.progress_manager.stop()
+        d1.addBoth(check_stop, "progress manager")
+        d2 = self.connection_manager.stop()
+        d2.addBoth(check_stop, "connection manager")
+        dl = defer.DeferredList([d1, d2])
+        dl.addCallback(lambda xs: False not in xs)
+        return dl
+
+    def add_blobs_to_download(self, blob_infos):
+
+        logging.debug("Adding %s to blobs", str(blob_infos))
+
+        def add_blob_to_list(blob, blob_num):
+            self.blobs[blob_num] = blob
+            logging.info("Added blob (hash: %s, number %s) to the list", str(blob.blob_hash), str(blob_num))
+
+        def error_during_add(err):
+            logging.warning("An error occurred adding the blob to blobs. Error:%s", err.getErrorMessage())
+            return err
+
+        ds = []
+        for blob_info in blob_infos:
+            if not blob_info.blob_num in self.blobs:
+                self.blob_infos[blob_info.blob_num] = blob_info
+                logging.debug("Trying to get the blob associated with blob hash %s", str(blob_info.blob_hash))
+                d = self.blob_manager.get_blob(blob_info.blob_hash, self.upload_allowed, blob_info.length)
+                d.addCallback(add_blob_to_list, blob_info.blob_num)
+                d.addErrback(error_during_add)
+                ds.append(d)
+
+        dl = defer.DeferredList(ds)
+        return dl
+
+    def stream_position(self):
+        return self.progress_manager.stream_position()
+
+    def needed_blobs(self):
+        return self.progress_manager.needed_blobs()
+
+    def final_blob_num(self):
+        return self.blob_info_finder.final_blob_num()
+
+    def handle_blob(self, blob_num):
+        return self.blob_handler.handle_blob(self.blobs[blob_num], self.blob_infos[blob_num])
+
+    def calculate_total_bytes(self):
+        return sum([bi.length for bi in self.blob_infos.itervalues()])
+
+    def calculate_bytes_left_to_output(self):
+        if not self.blobs:
+            return self.calculate_total_bytes()
+        else:
+            to_be_outputted = [b for n, b in self.blobs.iteritems() if n >= self.progress_manager.last_blob_outputted]
+            return sum([b.length for b in to_be_outputted if b.length is not None])
+
+    def calculate_bytes_left_to_download(self):
+        if not self.blobs:
+            return self.calculate_total_bytes()
+        else:
+            return sum([b.length for b in self.needed_blobs() if b.length is not None])

lbrynet/core/client/StandaloneBlobDownloader.py

@@ -0,0 +1,133 @@
+import logging
+from zope.interface import implements
+from lbrynet import interfaces
+from lbrynet.core.BlobInfo import BlobInfo
+from lbrynet.core.client.BlobRequester import BlobRequester
+from lbrynet.core.client.ConnectionManager import ConnectionManager
+from lbrynet.core.client.DownloadManager import DownloadManager
+from twisted.internet import defer
+
+
+class SingleBlobMetadataHandler(object):
+    implements(interfaces.IMetadataHandler)
+
+    def __init__(self, blob_hash, download_manager):
+        self.blob_hash = blob_hash
+        self.download_manager = download_manager
+
+    ######## IMetadataHandler #########
+
+    def get_initial_blobs(self):
+        logging.debug("Returning the blob info")
+        return defer.succeed([BlobInfo(self.blob_hash, 0, None)])
+
+    def final_blob_num(self):
+        return 0
+
+
+class SingleProgressManager(object):
+    def __init__(self, finished_callback, download_manager):
+        self.finished_callback = finished_callback
+        self.finished = False
+        self.download_manager = download_manager
+        self._next_check_if_finished = None
+
+    def start(self):
+
+        from twisted.internet import reactor
+
+        assert self._next_check_if_finished is None
+        self._next_check_if_finished = reactor.callLater(0, self._check_if_finished)
+        return defer.succeed(True)
+
+    def stop(self):
+        if self._next_check_if_finished is not None:
+            self._next_check_if_finished.cancel()
+            self._next_check_if_finished = None
+        return defer.succeed(True)
+
+    def _check_if_finished(self):
+
+        from twisted.internet import reactor
+
+        self._next_check_if_finished = None
+        if self.finished is False:
+            if self.stream_position() == 1:
+                self.blob_downloaded(self.download_manager.blobs[0], 0)
+            else:
+                self._next_check_if_finished = reactor.callLater(1, self._check_if_finished)
+
+    def stream_position(self):
+        blobs = self.download_manager.blobs
+        if blobs and blobs[0].is_validated():
+            return 1
+        return 0
+
+    def needed_blobs(self):
+        blobs = self.download_manager.blobs
+        assert len(blobs) == 1
+        return [b for b in blobs.itervalues() if not b.is_validated()]
+
+    def blob_downloaded(self, blob, blob_num):
+
+        from twisted.internet import reactor
+
+        logging.debug("The blob %s has been downloaded. Calling the finished callback", str(blob))
+        if self.finished is False:
+            self.finished = True
+            reactor.callLater(0, self.finished_callback, blob)
+
+
+class DummyBlobHandler(object):
+    def __init__(self):
+        pass
+
+    def handle_blob(self, blob, blob_info):
+        pass
+
+
+class StandaloneBlobDownloader(object):
+
+    def __init__(self, blob_hash, blob_manager, peer_finder, rate_limiter, payment_rate_manager, wallet):
+        self.blob_hash = blob_hash
+        self.blob_manager = blob_manager
+        self.peer_finder = peer_finder
+        self.rate_limiter = rate_limiter
+        self.payment_rate_manager = payment_rate_manager
+        self.wallet = wallet
+        self.download_manager = None
+        self.finished_deferred = None
+
+    def download(self):
+        def cancel_download(d):
+            self.stop()
+
+        self.finished_deferred = defer.Deferred(canceller=cancel_download)
+        self.download_manager = DownloadManager(self.blob_manager, True)
+        self.download_manager.blob_requester = BlobRequester(self.blob_manager, self.peer_finder,
+                                                             self.payment_rate_manager, self.wallet,
+                                                             self.download_manager)
+        self.download_manager.blob_info_finder = SingleBlobMetadataHandler(self.blob_hash,
+                                                                           self.download_manager)
+        self.download_manager.progress_manager = SingleProgressManager(self._blob_downloaded,
+                                                                       self.download_manager)
+        self.download_manager.blob_handler = DummyBlobHandler()
+        self.download_manager.wallet_info_exchanger = self.wallet.get_info_exchanger()
+        self.download_manager.connection_manager = ConnectionManager(
+            self, self.rate_limiter,
+            [self.download_manager.blob_requester],
+            [self.download_manager.wallet_info_exchanger]
+        )
+        d = self.download_manager.start_downloading()
+        d.addCallback(lambda _: self.finished_deferred)
+        return d
+
+    def stop(self):
+        return self.download_manager.stop_downloading()
+
+    def _blob_downloaded(self, blob):
+        self.stop()
+        self.finished_deferred.callback(blob)
+
+    def insufficient_funds(self):
+        return self.stop()

lbrynet/core/client/StreamProgressManager.py

@@ -0,0 +1,141 @@
+import logging
+from lbrynet.interfaces import IProgressManager
+from twisted.internet import defer
+from zope.interface import implements
+
+
+class StreamProgressManager(object):
+    implements(IProgressManager)
+
+    def __init__(self, finished_callback, blob_manager, download_manager, delete_blob_after_finished=False):
+        self.finished_callback = finished_callback
+        self.blob_manager = blob_manager
+        self.delete_blob_after_finished = delete_blob_after_finished
+        self.download_manager = download_manager
+        self.provided_blob_nums = []
+        self.last_blob_outputted = -1
+        self.stopped = True
+        self._next_try_to_output_call = None
+        self.outputting_d = None
+
+    ######### IProgressManager #########
+
+    def start(self):
+
+        from twisted.internet import reactor
+
+        self.stopped = False
+        self._next_try_to_output_call = reactor.callLater(0, self._try_to_output)
+        return defer.succeed(True)
+
+    def stop(self):
+        self.stopped = True
+        if self._next_try_to_output_call is not None and self._next_try_to_output_call.active():
+            self._next_try_to_output_call.cancel()
+        self._next_try_to_output_call = None
+        return self._stop_outputting()
+
+    def blob_downloaded(self, blob, blob_num):
+        if self.outputting_d is None:
+            self._output_loop()
+
+    ######### internal #########
+
+    def _finished_outputting(self):
+        self.finished_callback(True)
+
+    def _try_to_output(self):
+
+        from twisted.internet import reactor
+
+        self._next_try_to_output_call = reactor.callLater(1, self._try_to_output)
+        if self.outputting_d is None:
+            self._output_loop()
+
+    def _output_loop(self):
+        pass
+
+    def _stop_outputting(self):
+        if self.outputting_d is not None:
+            return self.outputting_d
+        return defer.succeed(None)
+
+    def _finished_with_blob(self, blob_num):
+        logging.debug("In _finished_with_blob, blob_num = %s", str(blob_num))
+        if self.delete_blob_after_finished is True:
+            logging.debug("delete_blob_after_finished is True")
+            blobs = self.download_manager.blobs
+            if blob_num in blobs:
+                logging.debug("Telling the blob manager, %s,  to delete blob %s", str(self.blob_manager),
+                              blobs[blob_num].blob_hash)
+                self.blob_manager.delete_blobs([blobs[blob_num].blob_hash])
+            else:
+                logging.debug("Blob number %s was not in blobs", str(blob_num))
+        else:
+            logging.debug("delete_blob_after_finished is False")
+
+
+class FullStreamProgressManager(StreamProgressManager):
+    def __init__(self, finished_callback, blob_manager, download_manager, delete_blob_after_finished=False):
+        StreamProgressManager.__init__(self, finished_callback, blob_manager, download_manager,
+                                       delete_blob_after_finished)
+        self.outputting_d = None
+
+    ######### IProgressManager #########
+
+    def stream_position(self):
+        blobs = self.download_manager.blobs
+        if not blobs:
+            return 0
+        else:
+            for i in xrange(max(blobs.iterkeys())):
+                if not i in blobs or (not blobs[i].is_validated() and not i in self.provided_blob_nums):
+                    return i
+            return max(blobs.iterkeys()) + 1
+
+    def needed_blobs(self):
+        blobs = self.download_manager.blobs
+        return [b for n, b in blobs.iteritems() if not b.is_validated() and not n in self.provided_blob_nums]
+
+    ######### internal #########
+
+    def _output_loop(self):
+
+        from twisted.internet import reactor
+
+        if self.stopped:
+            if self.outputting_d is not None:
+                self.outputting_d.callback(True)
+                self.outputting_d = None
+            return
+
+        if self.outputting_d is None:
+            self.outputting_d = defer.Deferred()
+        blobs = self.download_manager.blobs
+
+        def finished_outputting_blob():
+            self.last_blob_outputted += 1
+            final_blob_num = self.download_manager.final_blob_num()
+            if final_blob_num is not None and final_blob_num == self.last_blob_outputted:
+                self._finished_outputting()
+                self.outputting_d.callback(True)
+                self.outputting_d = None
+            else:
+                reactor.callLater(0, self._output_loop)
+
+        current_blob_num = self.last_blob_outputted + 1
+
+        if current_blob_num in blobs and blobs[current_blob_num].is_validated():
+            logging.info("Outputting blob %s", str(self.last_blob_outputted + 1))
+            self.provided_blob_nums.append(self.last_blob_outputted + 1)
+            d = self.download_manager.handle_blob(self.last_blob_outputted + 1)
+            d.addCallback(lambda _: finished_outputting_blob())
+            d.addCallback(lambda _: self._finished_with_blob(current_blob_num))
+
+            def log_error(err):
+                logging.warning("Error occurred in the output loop. Error: %s", err.getErrorMessage())
+
+            d.addErrback(log_error)
+        else:
+            self.outputting_d.callback(True)
+            self.outputting_d = None

lbrynet/core/cryptoutils.py

@@ -0,0 +1,18 @@
+from Crypto.Hash import SHA384
+import seccure
+
+
+def get_lbry_hash_obj():
+    return SHA384.new()
+
+
+def get_pub_key(pass_phrase):
+    return str(seccure.passphrase_to_pubkey(pass_phrase, curve="brainpoolp384r1"))
+
+
+def sign_with_pass_phrase(m, pass_phrase):
+    return seccure.sign(m, pass_phrase, curve="brainpoolp384r1")
+
+
+def verify_signature(m, signature, pub_key):
+    return seccure.verify(m, signature, pub_key, curve="brainpoolp384r1")

lbrynet/core/server/BlobAvailabilityHandler.py

@@ -0,0 +1,55 @@
+import logging
+from twisted.internet import defer
+from zope.interface import implements
+from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler
+
+
+class BlobAvailabilityHandlerFactory(object):
+    implements(IQueryHandlerFactory)
+
+    def __init__(self, blob_manager):
+        self.blob_manager = blob_manager
+
+    ######### IQueryHandlerFactory #########
+
+    def build_query_handler(self):
+        q_h = BlobAvailabilityHandler(self.blob_manager)
+        return q_h
+
+    def get_primary_query_identifier(self):
+        return 'requested_blobs'
+
+    def get_description(self):
+        return "Blob Availability - blobs that are available to be uploaded"
+
+
+class BlobAvailabilityHandler(object):
+    implements(IQueryHandler)
+
+    def __init__(self, blob_manager):
+        self.blob_manager = blob_manager
+        self.query_identifiers = ['requested_blobs']
+
+    ######### IQueryHandler #########
+
+    def register_with_request_handler(self, request_handler, peer):
+        request_handler.register_query_handler(self, self.query_identifiers)
+
+    def handle_queries(self, queries):
+        if self.query_identifiers[0] in queries:
+            logging.debug("Received the client's list of requested blobs")
+            d = self._get_available_blobs(queries[self.query_identifiers[0]])
+
+            def set_field(available_blobs):
+                return {'available_blobs': available_blobs}
+
+            d.addCallback(set_field)
+            return d
+        return defer.succeed({})
+
+    ######### internal #########
+
+    def _get_available_blobs(self, requested_blobs):
+        d = self.blob_manager.completed_blobs(requested_blobs)
+
+        return d

lbrynet/core/server/BlobRequestHandler.py

@@ -0,0 +1,156 @@
+import logging
+from twisted.internet import defer
+from twisted.protocols.basic import FileSender
+from twisted.python.failure import Failure
+from zope.interface import implements
+from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler, IBlobSender
+
+
+class BlobRequestHandlerFactory(object):
+    implements(IQueryHandlerFactory)
+
+    def __init__(self, blob_manager, wallet, payment_rate_manager):
+        self.blob_manager = blob_manager
+        self.wallet = wallet
+        self.payment_rate_manager = payment_rate_manager
+
+    ######### IQueryHandlerFactory #########
+
+    def build_query_handler(self):
+        q_h = BlobRequestHandler(self.blob_manager, self.wallet, self.payment_rate_manager)
+        return q_h
+
+    def get_primary_query_identifier(self):
+        return 'requested_blob'
+
+    def get_description(self):
+        return "Blob Uploader - uploads blobs"
+
+
+class BlobRequestHandler(object):
+    implements(IQueryHandler, IBlobSender)
+
+    def __init__(self, blob_manager, wallet, payment_rate_manager):
+        self.blob_manager = blob_manager
+        self.payment_rate_manager = payment_rate_manager
+        self.wallet = wallet
+        self.query_identifiers = ['blob_data_payment_rate', 'requested_blob']
+        self.peer = None
+        self.blob_data_payment_rate = None
+        self.read_handle = None
+        self.currently_uploading = None
+        self.file_sender = None
+        self.blob_bytes_uploaded = 0
+
+    ######### IQueryHandler #########
+
+    def register_with_request_handler(self, request_handler, peer):
+        self.peer = peer
+        request_handler.register_query_handler(self, self.query_identifiers)
+        request_handler.register_blob_sender(self)
+
+    def handle_queries(self, queries):
+        response = {}
+        if self.query_identifiers[0] in queries:
+            if not self.handle_blob_data_payment_rate(queries[self.query_identifiers[0]]):
+                    response['blob_data_payment_rate'] = "RATE_TOO_LOW"
+            else:
+                response['blob_data_payment_rate'] = 'RATE_ACCEPTED'
+
+        if self.query_identifiers[1] in queries:
+            logging.debug("Received the client's request to send a blob")
+            response_fields = {}
+            response['incoming_blob'] = response_fields
+
+            if self.blob_data_payment_rate is None:
+                response_fields['error'] = "RATE_UNSET"
+                return defer.succeed(response)
+            else:
+
+                d = self.blob_manager.get_blob(queries[self.query_identifiers[1]], True)
+
+                def open_blob_for_reading(blob):
+                    if blob.is_validated():
+                        read_handle = blob.open_for_reading()
+                        if read_handle is not None:
+                            self.currently_uploading = blob
+                            self.read_handle = read_handle
+                            logging.debug("Sending %s to client", str(blob))
+                            response_fields['blob_hash'] = blob.blob_hash
+                            response_fields['length'] = blob.length
+                            return response
+                    logging.debug("We can not send %s", str(blob))
+                    response_fields['error'] = "BLOB_UNAVAILABLE"
+                    return response
+
+                d.addCallback(open_blob_for_reading)
+
+                return d
+        else:
+            return defer.succeed(response)
+
+    ######### IBlobSender #########
+
+    def send_blob_if_requested(self, consumer):
+        if self.currently_uploading is not None:
+            return self.send_file(consumer)
+        return defer.succeed(True)
+
+    def cancel_send(self, err):
+        if self.currently_uploading is not None:
+            self.currently_uploading.close_read_handle(self.read_handle)
+        self.read_handle = None
+        self.currently_uploading = None
+        return err
+
+    ######### internal #########
+
+    def handle_blob_data_payment_rate(self, requested_payment_rate):
+        if not self.payment_rate_manager.accept_rate_blob_data(self.peer, requested_payment_rate):
+            return False
+        else:
+            self.blob_data_payment_rate = requested_payment_rate
+            return True
+
+    def send_file(self, consumer):
+
+        def _send_file():
+            inner_d = start_transfer()
+            # TODO: if the transfer fails, check if it's because the connection was cut off.
+            # TODO: if so, perhaps bill the client
+            inner_d.addCallback(lambda _: set_expected_payment())
+            inner_d.addBoth(set_not_uploading)
+            return inner_d
+
+        def count_bytes(data):
+            self.blob_bytes_uploaded += len(data)
+            self.peer.update_stats('blob_bytes_uploaded', len(data))
+            return data
+
+        def start_transfer():
+            self.file_sender = FileSender()
+            logging.info("Starting the file upload")
+            assert self.read_handle is not None, "self.read_handle was None when trying to start the transfer"
+            d = self.file_sender.beginFileTransfer(self.read_handle, consumer, count_bytes)
+            return d
+
+        def set_expected_payment():
+            logging.info("Setting expected payment")
+            if self.blob_bytes_uploaded != 0 and self.blob_data_payment_rate is not None:
+                self.wallet.add_expected_payment(self.peer,
+                                                 self.currently_uploading.length * 1.0 *
+                                                 self.blob_data_payment_rate / 2**20)
+                self.blob_bytes_uploaded = 0
+            self.peer.update_stats('blobs_uploaded', 1)
+            return None
+
+        def set_not_uploading(reason=None):
+            if self.currently_uploading is not None:
+                self.currently_uploading.close_read_handle(self.read_handle)
+                self.read_handle = None
+                self.currently_uploading = None
+            self.file_sender = None
+            if reason is not None and isinstance(reason, Failure):
+                logging.warning("Upload has failed. Reason: %s", reason.getErrorMessage())
+
+        return _send_file()

lbrynet/core/server/DHTHashAnnouncer.py

@@ -0,0 +1,81 @@
+import binascii
+from twisted.internet import defer, task, reactor
+import collections
+
+
+class DHTHashAnnouncer(object):
+    """This class announces to the DHT that this peer has certain blobs"""
+    def __init__(self, dht_node, peer_port):
+        self.dht_node = dht_node
+        self.peer_port = peer_port
+        self.suppliers = []
+        self.next_manage_call = None
+        self.hash_queue = collections.deque()
+        self._concurrent_announcers = 0
+
+    def run_manage_loop(self):
+
+        from twisted.internet import reactor
+
+        if self.peer_port is not None:
+            self._announce_available_hashes()
+        self.next_manage_call = reactor.callLater(60, self.run_manage_loop)
+
+    def stop(self):
+        if self.next_manage_call is not None:
+            self.next_manage_call.cancel()
+            self.next_manage_call = None
+
+    def add_supplier(self, supplier):
+        self.suppliers.append(supplier)
+
+    def immediate_announce(self, blob_hashes):
+        if self.peer_port is not None:
+            return self._announce_hashes(blob_hashes)
+        else:
+            return defer.succeed(False)
+
+    def _announce_available_hashes(self):
+        ds = []
+        for supplier in self.suppliers:
+            d = supplier.hashes_to_announce()
+            d.addCallback(self._announce_hashes)
+            ds.append(d)
+        dl = defer.DeferredList(ds)
+        return dl
+
+    def _announce_hashes(self, hashes):
+
+        ds = []
+
+        for h in hashes:
+            announce_deferred = defer.Deferred()
+            ds.append(announce_deferred)
+            self.hash_queue.append((h, announce_deferred))
+
+        def announce():
+            if len(self.hash_queue):
+                h, announce_deferred = self.hash_queue.popleft()
+                d = self.dht_node.announceHaveBlob(binascii.unhexlify(h), self.peer_port)
+                d.chainDeferred(announce_deferred)
+                d.addBoth(lambda _: reactor.callLater(0, announce))
+            else:
+                self._concurrent_announcers -= 1
+
+        for i in range(self._concurrent_announcers, 5):
+            # TODO: maybe make the 5 configurable
+            self._concurrent_announcers += 1
+            announce()
+        return defer.DeferredList(ds)
+
+
+class DHTHashSupplier(object):
+    """Classes derived from this class give hashes to a hash announcer"""
+    def __init__(self, announcer):
+        if announcer is not None:
+            announcer.add_supplier(self)
+        self.hash_announcer = announcer
+        self.hash_reannounce_time = 60 * 60  # 1 hour
+
+    def hashes_to_announce(self):
+        pass

lbrynet/core/server/ServerProtocol.py

@@ -0,0 +1,91 @@
+import logging
+from twisted.internet import interfaces, error
+from twisted.internet.protocol import Protocol, ServerFactory
+from twisted.python import failure
+from zope.interface import implements
+from lbrynet.core.server.ServerRequestHandler import ServerRequestHandler
+
+
+class ServerProtocol(Protocol):
+    """ServerProtocol needs to:
+
+    1) Receive requests from its transport
+    2) Pass those requests on to its request handler
+    3) Tell the request handler to pause/resume producing
+    4) Tell its transport to pause/resume producing
+    5) Hang up when the request handler is done producing
+    6) Tell the request handler to stop producing if the connection is lost
+    7) Upon creation, register with the rate limiter
+    8) Upon connection loss, unregister with the rate limiter
+    9) Report all uploaded and downloaded bytes to the rate limiter
+    10) Pause/resume production when told by the rate limiter
+    """
+
+    implements(interfaces.IConsumer)
+
+    #Protocol stuff
+
+    def connectionMade(self):
+        logging.debug("Got a connection")
+        peer_info = self.transport.getPeer()
+        self.peer = self.factory.peer_manager.get_peer(peer_info.host, peer_info.port)
+        self.request_handler = ServerRequestHandler(self)
+        for query_handler_factory, enabled in self.factory.query_handler_factories.iteritems():
+            if enabled is True:
+                query_handler = query_handler_factory.build_query_handler()
+                query_handler.register_with_request_handler(self.request_handler, self.peer)
+        logging.debug("Setting the request handler")
+        self.factory.rate_limiter.register_protocol(self)
+
+    def connectionLost(self, reason=failure.Failure(error.ConnectionDone())):
+        if self.request_handler is not None:
+            self.request_handler.stopProducing()
+        self.factory.rate_limiter.unregister_protocol(self)
+        if not reason.check(error.ConnectionDone):
+            logging.warning("Closing a connection. Reason: %s", reason.getErrorMessage())
+
+    def dataReceived(self, data):
+        logging.debug("Receiving %s bytes of data from the transport", str(len(data)))
+        self.factory.rate_limiter.report_dl_bytes(len(data))
+        if self.request_handler is not None:
+            self.request_handler.data_received(data)
+
+    #IConsumer stuff
+
+    def registerProducer(self, producer, streaming):
+        logging.debug("Registering the producer")
+        assert streaming is True
+
+    def unregisterProducer(self):
+        self.request_handler = None
+        self.transport.loseConnection()
+
+    def write(self, data):
+        logging.debug("Writing %s bytes of data to the transport", str(len(data)))
+        self.transport.write(data)
+        self.factory.rate_limiter.report_ul_bytes(len(data))
+
+    #Rate limiter stuff
+
+    def throttle_upload(self):
+        if self.request_handler is not None:
+            self.request_handler.pauseProducing()
+
+    def unthrottle_upload(self):
+        if self.request_handler is not None:
+            self.request_handler.resumeProducing()
+
+    def throttle_download(self):
+        self.transport.pauseProducing()
+
+    def unthrottle_download(self):
+        self.transport.resumeProducing()
+
+
+class ServerProtocolFactory(ServerFactory):
+    protocol = ServerProtocol
+
+    def __init__(self, rate_limiter, query_handler_factories, peer_manager):
+        self.rate_limiter = rate_limiter
+        self.query_handler_factories = query_handler_factories
+        self.peer_manager = peer_manager

lbrynet/core/server/ServerRequestHandler.py

@@ -0,0 +1,171 @@
+import json
+import logging
+from twisted.internet import interfaces, defer
+from zope.interface import implements
+from lbrynet.interfaces import IRequestHandler
+
+
+class ServerRequestHandler(object):
+    """This class handles requests from clients. It can upload blobs and return request for information about
+    more blobs that are associated with streams"""
+
+    implements(interfaces.IPushProducer, interfaces.IConsumer, IRequestHandler)
+
+    def __init__(self, consumer):
+        self.consumer = consumer
+        self.production_paused = False
+        self.request_buff = ''
+        self.response_buff = ''
+        self.producer = None
+        self.request_received = False
+        self.CHUNK_SIZE = 2**14
+        self.query_handlers = {}  # {IQueryHandler: [query_identifiers]}
+        self.blob_sender = None
+        self.consumer.registerProducer(self, True)
+
+    #IPushProducer stuff
+
+    def pauseProducing(self):
+        self.production_paused = True
+
+    def stopProducing(self):
+        if self.producer is not None:
+            self.producer.stopProducing()
+            self.producer = None
+        self.production_paused = True
+        self.consumer.unregisterProducer()
+
+    def resumeProducing(self):
+
+        from twisted.internet import reactor
+
+        self.production_paused = False
+        self._produce_more()
+        if self.producer is not None:
+            reactor.callLater(0, self.producer.resumeProducing)
+
+    def _produce_more(self):
+
+        from twisted.internet import reactor
+
+        if self.production_paused is False:
+            chunk = self.response_buff[:self.CHUNK_SIZE]
+            self.response_buff = self.response_buff[self.CHUNK_SIZE:]
+            if chunk != '':
+                logging.debug("writing %s bytes to the client", str(len(chunk)))
+                self.consumer.write(chunk)
+                reactor.callLater(0, self._produce_more)
+
+    #IConsumer stuff
+
+    def registerProducer(self, producer, streaming):
+        #assert self.file_sender == producer
+        self.producer = producer
+        assert streaming is False
+        producer.resumeProducing()
+
+    def unregisterProducer(self):
+        self.producer = None
+
+    def write(self, data):
+
+        from twisted.internet import reactor
+
+        self.response_buff = self.response_buff + data
+        self._produce_more()
+
+        def get_more_data():
+            if self.producer is not None:
+                logging.debug("Requesting more data from the producer")
+                self.producer.resumeProducing()
+
+        reactor.callLater(0, get_more_data)
+
+    #From Protocol
+
+    def data_received(self, data):
+        logging.debug("Received data")
+        logging.debug("%s", str(data))
+        if self.request_received is False:
+            self.request_buff = self.request_buff + data
+            msg = self.try_to_parse_request(self.request_buff)
+            if msg is not None:
+                self.request_buff = ''
+                d = self.handle_request(msg)
+                if self.blob_sender is not None:
+                    d.addCallback(lambda _: self.blob_sender.send_blob_if_requested(self))
+                d.addCallbacks(lambda _: self.finished_response(), self.request_failure_handler)
+            else:
+                logging.info("Request buff not a valid json message")
+                logging.info("Request buff: %s", str(self.request_buff))
+        else:
+            logging.warning("The client sent data when we were uploading a file. This should not happen")
+
+    ######### IRequestHandler #########
+
+    def register_query_handler(self, query_handler, query_identifiers):
+        self.query_handlers[query_handler] = query_identifiers
+
+    def register_blob_sender(self, blob_sender):
+        self.blob_sender = blob_sender
+
+    #response handling
+
+    def request_failure_handler(self, err):
+        logging.warning("An error occurred handling a request. Error: %s", err.getErrorMessage())
+        self.stopProducing()
+        return err
+
+    def finished_response(self):
+        self.request_received = False
+        self._produce_more()
+
+    def send_response(self, msg):
+        m = json.dumps(msg)
+        logging.info("Sending a response of length %s", str(len(m)))
+        logging.debug("Response: %s", str(m))
+        self.response_buff = self.response_buff + m
+        self._produce_more()
+        return True
+
+    def handle_request(self, msg):
+        logging.debug("Handling a request")
+        logging.debug(str(msg))
+
+        def create_response_message(results):
+            response = {}
+            for success, result in results:
+                if success is True:
+                    response.update(result)
+                else:
+                    # result is a Failure
+                    return result
+            logging.debug("Finished making the response message. Response: %s", str(response))
+            return response
+
+        def log_errors(err):
+            logging.warning("An error occurred handling a client request. Error message: %s", err.getErrorMessage())
+            return err
+
+        def send_response(response):
+            self.send_response(response)
+            return True
+
+        ds = []
+        for query_handler, query_identifiers in self.query_handlers.iteritems():
+            queries = {q_i: msg[q_i] for q_i in query_identifiers if q_i in msg}
+            d = query_handler.handle_queries(queries)
+            d.addErrback(log_errors)
+            ds.append(d)
+
+        dl = defer.DeferredList(ds)
+        dl.addCallback(create_response_message)
+        dl.addCallback(send_response)
+        return dl
+
+    def try_to_parse_request(self, request_buff):
+        try:
+            msg = json.loads(request_buff)
+            return msg
+        except ValueError:
+            return None

lbrynet/core/utils.py

@@ -0,0 +1,28 @@
+from lbrynet.core.cryptoutils import get_lbry_hash_obj
+import random
+
+
+blobhash_length = get_lbry_hash_obj().digest_size * 2  # digest_size is in bytes, and blob hashes are hex encoded
+
+
+def generate_id(num=None):
+    h = get_lbry_hash_obj()
+    if num is not None:
+        h.update(str(num))
+    else:
+        h.update(str(random.getrandbits(512)))
+    return h.digest()
+
+
+def is_valid_blobhash(blobhash):
+    """
+    @param blobhash: string, the blobhash to check
+
+    @return: Whether the blobhash is the correct length and contains only valid characters (0-9, a-f)
+    """
+    if len(blobhash) != blobhash_length:
+        return False
+    for l in blobhash:
+        if l not in "0123456789abcdef":
+            return False
+    return True

lbrynet/create_network.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+
+# Thanks to Paul Cannon for IP-address resolution functions (taken from aspn.activestate.com)
+
+import argparse
+import os, sys, time, signal
+
+amount = 0
+
+
+def destroyNetwork(nodes):
+    print 'Destroying Kademlia network...'
+    i = 0
+    for node in nodes:
+        i += 1
+        hashAmount = i*50/amount
+        hashbar = '#'*hashAmount
+        output = '\r[%-50s] %d/%d' % (hashbar, i, amount)
+        sys.stdout.write(output)
+        time.sleep(0.15)
+        os.kill(node, signal.SIGTERM)
+    print
+
+
+def main():
+
+    parser = argparse.ArgumentParser(description="Launch a network of dht nodes")
+
+    parser.add_argument("amount_of_nodes",
+                        help="The number of nodes to create",
+                        type=int)
+    parser.add_argument("--nic_ip_address",
+                        help="The network interface on which these nodes will listen for connections "
+                             "from each other and from other nodes. If omitted, an attempt will be "
+                             "made to automatically determine the system's IP address, but this may "
+                             "result in the nodes being reachable only from this system")
+
+    args = parser.parse_args()
+
+    global amount
+    amount = args.amount_of_nodes
+    if args.nic_ip_address:
+        ipAddress = args.nic_ip_address
+    else:
+        import socket
+        ipAddress = socket.gethostbyname(socket.gethostname())
+        print 'Network interface IP address omitted; using %s...' % ipAddress
+
+    startPort = 4000
+    port = startPort+1
+    nodes = []
+    print 'Creating Kademlia network...'
+    try:
+        nodes.append(os.spawnlp(os.P_NOWAIT, 'lbrynet-launch-node', 'lbrynet-launch-node', str(startPort)))
+        for i in range(amount-1):
+            time.sleep(0.15)
+            hashAmount = i*50/amount
+            hashbar = '#'*hashAmount
+            output = '\r[%-50s] %d/%d' % (hashbar, i, amount)
+            sys.stdout.write(output)
+            nodes.append(os.spawnlp(os.P_NOWAIT, 'lbrynet-launch-node', 'lbrynet-launch-node', str(port), ipAddress, str(startPort)))
+            port += 1
+    except KeyboardInterrupt:
+        '\nNetwork creation cancelled.'
+        destroyNetwork(nodes)
+        sys.exit(1)
+
+    print '\n\n---------------\nNetwork running\n---------------\n'
+    try:
+        while 1:
+            time.sleep(1)
+    except KeyboardInterrupt:
+        pass
+    finally:
+        destroyNetwork(nodes)
+
+if __name__ == '__main__':
+    main()

lbrynet/cryptstream/CryptBlob.py

@@ -0,0 +1,106 @@
+import binascii
+import logging
+from Crypto.Cipher import AES
+from lbrynet.conf import BLOB_SIZE
+from lbrynet.core.BlobInfo import BlobInfo
+
+
+class CryptBlobInfo(BlobInfo):
+    def __init__(self, blob_hash, blob_num, length, iv):
+        BlobInfo.__init__(self, blob_hash, blob_num, length)
+        self.iv = iv
+
+
+class StreamBlobDecryptor(object):
+    def __init__(self, blob, key, iv, length):
+        self.blob = blob
+        self.key = key
+        self.iv = iv
+        self.length = length
+        self.buff = b''
+        self.len_read = 0
+        self.cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
+
+    def decrypt(self, write_func):
+
+        def remove_padding(data):
+            pad_len = ord(data[-1])
+            data, padding = data[:-1 * pad_len], data[-1 * pad_len:]
+            for c in padding:
+                assert ord(c) == pad_len
+            return data
+
+        def write_bytes():
+            if self.len_read < self.length:
+                num_bytes_to_decrypt = (len(self.buff) // self.cipher.block_size) * self.cipher.block_size
+                data_to_decrypt, self.buff = self.buff[:num_bytes_to_decrypt], self.buff[num_bytes_to_decrypt:]
+                write_func(self.cipher.decrypt(data_to_decrypt))
+
+        def finish_decrypt():
+            assert len(self.buff) % self.cipher.block_size == 0
+            data_to_decrypt, self.buff = self.buff, b''
+            write_func(remove_padding(self.cipher.decrypt(data_to_decrypt)))
+
+        def decrypt_bytes(data):
+            self.buff += data
+            self.len_read += len(data)
+            write_bytes()
+            #write_func(remove_padding(self.cipher.decrypt(self.buff)))
+
+        d = self.blob.read(decrypt_bytes)
+        d.addCallback(lambda _: finish_decrypt())
+        return d
+
+
+class CryptStreamBlobMaker(object):
+    """This class encrypts data and writes it to a new blob"""
+    def __init__(self, key, iv, blob_num, blob):
+        self.key = key
+        self.iv = iv
+        self.blob_num = blob_num
+        self.blob = blob
+        self.cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
+        self.buff = b''
+        self.length = 0
+
+    def write(self, data):
+        max_bytes_to_write = BLOB_SIZE - self.length - 1
+        done = False
+        if max_bytes_to_write <= len(data):
+            num_bytes_to_write = max_bytes_to_write
+            done = True
+        else:
+            num_bytes_to_write = len(data)
+        self.length += num_bytes_to_write
+        data_to_write = data[:num_bytes_to_write]
+        self.buff += data_to_write
+        self._write_buffer()
+        return done, num_bytes_to_write
+
+    def close(self):
+        logging.debug("closing blob %s with plaintext len %s", str(self.blob_num), str(self.length))
+        if self.length != 0:
+            self._close_buffer()
+        d = self.blob.close()
+        d.addCallback(self._return_info)
+        logging.debug("called the finished_callback from CryptStreamBlobMaker.close")
+        return d
+
+    def _write_buffer(self):
+        num_bytes_to_encrypt = (len(self.buff) // AES.block_size) * AES.block_size
+        data_to_encrypt,  self.buff = self.buff[:num_bytes_to_encrypt], self.buff[num_bytes_to_encrypt:]
+        encrypted_data = self.cipher.encrypt(data_to_encrypt)
+        self.blob.write(encrypted_data)
+
+    def _close_buffer(self):
+        data_to_encrypt, self.buff = self.buff, b''
+        assert len(data_to_encrypt) < AES.block_size
+        pad_len = AES.block_size - len(data_to_encrypt)
+        padded_data = data_to_encrypt + chr(pad_len) * pad_len
+        self.length += pad_len
+        assert len(padded_data) == AES.block_size
+        encrypted_data = self.cipher.encrypt(padded_data)
+        self.blob.write(encrypted_data)
+
+    def _return_info(self, blob_hash):
+        return CryptBlobInfo(blob_hash, self.blob_num, self.length, binascii.hexlify(self.iv))

lbrynet/cryptstream/CryptStreamCreator.py

@@ -0,0 +1,94 @@
+"""
+Utility for creating Crypt Streams, which are encrypted blobs and associated metadata.
+"""
+
+import logging
+
+from Crypto import Random
+from Crypto.Cipher import AES
+
+from twisted.internet import defer
+from lbrynet.core.StreamCreator import StreamCreator
+from lbrynet.cryptstream.CryptBlob import CryptStreamBlobMaker
+
+
+class CryptStreamCreator(StreamCreator):
+    """Create a new stream with blobs encrypted by a symmetric cipher.
+
+    Each blob is encrypted with the same key, but each blob has its own initialization vector
+    which is associated with the blob when the blob is associated with the stream."""
+    def __init__(self, blob_manager, name=None, key=None, iv_generator=None):
+        """
+        @param blob_manager: Object that stores and provides access to blobs.
+        @type blob_manager: BlobManager
+
+        @param name: the name of the stream, which will be presented to the user
+        @type name: string
+
+        @param key: the raw AES key which will be used to encrypt the blobs. If None, a random key will
+            be generated.
+        @type key: string
+
+        @param iv_generator: a generator which yields initialization vectors for the blobs. Will be called
+            once for each blob.
+        @type iv_generator: a generator function which yields strings
+
+        @return: None
+        """
+        StreamCreator.__init__(self, name)
+        self.blob_manager = blob_manager
+        self.key = key
+        if iv_generator is None:
+            self.iv_generator = self.random_iv_generator()
+        else:
+            self.iv_generator = iv_generator
+
+    @staticmethod
+    def random_iv_generator():
+        while 1:
+            yield Random.new().read(AES.block_size)
+
+    def setup(self):
+        """Create the symmetric key if it wasn't provided"""
+
+        if self.key is None:
+            self.key = Random.new().read(AES.block_size)
+
+        return defer.succeed(True)
+
+    def _finalize(self):
+        logging.debug("_finalize has been called")
+        self.blob_count += 1
+        iv = self.iv_generator.next()
+        final_blob_creator = self.blob_manager.get_blob_creator()
+        logging.debug("Created the finished_deferred")
+        final_blob = self._get_blob_maker(iv, final_blob_creator)
+        logging.debug("Created the final blob")
+        logging.debug("Calling close on final blob")
+        d = final_blob.close()
+        d.addCallback(self._blob_finished)
+        self.finished_deferreds.append(d)
+        logging.debug("called close on final blob, returning from make_final_blob")
+        return d
+
+    def _write(self, data):
+
+        def close_blob(blob):
+            d = blob.close()
+            d.addCallback(self._blob_finished)
+            self.finished_deferreds.append(d)
+
+        while len(data) > 0:
+            if self.current_blob is None:
+                next_blob_creator = self.blob_manager.get_blob_creator()
+                self.blob_count += 1
+                iv = self.iv_generator.next()
+                self.current_blob = self._get_blob_maker(iv, next_blob_creator)
+            done, num_bytes_written = self.current_blob.write(data)
+            data = data[num_bytes_written:]
+            if done is True:
+                close_blob(self.current_blob)
+                self.current_blob = None
+
+    def _get_blob_maker(self, iv, blob_creator):
+        return CryptStreamBlobMaker(self.key, iv, self.blob_count, blob_creator)

lbrynet/cryptstream/__init__.py

@@ -0,0 +1,8 @@
+"""
+Classes and functions for dealing with Crypt Streams.
+
+Crypt Streams are encrypted blobs and metadata tying those blobs together. At least some of the
+metadata is generally stored in a Stream Descriptor File, for example containing a public key
+used to bind blobs to the stream and a symmetric key used to encrypt the blobs. The list of blobs
+may or may not be present.
+"""

lbrynet/cryptstream/client/CryptBlobHandler.py

@@ -0,0 +1,19 @@
+import binascii
+from zope.interface import implements
+from lbrynet.cryptstream.CryptBlob import StreamBlobDecryptor
+from lbrynet.interfaces import IBlobHandler
+
+
+class CryptBlobHandler(object):
+    implements(IBlobHandler)
+
+    def __init__(self, key, write_func):
+        self.key = key
+        self.write_func = write_func
+
+    ######## IBlobHandler #########
+
+    def handle_blob(self, blob, blob_info):
+        blob_decryptor = StreamBlobDecryptor(blob, self.key, binascii.unhexlify(blob_info.iv), blob_info.length)
+        d = blob_decryptor.decrypt(self.write_func)
+        return d

lbrynet/cryptstream/client/CryptStreamDownloader.py

@@ -0,0 +1,213 @@
+from zope.interface import implements
+from lbrynet.interfaces import IStreamDownloader
+from lbrynet.core.client.BlobRequester import BlobRequester
+from lbrynet.core.client.ConnectionManager import ConnectionManager
+from lbrynet.core.client.DownloadManager import DownloadManager
+from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
+from lbrynet.cryptstream.client.CryptBlobHandler import CryptBlobHandler
+from twisted.internet import defer
+from twisted.python.failure import Failure
+
+
+class StartFailedError(Exception):
+    pass
+
+
+class AlreadyRunningError(Exception):
+    pass
+
+
+class AlreadyStoppedError(Exception):
+    pass
+
+
+class CurrentlyStoppingError(Exception):
+    pass
+
+
+class CurrentlyStartingError(Exception):
+    pass
+
+
+class CryptStreamDownloader(object):
+
+    implements(IStreamDownloader)
+
+    def __init__(self, peer_finder, rate_limiter, blob_manager,
+                 payment_rate_manager, wallet, upload_allowed):
+        """
+        Initialize a CryptStreamDownloader
+
+        @param peer_finder: An object which implements the IPeerFinder interface. Used to look up peers by a hashsum.
+
+        @param rate_limiter: An object which implements the IRateLimiter interface
+
+        @param blob_manager: A BlobManager object
+
+        @param payment_rate_manager: A PaymentRateManager object
+
+        @param wallet: An object which implements the ILBRYWallet interface
+
+        @return:
+        """
+
+        self.peer_finder = peer_finder
+        self.rate_limiter = rate_limiter
+        self.blob_manager = blob_manager
+        self.payment_rate_manager = payment_rate_manager
+        self.wallet = wallet
+        self.upload_allowed = upload_allowed
+
+        self.key = None
+        self.stream_name = None
+
+        self.completed = False
+        self.stopped = True
+        self.stopping = False
+        self.starting = False
+
+        self.download_manager = None
+        self.finished_deferred = None
+
+        self.points_paid = 0.0
+
+    def toggle_running(self):
+        if self.stopped is True:
+            return self.start()
+        else:
+            return self.stop()
+
+    def start(self):
+
+        def set_finished_deferred():
+            self.finished_deferred = defer.Deferred()
+            return self.finished_deferred
+
+        if self.starting is True:
+            raise CurrentlyStartingError()
+        if self.stopping is True:
+            raise CurrentlyStoppingError()
+        if self.stopped is False:
+            raise AlreadyRunningError()
+        assert self.download_manager is None
+        self.starting = True
+        self.completed = False
+        d = self._start()
+        d.addCallback(lambda _: set_finished_deferred())
+        return d
+
+    def stop(self):
+
+        def check_if_stop_succeeded(success):
+            self.stopping = False
+            if success is True:
+                self.stopped = True
+                self._remove_download_manager()
+            return success
+
+        if self.stopped is True:
+            raise AlreadyStoppedError()
+        if self.stopping is True:
+            raise CurrentlyStoppingError()
+        assert self.download_manager is not None
+        self.stopping = True
+        d = self.download_manager.stop_downloading()
+        self._fire_completed_deferred()
+        d.addCallback(check_if_stop_succeeded)
+        return d
+
+    def _start_failed(self):
+
+        def set_stopped():
+            self.stopped = True
+            self.stopping = False
+            self.starting = False
+
+        if self.download_manager is not None:
+            d = self.download_manager.stop_downloading()
+            d.addCallback(lambda _: self._remove_download_manager())
+        else:
+            d = defer.succeed(True)
+        d.addCallback(lambda _: set_stopped())
+        d.addCallback(lambda _: Failure(StartFailedError()))
+        return d
+
+    def _start(self):
+
+        def check_start_succeeded(success):
+            if success:
+                self.starting = False
+                self.stopped = False
+                self.completed = False
+                return True
+            else:
+                return self._start_failed()
+
+        self.download_manager = self._get_download_manager()
+        d = self.download_manager.start_downloading()
+        d.addCallbacks(check_start_succeeded)
+        return d
+
+    def _get_download_manager(self):
+        download_manager = DownloadManager(self.blob_manager, self.upload_allowed)
+        download_manager.blob_info_finder = self._get_metadata_handler(download_manager)
+        download_manager.blob_requester = self._get_blob_requester(download_manager)
+        download_manager.progress_manager = self._get_progress_manager(download_manager)
+        download_manager.blob_handler = self._get_blob_handler(download_manager)
+        download_manager.wallet_info_exchanger = self.wallet.get_info_exchanger()
+        download_manager.connection_manager = self._get_connection_manager(download_manager)
+        #return DownloadManager(self.blob_manager, self.blob_requester, self.metadata_handler,
+        #                       self.progress_manager, self.blob_handler, self.connection_manager)
+        return download_manager
+
+    def _remove_download_manager(self):
+        self.download_manager.blob_info_finder = None
+        self.download_manager.blob_requester = None
+        self.download_manager.progress_manager = None
+        self.download_manager.blob_handler = None
+        self.download_manager.wallet_info_exchanger = None
+        self.download_manager.connection_manager = None
+        self.download_manager = None
+
+    def _get_primary_request_creators(self, download_manager):
+        return [download_manager.blob_requester]
+
+    def _get_secondary_request_creators(self, download_manager):
+        return [download_manager.wallet_info_exchanger]
+
+    def _get_metadata_handler(self, download_manager):
+        pass
+
+    def _get_blob_requester(self, download_manager):
+        return BlobRequester(self.blob_manager, self.peer_finder, self.payment_rate_manager, self.wallet,
+                             download_manager)
+
+    def _get_progress_manager(self, download_manager):
+        return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
+
+    def _get_write_func(self):
+        pass
+
+    def _get_blob_handler(self, download_manager):
+        return CryptBlobHandler(self.key, self._get_write_func())
+
+    def _get_connection_manager(self, download_manager):
+        return ConnectionManager(self, self.rate_limiter,
+                                 self._get_primary_request_creators(download_manager),
+                                 self._get_secondary_request_creators(download_manager))
+
+    def _fire_completed_deferred(self):
+        self.finished_deferred, d = None, self.finished_deferred
+        if d is not None:
+            d.callback(self._get_finished_deferred_callback_value())
+
+    def _get_finished_deferred_callback_value(self):
+        return None
+
+    def _finished_downloading(self, finished):
+        if finished is True:
+            self.completed = True
+        return self.stop()
+
+    def insufficient_funds(self):
+        return self.stop()

lbrynet/dht/AUTHORS

@@ -0,0 +1,7 @@
+Francois Aucamp <faucamp@csir.co.za>
+
+Thanks goes to the following people for providing patches/suggestions/tests:
+
+Neil Kleynhans <ntkleynhans@csir.co.za>
+Haiyang Ma <haiyang.ma@maidsafe.net>
+Bryan McAlister <bmcalister@csir.co.za>

lbrynet/dht/COPYING

@@ -0,0 +1,165 @@
+		   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions. 
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version. 
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

lbrynet/dht/constants.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+""" This module defines the charaterizing constants of the Kademlia network
+
+C{checkRefreshInterval} and C{udpDatagramMaxSize} are implementation-specific
+constants, and do not affect general Kademlia operation. 
+"""
+
+######### KADEMLIA CONSTANTS ###########
+
+#: Small number Representing the degree of parallelism in network calls
+alpha = 3
+
+#: Maximum number of contacts stored in a bucket; this should be an even number
+k = 8
+
+#: Timeout for network operations (in seconds)
+rpcTimeout = 5
+
+# Delay between iterations of iterative node lookups (for loose parallelism)  (in seconds)
+iterativeLookupDelay = rpcTimeout / 2
+
+#: If a k-bucket has not been used for this amount of time, refresh it (in seconds)
+refreshTimeout = 3600 # 1 hour
+#: The interval at which nodes replicate (republish/refresh) data they are holding
+replicateInterval = refreshTimeout
+# The time it takes for data to expire in the network; the original publisher of the data
+# will also republish the data at this time if it is still valid
+dataExpireTimeout = 86400 # 24 hours
+
+tokenSecretChangeInterval = 300 # 5 minutes
+
+peer_request_timeout = 10
+
+######## IMPLEMENTATION-SPECIFIC CONSTANTS ###########
+
+#: The interval in which the node should check its whether any buckets need refreshing,
+#: or whether any data needs to be republished (in seconds)
+checkRefreshInterval = refreshTimeout/5
+
+#: Max size of a single UDP datagram, in bytes. If a message is larger than this, it will
+#: be spread accross several UDP packets.
+udpDatagramMaxSize = 8192 # 8 KB
+
+key_bits = 384

lbrynet/dht/contact.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+
+class Contact(object):
+    """ Encapsulation for remote contact
+    
+    This class contains information on a single remote contact, and also
+    provides a direct RPC API to the remote node which it represents
+    """
+    def __init__(self, id, ipAddress, udpPort, networkProtocol, firstComm=0):
+        self.id = id
+        self.address = ipAddress
+        self.port = udpPort
+        self._networkProtocol = networkProtocol
+        self.commTime = firstComm
+        
+    def __eq__(self, other):
+        if isinstance(other, Contact):
+            return self.id == other.id
+        elif isinstance(other, str):
+            return self.id == other
+        else:
+            return False
+    
+    def __ne__(self, other):
+        if isinstance(other, Contact):
+            return self.id != other.id
+        elif isinstance(other, str):
+            return self.id != other
+        else:
+            return True
+
+    def compact_ip(self):
+        compact_ip = reduce(lambda buff, x: buff + bytearray([int(x)]), self.address.split('.'), bytearray())
+        return str(compact_ip)
+        
+    def __str__(self):
+        return '<%s.%s object; IP address: %s, UDP port: %d>' % (self.__module__, self.__class__.__name__, self.address, self.port)
+    
+    def __getattr__(self, name):
+        """ This override allows the host node to call a method of the remote
+        node (i.e. this contact) as if it was a local function.
+        
+        For instance, if C{remoteNode} is a instance of C{Contact}, the
+        following will result in C{remoteNode}'s C{test()} method to be
+        called with argument C{123}::
+         remoteNode.test(123)
+        
+        Such a RPC method call will return a Deferred, which will callback
+        when the contact responds with the result (or an error occurs).
+        This happens via this contact's C{_networkProtocol} object (i.e. the
+        host Node's C{_protocol} object).
+        """
+        def _sendRPC(*args, **kwargs):
+            return self._networkProtocol.sendRPC(self, name, args, **kwargs)
+        return _sendRPC

lbrynet/dht/datastore.py

@@ -0,0 +1,213 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import UserDict
+#import sqlite3
+import cPickle as pickle
+import time
+import os
+import constants
+
+
+
+class DataStore(UserDict.DictMixin):
+    """ Interface for classes implementing physical storage (for data
+    published via the "STORE" RPC) for the Kademlia DHT
+    
+    @note: This provides an interface for a dict-like object
+    """
+    def keys(self):
+        """ Return a list of the keys in this data store """
+
+#    def lastPublished(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was last published """
+
+#    def originalPublisherID(self, key):
+#        """ Get the original publisher of the data's node ID
+#
+#        @param key: The key that identifies the stored data
+#        @type key: str
+#
+#        @return: Return the node ID of the original publisher of the
+#        C{(key, value)} pair identified by C{key}.
+#        """
+
+#    def originalPublishTime(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was originally published """
+
+#    def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
+#        """ Set the value of the (key, value) pair identified by C{key};
+#        this should set the "last published" value for the (key, value)
+#        pair to the current time
+#        """
+
+    def addPeerToBlob(self, key, value, lastPublished, originallyPublished, originalPublisherID):
+        pass
+
+#    def __getitem__(self, key):
+#        """ Get the value identified by C{key} """
+
+#    def __setitem__(self, key, value):
+#        """ Convenience wrapper to C{setItem}; this accepts a tuple in the
+#        format: (value, lastPublished, originallyPublished, originalPublisherID) """
+#        self.setItem(key, *value)
+
+#    def __delitem__(self, key):
+#        """ Delete the specified key (and its value) """
+
+class DictDataStore(DataStore):
+    """ A datastore using an in-memory Python dictionary """
+    def __init__(self):
+        # Dictionary format:
+        # { <key>: (<value>, <lastPublished>, <originallyPublished> <originalPublisherID>) }
+        self._dict = {}
+
+    def keys(self):
+        """ Return a list of the keys in this data store """
+        return self._dict.keys()
+
+#    def lastPublished(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was last published """
+#        return self._dict[key][1]
+
+#    def originalPublisherID(self, key):
+#        """ Get the original publisher of the data's node ID
+#
+#        @param key: The key that identifies the stored data
+#        @type key: str
+#
+#        @return: Return the node ID of the original publisher of the
+#        C{(key, value)} pair identified by C{key}.
+#        """
+#        return self._dict[key][3]
+
+#    def originalPublishTime(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was originally published """
+#        return self._dict[key][2]
+
+    def removeExpiredPeers(self):
+        now = int(time.time())
+        def notExpired(peer):
+            if (now - peer[2]) > constants.dataExpireTimeout:
+                return False
+            return True
+        for key in self._dict.keys():
+            unexpired_peers = filter(notExpired, self._dict[key])
+            self._dict[key] = unexpired_peers
+
+    def hasPeersForBlob(self, key):
+        if key in self._dict and len(self._dict[key]) > 0:
+            return True
+        return False
+
+    def addPeerToBlob(self, key, value, lastPublished, originallyPublished, originalPublisherID):
+        if key in self._dict:
+            self._dict[key].append((value, lastPublished, originallyPublished, originalPublisherID))
+        else:
+            self._dict[key] = [(value, lastPublished, originallyPublished, originalPublisherID)]
+
+    def getPeersForBlob(self, key):
+        if key in self._dict:
+            return [val[0] for val in self._dict[key]]
+
+#    def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
+#        """ Set the value of the (key, value) pair identified by C{key};
+#        this should set the "last published" value for the (key, value)
+#        pair to the current time
+#        """
+#        self._dict[key] = (value, lastPublished, originallyPublished, originalPublisherID)
+
+#    def __getitem__(self, key):
+#        """ Get the value identified by C{key} """
+#        return self._dict[key][0]
+
+#    def __delitem__(self, key):
+#        """ Delete the specified key (and its value) """
+#        del self._dict[key]
+
+
+#class SQLiteDataStore(DataStore):
+#    """ Example of a SQLite database-based datastore
+#    """
+#    def __init__(self, dbFile=':memory:'):
+#        """
+#        @param dbFile: The name of the file containing the SQLite database; if
+#                       unspecified, an in-memory database is used.
+#        @type dbFile: str
+#        """
+#        createDB = not os.path.exists(dbFile)
+#        self._db = sqlite3.connect(dbFile)
+#        self._db.isolation_level = None
+#        self._db.text_factory = str
+#        if createDB:
+#            self._db.execute('CREATE TABLE data(key, value, lastPublished, originallyPublished, originalPublisherID)')
+#        self._cursor = self._db.cursor()
+
+#    def keys(self):
+#        """ Return a list of the keys in this data store """
+#        keys = []
+#        try:
+#            self._cursor.execute("SELECT key FROM data")
+#            for row in self._cursor:
+#                keys.append(row[0].decode('hex'))
+#        finally:
+#            return keys
+
+#    def lastPublished(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was last published """
+#        return int(self._dbQuery(key, 'lastPublished'))
+
+#    def originalPublisherID(self, key):
+#        """ Get the original publisher of the data's node ID
+
+#        @param key: The key that identifies the stored data
+#        @type key: str
+        
+#        @return: Return the node ID of the original publisher of the
+#        C{(key, value)} pair identified by C{key}.
+#        """
+#        return self._dbQuery(key, 'originalPublisherID')
+
+#    def originalPublishTime(self, key):
+#        """ Get the time the C{(key, value)} pair identified by C{key}
+#        was originally published """
+#        return int(self._dbQuery(key, 'originallyPublished'))
+
+#    def setItem(self, key, value, lastPublished, originallyPublished, originalPublisherID):
+#        # Encode the key so that it doesn't corrupt the database
+#        encodedKey = key.encode('hex')
+#        self._cursor.execute("select key from data where key=:reqKey", {'reqKey': encodedKey})
+#        if self._cursor.fetchone() == None:
+#            self._cursor.execute('INSERT INTO data(key, value, lastPublished, originallyPublished, originalPublisherID) VALUES (?, ?, ?, ?, ?)', (encodedKey, buffer(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)), lastPublished, originallyPublished, originalPublisherID))
+#        else:
+#            self._cursor.execute('UPDATE data SET value=?, lastPublished=?, originallyPublished=?, originalPublisherID=? WHERE key=?', (buffer(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)), lastPublished, originallyPublished, originalPublisherID, encodedKey))
+        
+#    def _dbQuery(self, key, columnName, unpickle=False):
+#        try:
+#            self._cursor.execute("SELECT %s FROM data WHERE key=:reqKey" % columnName, {'reqKey': key.encode('hex')})
+#            row = self._cursor.fetchone()
+#            value = str(row[0])
+#        except TypeError:
+#            raise KeyError, key
+#        else:
+#            if unpickle:
+#                return pickle.loads(value)
+#            else:
+#                return value
+
+#    def __getitem__(self, key):
+#        return self._dbQuery(key, 'value', unpickle=True)
+
+#    def __delitem__(self, key):
+#        self._cursor.execute("DELETE FROM data WHERE key=:reqKey", {'reqKey': key.encode('hex')})

lbrynet/dht/encoding.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+class DecodeError(Exception):
+    """ Should be raised by an C{Encoding} implementation if decode operation
+    fails
+    """
+
+class Encoding(object):
+    """ Interface for RPC message encoders/decoders
+    
+    All encoding implementations used with this library should inherit and
+    implement this.
+    """
+    def encode(self, data):
+        """ Encode the specified data
+        
+        @param data: The data to encode
+                     This method has to support encoding of the following
+                     types: C{str}, C{int} and C{long}
+                     Any additional data types may be supported as long as the
+                     implementing class's C{decode()} method can successfully
+                     decode them.
+        
+        @return: The encoded data
+        @rtype: str
+        """
+    def decode(self, data):
+        """ Decode the specified data string
+        
+        @param data: The data (byte string) to decode.
+        @type data: str
+        
+        @return: The decoded data (in its correct type)
+        """
+
+class Bencode(Encoding):
+    """ Implementation of a Bencode-based algorithm (Bencode is the encoding
+    algorithm used by Bittorrent).
+    
+    @note: This algorithm differs from the "official" Bencode algorithm in
+           that it can encode/decode floating point values in addition to
+           integers.
+    """
+    
+    def encode(self, data):
+        """ Encoder implementation of the Bencode algorithm
+        
+        @param data: The data to encode
+        @type data: int, long, tuple, list, dict or str
+        
+        @return: The encoded data
+        @rtype: str
+        """
+        if type(data) in (int, long):
+            return 'i%de' % data
+        elif type(data) == str:
+            return '%d:%s' % (len(data), data)
+        elif type(data) in (list, tuple):
+            encodedListItems = ''
+            for item in data:
+                encodedListItems += self.encode(item)
+            return 'l%se' % encodedListItems
+        elif type(data) == dict:
+            encodedDictItems = ''
+            keys = data.keys()
+            keys.sort()
+            for key in keys:
+                encodedDictItems += self.encode(key)
+                encodedDictItems += self.encode(data[key])
+            return 'd%se' % encodedDictItems
+        elif type(data) == float:
+            # This (float data type) is a non-standard extension to the original Bencode algorithm 
+            return 'f%fe' % data
+        elif data == None:
+            # This (None/NULL data type) is a non-standard extension to the original Bencode algorithm 
+            return 'n'
+        else:
+            print data
+            raise TypeError, "Cannot bencode '%s' object" % type(data)
+    
+    def decode(self, data):
+        """ Decoder implementation of the Bencode algorithm 
+        
+        @param data: The encoded data
+        @type data: str
+        
+        @note: This is a convenience wrapper for the recursive decoding
+               algorithm, C{_decodeRecursive}
+       
+        @return: The decoded data, as a native Python type
+        @rtype:  int, list, dict or str
+        """
+        if len(data) == 0:
+            raise DecodeError, 'Cannot decode empty string'
+        return self._decodeRecursive(data)[0]
+    
+    @staticmethod
+    def _decodeRecursive(data, startIndex=0):
+        """ Actual implementation of the recursive Bencode algorithm
+        
+        Do not call this; use C{decode()} instead
+        """
+        if data[startIndex] == 'i':
+            endPos = data[startIndex:].find('e')+startIndex
+            return (int(data[startIndex+1:endPos]), endPos+1)
+        elif data[startIndex] == 'l':
+            startIndex += 1
+            decodedList = []
+            while data[startIndex] != 'e':
+                listData, startIndex = Bencode._decodeRecursive(data, startIndex)
+                decodedList.append(listData)
+            return (decodedList, startIndex+1)
+        elif data[startIndex] == 'd':
+            startIndex += 1
+            decodedDict = {}
+            while data[startIndex] != 'e':
+                key, startIndex = Bencode._decodeRecursive(data, startIndex)
+                value, startIndex = Bencode._decodeRecursive(data, startIndex)
+                decodedDict[key] = value
+            return (decodedDict, startIndex)
+        elif data[startIndex] == 'f':
+            # This (float data type) is a non-standard extension to the original Bencode algorithm
+            endPos = data[startIndex:].find('e')+startIndex
+            return (float(data[startIndex+1:endPos]), endPos+1)
+        elif data[startIndex] == 'n':
+            # This (None/NULL data type) is a non-standard extension to the original Bencode algorithm 
+            return (None, startIndex+1)
+        else:
+            splitPos = data[startIndex:].find(':')+startIndex
+            try:
+                length = int(data[startIndex:splitPos])
+            except ValueError, e:
+                raise DecodeError, e
+            startIndex = splitPos+1
+            endPos = startIndex+length
+            bytes = data[startIndex:endPos]
+            return (bytes, endPos)

lbrynet/dht/hashwatcher.py

@@ -0,0 +1,35 @@
+
+from collections import Counter
+import datetime
+
+
+class HashWatcher():
+    def __init__(self, ttl=600):
+        self.ttl = 600
+        self.hashes = []
+        self.next_tick = None
+
+    def tick(self):
+
+        from twisted.internet import reactor
+
+        self._remove_old_hashes()
+        self.next_tick = reactor.callLater(10, self.tick)
+
+    def stop(self):
+        if self.next_tick is not None:
+            self.next_tick.cancel()
+            self.next_tick = None
+
+    def add_requested_hash(self, hashsum, from_ip):
+        matching_hashes = [h for h in self.hashes if h[0] == hashsum and h[2] == from_ip]
+        if len(matching_hashes) == 0:
+            self.hashes.append((hashsum, datetime.datetime.now(), from_ip))
+
+    def most_popular_hashes(self, num_to_return=10):
+        hash_counter = Counter([h[0] for h in self.hashes])
+        return hash_counter.most_common(num_to_return)
+
+    def _remove_old_hashes(self):
+        remove_time = datetime.datetime.now() - datetime.timedelta(minutes=10)
+        self.hashes = [h for h in self.hashes if h[1] < remove_time]

lbrynet/dht/kbucket.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import constants
+
+class BucketFull(Exception):
+    """ Raised when the bucket is full """
+
+
+class KBucket(object):
+    """ Description - later
+    """
+    def __init__(self, rangeMin, rangeMax):
+        """
+        @param rangeMin: The lower boundary for the range in the n-bit ID
+                         space covered by this k-bucket
+        @param rangeMax: The upper boundary for the range in the ID space
+                         covered by this k-bucket
+        """
+        self.lastAccessed = 0
+        self.rangeMin = rangeMin
+        self.rangeMax = rangeMax
+        self._contacts = list()
+
+    def addContact(self, contact):
+        """ Add contact to _contact list in the right order. This will move the
+        contact to the end of the k-bucket if it is already present.
+        
+        @raise kademlia.kbucket.BucketFull: Raised when the bucket is full and
+                                            the contact isn't in the bucket
+                                            already
+        
+        @param contact: The contact to add
+        @type contact: kademlia.contact.Contact
+        """
+        if contact in self._contacts:
+            # Move the existing contact to the end of the list
+            # - using the new contact to allow add-on data (e.g. optimization-specific stuff) to pe updated as well
+            self._contacts.remove(contact)
+            self._contacts.append(contact)
+        elif len(self._contacts) < constants.k:
+            self._contacts.append(contact)
+        else:
+            raise BucketFull("No space in bucket to insert contact")
+
+    def getContact(self, contactID):
+        """ Get the contact specified node ID"""
+        index = self._contacts.index(contactID)
+        return self._contacts[index]
+
+    def getContacts(self, count=-1, excludeContact=None):
+        """ Returns a list containing up to the first count number of contacts
+        
+        @param count: The amount of contacts to return (if 0 or less, return
+                      all contacts)
+        @type count: int
+        @param excludeContact: A contact to exclude; if this contact is in
+                               the list of returned values, it will be
+                               discarded before returning. If a C{str} is
+                               passed as this argument, it must be the
+                               contact's ID. 
+        @type excludeContact: kademlia.contact.Contact or str
+        
+        
+        @raise IndexError: If the number of requested contacts is too large
+        
+        @return: Return up to the first count number of contacts in a list
+                If no contacts are present an empty is returned
+        @rtype: list
+        """
+        # Return all contacts in bucket
+        if count <= 0:
+            count = len(self._contacts)
+
+        # Get current contact number
+        currentLen = len(self._contacts)
+
+        # If count greater than k - return only k contacts
+        if count > constants.k:
+            count = constants.k
+
+        # Check if count value in range and,
+        # if count number of contacts are available
+        if not currentLen:
+            contactList = list()
+
+        # length of list less than requested amount
+        elif currentLen < count:
+            contactList = self._contacts[0:currentLen]
+        # enough contacts in list
+        else:
+            contactList = self._contacts[0:count]
+            
+        if excludeContact in contactList:
+            contactList.remove(excludeContact)
+
+        return contactList
+
+    def removeContact(self, contact):
+        """ Remove given contact from list
+        
+        @param contact: The contact to remove, or a string containing the
+                        contact's node ID
+        @type contact: kademlia.contact.Contact or str
+        
+        @raise ValueError: The specified contact is not in this bucket
+        """
+        self._contacts.remove(contact)
+    
+    def keyInRange(self, key):
+        """ Tests whether the specified key (i.e. node ID) is in the range
+        of the n-bit ID space covered by this k-bucket (in otherwords, it
+        returns whether or not the specified key should be placed in this
+        k-bucket)
+        
+        @param key: The key to test
+        @type key: str or int
+        
+        @return: C{True} if the key is in this k-bucket's range, or C{False}
+                 if not.
+        @rtype: bool
+        """
+        if isinstance(key, str):
+            key = long(key.encode('hex'), 16)
+        return self.rangeMin <= key < self.rangeMax
+
+    def __len__(self):
+        return len(self._contacts)

lbrynet/dht/msgformat.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import msgtypes
+
+class MessageTranslator(object):
+    """ Interface for RPC message translators/formatters
+    
+    Classes inheriting from this should provide a translation services between
+    the classes used internally by this Kademlia implementation and the actual
+    data that is transmitted between nodes.
+    """
+    def fromPrimitive(self, msgPrimitive):
+        """ Create an RPC Message from a message's string representation
+        
+        @param msgPrimitive: The unencoded primitive representation of a message
+        @type msgPrimitive: str, int, list or dict
+        
+        @return: The translated message object
+        @rtype: entangled.kademlia.msgtypes.Message
+        """
+    
+    def toPrimitive(self, message):
+        """ Create a string representation of a message
+        
+        @param message: The message object
+        @type message: msgtypes.Message
+        
+        @return: The message's primitive representation in a particular
+                 messaging format
+        @rtype: str, int, list or dict
+        """
+        
+class DefaultFormat(MessageTranslator):
+    """ The default on-the-wire message format for this library """
+    typeRequest, typeResponse, typeError = range(3)
+    headerType, headerMsgID, headerNodeID, headerPayload, headerArgs = range(5)
+    
+    def fromPrimitive(self, msgPrimitive):
+        msgType = msgPrimitive[self.headerType]
+        if msgType == self.typeRequest:
+            msg = msgtypes.RequestMessage(msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload], msgPrimitive[self.headerArgs], msgPrimitive[self.headerMsgID])
+        elif msgType == self.typeResponse:
+            msg = msgtypes.ResponseMessage(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload])
+        elif msgType == self.typeError:
+            msg = msgtypes.ErrorMessage(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID], msgPrimitive[self.headerPayload], msgPrimitive[self.headerArgs])
+        else:
+            # Unknown message, no payload
+            msg = msgtypes.Message(msgPrimitive[self.headerMsgID], msgPrimitive[self.headerNodeID])
+        return msg
+    
+    def toPrimitive(self, message):    
+        msg = {self.headerMsgID:  message.id,
+               self.headerNodeID: message.nodeID}
+        if isinstance(message, msgtypes.RequestMessage):
+            msg[self.headerType] = self.typeRequest
+            msg[self.headerPayload] = message.request
+            msg[self.headerArgs] = message.args
+        elif isinstance(message, msgtypes.ErrorMessage):
+            msg[self.headerType] = self.typeError
+            msg[self.headerPayload] = message.exceptionType
+            msg[self.headerArgs] = message.response
+        elif isinstance(message, msgtypes.ResponseMessage):
+            msg[self.headerType] = self.typeResponse
+            msg[self.headerPayload] = message.response
+        return msg

lbrynet/dht/msgtypes.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import hashlib
+import random
+
+class Message(object):
+    """ Base class for messages - all "unknown" messages use this class """
+    def __init__(self, rpcID, nodeID):
+        self.id = rpcID
+        self.nodeID = nodeID
+
+
+class RequestMessage(Message):
+    """ Message containing an RPC request """
+    def __init__(self, nodeID, method, methodArgs, rpcID=None):
+        if rpcID == None:
+            hash = hashlib.sha384()
+            hash.update(str(random.getrandbits(255)))  
+            rpcID = hash.digest()
+        Message.__init__(self, rpcID, nodeID)
+        self.request = method
+        self.args = methodArgs
+
+
+class ResponseMessage(Message):
+    """ Message containing the result from a successful RPC request """
+    def __init__(self, rpcID, nodeID, response):
+        Message.__init__(self, rpcID, nodeID)
+        self.response = response
+
+
+class ErrorMessage(ResponseMessage):
+    """ Message containing the error from an unsuccessful RPC request """
+    def __init__(self, rpcID, nodeID, exceptionType, errorMessage):
+        ResponseMessage.__init__(self, rpcID, nodeID, errorMessage)
+        if isinstance(exceptionType, type):
+            self.exceptionType = '%s.%s' % (exceptionType.__module__, exceptionType.__name__)
+        else:
+            self.exceptionType = exceptionType

lbrynet/dht/protocol.py

@@ -0,0 +1,305 @@
+#!/usr/bin/env python
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import time
+
+from twisted.internet import protocol, defer
+from twisted.python import failure
+import twisted.internet.reactor
+
+import constants
+import encoding
+import msgtypes
+import msgformat
+from contact import Contact
+
+reactor = twisted.internet.reactor
+
+class TimeoutError(Exception):
+    """ Raised when a RPC times out """
+
+class KademliaProtocol(protocol.DatagramProtocol):
+    """ Implements all low-level network-related functions of a Kademlia node """
+    msgSizeLimit = constants.udpDatagramMaxSize-26
+    maxToSendDelay = 10**-3#0.05
+    minToSendDelay = 10**-5#0.01
+
+    def __init__(self, node, msgEncoder=encoding.Bencode(), msgTranslator=msgformat.DefaultFormat()):
+        self._node = node
+        self._encoder = msgEncoder
+        self._translator = msgTranslator
+        self._sentMessages = {}
+        self._partialMessages = {}
+        self._partialMessagesProgress = {}
+        self._next = 0
+        self._callLaterList = {}
+
+    def sendRPC(self, contact, method, args, rawResponse=False):
+        """ Sends an RPC to the specified contact
+
+        @param contact: The contact (remote node) to send the RPC to
+        @type contact: kademlia.contacts.Contact
+        @param method: The name of remote method to invoke
+        @type method: str
+        @param args: A list of (non-keyword) arguments to pass to the remote
+                    method, in the correct order
+        @type args: tuple
+        @param rawResponse: If this is set to C{True}, the caller of this RPC
+                            will receive a tuple containing the actual response
+                            message object and the originating address tuple as
+                            a result; in other words, it will not be
+                            interpreted by this class. Unless something special
+                            needs to be done with the metadata associated with
+                            the message, this should remain C{False}.
+        @type rawResponse: bool
+
+        @return: This immediately returns a deferred object, which will return
+                 the result of the RPC call, or raise the relevant exception
+                 if the remote node raised one. If C{rawResponse} is set to
+                 C{True}, however, it will always return the actual response
+                 message (which may be a C{ResponseMessage} or an
+                 C{ErrorMessage}).
+        @rtype: twisted.internet.defer.Deferred
+        """
+        msg = msgtypes.RequestMessage(self._node.id, method, args)
+        msgPrimitive = self._translator.toPrimitive(msg)
+        encodedMsg = self._encoder.encode(msgPrimitive)
+
+        df = defer.Deferred()
+        if rawResponse:
+            df._rpcRawResponse = True
+
+        # Set the RPC timeout timer
+        timeoutCall = reactor.callLater(constants.rpcTimeout, self._msgTimeout, msg.id) #IGNORE:E1101
+        # Transmit the data
+        self._send(encodedMsg, msg.id, (contact.address, contact.port))
+        self._sentMessages[msg.id] = (contact.id, df, timeoutCall)
+        return df
+
+    def datagramReceived(self, datagram, address):
+        """ Handles and parses incoming RPC messages (and responses)
+
+        @note: This is automatically called by Twisted when the protocol
+               receives a UDP datagram
+        """
+        if datagram[0] == '\x00' and datagram[25] == '\x00':
+            totalPackets = (ord(datagram[1]) << 8) | ord(datagram[2])
+            msgID = datagram[5:25]
+            seqNumber = (ord(datagram[3]) << 8) | ord(datagram[4])
+            if msgID not in self._partialMessages:
+                self._partialMessages[msgID] = {}
+            self._partialMessages[msgID][seqNumber] = datagram[26:]
+            if len(self._partialMessages[msgID]) == totalPackets:
+                keys = self._partialMessages[msgID].keys()
+                keys.sort()
+                data = ''
+                for key in keys:
+                    data += self._partialMessages[msgID][key]
+                    datagram = data
+                del self._partialMessages[msgID]
+            else:
+                return
+        try:
+            msgPrimitive = self._encoder.decode(datagram)
+        except encoding.DecodeError:
+            # We received some rubbish here
+            return
+        
+        message = self._translator.fromPrimitive(msgPrimitive)
+        remoteContact = Contact(message.nodeID, address[0], address[1], self)
+        
+        # Refresh the remote node's details in the local node's k-buckets
+        self._node.addContact(remoteContact)
+
+        if isinstance(message, msgtypes.RequestMessage):
+            # This is an RPC method request
+            self._handleRPC(remoteContact, message.id, message.request, message.args)
+        elif isinstance(message, msgtypes.ResponseMessage):
+            # Find the message that triggered this response
+            if self._sentMessages.has_key(message.id):
+                # Cancel timeout timer for this RPC
+                df, timeoutCall = self._sentMessages[message.id][1:3]
+                timeoutCall.cancel()
+                del self._sentMessages[message.id]
+
+                if hasattr(df, '_rpcRawResponse'):
+                    # The RPC requested that the raw response message and originating address be returned; do not interpret it
+                    df.callback((message, address))
+                elif isinstance(message, msgtypes.ErrorMessage):
+                    # The RPC request raised a remote exception; raise it locally
+                    if message.exceptionType.startswith('exceptions.'):
+                        exceptionClassName = message.exceptionType[11:]
+                    else:
+                        localModuleHierarchy = self.__module__.split('.')
+                        remoteHierarchy = message.exceptionType.split('.')
+                        #strip the remote hierarchy
+                        while remoteHierarchy[0] == localModuleHierarchy[0]:
+                            remoteHierarchy.pop(0)
+                            localModuleHierarchy.pop(0)
+                        exceptionClassName = '.'.join(remoteHierarchy)
+                    remoteException = None
+                    try:
+                        exec 'remoteException = %s("%s")' % (exceptionClassName, message.response)
+                    except Exception:
+                        # We could not recreate the exception; create a generic one
+                        remoteException = Exception(message.response)
+                    df.errback(remoteException)
+                else:
+                    # We got a result from the RPC
+                    df.callback(message.response)
+            else:
+                # If the original message isn't found, it must have timed out
+                #TODO: we should probably do something with this...
+                pass
+
+    def _send(self, data, rpcID, address):
+        """ Transmit the specified data over UDP, breaking it up into several
+        packets if necessary
+        
+        If the data is spread over multiple UDP datagrams, the packets have the
+        following structure::
+            |           |     |      |      |        ||||||||||||   0x00   |
+            |Transmision|Total number|Sequence number| RPC ID   |Header end|
+            | type ID   | of packets |of this packet |          | indicator|
+            | (1 byte)  | (2 bytes)  |  (2 bytes)    |(20 bytes)| (1 byte) |
+            |           |     |      |      |        ||||||||||||          |
+        
+        @note: The header used for breaking up large data segments will
+               possibly be moved out of the KademliaProtocol class in the
+               future, into something similar to a message translator/encoder
+               class (see C{kademlia.msgformat} and C{kademlia.encoding}). 
+        """
+        if len(data) > self.msgSizeLimit:
+            # We have to spread the data over multiple UDP datagrams, and provide sequencing information
+            # 1st byte is transmission type id, bytes 2 & 3 are the total number of packets in this transmission, bytes 4 & 5 are the sequence number for this specific packet
+            totalPackets = len(data) / self.msgSizeLimit
+            if len(data) % self.msgSizeLimit > 0:
+                totalPackets += 1
+            encTotalPackets = chr(totalPackets >> 8) + chr(totalPackets & 0xff)
+            seqNumber = 0
+            startPos = 0
+            while seqNumber < totalPackets:
+                #reactor.iterate() #IGNORE:E1101
+                packetData = data[startPos:startPos+self.msgSizeLimit]
+                encSeqNumber = chr(seqNumber >> 8) + chr(seqNumber & 0xff)
+                txData = '\x00%s%s%s\x00%s' % (encTotalPackets, encSeqNumber, rpcID, packetData)
+                self._sendNext(txData, address)
+
+                startPos += self.msgSizeLimit
+                seqNumber += 1
+        else:
+            self._sendNext(data, address)
+
+    def _sendNext(self, txData, address):
+        """ Send the next UDP packet """
+        ts = time.time()
+        delay = 0
+        if ts >= self._next:
+            delay = self.minToSendDelay
+            self._next = ts + self.minToSendDelay
+        else:
+            delay = (self._next-ts) + self.maxToSendDelay
+            self._next += self.maxToSendDelay
+        if self.transport:
+            laterCall = reactor.callLater(delay, self.transport.write, txData, address)
+            for key in self._callLaterList.keys():
+                if key <= ts:
+                    del self._callLaterList[key]
+            self._callLaterList[self._next] = laterCall
+
+    def _sendResponse(self, contact, rpcID, response):
+        """ Send a RPC response to the specified contact
+        """
+        msg = msgtypes.ResponseMessage(rpcID, self._node.id, response)
+        msgPrimitive = self._translator.toPrimitive(msg)
+        encodedMsg = self._encoder.encode(msgPrimitive)
+        self._send(encodedMsg, rpcID, (contact.address, contact.port))
+
+    def _sendError(self, contact, rpcID, exceptionType, exceptionMessage):
+        """ Send an RPC error message to the specified contact
+        """
+        msg = msgtypes.ErrorMessage(rpcID, self._node.id, exceptionType, exceptionMessage)
+        msgPrimitive = self._translator.toPrimitive(msg)
+        encodedMsg = self._encoder.encode(msgPrimitive)
+        self._send(encodedMsg, rpcID, (contact.address, contact.port))
+
+    def _handleRPC(self, senderContact, rpcID, method, args):
+        """ Executes a local function in response to an RPC request """
+        # Set up the deferred callchain
+        def handleError(f):
+            self._sendError(senderContact, rpcID, f.type, f.getErrorMessage())
+
+        def handleResult(result):
+            self._sendResponse(senderContact, rpcID, result)
+
+        df = defer.Deferred()
+        df.addCallback(handleResult)
+        df.addErrback(handleError)
+
+        # Execute the RPC
+        func = getattr(self._node, method, None)
+        if callable(func) and hasattr(func, 'rpcmethod'):
+            # Call the exposed Node method and return the result to the deferred callback chain
+            try:
+                ##try:
+                ##    # Try to pass the sender's node id to the function...
+                result = func(*args, **{'_rpcNodeID': senderContact.id, '_rpcNodeContact': senderContact})
+                ##except TypeError:
+                ##    # ...or simply call it if that fails
+                ##    result = func(*args)
+            except Exception, e:
+                df.errback(failure.Failure(e))
+            else:
+                df.callback(result)
+        else:
+            # No such exposed method
+            df.errback( failure.Failure( AttributeError('Invalid method: %s' % method) ) )
+
+    def _msgTimeout(self, messageID):
+        """ Called when an RPC request message times out """
+        # Find the message that timed out
+        if self._sentMessages.has_key(messageID):
+            remoteContactID, df = self._sentMessages[messageID][0:2]
+            if self._partialMessages.has_key(messageID):
+                # We are still receiving this message
+                # See if any progress has been made; if not, kill the message
+                if self._partialMessagesProgress.has_key(messageID):
+                    if len(self._partialMessagesProgress[messageID]) == len(self._partialMessages[messageID]):
+                        # No progress has been made
+                        del self._partialMessagesProgress[messageID]
+                        del self._partialMessages[messageID]
+                        df.errback(failure.Failure(TimeoutError(remoteContactID)))
+                        return
+                # Reset the RPC timeout timer
+                timeoutCall = reactor.callLater(constants.rpcTimeout, self._msgTimeout, messageID) #IGNORE:E1101
+                self._sentMessages[messageID] = (remoteContactID, df, timeoutCall)
+                return
+            del self._sentMessages[messageID]
+            # The message's destination node is now considered to be dead;
+            # raise an (asynchronous) TimeoutError exception and update the host node
+            self._node.removeContact(remoteContactID)
+            df.errback(failure.Failure(TimeoutError(remoteContactID)))
+        else:
+            # This should never be reached
+            print "ERROR: deferred timed out, but is not present in sent messages list!"
+
+    def stopProtocol(self):
+        """ Called when the transport is disconnected.
+        
+        Will only be called once, after all ports are disconnected.
+        """
+        for key in self._callLaterList.keys():
+            try:
+                if key > time.time():
+                    self._callLaterList[key].cancel()
+            except Exception, e:
+                print e
+            del self._callLaterList[key]
+            #TODO: test: do we really need the reactor.iterate() call?
+            reactor.iterate()

lbrynet/dht/routingtable.py

@@ -0,0 +1,422 @@
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+# The docstrings in this module contain epytext markup; API documentation
+# may be created by processing this file with epydoc: http://epydoc.sf.net
+
+import time, random
+
+import constants
+import kbucket
+from protocol import TimeoutError
+
+class RoutingTable(object):
+    """ Interface for RPC message translators/formatters
+    
+    Classes inheriting from this should provide a suitable routing table for
+    a parent Node object (i.e. the local entity in the Kademlia network)
+    """
+    def __init__(self, parentNodeID):
+        """
+        @param parentNodeID: The n-bit node ID of the node to which this
+                             routing table belongs
+        @type parentNodeID: str
+        """
+    def addContact(self, contact):
+        """ Add the given contact to the correct k-bucket; if it already
+        exists, its status will be updated
+
+        @param contact: The contact to add to this node's k-buckets
+        @type contact: kademlia.contact.Contact
+        """
+    
+    def distance(self, keyOne, keyTwo):
+        """ Calculate the XOR result between two string variables
+        
+        @return: XOR result of two long variables
+        @rtype: long
+        """
+        valKeyOne = long(keyOne.encode('hex'), 16)
+        valKeyTwo = long(keyTwo.encode('hex'), 16)
+        return valKeyOne ^ valKeyTwo
+
+    def findCloseNodes(self, key, count, _rpcNodeID=None):
+        """ Finds a number of known nodes closest to the node/value with the
+        specified key.
+        
+        @param key: the n-bit key (i.e. the node or value ID) to search for
+        @type key: str
+        @param count: the amount of contacts to return
+        @type count: int
+        @param _rpcNodeID: Used during RPC, this is be the sender's Node ID
+                           Whatever ID is passed in the paramater will get
+                           excluded from the list of returned contacts.
+        @type _rpcNodeID: str
+        
+        @return: A list of node contacts (C{kademlia.contact.Contact instances})
+                 closest to the specified key. 
+                 This method will return C{k} (or C{count}, if specified)
+                 contacts if at all possible; it will only return fewer if the
+                 node is returning all of the contacts that it knows of.
+        @rtype: list
+        """
+    def getContact(self, contactID):
+        """ Returns the (known) contact with the specified node ID
+        
+        @raise ValueError: No contact with the specified contact ID is known
+                           by this node
+        """
+    def getRefreshList(self, startIndex=0, force=False):
+        """ Finds all k-buckets that need refreshing, starting at the
+        k-bucket with the specified index, and returns IDs to be searched for
+        in order to refresh those k-buckets
+
+        @param startIndex: The index of the bucket to start refreshing at;
+                           this bucket and those further away from it will
+                           be refreshed. For example, when joining the
+                           network, this node will set this to the index of
+                           the bucket after the one containing it's closest
+                           neighbour.
+        @type startIndex: index
+        @param force: If this is C{True}, all buckets (in the specified range)
+                      will be refreshed, regardless of the time they were last
+                      accessed.
+        @type force: bool
+        
+        @return: A list of node ID's that the parent node should search for
+                 in order to refresh the routing Table
+        @rtype: list
+        """
+    def removeContact(self, contactID):
+        """ Remove the contact with the specified node ID from the routing
+        table
+        
+        @param contactID: The node ID of the contact to remove
+        @type contactID: str
+        """
+    def touchKBucket(self, key):
+        """ Update the "last accessed" timestamp of the k-bucket which covers
+        the range containing the specified key in the key/ID space
+        
+        @param key: A key in the range of the target k-bucket
+        @type key: str
+        """
+
+
+class TreeRoutingTable(RoutingTable):
+    """ This class implements a routing table used by a Node class.
+    
+    The Kademlia routing table is a binary tree whose leaves are k-buckets,
+    where each k-bucket contains nodes with some common prefix of their IDs.
+    This prefix is the k-bucket's position in the binary tree; it therefore
+    covers some range of ID values, and together all of the k-buckets cover
+    the entire n-bit ID (or key) space (with no overlap).
+    
+    @note: In this implementation, nodes in the tree (the k-buckets) are
+    added dynamically, as needed; this technique is described in the 13-page
+    version of the Kademlia paper, in section 2.4. It does, however, use the
+    C{PING} RPC-based k-bucket eviction algorithm described in section 2.2 of
+    that paper.
+    """
+    def __init__(self, parentNodeID):
+        """
+        @param parentNodeID: The n-bit node ID of the node to which this
+                             routing table belongs
+        @type parentNodeID: str
+        """
+        # Create the initial (single) k-bucket covering the range of the entire n-bit ID space
+        self._buckets = [kbucket.KBucket(rangeMin=0, rangeMax=2**constants.key_bits)]
+        self._parentNodeID = parentNodeID
+
+    def addContact(self, contact):
+        """ Add the given contact to the correct k-bucket; if it already
+        exists, its status will be updated
+
+        @param contact: The contact to add to this node's k-buckets
+        @type contact: kademlia.contact.Contact
+        """
+        if contact.id == self._parentNodeID:
+            return
+
+        bucketIndex = self._kbucketIndex(contact.id)
+        try:
+            self._buckets[bucketIndex].addContact(contact)
+        except kbucket.BucketFull:
+            # The bucket is full; see if it can be split (by checking if its range includes the host node's id)
+            if self._buckets[bucketIndex].keyInRange(self._parentNodeID):
+                self._splitBucket(bucketIndex)
+                # Retry the insertion attempt
+                self.addContact(contact)
+            else:
+                # We can't split the k-bucket
+                # NOTE:
+                # In section 2.4 of the 13-page version of the Kademlia paper, it is specified that
+                # in this case, the new contact should simply be dropped. However, in section 2.2,
+                # it states that the head contact in the k-bucket (i.e. the least-recently seen node)
+                # should be pinged - if it does not reply, it should be dropped, and the new contact
+                # added to the tail of the k-bucket. This implementation follows section 2.2 regarding
+                # this point.            
+                headContact = self._buckets[bucketIndex]._contacts[0]
+    
+                def replaceContact(failure):
+                    """ Callback for the deferred PING RPC to see if the head
+                    node in the k-bucket is still responding
+                    
+                    @type failure: twisted.python.failure.Failure
+                    """
+                    failure.trap(TimeoutError)
+                    print '==replacing contact=='
+                    # Remove the old contact...
+                    deadContactID = failure.getErrorMessage()
+                    try:
+                        self._buckets[bucketIndex].removeContact(deadContactID)
+                    except ValueError:
+                        # The contact has already been removed (probably due to a timeout)
+                        pass
+                    # ...and add the new one at the tail of the bucket
+                    self.addContact(contact)
+                
+                # Ping the least-recently seen contact in this k-bucket
+                headContact = self._buckets[bucketIndex]._contacts[0]
+                df = headContact.ping()
+                # If there's an error (i.e. timeout), remove the head contact, and append the new one
+                df.addErrback(replaceContact)
+                
+    def findCloseNodes(self, key, count, _rpcNodeID=None):
+        """ Finds a number of known nodes closest to the node/value with the
+        specified key.
+        
+        @param key: the n-bit key (i.e. the node or value ID) to search for
+        @type key: str
+        @param count: the amount of contacts to return
+        @type count: int
+        @param _rpcNodeID: Used during RPC, this is be the sender's Node ID
+                           Whatever ID is passed in the paramater will get
+                           excluded from the list of returned contacts.
+        @type _rpcNodeID: str
+        
+        @return: A list of node contacts (C{kademlia.contact.Contact instances})
+                 closest to the specified key. 
+                 This method will return C{k} (or C{count}, if specified)
+                 contacts if at all possible; it will only return fewer if the
+                 node is returning all of the contacts that it knows of.
+        @rtype: list
+        """
+        #if key == self.id:
+        #    bucketIndex = 0 #TODO: maybe not allow this to continue?
+        #else:
+        bucketIndex = self._kbucketIndex(key)
+        closestNodes = self._buckets[bucketIndex].getContacts(constants.k, _rpcNodeID)
+        # This method must return k contacts (even if we have the node with the specified key as node ID), 
+        # unless there is less than k remote nodes in the routing table
+        i = 1
+        canGoLower = bucketIndex-i >= 0
+        canGoHigher = bucketIndex+i < len(self._buckets)
+        # Fill up the node list to k nodes, starting with the closest neighbouring nodes known 
+        while len(closestNodes) < constants.k and (canGoLower or canGoHigher):
+            #TODO: this may need to be optimized
+            if canGoLower:
+                closestNodes.extend(self._buckets[bucketIndex-i].getContacts(constants.k - len(closestNodes), _rpcNodeID))
+                canGoLower = bucketIndex-(i+1) >= 0
+            if canGoHigher:
+                closestNodes.extend(self._buckets[bucketIndex+i].getContacts(constants.k - len(closestNodes), _rpcNodeID))
+                canGoHigher = bucketIndex+(i+1) < len(self._buckets)
+            i += 1
+        return closestNodes
+
+    def getContact(self, contactID):
+        """ Returns the (known) contact with the specified node ID
+        
+        @raise ValueError: No contact with the specified contact ID is known
+                           by this node
+        """
+        bucketIndex = self._kbucketIndex(contactID)
+        try:
+            contact = self._buckets[bucketIndex].getContact(contactID)
+        except ValueError:
+            raise
+        else:
+            return contact
+
+    def getRefreshList(self, startIndex=0, force=False):
+        """ Finds all k-buckets that need refreshing, starting at the
+        k-bucket with the specified index, and returns IDs to be searched for
+        in order to refresh those k-buckets
+
+        @param startIndex: The index of the bucket to start refreshing at;
+                           this bucket and those further away from it will
+                           be refreshed. For example, when joining the
+                           network, this node will set this to the index of
+                           the bucket after the one containing it's closest
+                           neighbour.
+        @type startIndex: index
+        @param force: If this is C{True}, all buckets (in the specified range)
+                      will be refreshed, regardless of the time they were last
+                      accessed.
+        @type force: bool
+        
+        @return: A list of node ID's that the parent node should search for
+                 in order to refresh the routing Table
+        @rtype: list
+        """
+        bucketIndex = startIndex
+        refreshIDs = []
+        for bucket in self._buckets[startIndex:]:
+            if force or (int(time.time()) - bucket.lastAccessed >= constants.refreshTimeout):
+                searchID = self._randomIDInBucketRange(bucketIndex)
+                refreshIDs.append(searchID)
+            bucketIndex += 1
+        return refreshIDs
+
+    def removeContact(self, contactID):
+        """ Remove the contact with the specified node ID from the routing
+        table
+        
+        @param contactID: The node ID of the contact to remove
+        @type contactID: str
+        """
+        bucketIndex = self._kbucketIndex(contactID)
+        try:
+            self._buckets[bucketIndex].removeContact(contactID)
+        except ValueError:
+            #print 'removeContact(): Contact not in routing table'
+            return
+
+    def touchKBucket(self, key):
+        """ Update the "last accessed" timestamp of the k-bucket which covers
+        the range containing the specified key in the key/ID space
+        
+        @param key: A key in the range of the target k-bucket
+        @type key: str
+        """
+        bucketIndex = self._kbucketIndex(key)
+        self._buckets[bucketIndex].lastAccessed = int(time.time())
+
+    def _kbucketIndex(self, key):
+        """ Calculate the index of the k-bucket which is responsible for the
+        specified key (or ID)
+        
+        @param key: The key for which to find the appropriate k-bucket index
+        @type key: str
+        
+        @return: The index of the k-bucket responsible for the specified key
+        @rtype: int
+        """
+        valKey = long(key.encode('hex'), 16)
+        i = 0
+        for bucket in self._buckets:
+            if bucket.keyInRange(valKey):
+                return i
+            else:
+                i += 1
+        return i
+
+    def _randomIDInBucketRange(self, bucketIndex):
+        """ Returns a random ID in the specified k-bucket's range
+        
+        @param bucketIndex: The index of the k-bucket to use
+        @type bucketIndex: int
+        """
+        idValue = random.randrange(self._buckets[bucketIndex].rangeMin, self._buckets[bucketIndex].rangeMax)
+        randomID = hex(idValue)[2:]
+        if randomID[-1] == 'L':
+            randomID = randomID[:-1]
+        if len(randomID) % 2 != 0:
+            randomID = '0' + randomID
+        randomID = randomID.decode('hex')
+        randomID = (constants.key_bits/8 - len(randomID))*'\x00' + randomID
+        return randomID
+
+    def _splitBucket(self, oldBucketIndex):
+        """ Splits the specified k-bucket into two new buckets which together
+        cover the same range in the key/ID space
+        
+        @param oldBucketIndex: The index of k-bucket to split (in this table's
+                               list of k-buckets)
+        @type oldBucketIndex: int
+        """
+        # Resize the range of the current (old) k-bucket
+        oldBucket = self._buckets[oldBucketIndex]
+        splitPoint = oldBucket.rangeMax - (oldBucket.rangeMax - oldBucket.rangeMin)/2
+        # Create a new k-bucket to cover the range split off from the old bucket
+        newBucket = kbucket.KBucket(splitPoint, oldBucket.rangeMax)
+        oldBucket.rangeMax = splitPoint
+        # Now, add the new bucket into the routing table tree
+        self._buckets.insert(oldBucketIndex + 1, newBucket)
+        # Finally, copy all nodes that belong to the new k-bucket into it...
+        for contact in oldBucket._contacts:
+            if newBucket.keyInRange(contact.id):
+                newBucket.addContact(contact)
+        # ...and remove them from the old bucket
+        for contact in newBucket._contacts:
+            oldBucket.removeContact(contact)
+
+class OptimizedTreeRoutingTable(TreeRoutingTable):
+    """ A version of the "tree"-type routing table specified by Kademlia,
+    along with contact accounting optimizations specified in section 4.1 of
+    of the 13-page version of the Kademlia paper.
+    """
+    def __init__(self, parentNodeID):
+        TreeRoutingTable.__init__(self, parentNodeID)
+        # Cache containing nodes eligible to replace stale k-bucket entries
+        self._replacementCache = {}
+        
+    def addContact(self, contact):
+        """ Add the given contact to the correct k-bucket; if it already
+        exists, its status will be updated
+
+        @param contact: The contact to add to this node's k-buckets
+        @type contact: kademlia.contact.Contact
+        """
+        if contact.id == self._parentNodeID:
+            return
+
+        # Initialize/reset the "successively failed RPC" counter
+        contact.failedRPCs = 0
+
+        bucketIndex = self._kbucketIndex(contact.id)
+        try:
+            self._buckets[bucketIndex].addContact(contact)
+        except kbucket.BucketFull:
+            # The bucket is full; see if it can be split (by checking if its range includes the host node's id)
+            if self._buckets[bucketIndex].keyInRange(self._parentNodeID):
+                self._splitBucket(bucketIndex)
+                # Retry the insertion attempt
+                self.addContact(contact)
+            else:
+                # We can't split the k-bucket
+                # NOTE: This implementation follows section 4.1 of the 13 page version
+                # of the Kademlia paper (optimized contact accounting without PINGs
+                #- results in much less network traffic, at the expense of some memory)
+
+                # Put the new contact in our replacement cache for the corresponding k-bucket (or update it's position if it exists already)
+                if not self._replacementCache.has_key(bucketIndex):
+                    self._replacementCache[bucketIndex] = []
+                if contact in self._replacementCache[bucketIndex]:
+                    self._replacementCache[bucketIndex].remove(contact)
+                #TODO: Using k to limit the size of the contact replacement cache - maybe define a seperate value for this in constants.py?
+                elif len(self._replacementCache) >= constants.k:
+                    self._replacementCache.pop(0)
+                self._replacementCache[bucketIndex].append(contact)
+    
+    def removeContact(self, contactID):
+        """ Remove the contact with the specified node ID from the routing
+        table
+        
+        @param contactID: The node ID of the contact to remove
+        @type contactID: str
+        """
+        bucketIndex = self._kbucketIndex(contactID)
+        try:
+            contact = self._buckets[bucketIndex].getContact(contactID)
+        except ValueError:
+            #print 'removeContact(): Contact not in routing table'
+            return
+        contact.failedRPCs += 1
+        if contact.failedRPCs >= 5:        
+            self._buckets[bucketIndex].removeContact(contactID)
+            # Replace this stale contact with one from our replacemnent cache, if we have any
+            if self._replacementCache.has_key(bucketIndex):
+                if len(self._replacementCache[bucketIndex]) > 0:
+                    self._buckets[bucketIndex].addContact( self._replacementCache[bucketIndex].pop() )

lbrynet/dht_scripts.py

@@ -0,0 +1,100 @@
+from lbrynet.dht.node import Node
+import binascii
+from twisted.internet import reactor, task
+import logging
+import sys
+from lbrynet.core.utils import generate_id
+
+
+def print_usage():
+    print "Usage:\n%s UDP_PORT KNOWN_NODE_IP KNOWN_NODE_PORT HASH"
+
+
+def join_network(udp_port, known_nodes):
+    lbryid = generate_id()
+
+    logging.info('Creating Node...')
+    node = Node(udpPort=udp_port, lbryid=lbryid)
+
+    logging.info('Joining network...')
+    d = node.joinNetwork(known_nodes)
+
+    def log_network_size():
+        logging.info("Approximate number of nodes in DHT: %s", str(node.getApproximateTotalDHTNodes()))
+        logging.info("Approximate number of blobs in DHT: %s", str(node.getApproximateTotalHashes()))
+
+    d.addCallback(lambda _: log_network_size())
+
+    d.addCallback(lambda _: node)
+
+    return d
+
+
+def get_hosts(node, h):
+
+    def print_hosts(hosts):
+        print "Hosts returned from the DHT: "
+        print hosts
+
+    logging.info("Looking up %s", h)
+    d = node.getPeersForBlob(h)
+    d.addCallback(print_hosts)
+    return d
+
+
+def announce_hash(node, h):
+    d = node.announceHaveBlob(h, 34567)
+
+    def log_results(results):
+        for success, result in results:
+            if success:
+                logging.info("Succeeded: %s", str(result))
+            else:
+                logging.info("Failed: %s", str(result.getErrorMessage()))
+
+    d.addCallback(log_results)
+    return d
+
+
+def get_args():
+    if len(sys.argv) < 5:
+        print_usage()
+        sys.exit(1)
+    udp_port = int(sys.argv[1])
+    known_nodes = [(sys.argv[2], int(sys.argv[3]))]
+    h = binascii.unhexlify(sys.argv[4])
+    return udp_port, known_nodes, h
+
+
+def run_dht_script(dht_func):
+    log_format = "(%(asctime)s)[%(filename)s:%(lineno)s] %(funcName)s(): %(message)s"
+    logging.basicConfig(level=logging.DEBUG, format=log_format)
+
+    udp_port, known_nodes, h = get_args()
+
+    d = task.deferLater(reactor, 0, join_network, udp_port, known_nodes)
+
+    def run_dht_func(node):
+        return dht_func(node, h)
+
+    d.addCallback(run_dht_func)
+
+    def log_err(err):
+        logging.error("An error occurred: %s", err.getTraceback())
+        return err
+
+    def shut_down():
+        logging.info("Shutting down")
+        reactor.stop()
+
+    d.addErrback(log_err)
+    d.addBoth(lambda _: shut_down())
+    reactor.run()
+
+
+def get_hosts_for_hash_in_dht():
+    run_dht_script(get_hosts)
+
+
+def announce_hash_to_dht():
+    run_dht_script(announce_hash)

lbrynet/dhttest.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python
+#
+# This is a basic single-node example of how to use the Entangled DHT. It creates a Node and (optionally) joins an existing DHT.
+# It then does a Kademlia store and find, and then it deletes the stored value (non-Kademlia method).
+#
+# No tuple space functionality is demonstrated by this script.
+# 
+# To test it properly, start a multi-node Kademlia DHT with the "create_network.py" 
+# script and point this node to that, e.g.:
+# $python create_network.py 10 127.0.0.1
+# 
+# $python basic_example.py 5000 127.0.0.1 4000
+#
+# This library is free software, distributed under the terms of
+# the GNU Lesser General Public License Version 3, or any later version.
+# See the COPYING file included in this archive
+#
+
+# Thanks to Paul Cannon for IP-address resolution functions (taken from aspn.activestate.com)
+
+
+
+import os, sys, time, signal, hashlib, random
+import twisted.internet.reactor
+from lbrynet.dht.node import Node
+#from entangled.kademlia.datastore import SQLiteDataStore
+
+# The Entangled DHT node; instantiated in the main() method
+node = None
+
+# The key to use for this example when storing/retrieving data
+hash = hashlib.sha384()
+hash.update("key")
+KEY = hash.digest()
+# The value to store
+VALUE = random.randint(10000, 20000)
+import binascii
+lbryid = KEY
+
+
+def storeValue(key, value):
+    """ Stores the specified value in the DHT using the specified key """
+    global node
+    print '\nStoring value; Key: %s, Value: %s' % (key, value)
+    # Store the value in the DHT. This method returns a Twisted Deferred result, which we then add callbacks to
+    deferredResult = node.announceHaveHash(key, value)
+    # Add our callback; this method is called when the operation completes...
+    deferredResult.addCallback(storeValueCallback)
+    # ...and for error handling, add an "error callback" as well.
+    # For this example script, I use a generic error handler; usually you would need something more specific
+    deferredResult.addErrback(genericErrorCallback)
+
+
+def storeValueCallback(*args, **kwargs):
+    """ Callback function that is invoked when the storeValue() operation succeeds """
+    print 'Value has been stored in the DHT'
+    # Now that the value has been stored, schedule that the value is read again after 2.5 seconds
+    print 'Scheduling retrieval in 2.5 seconds...'
+    twisted.internet.reactor.callLater(2.5, getValue)
+
+
+def genericErrorCallback(failure):
+    """ Callback function that is invoked if an error occurs during any of the DHT operations """
+    print 'An error has occurred:', failure.getErrorMessage()
+    twisted.internet.reactor.callLater(0, stop)
+
+def getValue():
+    """ Retrieves the value of the specified key (KEY) from the DHT """
+    global node, KEY
+    # Get the value for the specified key (immediately returns a Twisted deferred result)
+    print '\nRetrieving value from DHT for key "%s"...' % binascii.unhexlify("f7d9dc4de674eaa2c5a022eb95bc0d33ec2e75c6")
+    deferredResult = node.iterativeFindValue(binascii.unhexlify("f7d9dc4de674eaa2c5a022eb95bc0d33ec2e75c6"))
+    #deferredResult = node.iterativeFindValue(KEY)
+    # Add a callback to this result; this will be called as soon as the operation has completed
+    deferredResult.addCallback(getValueCallback)
+    # As before, add the generic error callback
+    deferredResult.addErrback(genericErrorCallback)
+
+
+def getValueCallback(result):
+    """ Callback function that is invoked when the getValue() operation succeeds """
+    # Check if the key was found (result is a dict of format {key: value}) or not (in which case a list of "closest" Kademlia contacts would be returned instead")
+    print "Got the value"
+    print result
+    #if type(result) == dict:
+    #    for v in result[binascii.unhexlify("5292fa9c426621f02419f5050900392bdff5036c")]:
+    #        print "v:", v
+    #        print "v[6:", v[6:]
+    #        print "lbryid:",lbryid
+    #        print "lbryid == v[6:]:", lbryid == v[6:]
+    #    print 'Value successfully retrieved: %s' % result[KEY]
+
+    #else:
+    #    print 'Value not found'
+    # Either way, schedule a "delete" operation for the key
+    #print 'Scheduling removal in 2.5 seconds...'
+    #twisted.internet.reactor.callLater(2.5, deleteValue)
+    print 'Scheduling shutdown in 2.5 seconds...'
+    twisted.internet.reactor.callLater(2.5, stop)
+
+
+def stop():
+    """ Stops the Twisted reactor, and thus the script """
+    print '\nStopping Kademlia node and terminating script...'
+    twisted.internet.reactor.stop()
+
+if __name__ == '__main__':
+    
+    import sys, os
+    if len(sys.argv) < 2:
+        print 'Usage:\n%s UDP_PORT [KNOWN_NODE_IP  KNOWN_NODE_PORT]' % sys.argv[0]
+        print 'or:\n%s UDP_PORT [FILE_WITH_KNOWN_NODES]' % sys.argv[0]
+        print '\nIf a file is specified, it should containg one IP address and UDP port\nper line, seperated by a space.'
+        sys.exit(1)
+    try:
+        int(sys.argv[1])
+    except ValueError:
+        print '\nUDP_PORT must be an integer value.\n'
+        print 'Usage:\n%s UDP_PORT  [KNOWN_NODE_IP  KNOWN_NODE_PORT]' % sys.argv[0]
+        print 'or:\n%s UDP_PORT  [FILE_WITH_KNOWN_NODES]' % sys.argv[0]
+        print '\nIf a file is specified, it should contain one IP address and UDP port\nper line, seperated by a space.'
+        sys.exit(1)
+
+    if len(sys.argv) == 4:
+        knownNodes = [(sys.argv[2], int(sys.argv[3]))]
+    elif len(sys.argv) == 3:
+        knownNodes = []
+        f = open(sys.argv[2], 'r')
+        lines = f.readlines()
+        f.close()
+        for line in lines:
+            ipAddress, udpPort = line.split()
+            knownNodes.append((ipAddress, int(udpPort)))
+    else:
+        knownNodes = None
+        print '\nNOTE: You have not specified any remote DHT node(s) to connect to'
+        print 'It will thus not be aware of any existing DHT, but will still function as a self-contained DHT (until another node contacts it).'
+        print 'Run this script without any arguments for info.\n'
+
+    # Set up SQLite-based data store (you could use an in-memory store instead, for example)
+    #if os.path.isfile('/tmp/dbFile%s.db' % sys.argv[1]):
+    #    os.remove('/tmp/dbFile%s.db' % sys.argv[1])
+    #dataStore = SQLiteDataStore(dbFile = '/tmp/dbFile%s.db' % sys.argv[1])
+    # Create the Entangled node. It extends the functionality of a basic Kademlia node (but is fully backwards-compatible with a Kademlia-only network)
+    # If you wish to have a pure Kademlia network, use the entangled.kademlia.node.Node class instead
+    print 'Creating Node...'
+    #node = EntangledNode( udpPort=int(sys.argv[1]), dataStore=dataStore )
+    node = Node( udpPort=int(sys.argv[1]), lbryid=lbryid)
+
+    # Schedule the node to join the Kademlia/Entangled DHT 
+    node.joinNetwork(knownNodes)
+    # Schedule the "storeValue() call to be invoked after 2.5 seconds, using KEY and VALUE as arguments
+    #twisted.internet.reactor.callLater(2.5, storeValue, KEY, VALUE)
+    twisted.internet.reactor.callLater(2.5, getValue)
+    # Start the Twisted reactor - this fires up all networking, and allows the scheduled join operation to take place
+    print 'Twisted reactor started (script will commence in 2.5 seconds)'
+    twisted.internet.reactor.run()
+

lbrynet/interfaces.py

@@ -0,0 +1,633 @@
+"""
+Interfaces which are implemented by various classes within LBRYnet.
+"""
+from zope.interface import Interface
+
+
+class IPeerFinder(Interface):
+    """
+    Used to find peers by sha384 hashes which they claim to be associated with.
+    """
+    def find_peers_for_blob(self, blob_hash):
+        """
+        Look for peers claiming to be associated with a sha384 hashsum.
+
+        @param blob_hash: The sha384 hashsum to use to look up peers.
+        @type blob_hash: string, hex encoded
+
+        @return: a Deferred object which fires with a list of Peer objects
+        @rtype: Deferred which fires with [Peer]
+        """
+
+
+class IRequestSender(Interface):
+    """
+    Used to connect to a peer, send requests to it, and return the responses to those requests.
+    """
+    def add_request(self, request):
+        """
+        Add a request to the next message that will be sent to the peer
+
+        @param request: a request to be sent to the peer in the next message
+        @type request: ClientRequest
+
+        @return: Deferred object which will callback with the response to this request, a dict
+        @rtype: Deferred which fires with dict
+        """
+
+    def add_blob_request(self, blob_request):
+        """
+        Add a request for a blob to the next message that will be sent to the peer.
+
+        This will cause the protocol to call blob_request.write(data) for all incoming
+        data, after the response message has been parsed out, until blob_request.finished_deferred fires.
+
+        @param blob_request: the request for the blob
+        @type blob_request: ClientBlobRequest
+
+        @return: Deferred object which will callback with the response to this request
+        @rtype: Deferred which fires with dict
+        """
+
+
+class IRequestCreator(Interface):
+    """
+    Send requests, via an IRequestSender, to peers.
+    """
+
+    def send_next_request(self, peer, protocol):
+        """
+        Create a Request object for the peer and then give the protocol that request.
+
+        @param peer: the Peer object which the request will be sent to.
+        @type peer: Peer
+
+        @param protocol: the protocol to pass the request to.
+        @type protocol: object which implements IRequestSender
+
+        @return: Deferred object which will callback with True or False depending on whether a Request was sent
+        @rtype: Deferred which fires with boolean
+        """
+
+    def get_new_peers(self):
+        """
+        Get some new peers which the request creator wants to send requests to.
+
+        @return: Deferred object which will callback with [Peer]
+        @rtype: Deferred which fires with [Peer]
+        """
+
+
+class IMetadataHandler(Interface):
+    """
+    Get metadata for the IDownloadManager.
+    """
+    def get_initial_blobs(self):
+        """
+        Return metadata about blobs that are known to be associated with the stream at the time that the
+        stream is set up.
+
+        @return: Deferred object which will call back with a list of BlobInfo objects
+        @rtype: Deferred which fires with [BlobInfo]
+        """
+
+    def final_blob_num(self):
+        """
+        If the last blob in the stream is known, return its blob_num. Otherwise, return None.
+
+        @return: integer representing the final blob num in the stream, or None
+        @rtype: integer or None
+        """
+
+
+class IDownloadManager(Interface):
+    """
+    Manage the downloading of an associated group of blobs, referred to as a stream.
+
+    These objects keep track of metadata about the stream, are responsible for starting and stopping
+    other components, and handle communication between other components.
+    """
+
+    def start_downloading(self):
+        """
+        Load the initial metadata about the stream and then start the other components.
+
+        @return: Deferred which fires when the other components have been started.
+        @rtype: Deferred which fires with boolean
+        """
+
+    def resume_downloading(self):
+        """
+        Start the other components after they have been stopped.
+
+        @return: Deferred which fires when the other components have been started.
+        @rtype: Deferred which fires with boolean
+        """
+
+    def pause_downloading(self):
+        """
+        Stop the other components.
+
+        @return: Deferred which fires when the other components have been stopped.
+        @rtype: Deferred which fires with boolean
+        """
+
+    def add_blobs_to_download(self, blobs):
+        """
+        Add blobs to the list of blobs that should be downloaded
+
+        @param blobs: list of BlobInfos that are associated with the stream being downloaded
+        @type blobs: [BlobInfo]
+
+        @return: DeferredList which fires with the result of adding each previously unknown BlobInfo
+            to the list of known BlobInfos.
+        @rtype: DeferredList which fires with [(boolean, Failure/None)]
+        """
+
+    def stream_position(self):
+        """
+        Returns the blob_num of the next blob needed in the stream.
+
+        If the stream already has all of the blobs it needs, then this will return the blob_num
+        of the last blob in the stream plus 1.
+
+        @return: the blob_num of the next blob needed, or the last blob_num + 1.
+        @rtype: integer
+        """
+
+    def needed_blobs(self):
+        """
+        Returns a list of BlobInfos representing all of the blobs that the stream still needs to download.
+
+        @return: the list of BlobInfos representing blobs that the stream still needs to download.
+        @rtype: [BlobInfo]
+        """
+
+    def final_blob_num(self):
+        """
+        If the last blob in the stream is known, return its blob_num. If not, return None.
+
+        @return: The blob_num of the last blob in the stream, or None if it is unknown.
+        @rtype: integer or None
+        """
+
+    def handle_blob(self, blob_num):
+        """
+        This function is called when the next blob in the stream is ready to be handled, whatever that may mean.
+
+        @param blob_num: The blob_num of the blob that is ready to be handled.
+        @type blob_num: integer
+
+        @return: A Deferred which fires when the blob has been 'handled'
+        @rtype: Deferred which can fire with anything
+        """
+
+
+class IConnectionManager(Interface):
+    """
+    Connects to peers so that IRequestCreators can send their requests.
+    """
+    def get_next_request(self, peer, protocol):
+        """
+        Ask all IRequestCreators belonging to this object to create a Request for peer and give it to protocol
+
+        @param peer: the peer which the request will be sent to.
+        @type peer: Peer
+
+        @param protocol: the protocol which the request should be sent to by the IRequestCreator.
+        @type protocol: IRequestSender
+
+        @return: Deferred object which will callback with True or False depending on whether the IRequestSender
+            should send the request or hang up
+        @rtype: Deferred which fires with boolean
+        """
+
+    def protocol_disconnected(self, peer, protocol):
+        """
+        Inform the IConnectionManager that the protocol has been disconnected
+
+        @param peer: The peer which the connection was to.
+        @type peer: Peer
+
+        @param protocol: The protocol which was disconnected.
+        @type protocol: Protocol
+
+        @return: None
+        """
+
+
+class IProgressManager(Interface):
+    """
+    Responsible for keeping track of the progress of the download.
+
+    Specifically, it is their responsibility to decide which blobs need to be downloaded and keep track of
+    the progress of the download
+    """
+    def stream_position(self):
+        """
+        Returns the blob_num of the next blob needed in the stream.
+
+        If the stream already has all of the blobs it needs, then this will return the blob_num
+        of the last blob in the stream plus 1.
+
+        @return: the blob_num of the next blob needed, or the last blob_num + 1.
+        @rtype: integer
+        """
+
+    def needed_blobs(self):
+        """
+        Returns a list of BlobInfos representing all of the blobs that the stream still needs to download.
+
+        @return: the list of BlobInfos representing blobs that the stream still needs to download.
+        @rtype: [BlobInfo]
+        """
+
+    def blob_downloaded(self, blob, blob_info):
+        """
+        Mark that a blob has been downloaded and does not need to be downloaded again
+
+        @param blob: the blob that has been downloaded.
+        @type blob: Blob
+
+        @param blob_info: the metadata of the blob that has been downloaded.
+        @type blob_info: BlobInfo
+
+        @return: None
+        """
+
+
+class IBlobHandler(Interface):
+    """
+    Responsible for doing whatever should be done with blobs that have been downloaded.
+    """
+    def blob_downloaded(self, blob, blob_info):
+        """
+        Do whatever the downloader is supposed to do when a blob has been downloaded
+
+        @param blob: The downloaded blob
+        @type blob: Blob
+
+        @param blob_info: The metadata of the downloaded blob
+        @type blob_info: BlobInfo
+
+        @return: A Deferred which fires when the blob has been handled.
+        @rtype: Deferred which can fire with anything
+        """
+
+
+class IRateLimited(Interface):
+    """
+    Have the ability to be throttled (temporarily stopped).
+    """
+    def throttle_upload(self):
+        """
+        Stop uploading data until unthrottle_upload is called.
+
+        @return: None
+        """
+
+    def throttle_download(self):
+        """
+        Stop downloading data until unthrottle_upload is called.
+
+        @return: None
+        """
+
+    def unthrottle_upload(self):
+        """
+        Resume uploading data at will until throttle_upload is called.
+
+        @return: None
+        """
+
+    def unthrottle_downlad(self):
+        """
+        Resume downloading data at will until throttle_download is called.
+
+        @return: None
+        """
+
+
+class IRateLimiter(Interface):
+    """
+    Can keep track of download and upload rates and can throttle objects which implement the
+    IRateLimited interface.
+    """
+    def report_dl_bytes(self, num_bytes):
+        """
+        Inform the IRateLimiter that num_bytes have been downloaded.
+
+        @param num_bytes: the number of bytes that have been downloaded
+        @type num_bytes: integer
+
+        @return: None
+        """
+
+    def report_ul_bytes(self, num_bytes):
+        """
+        Inform the IRateLimiter that num_bytes have been uploaded.
+
+        @param num_bytes: the number of bytes that have been uploaded
+        @type num_bytes: integer
+
+        @return: None
+        """
+
+    def register_protocol(self, protocol):
+        """
+        Register an IRateLimited object with the IRateLimiter so that the IRateLimiter can throttle it
+
+        @param protocol: An object implementing the interface IRateLimited
+        @type protocol: Object implementing IRateLimited
+
+        @return: None
+        """
+
+    def unregister_protocol(self, protocol):
+        """
+        Unregister an IRateLimited object so that it won't be throttled any more.
+
+        @param protocol: An object implementing the interface IRateLimited, which was previously registered with this
+            IRateLimiter via "register_protocol"
+        @type protocol: Object implementing IRateLimited
+
+        @return: None
+        """
+
+
+class IRequestHandler(Interface):
+    """
+    Pass client queries on to IQueryHandlers
+    """
+    def register_query_handler(self, query_handler, query_identifiers):
+        """
+        Register a query handler, which will be passed any queries that
+        match any of the identifiers in query_identifiers
+
+        @param query_handler: the object which will handle queries matching the given query_identifiers
+        @type query_handler: Object implementing IQueryHandler
+
+        @param query_identifiers: A list of strings representing the query identifiers
+            for queries that should be passed to this handler
+        @type query_identifiers: [string]
+
+        @return: None
+        """
+
+    def register_blob_sender(self, blob_sender):
+        """
+        Register a blob sender which will be called after the response has
+        finished to see if it wants to send a blob
+
+        @param blob_sender: the object which will upload the blob to the client.
+        @type blob_sender: IBlobSender
+
+        @return: None
+        """
+
+
+class IBlobSender(Interface):
+    """
+    Upload blobs to clients.
+    """
+    def send_blob_if_requested(self, consumer):
+        """
+        If a blob has been requested, write it to 'write' func of the consumer and then
+        callback the returned deferred when it has all been written
+
+        @param consumer: the object implementing IConsumer which the file will be written to
+        @type consumer: object which implements IConsumer
+
+        @return: Deferred which will fire when the blob sender is done, which will be
+            immediately if no blob should be sent.
+        @rtype: Deferred which fires with anything
+        """
+
+
+class IQueryHandler(Interface):
+    """
+    Respond to requests from clients.
+    """
+    def register_with_request_handler(self, request_handler, peer):
+        """
+        Register with the request handler to receive queries
+
+        @param request_handler: the object implementing IRequestHandler to register with
+        @type request_handler: object implementing IRequestHandler
+
+        @param peer: the Peer which this query handler will be answering requests from
+        @type peer: Peer
+
+        @return: None
+        """
+
+    def handle_queries(self, queries):
+        """
+        Return responses to queries from the client.
+
+        @param queries: a dict representing the query_identifiers:queries that should be handled
+        @type queries: {string: dict}
+
+        @return: a Deferred object which will callback with a dict of query responses
+        @rtype: Deferred which fires with {string: dict}
+        """
+
+
+class IQueryHandlerFactory(Interface):
+    """
+    Construct IQueryHandlers to handle queries from each new client that connects.
+    """
+    def build_query_handler(self):
+        """
+        Create an object that implements the IQueryHandler interface
+
+        @return: object that implements IQueryHandler
+        """
+
+
+class IStreamDownloaderFactory(Interface):
+    """
+    Construct IStreamDownloaders and provide options that will be passed to those IStreamDownloaders.
+    """
+    def get_downloader_options(self, sd_validator, payment_rate_manager):
+        """
+        Return the list of options that can be used to modify IStreamDownloader behavior
+
+        @param sd_validator: object containing stream metadata, which the options may depend on
+        @type sd_validator: object which implements IStreamDescriptorValidator interface
+
+        @param payment_rate_manager: The payment rate manager currently in effect for the downloader
+        @type payment_rate_manager: PaymentRateManager
+
+        @return: [(option_description, default)]
+        @rtype: [(string, string)]
+        """
+
+    def make_downloader(self, sd_validator, options, payment_rate_manager):
+        """
+        Create an object that implements the IStreamDownloader interface
+
+        @param sd_validator: object containing stream metadata which will be given to the IStreamDownloader
+        @type sd_validator: object which implements IStreamDescriptorValidator interface
+
+        @param options: a list of strings that will be used by the IStreamDownloaderFactory to
+            construct the IStreamDownloader. the options are in the same order as they were given
+            by get_downloader_options.
+        @type options: [string]
+
+        @param payment_rate_manager: the PaymentRateManager which the IStreamDownloader should use.
+        @type payment_rate_manager: PaymentRateManager
+
+        @return: a Deferred which fires with the downloader object
+        @rtype: Deferred which fires with IStreamDownloader
+        """
+
+    def get_description(self):
+        """
+        Return a string detailing what this downloader does with streams
+
+        @return: short description of what the IStreamDownloader does.
+        @rtype: string
+        """
+
+
+class IStreamDownloader(Interface):
+    """
+    Use metadata and data from the network for some useful purpose.
+    """
+    def start(self):
+        """
+        start downloading the stream
+
+        @return: a Deferred which fires when the stream is finished downloading, or errbacks when the stream is
+            cancelled.
+        @rtype: Deferred which fires with anything
+        """
+
+    def insufficient_funds(self):
+        """
+        this function informs the stream downloader that funds are too low to finish downloading.
+
+        @return: None
+        """
+
+
+class IStreamDescriptorValidator(Interface):
+    """
+    Pull metadata out of Stream Descriptor Files and perform some
+    validation on the metadata.
+    """
+    def validate(self):
+        """
+        @return: whether the stream descriptor passes validation checks
+        @rtype: boolean
+        """
+
+    def info_to_show(self):
+        """
+        @return: A list of tuples representing metadata that should be presented to the user before starting the
+            download
+        @rtype: [(string, string)]
+        """
+
+
+class ILBRYWallet(Interface):
+    """
+    Send and receive payments.
+
+    To send a payment, a payment reservation must be obtained first. This guarantees that a payment
+    isn't promised if it can't be paid. When the service in question is rendered, the payment
+    reservation must be given to the ILBRYWallet along with the final price. The reservation can also
+    be canceled.
+    """
+    def stop(self):
+        """
+        Send out any unsent payments, close any connections, and stop checking for incoming payments.
+
+        @return: None
+        """
+
+    def start(self):
+        """
+        Set up any connections and start checking for incoming payments
+
+        @return: None
+        """
+    def get_info_exchanger(self):
+        """
+        Get the object that will be used to find the payment addresses of peers.
+
+        @return: The object that will be used to find the payment addresses of peers.
+        @rtype: An object implementing IRequestCreator
+        """
+
+    def get_wallet_info_query_handler_factory(self):
+        """
+        Get the object that will be used to give our payment address to peers.
+
+        This must return an object implementing IQueryHandlerFactory. It will be used to
+        create IQueryHandler objects that will be registered with an IRequestHandler.
+
+        @return: The object that will be used to give our payment address to peers.
+        @rtype: An object implementing IQueryHandlerFactory
+        """
+
+    def reserve_points(self, peer, amount):
+        """
+        Ensure a certain amount of points are available to be sent as payment, before the service is rendered
+
+        @param peer: The peer to which the payment will ultimately be sent
+        @type peer: Peer
+
+        @param amount: The amount of points to reserve
+        @type amount: float
+
+        @return: A ReservedPoints object which is given to send_points once the service has been rendered
+        @rtype: ReservedPoints
+        """
+
+    def cancel_point_reservation(self, reserved_points):
+        """
+        Return all of the points that were reserved previously for some ReservedPoints object
+
+        @param reserved_points: ReservedPoints previously returned by reserve_points
+        @type reserved_points: ReservedPoints
+
+        @return: None
+        """
+
+    def send_points(self, reserved_points, amount):
+        """
+        Schedule a payment to be sent to a peer
+
+        @param reserved_points: ReservedPoints object previously returned by reserve_points.
+        @type reserved_points: ReservedPoints
+
+        @param amount: amount of points to actually send, must be less than or equal to the
+            amount reserved in reserved_points
+        @type amount: float
+
+        @return: Deferred which fires when the payment has been scheduled
+        @rtype: Deferred which fires with anything
+        """
+
+    def get_balance(self):
+        """
+        Return the balance of this wallet
+
+        @return: Deferred which fires with the balance of the wallet
+        @rtype: Deferred which fires with float
+        """
+
+    def add_expected_payment(self, peer, amount):
+        """
+        Increase the number of points expected to be paid by a peer
+
+        @param peer: the peer which is expected to pay the points
+        @type peer: Peer
+
+        @param amount: the amount of points expected to be paid
+        @type amount: float
+
+        @return: None
+        """

lbrynet/lbryfile/LBRYFileMetadataManager.py

@@ -0,0 +1,268 @@
+import logging
+import leveldb
+import json
+import os
+from twisted.internet import threads, defer
+from lbrynet.core.Error import DuplicateStreamHashError
+
+
+class DBLBRYFileMetadataManager(object):
+    """Store and provide access to LBRY file metadata using leveldb files"""
+
+    def __init__(self, db_dir):
+        self.db_dir = db_dir
+        self.stream_info_db = None
+        self.stream_blob_db = None
+        self.stream_desc_db = None
+
+    def setup(self):
+        return threads.deferToThread(self._open_db)
+
+    def stop(self):
+        self.stream_info_db = None
+        self.stream_blob_db = None
+        self.stream_desc_db = None
+        return defer.succeed(True)
+
+    def get_all_streams(self):
+        return threads.deferToThread(self._get_all_streams)
+
+    def save_stream(self, stream_hash, file_name, key, suggested_file_name, blobs):
+        d = threads.deferToThread(self._store_stream, stream_hash, file_name, key, suggested_file_name)
+        d.addCallback(lambda _: self.add_blobs_to_stream(stream_hash, blobs))
+        return d
+
+    def get_stream_info(self, stream_hash):
+        return threads.deferToThread(self._get_stream_info, stream_hash)
+
+    def check_if_stream_exists(self, stream_hash):
+        return threads.deferToThread(self._check_if_stream_exists, stream_hash)
+
+    def delete_stream(self, stream_hash):
+        return threads.deferToThread(self._delete_stream, stream_hash)
+
+    def add_blobs_to_stream(self, stream_hash, blobs):
+
+        def add_blobs():
+            self._add_blobs_to_stream(stream_hash, blobs, ignore_duplicate_error=True)
+
+        return threads.deferToThread(add_blobs)
+
+    def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
+        logging.info("Getting blobs for a stream. Count is %s", str(count))
+
+        def get_positions_of_start_and_end():
+            if start_blob is not None:
+                start_num = self._get_blob_num_by_hash(stream_hash, start_blob)
+            else:
+                start_num = None
+            if end_blob is not None:
+                end_num = self._get_blob_num_by_hash(stream_hash, end_blob)
+            else:
+                end_num = None
+            return start_num, end_num
+
+        def get_blob_infos(nums):
+            start_num, end_num = nums
+            return threads.deferToThread(self._get_further_blob_infos, stream_hash, start_num, end_num,
+                                         count, reverse)
+
+        d = threads.deferToThread(get_positions_of_start_and_end)
+        d.addCallback(get_blob_infos)
+        return d
+
+    def get_stream_of_blob(self, blob_hash):
+        return threads.deferToThread(self._get_stream_of_blobhash, blob_hash)
+
+    def save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        return threads.deferToThread(self._save_sd_blob_hash_to_stream, stream_hash, sd_blob_hash)
+
+    def get_sd_blob_hashes_for_stream(self, stream_hash):
+        return threads.deferToThread(self._get_sd_blob_hashes_for_stream, stream_hash)
+
+    def _open_db(self):
+        self.stream_info_db = leveldb.LevelDB(os.path.join(self.db_dir, "lbryfile_info.db"))
+        self.stream_blob_db = leveldb.LevelDB(os.path.join(self.db_dir, "lbryfile_blob.db"))
+        self.stream_desc_db = leveldb.LevelDB(os.path.join(self.db_dir, "lbryfile_desc.db"))
+
+    def _delete_stream(self, stream_hash):
+        desc_batch = leveldb.WriteBatch()
+        for sd_blob_hash, s_h in self.stream_desc_db.RangeIter():
+            if stream_hash == s_h:
+                desc_batch.Delete(sd_blob_hash)
+        self.stream_desc_db.Write(desc_batch, sync=True)
+
+        blob_batch = leveldb.WriteBatch()
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if stream_hash == s_h:
+                blob_batch.Delete(blob_hash_stream_hash)
+        self.stream_blob_db.Write(blob_batch, sync=True)
+
+        stream_batch = leveldb.WriteBatch()
+        for s_h, stream_info in self.stream_info_db.RangeIter():
+            if stream_hash == s_h:
+                stream_batch.Delete(s_h)
+        self.stream_info_db.Write(stream_batch, sync=True)
+
+    def _store_stream(self, stream_hash, name, key, suggested_file_name):
+        try:
+            self.stream_info_db.Get(stream_hash)
+            raise DuplicateStreamHashError("Stream hash %s already exists" % stream_hash)
+        except KeyError:
+            pass
+        self.stream_info_db.Put(stream_hash, json.dumps((key, name, suggested_file_name)), sync=True)
+
+    def _get_all_streams(self):
+        return [stream_hash for stream_hash, stream_info in self.stream_info_db.RangeIter()]
+
+    def _get_stream_info(self, stream_hash):
+        return json.loads(self.stream_info_db.Get(stream_hash))[:3]
+
+    def _check_if_stream_exists(self, stream_hash):
+        try:
+            self.stream_info_db.Get(stream_hash)
+            return True
+        except KeyError:
+            return False
+
+    def _get_blob_num_by_hash(self, stream_hash, blob_hash):
+        blob_hash_stream_hash = json.dumps((blob_hash, stream_hash))
+        return json.loads(self.stream_blob_db.Get(blob_hash_stream_hash))[0]
+
+    def _get_further_blob_infos(self, stream_hash, start_num, end_num, count=None, reverse=False):
+        blob_infos = []
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if stream_hash == s_h:
+                position, iv, length = json.loads(blob_info)
+                if (start_num is None) or (position > start_num):
+                    if (end_num is None) or (position < end_num):
+                        blob_infos.append((b_h, position, iv, length))
+        blob_infos.sort(key=lambda i: i[1], reverse=reverse)
+        if count is not None:
+            blob_infos = blob_infos[:count]
+        return blob_infos
+
+    def _add_blobs_to_stream(self, stream_hash, blob_infos, ignore_duplicate_error=False):
+        batch = leveldb.WriteBatch()
+        for blob_info in blob_infos:
+            blob_hash_stream_hash = json.dumps((blob_info.blob_hash, stream_hash))
+            try:
+                self.stream_blob_db.Get(blob_hash_stream_hash)
+                if ignore_duplicate_error is False:
+                    raise KeyError()  # TODO: change this to DuplicateStreamBlobError?
+                continue
+            except KeyError:
+                pass
+            batch.Put(blob_hash_stream_hash,
+                      json.dumps((blob_info.blob_num,
+                                  blob_info.iv,
+                                  blob_info.length)))
+        self.stream_blob_db.Write(batch, sync=True)
+
+    def _get_stream_of_blobhash(self, blob_hash):
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if blob_hash == b_h:
+                return s_h
+        return None
+
+    def _save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        self.stream_desc_db.Put(sd_blob_hash, stream_hash)
+
+    def _get_sd_blob_hashes_for_stream(self, stream_hash):
+        return [sd_blob_hash for sd_blob_hash, s_h in self.stream_desc_db.RangeIter() if stream_hash == s_h]
+
+
+class TempLBRYFileMetadataManager(object):
+    def __init__(self):
+        self.streams = {}
+        self.stream_blobs = {}
+        self.sd_files = {}
+
+    def setup(self):
+        return defer.succeed(True)
+
+    def stop(self):
+        return defer.succeed(True)
+
+    def get_all_streams(self):
+        return defer.succeed(self.streams.keys())
+
+    def save_stream(self, stream_hash, file_name, key, suggested_file_name, blobs):
+        self.streams[stream_hash] = {'suggested_file_name': suggested_file_name,
+                                     'stream_name': file_name,
+                                     'key': key}
+        d = self.add_blobs_to_stream(stream_hash, blobs)
+        d.addCallback(lambda _: stream_hash)
+        return d
+
+    def get_stream_info(self, stream_hash):
+        if stream_hash in self.streams:
+            stream_info = self.streams[stream_hash]
+            return defer.succeed([stream_info['key'], stream_info['stream_name'],
+                                  stream_info['suggested_file_name']])
+        return defer.succeed(None)
+
+    def delete_stream(self, stream_hash):
+        if stream_hash in self.streams:
+            del self.streams[stream_hash]
+            for (s_h, b_h) in self.stream_blobs.keys():
+                if s_h == stream_hash:
+                    del self.stream_blobs[(s_h, b_h)]
+        return defer.succeed(True)
+
+    def add_blobs_to_stream(self, stream_hash, blobs):
+        assert stream_hash in self.streams, "Can't add blobs to a stream that isn't known"
+        for blob in blobs:
+            info = {}
+            info['blob_num'] = blob.blob_num
+            info['length'] = blob.length
+            info['iv'] = blob.iv
+            self.stream_blobs[(stream_hash, blob.blob_hash)] = info
+        return defer.succeed(True)
+
+    def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
+
+        if start_blob is not None:
+            start_num = self._get_blob_num_by_hash(stream_hash, start_blob)
+        else:
+            start_num = None
+        if end_blob is not None:
+            end_num = self._get_blob_num_by_hash(stream_hash, end_blob)
+        else:
+            end_num = None
+        return self._get_further_blob_infos(stream_hash, start_num, end_num, count, reverse)
+
+    def get_stream_of_blob(self, blob_hash):
+        for (s_h, b_h) in self.stream_blobs.iterkeys():
+            if b_h == blob_hash:
+                return defer.succeed(s_h)
+        return defer.succeed(None)
+
+    def _get_further_blob_infos(self, stream_hash, start_num, end_num, count=None, reverse=False):
+        blob_infos = []
+        for (s_h, b_h), info in self.stream_blobs.iteritems():
+            if stream_hash == s_h:
+                position = info['blob_num']
+                length = info['length']
+                iv = info['iv']
+                if (start_num is None) or (position > start_num):
+                    if (end_num is None) or (position < end_num):
+                        blob_infos.append((b_h, position, iv, length))
+        blob_infos.sort(key=lambda i: i[1], reverse=reverse)
+        if count is not None:
+            blob_infos = blob_infos[:count]
+        return defer.succeed(blob_infos)
+
+    def _get_blob_num_by_hash(self, stream_hash, blob_hash):
+        if (stream_hash, blob_hash) in self.stream_blobs:
+            return defer.succeed(self.stream_blobs[(stream_hash, blob_hash)]['blob_num'])
+
+    def save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        self.sd_files[sd_blob_hash] = stream_hash
+        return defer.succeed(True)
+
+    def get_sd_blob_hashes_for_stream(self, stream_hash):
+        return defer.succeed([sd_hash for sd_hash, s_h in self.sd_files.iteritems() if stream_hash == s_h])

lbrynet/lbryfile/StreamDescriptor.py

@@ -0,0 +1,138 @@
+import binascii
+import logging
+from lbrynet.core.cryptoutils import get_lbry_hash_obj
+from lbrynet.cryptstream.CryptBlob import CryptBlobInfo
+from twisted.internet import defer
+from lbrynet.core.Error import DuplicateStreamHashError
+
+
+LBRYFileStreamType = "lbryfile"
+
+
+def save_sd_info(stream_info_manager, sd_info, ignore_duplicate=False):
+    logging.debug("Saving info for %s", str(sd_info['stream_name']))
+    hex_stream_name = sd_info['stream_name']
+    key = sd_info['key']
+    stream_hash = sd_info['stream_hash']
+    raw_blobs = sd_info['blobs']
+    suggested_file_name = sd_info['suggested_file_name']
+    crypt_blobs = []
+    for blob in raw_blobs:
+        length = blob['length']
+        if length != 0:
+            blob_hash = blob['blob_hash']
+        else:
+            blob_hash = None
+        blob_num = blob['blob_num']
+        iv = blob['iv']
+        crypt_blobs.append(CryptBlobInfo(blob_hash, blob_num, length, iv))
+    logging.debug("Trying to save stream info for %s", str(hex_stream_name))
+    d = stream_info_manager.save_stream(stream_hash, hex_stream_name, key,
+                                        suggested_file_name, crypt_blobs)
+
+    def check_if_duplicate(err):
+        if ignore_duplicate is True:
+            err.trap(DuplicateStreamHashError)
+
+    d.addErrback(check_if_duplicate)
+
+    d.addCallback(lambda _: stream_hash)
+    return d
+
+
+def get_sd_info(stream_info_manager, stream_hash, include_blobs):
+    d = stream_info_manager.get_stream_info(stream_hash)
+
+    def format_info(stream_info):
+        fields = {}
+        fields['stream_type'] = LBRYFileStreamType
+        fields['stream_name'] = stream_info[1]
+        fields['key'] = stream_info[0]
+        fields['suggested_file_name'] = stream_info[2]
+        fields['stream_hash'] = stream_hash
+
+        def format_blobs(blobs):
+            formatted_blobs = []
+            for blob_hash, blob_num, iv, length in blobs:
+                blob = {}
+                if length != 0:
+                    blob['blob_hash'] = blob_hash
+                blob['blob_num'] = blob_num
+                blob['iv'] = iv
+                blob['length'] = length
+                formatted_blobs.append(blob)
+            fields['blobs'] = formatted_blobs
+            return fields
+
+        if include_blobs is True:
+            d = stream_info_manager.get_blobs_for_stream(stream_hash)
+        else:
+            d = defer.succeed([])
+        d.addCallback(format_blobs)
+        return d
+
+    d.addCallback(format_info)
+    return d
+
+
+class LBRYFileStreamDescriptorValidator(object):
+    def __init__(self, raw_info):
+        self.raw_info = raw_info
+
+    def validate(self):
+        logging.debug("Trying to validate stream descriptor for %s", str(self.raw_info['stream_name']))
+        try:
+            hex_stream_name = self.raw_info['stream_name']
+            key = self.raw_info['key']
+            hex_suggested_file_name = self.raw_info['suggested_file_name']
+            stream_hash = self.raw_info['stream_hash']
+            blobs = self.raw_info['blobs']
+        except KeyError as e:
+            raise ValueError("Invalid stream descriptor. Missing '%s'" % (e.args[0]))
+        for c in hex_suggested_file_name:
+            if c not in '0123456789abcdef':
+                raise ValueError("Invalid stream descriptor: "
+                                 "suggested file name is not a hex-encoded string")
+        h = get_lbry_hash_obj()
+        h.update(hex_stream_name)
+        h.update(key)
+        h.update(hex_suggested_file_name)
+
+        def get_blob_hashsum(b):
+            length = b['length']
+            if length != 0:
+                blob_hash = b['blob_hash']
+            else:
+                blob_hash = None
+            blob_num = b['blob_num']
+            iv = b['iv']
+            blob_hashsum = get_lbry_hash_obj()
+            if length != 0:
+                blob_hashsum.update(blob_hash)
+            blob_hashsum.update(str(blob_num))
+            blob_hashsum.update(iv)
+            blob_hashsum.update(str(length))
+            return blob_hashsum.digest()
+
+        blobs_hashsum = get_lbry_hash_obj()
+        for blob in blobs:
+            blobs_hashsum.update(get_blob_hashsum(blob))
+        if blobs[-1]['length'] != 0:
+            raise ValueError("Improperly formed stream descriptor. Must end with a zero-length blob.")
+        h.update(blobs_hashsum.digest())
+        if h.hexdigest() != stream_hash:
+            raise ValueError("Stream hash does not match stream metadata")
+        return defer.succeed(True)
+
+    def info_to_show(self):
+        info = []
+        info.append(("stream_name", binascii.unhexlify(self.raw_info.get("stream_name"))))
+        size_so_far = 0
+        for blob_info in self.raw_info.get("blobs", []):
+            size_so_far += int(blob_info['length'])
+        info.append(("stream_size", str(size_so_far)))
+        suggested_file_name = self.raw_info.get("suggested_file_name", None)
+        if suggested_file_name is not None:
+            suggested_file_name = binascii.unhexlify(suggested_file_name)
+        info.append(("suggested_file_name", suggested_file_name))
+        return info

lbrynet/lbryfile/client/LBRYFileDownloader.py

@@ -0,0 +1,284 @@
+import subprocess
+import binascii
+
+from zope.interface import implements
+
+from lbrynet.core.DownloadOption import DownloadOption
+from lbrynet.lbryfile.StreamDescriptor import save_sd_info
+from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownloader
+from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
+from lbrynet.interfaces import IStreamDownloaderFactory
+from lbrynet.lbryfile.client.LBRYFileMetadataHandler import LBRYFileMetadataHandler
+import os
+from twisted.internet import defer, threads, reactor
+
+
+class LBRYFileDownloader(CryptStreamDownloader):
+    """Classes which inherit from this class download LBRY files"""
+
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
+                 stream_info_manager, payment_rate_manager, wallet, upload_allowed):
+        CryptStreamDownloader.__init__(self, peer_finder, rate_limiter, blob_manager,
+                                       payment_rate_manager, wallet, upload_allowed)
+        self.stream_hash = stream_hash
+        self.stream_info_manager = stream_info_manager
+        self.suggested_file_name = None
+        self._calculated_total_bytes = None
+
+    def set_stream_info(self):
+        if self.key is None:
+            d = self.stream_info_manager.get_stream_info(self.stream_hash)
+
+            def set_stream_info(stream_info):
+                key, stream_name, suggested_file_name = stream_info
+                self.key = binascii.unhexlify(key)
+                self.stream_name = binascii.unhexlify(stream_name)
+                self.suggested_file_name = binascii.unhexlify(suggested_file_name)
+
+            d.addCallback(set_stream_info)
+            return d
+        else:
+            return defer.succeed(True)
+
+    def stop(self):
+        d = self._close_output()
+        d.addCallback(lambda _: CryptStreamDownloader.stop(self))
+        return d
+
+    def _get_progress_manager(self, download_manager):
+        return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
+
+    def _start(self):
+        d = self._setup_output()
+        d.addCallback(lambda _: CryptStreamDownloader._start(self))
+        return d
+
+    def _setup_output(self):
+        pass
+
+    def _close_output(self):
+        pass
+
+    def get_total_bytes(self):
+        if self._calculated_total_bytes is None or self._calculated_total_bytes == 0:
+            if self.download_manager is None:
+                return 0
+            else:
+                self._calculated_total_bytes = self.download_manager.calculate_total_bytes()
+        return self._calculated_total_bytes
+
+    def get_bytes_left_to_output(self):
+        if self.download_manager is not None:
+            return self.download_manager.calculate_bytes_left_to_output()
+        else:
+            return 0
+
+    def get_bytes_left_to_download(self):
+        if self.download_manager is not None:
+            return self.download_manager.calculate_bytes_left_to_download()
+        else:
+            return 0
+
+    def _get_metadata_handler(self, download_manager):
+        return LBRYFileMetadataHandler(self.stream_hash, self.stream_info_manager, download_manager)
+
+
+class LBRYFileDownloaderFactory(object):
+    implements(IStreamDownloaderFactory)
+
+    def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 wallet):
+        self.peer_finder = peer_finder
+        self.rate_limiter = rate_limiter
+        self.blob_manager = blob_manager
+        self.stream_info_manager = stream_info_manager
+        self.wallet = wallet
+
+    def get_downloader_options(self, sd_validator, payment_rate_manager):
+        options = [
+            DownloadOption(
+                [float, None],
+                "rate which will be paid for data (None means use application default)",
+                "data payment rate",
+                None
+            ),
+            DownloadOption(
+                [bool],
+                "allow reuploading data downloaded for this file",
+                "allow upload",
+                True
+            ),
+        ]
+        return options
+
+    def make_downloader(self, sd_validator, options, payment_rate_manager, **kwargs):
+        if options[0] is not None:
+            payment_rate_manager.float(options[0])
+        upload_allowed = options[1]
+
+        def create_downloader(stream_hash):
+            downloader = self._make_downloader(stream_hash, payment_rate_manager, sd_validator.raw_info,
+                                               upload_allowed)
+            d = downloader.set_stream_info()
+            d.addCallback(lambda _: downloader)
+            return d
+
+        d = save_sd_info(self.stream_info_manager, sd_validator.raw_info)
+
+        d.addCallback(create_downloader)
+        return d
+
+    def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
+        pass
+
+
+class LBRYFileSaver(LBRYFileDownloader):
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 payment_rate_manager, wallet, download_directory, upload_allowed, file_name=None):
+        LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
+                                    stream_info_manager, payment_rate_manager, wallet, upload_allowed)
+        self.download_directory = download_directory
+        self.file_name = file_name
+        self.file_handle = None
+
+    def set_stream_info(self):
+        d = LBRYFileDownloader.set_stream_info(self)
+
+        def set_file_name():
+            if self.file_name is None:
+                if self.suggested_file_name:
+                    self.file_name = os.path.basename(self.suggested_file_name)
+                else:
+                    self.file_name = os.path.basename(self.stream_name)
+
+        d.addCallback(lambda _: set_file_name())
+        return d
+
+    def stop(self):
+        d = LBRYFileDownloader.stop(self)
+        d.addCallback(lambda _: self._delete_from_info_manager())
+        return d
+
+    def _get_progress_manager(self, download_manager):
+        return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager,
+                                         delete_blob_after_finished=True)
+
+    def _setup_output(self):
+        def open_file():
+            if self.file_handle is None:
+                file_name = self.file_name
+                if not file_name:
+                    file_name = "_"
+                if os.path.exists(os.path.join(self.download_directory, file_name)):
+                    ext_num = 1
+                    while os.path.exists(os.path.join(self.download_directory,
+                                                      file_name + "_" + str(ext_num))):
+                        ext_num += 1
+                    file_name = file_name + "_" + str(ext_num)
+                self.file_handle = open(os.path.join(self.download_directory, file_name), 'wb')
+        return threads.deferToThread(open_file)
+
+    def _close_output(self):
+        self.file_handle, file_handle = None, self.file_handle
+
+        def close_file():
+            if file_handle is not None:
+                name = file_handle.name
+                file_handle.close()
+                if self.completed is False:
+                    os.remove(name)
+
+        return threads.deferToThread(close_file)
+
+    def _get_write_func(self):
+        def write_func(data):
+            if self.stopped is False and self.file_handle is not None:
+                self.file_handle.write(data)
+        return write_func
+
+    def _delete_from_info_manager(self):
+        return self.stream_info_manager.delete_stream(self.stream_hash)
+
+
+class LBRYFileSaverFactory(LBRYFileDownloaderFactory):
+    def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 wallet, download_directory):
+        LBRYFileDownloaderFactory.__init__(self, peer_finder, rate_limiter, blob_manager,
+                                           stream_info_manager, wallet)
+        self.download_directory = download_directory
+
+    def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
+        return LBRYFileSaver(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
+                             self.stream_info_manager, payment_rate_manager, self.wallet,
+                             self.download_directory, upload_allowed)
+
+    def get_description(self):
+        return "Save"
+
+
+class LBRYFileOpener(LBRYFileDownloader):
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 payment_rate_manager, wallet, upload_allowed):
+        LBRYFileDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
+                                    stream_info_manager, payment_rate_manager, wallet, upload_allowed)
+        self.process = None
+        self.process_log = None
+
+    def stop(self):
+        d = LBRYFileDownloader.stop(self)
+        d.addCallback(lambda _: self._delete_from_info_manager())
+        return d
+
+    def _get_progress_manager(self, download_manager):
+        return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager,
+                                         delete_blob_after_finished=True)
+
+    def _setup_output(self):
+        def start_process():
+            if os.name == "nt":
+                paths = [r'C:\Program Files\VideoLAN\VLC\vlc.exe',
+                         r'C:\Program Files (x86)\VideoLAN\VLC\vlc.exe']
+                for p in paths:
+                    if os.path.exists(p):
+                        vlc_path = p
+                        break
+                else:
+                    raise ValueError("You must install VLC media player to stream files")
+            else:
+                vlc_path = 'vlc'
+            self.process_log = open("vlc.out", 'a')
+            try:
+                self.process = subprocess.Popen([vlc_path, '-'], stdin=subprocess.PIPE,
+                                                stdout=self.process_log, stderr=self.process_log)
+            except OSError:
+                raise ValueError("VLC media player could not be opened")
+
+        d = threads.deferToThread(start_process)
+        return d
+
+    def _close_output(self):
+        if self.process is not None:
+            self.process.stdin.close()
+        self.process = None
+        return defer.succeed(True)
+
+    def _get_write_func(self):
+        def write_func(data):
+            if self.stopped is False and self.process is not None:
+                try:
+                    self.process.stdin.write(data)
+                except IOError:
+                    reactor.callLater(0, self.stop)
+        return write_func
+
+    def _delete_from_info_manager(self):
+        return self.stream_info_manager.delete_stream(self.stream_hash)
+
+
+class LBRYFileOpenerFactory(LBRYFileDownloaderFactory):
+    def _make_downloader(self, stream_hash, payment_rate_manager, stream_info, upload_allowed):
+        return LBRYFileOpener(stream_hash, self.peer_finder, self.rate_limiter, self.blob_manager,
+                              self.stream_info_manager, payment_rate_manager, self.wallet, upload_allowed)
+
+    def get_description(self):
+        return "Stream"

lbrynet/lbryfile/client/LBRYFileMetadataHandler.py

@@ -0,0 +1,36 @@
+import logging
+from zope.interface import implements
+from lbrynet.cryptstream.CryptBlob import CryptBlobInfo
+from lbrynet.interfaces import IMetadataHandler
+
+
+class LBRYFileMetadataHandler(object):
+    implements(IMetadataHandler)
+
+    def __init__(self, stream_hash, stream_info_manager, download_manager):
+        self.stream_hash = stream_hash
+        self.stream_info_manager = stream_info_manager
+        self.download_manager = download_manager
+        self._final_blob_num = None
+
+    ######### IMetadataHandler #########
+
+    def get_initial_blobs(self):
+        d = self.stream_info_manager.get_blobs_for_stream(self.stream_hash)
+        d.addCallback(self._format_initial_blobs_for_download_manager)
+        return d
+
+    def final_blob_num(self):
+        return self._final_blob_num
+
+    ######### internal calls #########
+
+    def _format_initial_blobs_for_download_manager(self, blob_infos):
+        infos = []
+        for blob_hash, blob_num, iv, length in blob_infos:
+            if blob_hash is not None:
+                infos.append(CryptBlobInfo(blob_hash, blob_num, length, iv))
+            else:
+                logging.debug("Setting _final_blob_num to %s", str(blob_num - 1))
+                self._final_blob_num = blob_num - 1
+        return infos

lbrynet/lbryfilemanager/LBRYFileCreator.py

@@ -0,0 +1,159 @@
+"""
+Utilities for turning plain files into LBRY Files.
+"""
+
+import binascii
+import logging
+import os
+from lbrynet.core.StreamDescriptor import PlainStreamDescriptorWriter
+from lbrynet.cryptstream.CryptStreamCreator import CryptStreamCreator
+from lbrynet import conf
+from lbrynet.lbryfile.StreamDescriptor import get_sd_info
+from lbrynet.core.cryptoutils import get_lbry_hash_obj
+from twisted.protocols.basic import FileSender
+from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
+
+
+class LBRYFileStreamCreator(CryptStreamCreator):
+    """
+    A CryptStreamCreator which adds itself and its additional metadata to an LBRYFileManager
+    """
+    def __init__(self, blob_manager, lbry_file_manager, name=None,
+                 key=None, iv_generator=None, suggested_file_name=None):
+        CryptStreamCreator.__init__(self, blob_manager, name, key, iv_generator)
+        self.lbry_file_manager = lbry_file_manager
+        if suggested_file_name is None:
+            self.suggested_file_name = name
+        else:
+            self.suggested_file_name = suggested_file_name
+        self.stream_hash = None
+        self.blob_infos = []
+
+    def _blob_finished(self, blob_info):
+        logging.debug("length: %s", str(blob_info.length))
+        self.blob_infos.append(blob_info)
+
+    def _save_lbry_file_info(self):
+        stream_info_manager = self.lbry_file_manager.stream_info_manager
+        d = stream_info_manager.save_stream(self.stream_hash, binascii.hexlify(self.name),
+                                            binascii.hexlify(self.key),
+                                            binascii.hexlify(self.suggested_file_name),
+                                            self.blob_infos)
+        return d
+
+    def setup(self):
+        d = CryptStreamCreator.setup(self)
+        d.addCallback(lambda _: self.stream_hash)
+
+        return d
+
+    def _get_blobs_hashsum(self):
+        blobs_hashsum = get_lbry_hash_obj()
+        for blob_info in sorted(self.blob_infos, key=lambda b_i: b_i.blob_num):
+            length = blob_info.length
+            if length != 0:
+                blob_hash = blob_info.blob_hash
+            else:
+                blob_hash = None
+            blob_num = blob_info.blob_num
+            iv = blob_info.iv
+            blob_hashsum = get_lbry_hash_obj()
+            if length != 0:
+                blob_hashsum.update(blob_hash)
+            blob_hashsum.update(str(blob_num))
+            blob_hashsum.update(iv)
+            blob_hashsum.update(str(length))
+            blobs_hashsum.update(blob_hashsum.digest())
+        return blobs_hashsum.digest()
+
+    def _make_stream_hash(self):
+        hashsum = get_lbry_hash_obj()
+        hashsum.update(binascii.hexlify(self.name))
+        hashsum.update(binascii.hexlify(self.key))
+        hashsum.update(binascii.hexlify(self.suggested_file_name))
+        hashsum.update(self._get_blobs_hashsum())
+        self.stream_hash = hashsum.hexdigest()
+
+    def _finished(self):
+        self._make_stream_hash()
+        d = self._save_lbry_file_info()
+        d.addCallback(lambda _: self.lbry_file_manager.change_lbry_file_status(
+            self.stream_hash, ManagedLBRYFileDownloader.STATUS_FINISHED
+        ))
+        return d
+
+
+def create_lbry_file(session, lbry_file_manager, file_name, file_handle, key=None,
+                     iv_generator=None, suggested_file_name=None):
+    """
+    Turn a plain file into an LBRY File.
+
+    An LBRY File is a collection of encrypted blobs of data and the metadata that binds them
+    together which, when decrypted and put back together according to the metadata, results
+    in the original file.
+
+    The stream parameters that aren't specified are generated, the file is read and broken
+    into chunks and encrypted, and then a stream descriptor file with the stream parameters
+    and other metadata is written to disk.
+
+    @param session: An LBRYSession object.
+    @type session: LBRYSession
+
+    @param lbry_file_manager: The LBRYFileManager object this LBRY File will be added to.
+    @type lbry_file_manager: LBRYFileManager
+
+    @param file_name: The path to the plain file.
+    @type file_name: string
+
+    @param file_handle: The file-like object to read
+    @type file_handle: any file-like object which can be read by twisted.protocols.basic.FileSender
+
+    @param secret_pass_phrase: A string that will be used to generate the public key. If None, a
+        random string will be used.
+    @type secret_pass_phrase: string
+
+    @param key: the raw AES key which will be used to encrypt the blobs. If None, a random key will
+        be generated.
+    @type key: string
+
+    @param iv_generator: a generator which yields initialization vectors for the blobs. Will be called
+        once for each blob.
+    @type iv_generator: a generator function which yields strings
+
+    @param suggested_file_name: what the file should be called when the LBRY File is saved to disk.
+    @type suggested_file_name: string
+
+    @return: a Deferred which fires with the stream_hash of the LBRY File
+    @rtype: Deferred which fires with hex-encoded string
+    """
+
+    def stop_file(creator):
+        logging.debug("the file sender has triggered its deferred. stopping the stream writer")
+        return creator.stop()
+
+    def make_stream_desc_file(stream_hash):
+        logging.debug("creating the stream descriptor file")
+        descriptor_writer = PlainStreamDescriptorWriter(file_name + conf.CRYPTSD_FILE_EXTENSION)
+
+        d = get_sd_info(lbry_file_manager.stream_info_manager, stream_hash, True)
+
+        d.addCallback(descriptor_writer.create_descriptor)
+
+        return d
+
+    base_file_name = os.path.basename(file_name)
+
+    lbry_file_creator = LBRYFileStreamCreator(session.blob_manager, lbry_file_manager, base_file_name,
+                                              key, iv_generator, suggested_file_name)
+
+    def start_stream():
+        file_sender = FileSender()
+        d = file_sender.beginFileTransfer(file_handle, lbry_file_creator)
+        d.addCallback(lambda _: stop_file(lbry_file_creator))
+        d.addCallback(lambda _: make_stream_desc_file(lbry_file_creator.stream_hash))
+        d.addCallback(lambda _: lbry_file_creator.stream_hash)
+        return d
+
+    d = lbry_file_creator.setup()
+    d.addCallback(lambda _: start_stream())
+    return d

lbrynet/lbryfilemanager/LBRYFileDownloader.py

@@ -0,0 +1,149 @@
+"""
+Download LBRY Files from LBRYnet and save them to disk.
+"""
+
+from lbrynet.core.DownloadOption import DownloadOption
+from zope.interface import implements
+from lbrynet.core.client.StreamProgressManager import FullStreamProgressManager
+from lbrynet.lbryfile.client.LBRYFileDownloader import LBRYFileSaver, LBRYFileDownloader
+from lbrynet.lbryfilemanager.LBRYFileStatusReport import LBRYFileStatusReport
+from lbrynet.interfaces import IStreamDownloaderFactory
+from lbrynet.lbryfile.StreamDescriptor import save_sd_info
+from twisted.internet import defer
+
+
+class ManagedLBRYFileDownloader(LBRYFileSaver):
+
+    STATUS_RUNNING = "running"
+    STATUS_STOPPED = "stopped"
+    STATUS_FINISHED = "finished"
+
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 lbry_file_manager, payment_rate_manager, wallet, download_directory, upload_allowed,
+                 file_name=None):
+        LBRYFileSaver.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
+                               stream_info_manager, payment_rate_manager, wallet, download_directory,
+                               upload_allowed)
+        self.lbry_file_manager = lbry_file_manager
+        self.file_name = file_name
+        self.file_handle = None
+        self.saving_status = False
+
+    def restore(self):
+        d = self.lbry_file_manager.get_lbry_file_status(self.stream_hash)
+
+        def restore_status(status):
+            if status == ManagedLBRYFileDownloader.STATUS_RUNNING:
+                return self.start()
+            elif status == ManagedLBRYFileDownloader.STATUS_STOPPED:
+                return defer.succeed(False)
+            elif status == ManagedLBRYFileDownloader.STATUS_FINISHED:
+                self.completed = True
+                return defer.succeed(True)
+
+        d.addCallback(restore_status)
+        return d
+
+    def stop(self, change_status=True):
+
+        def set_saving_status_done():
+            self.saving_status = False
+
+        d = LBRYFileDownloader.stop(self)  # LBRYFileSaver deletes metadata when it's stopped. We don't want that here.
+        if change_status is True:
+            self.saving_status = True
+            d.addCallback(lambda _: self._save_status())
+            d.addCallback(lambda _: set_saving_status_done())
+        return d
+
+    def status(self):
+        def find_completed_blobhashes(blobs):
+            blobhashes = [b[0] for b in blobs if b[0] is not None]
+
+            def get_num_completed(completed_blobs):
+                return len(completed_blobs), len(blobhashes)
+
+            inner_d = self.blob_manager.completed_blobs(blobhashes)
+            inner_d.addCallback(get_num_completed)
+            return inner_d
+
+        def make_full_status(progress):
+            num_completed = progress[0]
+            num_known = progress[1]
+            if self.completed is True:
+                s = "completed"
+            elif self.stopped is True:
+                s = "stopped"
+            else:
+                s = "running"
+            status = LBRYFileStatusReport(self.file_name, num_completed, num_known, s)
+            return status
+
+        d = self.stream_info_manager.get_blobs_for_stream(self.stream_hash)
+        d.addCallback(find_completed_blobhashes)
+        d.addCallback(make_full_status)
+        return d
+
+    def _start(self):
+
+        d = LBRYFileSaver._start(self)
+
+        d.addCallback(lambda _: self._save_status())
+
+        return d
+
+    def _get_finished_deferred_callback_value(self):
+        if self.completed is True:
+            return "Download successful"
+        else:
+            return "Download stopped"
+
+    def _save_status(self):
+        if self.completed is True:
+            s = ManagedLBRYFileDownloader.STATUS_FINISHED
+        elif self.stopped is True:
+            s = ManagedLBRYFileDownloader.STATUS_STOPPED
+        else:
+            s = ManagedLBRYFileDownloader.STATUS_RUNNING
+        return self.lbry_file_manager.change_lbry_file_status(self.stream_hash, s)
+
+    def _get_progress_manager(self, download_manager):
+        return FullStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager)
+
+
+class ManagedLBRYFileDownloaderFactory(object):
+    implements(IStreamDownloaderFactory)
+
+    def __init__(self, lbry_file_manager):
+        self.lbry_file_manager = lbry_file_manager
+
+    def get_downloader_options(self, sd_validator, payment_rate_manager):
+        options = [
+            DownloadOption(
+                [float, None],
+                "rate which will be paid for data (None means use application default)",
+                "data payment rate",
+                None
+            ),
+            DownloadOption(
+                [bool],
+                "allow reuploading data downloaded for this file",
+                "allow upload",
+                True
+            ),
+        ]
+        return options
+
+    def make_downloader(self, sd_validator, options, payment_rate_manager):
+        data_rate = options[0]
+        upload_allowed = options[1]
+
+        d = save_sd_info(self.lbry_file_manager.stream_info_manager, sd_validator.raw_info)
+        d.addCallback(lambda stream_hash: self.lbry_file_manager.add_lbry_file(stream_hash,
+                                                                               payment_rate_manager,
+                                                                               data_rate,
+                                                                               upload_allowed))
+        return d
+
+    def get_description(self):
+        return "Save the file to disk"

lbrynet/lbryfilemanager/LBRYFileManager.py

@@ -0,0 +1,255 @@
+"""
+Keep track of which LBRY Files are downloading and store their LBRY File specific metadata
+"""
+
+import logging
+import json
+
+import leveldb
+
+from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamDescriptorValidator
+import os
+from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloader
+from lbrynet.lbryfilemanager.LBRYFileDownloader import ManagedLBRYFileDownloaderFactory
+from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType
+from lbrynet.core.PaymentRateManager import PaymentRateManager
+from twisted.internet import threads, defer, task, reactor
+from twisted.python.failure import Failure
+from lbrynet.cryptstream.client.CryptStreamDownloader import AlreadyStoppedError, CurrentlyStoppingError
+
+
+class LBRYFileManager(object):
+    """
+    Keeps track of currently opened LBRY Files, their options, and their LBRY File specific metadata.
+    """
+    SETTING = "s"
+    LBRYFILE_STATUS = "t"
+    LBRYFILE_OPTIONS = "o"
+
+    def __init__(self, session, stream_info_manager, sd_identifier):
+        self.session = session
+        self.stream_info_manager = stream_info_manager
+        self.sd_identifier = sd_identifier
+        self.lbry_files = []
+        self.db = None
+        self.download_directory = os.getcwd()
+
+    def setup(self):
+        d = threads.deferToThread(self._open_db)
+        d.addCallback(lambda _: self._add_to_sd_identifier())
+        d.addCallback(lambda _: self._start_lbry_files())
+        return d
+
+    def get_all_lbry_file_stream_hashes_and_options(self):
+        d = threads.deferToThread(self._get_all_lbry_file_stream_hashes)
+
+        def get_options(stream_hashes):
+            ds = []
+
+            def get_options_for_stream_hash(stream_hash):
+                d = self.get_lbry_file_options(stream_hash)
+                d.addCallback(lambda options: (stream_hash, options))
+                return d
+
+            for stream_hash in stream_hashes:
+                ds.append(get_options_for_stream_hash(stream_hash))
+            dl = defer.DeferredList(ds)
+            dl.addCallback(lambda results: [r[1] for r in results if r[0]])
+            return dl
+
+        d.addCallback(get_options)
+        return d
+
+    def get_lbry_file_status(self, stream_hash):
+        return threads.deferToThread(self._get_lbry_file_status, stream_hash)
+
+    def save_lbry_file_options(self, stream_hash, blob_data_rate):
+        return threads.deferToThread(self._save_lbry_file_options, stream_hash, blob_data_rate)
+
+    def get_lbry_file_options(self, stream_hash):
+        return threads.deferToThread(self._get_lbry_file_options, stream_hash)
+
+    def delete_lbry_file_options(self, stream_hash):
+        return threads.deferToThread(self._delete_lbry_file_options, stream_hash)
+
+    def set_lbry_file_data_payment_rate(self, stream_hash, new_rate):
+        return threads.deferToThread(self._set_lbry_file_payment_rate, stream_hash, new_rate)
+
+    def change_lbry_file_status(self, stream_hash, status):
+        logging.debug("Changing status of %s to %s", stream_hash, status)
+        return threads.deferToThread(self._change_file_status, stream_hash, status)
+
+    def delete_lbry_file_status(self, stream_hash):
+        return threads.deferToThread(self._delete_lbry_file_status, stream_hash)
+
+    def get_lbry_file_status_reports(self):
+        ds = []
+
+        for lbry_file in self.lbry_files:
+            ds.append(lbry_file.status())
+
+        dl = defer.DeferredList(ds)
+
+        def filter_failures(status_reports):
+            return [status_report for success, status_report in status_reports if success is True]
+
+        dl.addCallback(filter_failures)
+        return dl
+
+    def _add_to_sd_identifier(self):
+        downloader_factory = ManagedLBRYFileDownloaderFactory(self)
+        self.sd_identifier.add_stream_info_validator(LBRYFileStreamType, LBRYFileStreamDescriptorValidator)
+        self.sd_identifier.add_stream_downloader_factory(LBRYFileStreamType, downloader_factory)
+
+    def _start_lbry_files(self):
+
+        def set_options_and_restore(stream_hash, options):
+            payment_rate_manager = PaymentRateManager(self.session.base_payment_rate_manager)
+            d = self.add_lbry_file(stream_hash, payment_rate_manager, blob_data_rate=options[0])
+            d.addCallback(lambda downloader: downloader.restore())
+            return d
+
+        def log_error(err):
+            logging.error("An error occurred while starting a lbry file: %s", err.getErrorMessage())
+
+        def start_lbry_files(stream_hashes_and_options):
+            for stream_hash, options in stream_hashes_and_options:
+                d = set_options_and_restore(stream_hash, options)
+                d.addErrback(log_error)
+            return True
+
+        d = self.get_all_lbry_file_stream_hashes_and_options()
+        d.addCallback(start_lbry_files)
+        return d
+
+    def add_lbry_file(self, stream_hash, payment_rate_manager, blob_data_rate=None, upload_allowed=True):
+        payment_rate_manager.min_blob_data_payment_rate = blob_data_rate
+        lbry_file_downloader = ManagedLBRYFileDownloader(stream_hash, self.session.peer_finder,
+                                                         self.session.rate_limiter, self.session.blob_manager,
+                                                         self.stream_info_manager, self,
+                                                         payment_rate_manager, self.session.wallet,
+                                                         self.download_directory,
+                                                         upload_allowed)
+        self.lbry_files.append(lbry_file_downloader)
+        d = self.save_lbry_file_options(stream_hash, blob_data_rate)
+        d.addCallback(lambda _: lbry_file_downloader.set_stream_info())
+        d.addCallback(lambda _: lbry_file_downloader)
+        return d
+
+    def delete_lbry_file(self, stream_hash):
+        for l in self.lbry_files:
+            if l.stream_hash == stream_hash:
+                lbry_file = l
+                break
+        else:
+            return defer.fail(Failure(ValueError("Could not find an LBRY file with the given stream hash, " +
+                                                 stream_hash)))
+
+        def wait_for_finished(count=2):
+            if count <= 0 or lbry_file.saving_status is False:
+                return True
+            else:
+                return task.deferLater(reactor, 1, wait_for_finished, count=count - 1)
+
+        def ignore_stopped(err):
+            err.trap(AlreadyStoppedError, CurrentlyStoppingError)
+            return wait_for_finished()
+
+        d = lbry_file.stop()
+        d.addErrback(ignore_stopped)
+
+        def remove_from_list():
+            self.lbry_files.remove(lbry_file)
+
+        d.addCallback(lambda _: remove_from_list())
+        d.addCallback(lambda _: self.delete_lbry_file_options(stream_hash))
+        d.addCallback(lambda _: self.delete_lbry_file_status(stream_hash))
+        return d
+
+    def toggle_lbry_file_running(self, stream_hash):
+        """Toggle whether a stream reader is currently running"""
+        for l in self.lbry_files:
+            if l.stream_hash == stream_hash:
+                return l.toggle_running()
+        else:
+            return defer.fail(Failure(ValueError("Could not find an LBRY file with the given stream hash, " +
+                                                 stream_hash)))
+
+    def get_stream_hash_from_name(self, lbry_file_name):
+        for l in self.lbry_files:
+            if l.file_name == lbry_file_name:
+                return l.stream_hash
+        return None
+
+    def stop(self):
+        ds = []
+
+        def wait_for_finished(lbry_file, count=2):
+            if count <= 0 or lbry_file.saving_status is False:
+                return True
+            else:
+                return task.deferLater(reactor, 1, wait_for_finished, lbry_file, count=count - 1)
+
+        def ignore_stopped(err, lbry_file):
+            err.trap(AlreadyStoppedError, CurrentlyStoppingError)
+            return wait_for_finished(lbry_file)
+
+        for lbry_file in self.lbry_files:
+            d = lbry_file.stop(change_status=False)
+            d.addErrback(ignore_stopped, lbry_file)
+            ds.append(d)
+        dl = defer.DeferredList(ds)
+
+        def close_db():
+            self.db = None
+
+        dl.addCallback(lambda _: close_db())
+        return dl
+
+    ######### database calls #########
+
+    def _open_db(self):
+        self.db = leveldb.LevelDB(os.path.join(self.session.db_dir, "lbryfiles.db"))
+
+    def _save_payment_rate(self, rate_type, rate):
+        if rate is not None:
+            self.db.Put(json.dumps((self.SETTING, rate_type)), json.dumps(rate), sync=True)
+        else:
+            self.db.Delete(json.dumps((self.SETTING, rate_type)), sync=True)
+
+    def _save_lbry_file_options(self, stream_hash, blob_data_rate):
+        self.db.Put(json.dumps((self.LBRYFILE_OPTIONS, stream_hash)), json.dumps((blob_data_rate,)),
+                    sync=True)
+
+    def _get_lbry_file_options(self, stream_hash):
+        try:
+            return json.loads(self.db.Get(json.dumps((self.LBRYFILE_OPTIONS, stream_hash))))
+        except KeyError:
+            return None, None
+
+    def _delete_lbry_file_options(self, stream_hash):
+        self.db.Delete(json.dumps((self.LBRYFILE_OPTIONS, stream_hash)), sync=True)
+
+    def _set_lbry_file_payment_rate(self, stream_hash, new_rate):
+
+        self.db.Put(json.dumps((self.LBRYFILE_OPTIONS, stream_hash)), json.dumps((new_rate, )), sync=True)
+
+    def _get_all_lbry_file_stream_hashes(self):
+        hashes = []
+        for k, v in self.db.RangeIter():
+            key_type, stream_hash = json.loads(k)
+            if key_type == self.LBRYFILE_STATUS:
+                hashes.append(stream_hash)
+        return hashes
+
+    def _change_file_status(self, stream_hash, new_status):
+        self.db.Put(json.dumps((self.LBRYFILE_STATUS, stream_hash)), new_status, sync=True)
+
+    def _get_lbry_file_status(self, stream_hash):
+        try:
+            return self.db.Get(json.dumps((self.LBRYFILE_STATUS, stream_hash)))
+        except KeyError:
+            return ManagedLBRYFileDownloader.STATUS_STOPPED
+
+    def _delete_lbry_file_status(self, stream_hash):
+        self.db.Delete(json.dumps((self.LBRYFILE_STATUS, stream_hash)), sync=True)

lbrynet/lbryfilemanager/LBRYFileStatusReport.py

@@ -0,0 +1,6 @@
+class LBRYFileStatusReport(object):
+    def __init__(self, name, num_completed, num_known, running_status):
+        self.name = name
+        self.num_completed = num_completed
+        self.num_known = num_known
+        self.running_status = running_status

lbrynet/lbryfilemanager/__init__.py

@@ -0,0 +1,7 @@
+"""
+Classes and functions used to create and download LBRY Files.
+
+LBRY Files are Crypt Streams created from any regular file. The whole file is read
+at the time that the LBRY File is created, so all constituent blobs are known and
+included in the stream descriptor file.
+"""

lbrynet/lbrylive/LBRYStdinUploader.py

@@ -0,0 +1,117 @@
+import logging
+import sys
+from lbrynet.lbrylive.LiveStreamCreator import StdOutLiveStreamCreator
+from lbrynet.core.BlobManager import TempBlobManager
+from lbrynet.core.Session import LBRYSession
+from lbrynet.core.server.BlobAvailabilityHandler import BlobAvailabilityHandlerFactory
+from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
+from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
+from lbrynet.lbrylive.PaymentRateManager import BaseLiveStreamPaymentRateManager
+from lbrynet.lbrylive.LiveStreamMetadataManager import DBLiveStreamMetadataManager
+from lbrynet.lbrylive.server.LiveBlobInfoQueryHandler import CryptBlobInfoQueryHandlerFactory
+from lbrynet.dht.node import Node
+from twisted.internet import defer, task
+
+
+class LBRYStdinUploader():
+    """This class reads from standard in, creates a stream, and makes it available on the network."""
+    def __init__(self, peer_port, dht_node_port, known_dht_nodes):
+        """
+        @param peer_port: the network port on which to listen for peers
+
+        @param dht_node_port: the network port on which to listen for nodes in the DHT
+
+        @param known_dht_nodes: a list of (ip_address, dht_port) which will be used to join the DHT network
+        """
+        self.peer_port = peer_port
+        self.lbry_server_port = None
+        self.session = LBRYSession(blob_manager_class=TempBlobManager,
+                                   stream_info_manager_class=DBLiveStreamMetadataManager,
+                                   dht_node_class=Node, dht_node_port=dht_node_port,
+                                   known_dht_nodes=known_dht_nodes, peer_port=self.peer_port,
+                                   use_upnp=False)
+        self.payment_rate_manager = BaseLiveStreamPaymentRateManager()
+
+    def start(self):
+        """Initialize the session and start listening on the peer port"""
+        d = self.session.setup()
+        d.addCallback(lambda _: self._start())
+
+        return d
+
+    def _start(self):
+        self._start_server()
+        return True
+
+    def _start_server(self):
+        query_handler_factories = [
+            CryptBlobInfoQueryHandlerFactory(self.stream_info_manager, self.session.wallet,
+                                             self.payment_rate_manager),
+            BlobAvailabilityHandlerFactory(self.session.blob_manager),
+            BlobRequestHandlerFactory(self.session.blob_manager, self.session.wallet,
+                                      self.payment_rate_manager),
+            self.session.wallet.get_wallet_info_query_handler_factory()
+        ]
+
+        self.server_factory = ServerProtocolFactory(self.session.rate_limiter,
+                                                    query_handler_factories,
+                                                    self.session.peer_manager)
+        from twisted.internet import reactor
+        self.lbry_server_port = reactor.listenTCP(self.peer_port, self.server_factory)
+
+    def start_live_stream(self, stream_name):
+        """Create the stream and start reading from stdin
+
+        @param stream_name: a string, the suggested name of this stream
+        """
+        stream_creator_helper = StdOutLiveStreamCreator(stream_name, self.session.blob_manager,
+                                                        self.stream_info_manager)
+        d = stream_creator_helper.create_and_publish_stream_descriptor()
+
+        def print_sd_hash(sd_hash):
+            print "Stream descriptor hash:", sd_hash
+
+        d.addCallback(print_sd_hash)
+        d.addCallback(lambda _: stream_creator_helper.start_streaming())
+        return d
+
+    def shut_down(self):
+        """End the session and stop listening on the server port"""
+        d = self.session.shut_down()
+        d.addCallback(lambda _: self._shut_down())
+        return d
+
+    def _shut_down(self):
+        if self.lbry_server_port is not None:
+            d = defer.maybeDeferred(self.lbry_server_port.stopListening)
+        else:
+            d = defer.succeed(True)
+        return d
+
+
+def launch_stdin_uploader():
+
+    from twisted.internet import reactor
+
+    logging.basicConfig(level=logging.WARNING, filename="ul.log")
+    if len(sys.argv) == 4:
+        uploader = LBRYStdinUploader(int(sys.argv[2]), int(sys.argv[3]), [])
+    elif len(sys.argv) == 6:
+        uploader = LBRYStdinUploader(int(sys.argv[2]), int(sys.argv[3]), [(sys.argv[4], int(sys.argv[5]))])
+    else:
+        print "Usage: lbrynet-stdin-uploader <stream_name> <peer_port> <dht_node_port>" \
+              " [<dht_bootstrap_host> <dht_bootstrap port>]"
+        sys.exit(1)
+
+    def start_stdin_uploader():
+        return uploader.start_live_stream(sys.argv[1])
+
+    def shut_down():
+        logging.debug("Telling the reactor to stop in 60 seconds")
+        reactor.callLater(60, reactor.stop)
+
+    d = task.deferLater(reactor, 0, uploader.start)
+    d.addCallback(lambda _: start_stdin_uploader())
+    d.addCallback(lambda _: shut_down())
+    reactor.addSystemEventTrigger('before', 'shutdown', uploader.shut_down)
+    reactor.run()

lbrynet/lbrylive/LBRYStdoutDownloader.py

@@ -0,0 +1,96 @@
+import logging
+import sys
+
+from lbrynet.lbrynet_console.plugins.LBRYLive.LBRYLiveStreamDownloader import LBRYLiveStreamDownloader
+from lbrynet.core.BlobManager import TempBlobManager
+from lbrynet.core.Session import LBRYSession
+from lbrynet.core.client.StandaloneBlobDownloader import StandaloneBlobDownloader
+from lbrynet.core.StreamDescriptor import BlobStreamDescriptorReader
+from lbrynet.lbrylive.PaymentRateManager import BaseLiveStreamPaymentRateManager
+from lbrynet.lbrylive.LiveStreamMetadataManager import DBLiveStreamMetadataManager
+from lbrynet.lbrylive.StreamDescriptor import save_sd_info
+from lbrynet.dht.node import Node
+from twisted.internet import task
+
+
+class LBRYStdoutDownloader():
+    """This class downloads a live stream from the network and outputs it to standard out."""
+    def __init__(self, dht_node_port, known_dht_nodes):
+        """
+        @param dht_node_port: the network port on which to listen for DHT node requests
+
+        @param known_dht_nodes: a list of (ip_address, dht_port) which will be used to join the DHT network
+
+        """
+        self.session = LBRYSession(blob_manager_class=TempBlobManager,
+                                   stream_info_manager_class=DBLiveStreamMetadataManager,
+                                   dht_node_class=Node, dht_node_port=dht_node_port, known_dht_nodes=known_dht_nodes,
+                                   use_upnp=False)
+        self.payment_rate_manager = BaseLiveStreamPaymentRateManager()
+
+    def start(self):
+        """Initialize the session"""
+        d = self.session.setup()
+        return d
+
+    def read_sd_file(self, sd_blob):
+        reader = BlobStreamDescriptorReader(sd_blob)
+        return save_sd_info(self.stream_info_manager, reader, ignore_duplicate=True)
+
+    def download_sd_file_from_hash(self, sd_hash):
+        downloader = StandaloneBlobDownloader(sd_hash, self.session.blob_manager,
+                                              self.session.peer_finder, self.session.rate_limiter,
+                                              self.session.wallet)
+        d = downloader.download()
+        return d
+
+    def start_download(self, sd_hash):
+        """Start downloading the stream from the network and outputting it to standard out"""
+        d = self.download_sd_file_from_hash(sd_hash)
+        d.addCallbacks(self.read_sd_file)
+
+        def start_stream(stream_hash):
+            consumer = LBRYLiveStreamDownloader(stream_hash, self.session.peer_finder,
+                                                self.session.rate_limiter, self.session.blob_manager,
+                                                self.stream_info_manager, self.payment_rate_manager,
+                                                self.session.wallet)
+            return consumer.start()
+
+        d.addCallback(start_stream)
+        return d
+
+    def shut_down(self):
+        """End the session"""
+        d = self.session.shut_down()
+        return d
+
+
+def launch_stdout_downloader():
+
+    from twisted.internet import reactor
+
+    logging.basicConfig(level=logging.WARNING, filename="dl.log")
+    if len(sys.argv) == 3:
+        downloader = LBRYStdoutDownloader(int(sys.argv[2]), [])
+    elif len(sys.argv) == 5:
+        downloader = LBRYStdoutDownloader(int(sys.argv[2]), [(sys.argv[3], int(sys.argv[4]))])
+    else:
+        print "Usage: lbrynet-stdout-downloader <sd_hash> <peer_port> <dht_node_port>" \
+              " [<dht_bootstrap_host> <dht_bootstrap port>]"
+        sys.exit(1)
+
+    def start_stdout_downloader():
+        return downloader.start_download(sys.argv[1])
+
+    def print_error(err):
+        logging.warning(err.getErrorMessage())
+
+    def shut_down():
+        reactor.stop()
+
+    d = task.deferLater(reactor, 0, downloader.start)
+    d.addCallback(lambda _: start_stdout_downloader())
+    d.addErrback(print_error)
+    d.addCallback(lambda _: shut_down())
+    reactor.addSystemEventTrigger('before', 'shutdown', downloader.shut_down)
+    reactor.run()

lbrynet/lbrylive/LiveBlob.py

@@ -0,0 +1,23 @@
+from lbrynet.cryptstream.CryptBlob import CryptStreamBlobMaker, CryptBlobInfo
+import binascii
+
+
+class LiveBlobInfo(CryptBlobInfo):
+    def __init__(self, blob_hash, blob_num, length, iv, revision, signature):
+        CryptBlobInfo.__init__(self, blob_hash, blob_num, length, iv)
+        self.revision = revision
+        self.signature = signature
+
+
+class LiveStreamBlobMaker(CryptStreamBlobMaker):
+    def __init__(self, key, iv, blob_num, blob):
+        CryptStreamBlobMaker.__init__(self, key, iv, blob_num, blob)
+        # The following is a placeholder for a currently unimplemented feature.
+        # In the future it may be possible for the live stream creator to overwrite a blob
+        # with a newer revision. If that happens, the 0 will be incremented to the
+        # actual revision count
+        self.revision = 0
+
+    def _return_info(self, blob_hash):
+        return LiveBlobInfo(blob_hash, self.blob_num, self.length, binascii.hexlify(self.iv),
+                            self.revision, None)

lbrynet/lbrylive/LiveStreamCreator.py

@@ -0,0 +1,189 @@
+from lbrynet.core.StreamDescriptor import BlobStreamDescriptorWriter
+from lbrynet.lbrylive.StreamDescriptor import get_sd_info, LiveStreamType, LBRYLiveStreamDescriptorValidator
+from lbrynet.cryptstream.CryptStreamCreator import CryptStreamCreator
+from lbrynet.lbrylive.LiveBlob import LiveStreamBlobMaker
+from lbrynet.lbrylive.PaymentRateManager import BaseLiveStreamPaymentRateManager
+from lbrynet.core.cryptoutils import get_lbry_hash_obj, get_pub_key, sign_with_pass_phrase
+from Crypto import Random
+import binascii
+import logging
+from lbrynet.conf import CRYPTSD_FILE_EXTENSION
+from lbrynet.conf import MIN_BLOB_INFO_PAYMENT_RATE
+from lbrynet.lbrylive.client.LiveStreamDownloader import FullLiveStreamDownloaderFactory
+from twisted.internet import interfaces, defer
+from twisted.protocols.basic import FileSender
+from zope.interface import implements
+
+
+class LiveStreamCreator(CryptStreamCreator):
+    def __init__(self, blob_manager, stream_info_manager, name=None, key=None, iv_generator=None,
+                 delete_after_num=None, secret_pass_phrase=None):
+        CryptStreamCreator.__init__(self, blob_manager, name, key, iv_generator)
+        self.stream_hash = None
+        self.stream_info_manager = stream_info_manager
+        self.delete_after_num = delete_after_num
+        self.secret_pass_phrase = secret_pass_phrase
+        self.file_extension = CRYPTSD_FILE_EXTENSION
+        self.finished_blob_hashes = {}
+
+    def _save_stream(self):
+        d = self.stream_info_manager.save_stream(self.stream_hash, get_pub_key(self.secret_pass_phrase),
+                                                 binascii.hexlify(self.name), binascii.hexlify(self.key),
+                                                 [])
+        return d
+
+    def _blob_finished(self, blob_info):
+        logging.debug("In blob_finished")
+        logging.debug("length: %s", str(blob_info.length))
+        sig_hash = get_lbry_hash_obj()
+        sig_hash.update(self.stream_hash)
+        if blob_info.length != 0:
+            sig_hash.update(blob_info.blob_hash)
+        sig_hash.update(str(blob_info.blob_num))
+        sig_hash.update(str(blob_info.revision))
+        sig_hash.update(blob_info.iv)
+        sig_hash.update(str(blob_info.length))
+        signature = sign_with_pass_phrase(sig_hash.digest(), self.secret_pass_phrase)
+        blob_info.signature = signature
+        self.finished_blob_hashes[blob_info.blob_num] = blob_info.blob_hash
+        if self.delete_after_num is not None:
+            self._delete_old_blobs(blob_info.blob_num)
+        d = self.stream_info_manager.add_blobs_to_stream(self.stream_hash, [blob_info])
+
+        def log_add_error(err):
+            logging.error("An error occurred adding a blob info to the stream info manager: %s", err.getErrorMessage())
+            return err
+
+        d.addErrback(log_add_error)
+        logging.debug("returning from blob_finished")
+        return d
+
+    def setup(self):
+        """Create the secret pass phrase if it wasn't provided, compute the stream hash,
+        save the stream to the stream info manager, and return the stream hash
+        """
+        if self.secret_pass_phrase is None:
+            self.secret_pass_phrase = Random.new().read(512)
+
+        d = CryptStreamCreator.setup(self)
+
+        def make_stream_hash():
+            hashsum = get_lbry_hash_obj()
+            hashsum.update(binascii.hexlify(self.name))
+            hashsum.update(get_pub_key(self.secret_pass_phrase))
+            hashsum.update(binascii.hexlify(self.key))
+            self.stream_hash = hashsum.hexdigest()
+            return self.stream_hash
+
+        d.addCallback(lambda _: make_stream_hash())
+        d.addCallback(lambda _: self._save_stream())
+        d.addCallback(lambda _: self.stream_hash)
+        return d
+
+    def publish_stream_descriptor(self):
+        descriptor_writer = BlobStreamDescriptorWriter(self.blob_manager)
+        d = get_sd_info(self.stream_info_manager, self.stream_hash, False)
+        d.addCallback(descriptor_writer.create_descriptor)
+        return d
+
+    def _delete_old_blobs(self, newest_blob_num):
+        assert self.delete_after_num is not None, "_delete_old_blobs called with delete_after_num=None"
+        oldest_to_keep = newest_blob_num - self.delete_after_num + 1
+        nums_to_delete = [num for num in self.finished_blob_hashes.iterkeys() if num < oldest_to_keep]
+        for num in nums_to_delete:
+            self.blob_manager.delete_blobs([self.finished_blob_hashes[num]])
+            del self.finished_blob_hashes[num]
+
+    def _get_blob_maker(self, iv, blob_creator):
+        return LiveStreamBlobMaker(self.key, iv, self.blob_count, blob_creator)
+
+
+class StdOutLiveStreamCreator(LiveStreamCreator):
+    def __init__(self, stream_name, blob_manager, stream_info_manager):
+        LiveStreamCreator.__init__(self, blob_manager, stream_info_manager, stream_name,
+                                   delete_after_num=20)
+
+    def start_streaming(self):
+        stdin_producer = StdinStreamProducer(self)
+        d = stdin_producer.begin_producing()
+
+        def stop_stream():
+            d = self.stop()
+            return d
+
+        d.addCallback(lambda _: stop_stream())
+        return d
+
+
+class FileLiveStreamCreator(LiveStreamCreator):
+    def __init__(self, blob_manager, stream_info_manager, file_name, file_handle,
+                 secret_pass_phrase=None, key=None, iv_generator=None, stream_name=None):
+        if stream_name is None:
+            stream_name = file_name
+        LiveStreamCreator.__init__(self, blob_manager, stream_info_manager, stream_name,
+                                   secret_pass_phrase, key, iv_generator)
+        self.file_name = file_name
+        self.file_handle = file_handle
+
+    def start_streaming(self):
+        file_sender = FileSender()
+        d = file_sender.beginFileTransfer(self.file_handle, self)
+
+        def stop_stream():
+            d = self.stop()
+            return d
+
+        d.addCallback(lambda _: stop_stream())
+        return d
+
+
+class StdinStreamProducer(object):
+    """This class reads data from standard in and sends it to a stream creator"""
+
+    implements(interfaces.IPushProducer)
+
+    def __init__(self, consumer):
+        self.consumer = consumer
+        self.reader = None
+        self.finished_deferred = None
+
+    def begin_producing(self):
+
+        self.finished_deferred = defer.Deferred()
+        self.consumer.registerProducer(self, True)
+        #self.reader = process.ProcessReader(reactor, self, 'read', 0)
+        self.resumeProducing()
+        return self.finished_deferred
+
+    def resumeProducing(self):
+        if self.reader is not None:
+            self.reader.resumeProducing()
+
+    def stopProducing(self):
+        if self.reader is not None:
+            self.reader.stopReading()
+        self.consumer.unregisterProducer()
+        self.finished_deferred.callback(True)
+
+    def pauseProducing(self):
+        if self.reader is not None:
+            self.reader.pauseProducing()
+
+    def childDataReceived(self, fd, data):
+        self.consumer.write(data)
+
+    def childConnectionLost(self, fd, reason):
+        self.stopProducing()
+
+
+def add_live_stream_to_sd_identifier(session, stream_info_manager, sd_identifier):
+    downloader_factory = FullLiveStreamDownloaderFactory(session.peer_finder,
+                                                         session.rate_limiter,
+                                                         session.blob_manager,
+                                                         stream_info_manager,
+                                                         session.wallet,
+                                                         BaseLiveStreamPaymentRateManager(
+                                                             MIN_BLOB_INFO_PAYMENT_RATE
+                                                         ))
+    sd_identifier.add_stream_info_validator(LiveStreamType, LBRYLiveStreamDescriptorValidator)
+    sd_identifier.add_stream_downloader_factory(LiveStreamType, downloader_factory)

lbrynet/lbrylive/LiveStreamMetadataManager.py

@@ -0,0 +1,328 @@
+import time
+import logging
+import leveldb
+import json
+import os
+from twisted.internet import threads, defer
+from lbrynet.core.server.DHTHashAnnouncer import DHTHashSupplier
+from lbrynet.core.Error import DuplicateStreamHashError
+
+
+class DBLiveStreamMetadataManager(DHTHashSupplier):
+    """This class stores all stream info in a leveldb database stored in the same directory as the blobfiles"""
+
+    def __init__(self, db_dir, hash_announcer):
+        DHTHashSupplier.__init__(self, hash_announcer)
+        self.db_dir = db_dir
+        self.stream_info_db = None
+        self.stream_blob_db = None
+        self.stream_desc_db = None
+
+    def setup(self):
+        return threads.deferToThread(self._open_db)
+
+    def stop(self):
+        self.stream_info_db = None
+        self.stream_blob_db = None
+        self.stream_desc_db = None
+        return defer.succeed(True)
+
+    def get_all_streams(self):
+        return threads.deferToThread(self._get_all_streams)
+
+    def save_stream(self, stream_hash, pub_key, file_name, key, blobs):
+        next_announce_time = time.time() + self.hash_reannounce_time
+        d = threads.deferToThread(self._store_stream, stream_hash, pub_key, file_name, key,
+                                  next_announce_time=next_announce_time)
+
+        def save_blobs():
+            return self.add_blobs_to_stream(stream_hash, blobs)
+
+        def announce_have_stream():
+            if self.hash_announcer is not None:
+                self.hash_announcer.immediate_announce([stream_hash])
+            return stream_hash
+
+        d.addCallback(lambda _: save_blobs())
+        d.addCallback(lambda _: announce_have_stream())
+        return d
+
+    def get_stream_info(self, stream_hash):
+        return threads.deferToThread(self._get_stream_info, stream_hash)
+
+    def check_if_stream_exists(self, stream_hash):
+        return threads.deferToThread(self._check_if_stream_exists, stream_hash)
+
+    def delete_stream(self, stream_hash):
+        return threads.deferToThread(self._delete_stream, stream_hash)
+
+    def add_blobs_to_stream(self, stream_hash, blobs):
+
+        def add_blobs():
+            self._add_blobs_to_stream(stream_hash, blobs, ignore_duplicate_error=True)
+
+        return threads.deferToThread(add_blobs)
+
+    def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
+        logging.info("Getting blobs for a stream. Count is %s", str(count))
+
+        def get_positions_of_start_and_end():
+            if start_blob is not None:
+                start_num = self._get_blob_num_by_hash(stream_hash, start_blob)
+            else:
+                start_num = None
+            if end_blob is not None:
+                end_num = self._get_blob_num_by_hash(stream_hash, end_blob)
+            else:
+                end_num = None
+            return start_num, end_num
+
+        def get_blob_infos(nums):
+            start_num, end_num = nums
+            return threads.deferToThread(self._get_further_blob_infos, stream_hash, start_num, end_num,
+                                         count, reverse)
+
+        d = threads.deferToThread(get_positions_of_start_and_end)
+        d.addCallback(get_blob_infos)
+        return d
+
+    def get_stream_of_blob(self, blob_hash):
+        return threads.deferToThread(self._get_stream_of_blobhash, blob_hash)
+
+    def save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        return threads.deferToThread(self._save_sd_blob_hash_to_stream, stream_hash, sd_blob_hash)
+
+    def get_sd_blob_hashes_for_stream(self, stream_hash):
+        return threads.deferToThread(self._get_sd_blob_hashes_for_stream, stream_hash)
+
+    def hashes_to_announce(self):
+        next_announce_time = time.time() + self.hash_reannounce_time
+        return threads.deferToThread(self._get_streams_to_announce, next_announce_time)
+
+    ######### database calls #########
+
+    def _open_db(self):
+        self.stream_info_db = leveldb.LevelDB(os.path.join(self.db_dir, "stream_info.db"))
+        self.stream_blob_db = leveldb.LevelDB(os.path.join(self.db_dir, "stream_blob.db"))
+        self.stream_desc_db = leveldb.LevelDB(os.path.join(self.db_dir, "stream_desc.db"))
+
+    def _delete_stream(self, stream_hash):
+        desc_batch = leveldb.WriteBatch()
+        for sd_blob_hash, s_h in self.stream_desc_db.RangeIter():
+            if stream_hash == s_h:
+                desc_batch.Delete(sd_blob_hash)
+        self.stream_desc_db.Write(desc_batch, sync=True)
+
+        blob_batch = leveldb.WriteBatch()
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if stream_hash == s_h:
+                blob_batch.Delete(blob_hash_stream_hash)
+        self.stream_blob_db.Write(blob_batch, sync=True)
+
+        stream_batch = leveldb.WriteBatch()
+        for s_h, stream_info in self.stream_info_db.RangeIter():
+            if stream_hash == s_h:
+                stream_batch.Delete(s_h)
+        self.stream_info_db.Write(stream_batch, sync=True)
+
+    def _store_stream(self, stream_hash, public_key, name, key, next_announce_time=None):
+        try:
+            self.stream_info_db.Get(stream_hash)
+            raise DuplicateStreamHashError("Stream hash %s already exists" % stream_hash)
+        except KeyError:
+            pass
+        self.stream_info_db.Put(stream_hash, json.dumps((public_key, key, name, next_announce_time)), sync=True)
+
+    def _get_all_streams(self):
+        return [stream_hash for stream_hash, stream_info in self.stream_info_db.RangeIter()]
+
+    def _get_stream_info(self, stream_hash):
+        return json.loads(self.stream_info_db.Get(stream_hash))[:3]
+
+    def _check_if_stream_exists(self, stream_hash):
+        try:
+            self.stream_info_db.Get(stream_hash)
+            return True
+        except KeyError:
+            return False
+
+    def _get_streams_to_announce(self, next_announce_time):
+        # TODO: See if the following would be better for handling announce times:
+        # TODO:    Have a separate db for them, and read the whole thing into memory
+        # TODO:    on startup, and then write changes to db when they happen
+        stream_hashes = []
+        batch = leveldb.WriteBatch()
+        current_time = time.time()
+        for stream_hash, stream_info in self.stream_info_db.RangeIter():
+            public_key, key, name, announce_time = json.loads(stream_info)
+            if announce_time < current_time:
+                batch.Put(stream_hash, json.dumps((public_key, key, name, next_announce_time)))
+                stream_hashes.append(stream_hash)
+        self.stream_info_db.Write(batch, sync=True)
+        return stream_hashes
+
+    def _get_blob_num_by_hash(self, stream_hash, blob_hash):
+        blob_hash_stream_hash = json.dumps((blob_hash, stream_hash))
+        return json.loads(self.stream_blob_db.Get(blob_hash_stream_hash))[0]
+
+    def _get_further_blob_infos(self, stream_hash, start_num, end_num, count=None, reverse=False):
+        blob_infos = []
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if stream_hash == s_h:
+                position, revision, iv, length, signature = json.loads(blob_info)
+                if (start_num is None) or (position > start_num):
+                    if (end_num is None) or (position < end_num):
+                        blob_infos.append((b_h, position, revision, iv, length, signature))
+        blob_infos.sort(key=lambda i: i[1], reverse=reverse)
+        if count is not None:
+            blob_infos = blob_infos[:count]
+        return blob_infos
+
+    def _add_blobs_to_stream(self, stream_hash, blob_infos, ignore_duplicate_error=False):
+        batch = leveldb.WriteBatch()
+        for blob_info in blob_infos:
+            blob_hash_stream_hash = json.dumps((blob_info.blob_hash, stream_hash))
+            try:
+                self.stream_blob_db.Get(blob_hash_stream_hash)
+                if ignore_duplicate_error is False:
+                    raise KeyError()  # TODO: change this to DuplicateStreamBlobError?
+                continue
+            except KeyError:
+                pass
+            batch.Put(blob_hash_stream_hash,
+                      json.dumps((blob_info.blob_num,
+                                  blob_info.revision,
+                                  blob_info.iv,
+                                  blob_info.length,
+                                  blob_info.signature)))
+        self.stream_blob_db.Write(batch, sync=True)
+
+    def _get_stream_of_blobhash(self, blob_hash):
+        for blob_hash_stream_hash, blob_info in self.stream_blob_db.RangeIter():
+            b_h, s_h = json.loads(blob_hash_stream_hash)
+            if blob_hash == b_h:
+                return s_h
+        return None
+
+    def _save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        self.stream_desc_db.Put(sd_blob_hash, stream_hash)
+
+    def _get_sd_blob_hashes_for_stream(self, stream_hash):
+        return [sd_blob_hash for sd_blob_hash, s_h in self.stream_desc_db.RangeIter() if stream_hash == s_h]
+
+
+class TempLiveStreamMetadataManager(DHTHashSupplier):
+
+    def __init__(self, hash_announcer):
+        DHTHashSupplier.__init__(self, hash_announcer)
+        self.streams = {}
+        self.stream_blobs = {}
+        self.stream_desc = {}
+
+    def setup(self):
+        return defer.succeed(True)
+
+    def stop(self):
+        return defer.succeed(True)
+
+    def get_all_streams(self):
+        return defer.succeed(self.streams.keys())
+
+    def save_stream(self, stream_hash, pub_key, file_name, key, blobs):
+        next_announce_time = time.time() + self.hash_reannounce_time
+        self.streams[stream_hash] = {'public_key': pub_key, 'stream_name': file_name,
+                                     'key': key, 'next_announce_time': next_announce_time}
+        d = self.add_blobs_to_stream(stream_hash, blobs)
+
+        def announce_have_stream():
+            if self.hash_announcer is not None:
+                self.hash_announcer.immediate_announce([stream_hash])
+            return stream_hash
+
+        d.addCallback(lambda _: announce_have_stream())
+        return d
+
+    def get_stream_info(self, stream_hash):
+        if stream_hash in self.streams:
+            stream_info = self.streams[stream_hash]
+            return defer.succeed([stream_info['public_key'], stream_info['key'], stream_info['stream_name']])
+        return defer.succeed(None)
+
+    def delete_stream(self, stream_hash):
+        if stream_hash in self.streams:
+            del self.streams[stream_hash]
+            for (s_h, b_h) in self.stream_blobs.keys():
+                if s_h == stream_hash:
+                    del self.stream_blobs[(s_h, b_h)]
+        return defer.succeed(True)
+
+    def add_blobs_to_stream(self, stream_hash, blobs):
+        assert stream_hash in self.streams, "Can't add blobs to a stream that isn't known"
+        for blob in blobs:
+            info = {}
+            info['blob_num'] = blob.blob_num
+            info['length'] = blob.length
+            info['iv'] = blob.iv
+            info['revision'] = blob.revision
+            info['signature'] = blob.signature
+            self.stream_blobs[(stream_hash, blob.blob_hash)] = info
+        return defer.succeed(True)
+
+    def get_blobs_for_stream(self, stream_hash, start_blob=None, end_blob=None, count=None, reverse=False):
+
+        if start_blob is not None:
+            start_num = self._get_blob_num_by_hash(stream_hash, start_blob)
+        else:
+            start_num = None
+        if end_blob is not None:
+            end_num = self._get_blob_num_by_hash(stream_hash, end_blob)
+        else:
+            end_num = None
+        return self._get_further_blob_infos(stream_hash, start_num, end_num, count, reverse)
+
+    def get_stream_of_blob(self, blob_hash):
+        for (s_h, b_h) in self.stream_blobs.iterkeys():
+            if b_h == blob_hash:
+                return defer.succeed(s_h)
+        return defer.succeed(None)
+
+    def _get_further_blob_infos(self, stream_hash, start_num, end_num, count=None, reverse=False):
+        blob_infos = []
+        for (s_h, b_h), info in self.stream_blobs.iteritems():
+            if stream_hash == s_h:
+                position = info['blob_num']
+                length = info['length']
+                iv = info['iv']
+                revision = info['revision']
+                signature = info['signature']
+                if (start_num is None) or (position > start_num):
+                    if (end_num is None) or (position < end_num):
+                        blob_infos.append((b_h, position, revision, iv, length, signature))
+        blob_infos.sort(key=lambda i: i[1], reverse=reverse)
+        if count is not None:
+            blob_infos = blob_infos[:count]
+        return defer.succeed(blob_infos)
+
+    def _get_blob_num_by_hash(self, stream_hash, blob_hash):
+        if (stream_hash, blob_hash) in self.stream_blobs:
+            return self.stream_blobs[(stream_hash, blob_hash)]['blob_num']
+
+    def save_sd_blob_hash_to_stream(self, stream_hash, sd_blob_hash):
+        self.stream_desc[sd_blob_hash] = stream_hash
+        return defer.succeed(True)
+
+    def get_sd_blob_hashes_for_stream(self, stream_hash):
+        return defer.succeed([sd_hash for sd_hash, s_h in self.stream_desc.iteritems() if s_h == stream_hash])
+
+    def hashes_to_announce(self):
+        next_announce_time = time.time() + self.hash_reannounce_time
+        stream_hashes = []
+        current_time = time.time()
+        for stream_hash, stream_info in self.streams.iteritems():
+            announce_time = stream_info['announce_time']
+            if announce_time < current_time:
+                self.streams[stream_hash]['announce_time'] = next_announce_time
+                stream_hashes.append(stream_hash)
+        return stream_hashes

lbrynet/lbrylive/PaymentRateManager.py

@@ -0,0 +1,45 @@
+class BaseLiveStreamPaymentRateManager(object):
+    def __init__(self, blob_info_rate, blob_data_rate=None):
+        self.min_live_blob_info_payment_rate = blob_info_rate
+        self.min_blob_data_payment_rate = blob_data_rate
+
+
+class LiveStreamPaymentRateManager(object):
+    def __init__(self, base_live_stream_payment_rate_manager, payment_rate_manager,
+                 blob_info_rate=None, blob_data_rate=None):
+        self._base_live_stream_payment_rate_manager = base_live_stream_payment_rate_manager
+        self._payment_rate_manager = payment_rate_manager
+        self.min_live_blob_info_payment_rate = blob_info_rate
+        self.min_blob_data_payment_rate = blob_data_rate
+        self.points_paid = 0.0
+
+    def get_rate_live_blob_info(self, peer):
+        return self.get_effective_min_live_blob_info_payment_rate()
+
+    def accept_rate_live_blob_info(self, peer, payment_rate):
+        return payment_rate >= self.get_effective_min_live_blob_info_payment_rate()
+
+    def get_rate_blob_data(self, peer):
+        return self.get_effective_min_blob_data_payment_rate()
+
+    def accept_rate_blob_data(self, peer, payment_rate):
+        return payment_rate >= self.get_effective_min_blob_data_payment_rate()
+
+    def get_effective_min_blob_data_payment_rate(self):
+        rate = self.min_blob_data_payment_rate
+        if rate is None:
+            rate = self._payment_rate_manager.min_blob_data_payment_rate
+        if rate is None:
+            rate = self._base_live_stream_payment_rate_manager.min_blob_data_payment_rate
+        if rate is None:
+            rate = self._payment_rate_manager.get_effective_min_blob_data_payment_rate()
+        return rate
+
+    def get_effective_min_live_blob_info_payment_rate(self):
+        rate = self.min_live_blob_info_payment_rate
+        if rate is None:
+            rate = self._base_live_stream_payment_rate_manager.min_live_blob_info_payment_rate
+        return rate
+
+    def record_points_paid(self, amount):
+        self.points_paid += amount

lbrynet/lbrylive/StreamDescriptor.py

@@ -0,0 +1,131 @@
+import binascii
+import logging
+from lbrynet.core.cryptoutils import get_lbry_hash_obj, verify_signature
+from twisted.internet import defer, threads
+from lbrynet.core.Error import DuplicateStreamHashError
+from lbrynet.lbrylive.LiveBlob import LiveBlobInfo
+from lbrynet.interfaces import IStreamDescriptorValidator
+from zope.interface import implements
+
+
+LiveStreamType = "lbrylive"
+
+
+def save_sd_info(stream_info_manager, sd_info, ignore_duplicate=False):
+    logging.debug("Saving info for %s", str(sd_info['stream_name']))
+    hex_stream_name = sd_info['stream_name']
+    public_key = sd_info['public_key']
+    key = sd_info['key']
+    stream_hash = sd_info['stream_hash']
+    raw_blobs = sd_info['blobs']
+    crypt_blobs = []
+    for blob in raw_blobs:
+        length = blob['length']
+        if length != 0:
+            blob_hash = blob['blob_hash']
+        else:
+            blob_hash = None
+        blob_num = blob['blob_num']
+        revision = blob['revision']
+        iv = blob['iv']
+        signature = blob['signature']
+        crypt_blobs.append(LiveBlobInfo(blob_hash, blob_num, length, iv, revision, signature))
+    logging.debug("Trying to save stream info for %s", str(hex_stream_name))
+    d = stream_info_manager.save_stream(stream_hash, public_key, hex_stream_name,
+                                        key, crypt_blobs)
+
+    def check_if_duplicate(err):
+        if ignore_duplicate is True:
+            err.trap(DuplicateStreamHashError)
+
+    d.addErrback(check_if_duplicate)
+
+    d.addCallback(lambda _: stream_hash)
+    return d
+
+
+def get_sd_info(stream_info_manager, stream_hash, include_blobs):
+    d = stream_info_manager.get_stream_info(stream_hash)
+
+    def format_info(stream_info):
+        fields = {}
+        fields['stream_type'] = LiveStreamType
+        fields['stream_name'] = stream_info[2]
+        fields['public_key'] = stream_info[0]
+        fields['key'] = stream_info[1]
+        fields['stream_hash'] = stream_hash
+
+        def format_blobs(blobs):
+            formatted_blobs = []
+            for blob_hash, blob_num, revision, iv, length, signature in blobs:
+                blob = {}
+                if length != 0:
+                    blob['blob_hash'] = blob_hash
+                blob['blob_num'] = blob_num
+                blob['revision'] = revision
+                blob['iv'] = iv
+                blob['length'] = length
+                blob['signature'] = signature
+                formatted_blobs.append(blob)
+            fields['blobs'] = formatted_blobs
+            return fields
+
+        if include_blobs is True:
+            d = stream_info_manager.get_blobs_for_stream(stream_hash)
+        else:
+            d = defer.succeed([])
+        d.addCallback(format_blobs)
+        return d
+
+    d.addCallback(format_info)
+    return d
+
+
+class LBRYLiveStreamDescriptorValidator(object):
+    implements(IStreamDescriptorValidator)
+
+    def __init__(self, raw_info):
+        self.raw_info = raw_info
+
+    def validate(self):
+        logging.debug("Trying to validate stream descriptor for %s", str(self.raw_info['stream_name']))
+        hex_stream_name = self.raw_info['stream_name']
+        public_key = self.raw_info['public_key']
+        key = self.raw_info['key']
+        stream_hash = self.raw_info['stream_hash']
+        h = get_lbry_hash_obj()
+        h.update(hex_stream_name)
+        h.update(public_key)
+        h.update(key)
+        if h.hexdigest() != stream_hash:
+            raise ValueError("Stream hash does not match stream metadata")
+        blobs = self.raw_info['blobs']
+
+        def check_blob_signatures():
+            for blob in blobs:
+                length = blob['length']
+                if length != 0:
+                    blob_hash = blob['blob_hash']
+                else:
+                    blob_hash = None
+                blob_num = blob['blob_num']
+                revision = blob['revision']
+                iv = blob['iv']
+                signature = blob['signature']
+                hashsum = get_lbry_hash_obj()
+                hashsum.update(stream_hash)
+                if length != 0:
+                    hashsum.update(blob_hash)
+                hashsum.update(str(blob_num))
+                hashsum.update(str(revision))
+                hashsum.update(iv)
+                hashsum.update(str(length))
+                if not verify_signature(hashsum.digest(), signature, public_key):
+                    raise ValueError("Invalid signature in stream descriptor")
+
+        return threads.deferToThread(check_blob_signatures)
+
+    def info_to_show(self):
+        info = []
+        info.append(("stream_name", binascii.unhexlify(self.raw_info.get("stream_name"))))
+        return info

lbrynet/lbrylive/client/LiveStreamDownloader.py

@@ -0,0 +1,180 @@
+import binascii
+from lbrynet.core.DownloadOption import DownloadOption
+from lbrynet.cryptstream.client.CryptStreamDownloader import CryptStreamDownloader
+from zope.interface import implements
+from lbrynet.lbrylive.client.LiveStreamMetadataHandler import LiveStreamMetadataHandler
+from lbrynet.lbrylive.client.LiveStreamProgressManager import LiveStreamProgressManager
+import os
+from lbrynet.lbrylive.StreamDescriptor import save_sd_info
+from lbrynet.lbrylive.PaymentRateManager import LiveStreamPaymentRateManager
+from twisted.internet import defer, threads  # , process
+from lbrynet.interfaces import IStreamDownloaderFactory
+
+
+class LiveStreamDownloader(CryptStreamDownloader):
+
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 payment_rate_manager, wallet, upload_allowed):
+        CryptStreamDownloader.__init__(self, peer_finder, rate_limiter, blob_manager,
+                                       payment_rate_manager, wallet, upload_allowed)
+        self.stream_hash = stream_hash
+        self.stream_info_manager = stream_info_manager
+        self.public_key = None
+
+    def set_stream_info(self):
+        if self.public_key is None and self.key is None:
+
+            d = self.stream_info_manager.get_stream_info(self.stream_hash)
+
+            def set_stream_info(stream_info):
+                public_key, key, stream_name = stream_info
+                self.public_key = public_key
+                self.key = binascii.unhexlify(key)
+                self.stream_name = binascii.unhexlify(stream_name)
+
+            d.addCallback(set_stream_info)
+            return d
+        else:
+            return defer.succeed(True)
+
+
+class LBRYLiveStreamDownloader(LiveStreamDownloader):
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 payment_rate_manager, wallet, upload_allowed):
+        LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter, blob_manager,
+                                      stream_info_manager, payment_rate_manager, wallet, upload_allowed)
+
+        #self.writer = process.ProcessWriter(reactor, self, 'write', 1)
+
+    def _get_metadata_handler(self, download_manager):
+        return LiveStreamMetadataHandler(self.stream_hash, self.stream_info_manager,
+                                         self.peer_finder, self.public_key, False,
+                                         self.payment_rate_manager, self.wallet, download_manager, 10)
+
+    def _get_progress_manager(self, download_manager):
+        return LiveStreamProgressManager(self._finished_downloading, self.blob_manager, download_manager,
+                                         delete_blob_after_finished=True, download_whole=False,
+                                         max_before_skip_ahead=10)
+
+    def _get_write_func(self):
+        def write_func(data):
+            if self.stopped is False:
+                #self.writer.write(data)
+                pass
+        return write_func
+
+
+class FullLiveStreamDownloader(LiveStreamDownloader):
+    def __init__(self, stream_hash, peer_finder, rate_limiter, blob_manager, stream_info_manager,
+                 payment_rate_manager, wallet, upload_allowed):
+        LiveStreamDownloader.__init__(self, stream_hash, peer_finder, rate_limiter,
+                                      blob_manager, stream_info_manager, payment_rate_manager,
+                                      wallet, upload_allowed)
+        self.file_handle = None
+        self.file_name = None
+
+    def set_stream_info(self):
+        d = LiveStreamDownloader.set_stream_info(self)
+
+        def set_file_name_if_unset():
+            if not self.file_name:
+                if not self.stream_name:
+                    self.stream_name = "_"
+                self.file_name = os.path.basename(self.stream_name)
+
+        d.addCallback(lambda _: set_file_name_if_unset())
+        return d
+
+    def stop(self):
+        d = self._close_file()
+        d.addBoth(lambda _: LiveStreamDownloader.stop(self))
+        return d
+
+    def _start(self):
+        if self.file_handle is None:
+            d = self._open_file()
+        else:
+            d = defer.succeed(True)
+        d.addCallback(lambda _: LiveStreamDownloader._start(self))
+        return d
+
+    def _open_file(self):
+        def open_file():
+            self.file_handle = open(self.file_name, 'wb')
+        return threads.deferToThread(open_file)
+
+    def _get_metadata_handler(self, download_manager):
+        return LiveStreamMetadataHandler(self.stream_hash, self.stream_info_manager,
+                                         self.peer_finder, self.public_key, True,
+                                         self.payment_rate_manager, self.wallet, download_manager)
+
+    def _get_primary_request_creators(self, download_manager):
+        return [download_manager.blob_requester, download_manager.blob_info_finder]
+
+    def _get_write_func(self):
+        def write_func(data):
+            if self.stopped is False:
+                self.file_handle.write(data)
+        return write_func
+
+    def _close_file(self):
+        def close_file():
+            if self.file_handle is not None:
+                self.file_handle.close()
+                self.file_handle = None
+        return threads.deferToThread(close_file)
+
+
+class FullLiveStreamDownloaderFactory(object):
+
+    implements(IStreamDownloaderFactory)
+
+    def __init__(self, peer_finder, rate_limiter, blob_manager, stream_info_manager, wallet,
+                 default_payment_rate_manager):
+        self.peer_finder = peer_finder
+        self.rate_limiter = rate_limiter
+        self.blob_manager = blob_manager
+        self.stream_info_manager = stream_info_manager
+        self.wallet = wallet
+        self.default_payment_rate_manager = default_payment_rate_manager
+
+    def get_downloader_options(self, sd_validator, payment_rate_manager):
+        options = [
+            DownloadOption(
+                [float, None],
+                "rate which will be paid for data (None means use application default)",
+                "data payment rate",
+                None
+            ),
+            DownloadOption(
+                [float, None],
+                "rate which will be paid for metadata (None means use application default)",
+                "metadata payment rate",
+                None
+            ),
+            DownloadOption(
+                [bool],
+                "allow reuploading data downloaded for this file",
+                "allow upload",
+                True
+            ),
+        ]
+        return options
+
+    def make_downloader(self, sd_validator, options, payment_rate_manager):
+        # TODO: check options for payment rate manager parameters
+        payment_rate_manager = LiveStreamPaymentRateManager(self.default_payment_rate_manager,
+                                                            payment_rate_manager)
+        d = save_sd_info(self.stream_info_manager, sd_validator.raw_info)
+
+        def create_downloader(stream_hash):
+            stream_downloader = FullLiveStreamDownloader(stream_hash, self.peer_finder, self.rate_limiter,
+                                                         self.blob_manager, self.stream_info_manager,
+                                                         payment_rate_manager, self.wallet, True)
+            # TODO: change upload_allowed=True above to something better
+            d = stream_downloader.set_stream_info()
+            d.addCallback(lambda _: stream_downloader)
+            return d
+
+        d.addCallback(create_downloader)
+        return d

lbrynet/lbrylive/client/LiveStreamMetadataHandler.py

@@ -0,0 +1,342 @@
+from collections import defaultdict
+import logging
+from zope.interface import implements
+from twisted.internet import defer
+from twisted.python.failure import Failure
+from lbrynet.conf import MAX_BLOB_INFOS_TO_REQUEST
+from lbrynet.core.client.ClientRequest import ClientRequest, ClientPaidRequest
+from lbrynet.lbrylive.LiveBlob import LiveBlobInfo
+from lbrynet.core.cryptoutils import get_lbry_hash_obj, verify_signature
+from lbrynet.interfaces import IRequestCreator, IMetadataHandler
+from lbrynet.core.Error import InsufficientFundsError, InvalidResponseError, RequestCanceledError
+from lbrynet.core.Error import NoResponseError
+
+
+class LiveStreamMetadataHandler(object):
+    implements(IRequestCreator, IMetadataHandler)
+
+    def __init__(self, stream_hash, stream_info_manager, peer_finder, stream_pub_key, download_whole,
+                 payment_rate_manager, wallet, download_manager, max_before_skip_ahead=None):
+        self.stream_hash = stream_hash
+        self.stream_info_manager = stream_info_manager
+        self.payment_rate_manager = payment_rate_manager
+        self.wallet = wallet
+        self.peer_finder = peer_finder
+        self.stream_pub_key = stream_pub_key
+        self.download_whole = download_whole
+        self.max_before_skip_ahead = max_before_skip_ahead
+        if self.download_whole is False:
+            assert self.max_before_skip_ahead is not None, \
+                "If download whole is False, max_before_skip_ahead must be set"
+        self.download_manager = download_manager
+        self._peers = defaultdict(int)  # {Peer: score}
+        self._protocol_prices = {}
+        self._final_blob_num = None
+        self._price_disagreements = []  # [Peer]
+        self._incompatible_peers = []  # [Peer]
+
+    ######### IMetadataHandler #########
+
+    def get_initial_blobs(self):
+        d = self.stream_info_manager.get_blobs_for_stream(self.stream_hash)
+        d.addCallback(self._format_initial_blobs_for_download_manager)
+        return d
+
+    def final_blob_num(self):
+        return self._final_blob_num
+
+    ######## IRequestCreator #########
+
+    def send_next_request(self, peer, protocol):
+        if self._finished_discovery() is False and self._should_send_request_to(peer) is True:
+            p_r = None
+            if not self._price_settled(protocol):
+                p_r = self._get_price_request(peer, protocol)
+            d_r = self._get_discover_request(peer)
+            reserved_points = self._reserve_points(peer, protocol, d_r.max_pay_units)
+            if reserved_points is not None:
+                d1 = protocol.add_request(d_r)
+                d1.addCallback(self._handle_discover_response, peer, d_r)
+                d1.addBoth(self._pay_or_cancel_payment, protocol, reserved_points)
+                d1.addErrback(self._request_failed, peer)
+                if p_r is not None:
+                    d2 = protocol.add_request(p_r)
+                    d2.addCallback(self._handle_price_response, peer, p_r, protocol)
+                    d2.addErrback(self._request_failed, peer)
+                return defer.succeed(True)
+            else:
+                return defer.fail(InsufficientFundsError())
+        return defer.succeed(False)
+
+    def get_new_peers(self):
+        d = self._get_hash_for_peer_search()
+        d.addCallback(self._find_peers_for_hash)
+        return d
+
+    ######### internal calls #########
+
+    def _get_hash_for_peer_search(self):
+        r = None
+        if self._finished_discovery() is False:
+            r = self.stream_hash
+        logging.debug("Info finder peer search response for stream %s: %s", str(self.stream_hash), str(r))
+        return defer.succeed(r)
+
+    def _find_peers_for_hash(self, h):
+        if h is None:
+            return None
+        else:
+            d = self.peer_finder.find_peers_for_blob(h)
+
+            def choose_best_peers(peers):
+                bad_peers = self._get_bad_peers()
+                return [p for p in peers if not p in bad_peers]
+
+            d.addCallback(choose_best_peers)
+            return d
+
+    def _format_initial_blobs_for_download_manager(self, blob_infos):
+        infos = []
+        for blob_hash, blob_num, revision, iv, length, signature in blob_infos:
+            if blob_hash is not None:
+                infos.append(LiveBlobInfo(blob_hash, blob_num, length, iv, revision, signature))
+            else:
+                logging.debug("Setting _final_blob_num to %s", str(blob_num - 1))
+                self._final_blob_num = blob_num - 1
+        return infos
+
+    def _should_send_request_to(self, peer):
+        if self._peers[peer] < -5.0:
+            return False
+        if peer in self._price_disagreements:
+            return False
+        return True
+
+    def _get_bad_peers(self):
+        return [p for p in self._peers.iterkeys() if not self._should_send_request_to(p)]
+
+    def _finished_discovery(self):
+        if self._get_discovery_params() is None:
+            return True
+        return False
+
+    def _get_discover_request(self, peer):
+        discovery_params = self._get_discovery_params()
+        if discovery_params:
+            further_blobs_request = {}
+            reference, start, end, count = discovery_params
+            further_blobs_request['reference'] = reference
+            if start is not None:
+                further_blobs_request['start'] = start
+            if end is not None:
+                further_blobs_request['end'] = end
+            if count is not None:
+                further_blobs_request['count'] = count
+            else:
+                further_blobs_request['count'] = MAX_BLOB_INFOS_TO_REQUEST
+            logging.debug("Requesting %s blob infos from %s", str(further_blobs_request['count']), str(peer))
+            r_dict = {'further_blobs': further_blobs_request}
+            response_identifier = 'further_blobs'
+            request = ClientPaidRequest(r_dict, response_identifier, further_blobs_request['count'])
+            return request
+        return None
+
+    def _get_discovery_params(self):
+        logging.debug("In _get_discovery_params")
+        stream_position = self.download_manager.stream_position()
+        blobs = self.download_manager.blobs
+        if blobs:
+            last_blob_num = max(blobs.iterkeys())
+        else:
+            last_blob_num = -1
+        final_blob_num = self.final_blob_num()
+        if final_blob_num is not None:
+            last_blob_num = final_blob_num
+        if self.download_whole is False:
+            logging.debug("download_whole is False")
+            if final_blob_num is not None:
+                for i in xrange(stream_position, final_blob_num + 1):
+                    if not i in blobs:
+                        count = min(self.max_before_skip_ahead, (final_blob_num - i + 1))
+                        return self.stream_hash, None, 'end', count
+                return None
+            else:
+                if blobs:
+                    for i in xrange(stream_position, last_blob_num + 1):
+                        if not i in blobs:
+                            if i == 0:
+                                return self.stream_hash, 'beginning', 'end', -1 * self.max_before_skip_ahead
+                            else:
+                                return self.stream_hash, blobs[i-1].blob_hash, 'end', -1 * self.max_before_skip_ahead
+                    return self.stream_hash, blobs[last_blob_num].blob_hash, 'end', -1 * self.max_before_skip_ahead
+                else:
+                    return self.stream_hash, None, 'end', -1 * self.max_before_skip_ahead
+        logging.debug("download_whole is True")
+        beginning = None
+        end = None
+        for i in xrange(stream_position, last_blob_num + 1):
+            if not i in blobs:
+                if beginning is None:
+                    if i == 0:
+                        beginning = 'beginning'
+                    else:
+                        beginning = blobs[i-1].blob_hash
+            else:
+                if beginning is not None:
+                    end = blobs[i].blob_hash
+                    break
+        if beginning is None:
+            if final_blob_num is not None:
+                logging.debug("Discovery is finished. stream_position: %s, last_blob_num + 1: %s", str(stream_position),
+                              str(last_blob_num + 1))
+                return None
+            else:
+                logging.debug("Discovery is not finished. final blob num is unknown.")
+                if last_blob_num != -1:
+                    return self.stream_hash, blobs[last_blob_num].blob_hash, None, None
+                else:
+                    return self.stream_hash, 'beginning', None, None
+        else:
+            logging.info("Discovery is not finished. Not all blobs are known.")
+            return self.stream_hash, beginning, end, None
+
+    def _price_settled(self, protocol):
+        if protocol in self._protocol_prices:
+            return True
+        return False
+
+    def _update_local_score(self, peer, amount):
+        self._peers[peer] += amount
+
+    def _reserve_points(self, peer, protocol, max_infos):
+        assert protocol in self._protocol_prices
+        point_amount = 1.0 * max_infos * self._protocol_prices[protocol] / 1000.0
+        return self.wallet.reserve_points(peer, point_amount)
+
+    def _pay_or_cancel_payment(self, arg, protocol, reserved_points):
+        if isinstance(arg, Failure) or arg == 0:
+            self._cancel_points(reserved_points)
+        else:
+            self._pay_peer(protocol, arg, reserved_points)
+        return arg
+
+    def _pay_peer(self, protocol, num_infos, reserved_points):
+        assert num_infos != 0
+        assert protocol in self._protocol_prices
+        point_amount = 1.0 * num_infos * self._protocol_prices[protocol] / 1000.0
+        self.wallet.send_points(reserved_points, point_amount)
+        self.payment_rate_manager.record_points_paid(point_amount)
+
+    def _cancel_points(self, reserved_points):
+        return self.wallet.cancel_point_reservation(reserved_points)
+
+    def _get_price_request(self, peer, protocol):
+        self._protocol_prices[protocol] = self.payment_rate_manager.get_rate_live_blob_info(peer)
+        request_dict = {'blob_info_payment_rate': self._protocol_prices[protocol]}
+        request = ClientRequest(request_dict, 'blob_info_payment_rate')
+        return request
+
+    def _handle_price_response(self, response_dict, peer, request, protocol):
+        if not request.response_identifier in response_dict:
+            return InvalidResponseError("response identifier not in response")
+        assert protocol in self._protocol_prices
+        response = response_dict[request.response_identifier]
+        if response == "RATE_ACCEPTED":
+            return True
+        else:
+            logging.info("Rate offer has been rejected by %s", str(peer))
+            del self._protocol_prices[protocol]
+            self._price_disagreements.append(peer)
+        return True
+
+    def _handle_discover_response(self, response_dict, peer, request):
+        if not request.response_identifier in response_dict:
+            return InvalidResponseError("response identifier not in response")
+        response = response_dict[request.response_identifier]
+        blob_infos = []
+        if 'error' in response:
+            if response['error'] == 'RATE_UNSET':
+                return defer.succeed(0)
+            else:
+                return InvalidResponseError("Got an unknown error from the peer: %s" %
+                                            (response['error'],))
+        if not 'blob_infos' in response:
+            return InvalidResponseError("Missing the required field 'blob_infos'")
+        raw_blob_infos = response['blob_infos']
+        logging.info("Handling %s further blobs from %s", str(len(raw_blob_infos)), str(peer))
+        logging.debug("blobs: %s", str(raw_blob_infos))
+        for raw_blob_info in raw_blob_infos:
+            length = raw_blob_info['length']
+            if length != 0:
+                blob_hash = raw_blob_info['blob_hash']
+            else:
+                blob_hash = None
+            num = raw_blob_info['blob_num']
+            revision = raw_blob_info['revision']
+            iv = raw_blob_info['iv']
+            signature = raw_blob_info['signature']
+            blob_info = LiveBlobInfo(blob_hash, num, length, iv, revision, signature)
+            logging.debug("Learned about a potential blob: %s", str(blob_hash))
+            if self._verify_blob(blob_info):
+                if blob_hash is None:
+                    logging.info("Setting _final_blob_num to %s", str(num - 1))
+                    self._final_blob_num = num - 1
+                else:
+                    blob_infos.append(blob_info)
+            else:
+                raise ValueError("Peer sent an invalid blob info")
+        d = self.stream_info_manager.add_blobs_to_stream(self.stream_hash, blob_infos)
+
+        def add_blobs_to_download_manager():
+            blob_nums = [b.blob_num for b in blob_infos]
+            logging.info("Adding the following blob nums to the download manager: %s", str(blob_nums))
+            self.download_manager.add_blobs_to_download(blob_infos)
+
+        d.addCallback(lambda _: add_blobs_to_download_manager())
+
+        def pay_or_penalize_peer():
+            if len(blob_infos):
+                self._update_local_score(peer, len(blob_infos))
+                peer.update_stats('downloaded_crypt_blob_infos', len(blob_infos))
+                peer.update_score(len(blob_infos))
+            else:
+                self._update_local_score(peer, -.0001)
+            return len(blob_infos)
+
+        d.addCallback(lambda _: pay_or_penalize_peer())
+
+        return d
+
+    def _verify_blob(self, blob):
+        logging.debug("Got an unverified blob to check:")
+        logging.debug("blob_hash: %s", blob.blob_hash)
+        logging.debug("blob_num: %s", str(blob.blob_num))
+        logging.debug("revision: %s", str(blob.revision))
+        logging.debug("iv: %s", blob.iv)
+        logging.debug("length: %s", str(blob.length))
+        hashsum = get_lbry_hash_obj()
+        hashsum.update(self.stream_hash)
+        if blob.length != 0:
+            hashsum.update(blob.blob_hash)
+        hashsum.update(str(blob.blob_num))
+        hashsum.update(str(blob.revision))
+        hashsum.update(blob.iv)
+        hashsum.update(str(blob.length))
+        logging.debug("hexdigest to be verified: %s", hashsum.hexdigest())
+        if verify_signature(hashsum.digest(), blob.signature, self.stream_pub_key):
+            logging.debug("Blob info is valid")
+            return True
+        else:
+            logging.debug("The blob info is invalid")
+            return False
+
+    def _request_failed(self, reason, peer):
+        if reason.check(RequestCanceledError):
+            return
+        if reason.check(NoResponseError):
+            self._incompatible_peers.append(peer)
+            return
+        logging.warning("Crypt stream info finder: a request failed. Reason: %s", reason.getErrorMessage())
+        self._update_local_score(peer, -5.0)
+        peer.update_score(-10.0)
+        return reason

lbrynet/lbrylive/client/LiveStreamProgressManager.py

@@ -0,0 +1,87 @@
+import logging
+from lbrynet.core.client.StreamProgressManager import StreamProgressManager
+from twisted.internet import defer
+
+
+class LiveStreamProgressManager(StreamProgressManager):
+    def __init__(self, finished_callback, blob_manager, download_manager, delete_blob_after_finished=False,
+                 download_whole=True, max_before_skip_ahead=5):
+        self.download_whole = download_whole
+        self.max_before_skip_ahead = max_before_skip_ahead
+        StreamProgressManager.__init__(self, finished_callback, blob_manager, download_manager,
+                                       delete_blob_after_finished)
+
+    ######### IProgressManager #########
+
+    def stream_position(self):
+        blobs = self.download_manager.blobs
+        if not blobs:
+            return 0
+        else:
+            newest_known_blobnum = max(blobs.iterkeys())
+            position = newest_known_blobnum
+            oldest_relevant_blob_num = (max(0, newest_known_blobnum - self.max_before_skip_ahead + 1))
+            for i in xrange(newest_known_blobnum, oldest_relevant_blob_num - 1, -1):
+                if i in blobs and (not blobs[i].is_validated() and not i in self.provided_blob_nums):
+                    position = i
+            return position
+
+    def needed_blobs(self):
+        blobs = self.download_manager.blobs
+        stream_position = self.stream_position()
+        if blobs:
+            newest_known_blobnum = max(blobs.iterkeys())
+        else:
+            newest_known_blobnum = -1
+        blobs_needed = []
+        for i in xrange(stream_position, newest_known_blobnum + 1):
+            if i in blobs and not blobs[i].is_validated() and not i in self.provided_blob_nums:
+                blobs_needed.append(blobs[i])
+        return blobs_needed
+
+    ######### internal #########
+
+    def _output_loop(self):
+
+        from twisted.internet import reactor
+
+        if self.stopped is True:
+            if self.outputting_d is not None:
+                self.outputting_d.callback(True)
+                self.outputting_d = None
+            return
+
+        blobs = self.download_manager.blobs
+        logging.info("In _output_loop. last_blob_outputted: %s", str(self.last_blob_outputted))
+        if blobs:
+            logging.debug("Newest blob number: %s", str(max(blobs.iterkeys())))
+        if self.outputting_d is None:
+            self.outputting_d = defer.Deferred()
+
+        current_blob_num = self.last_blob_outputted + 1
+
+        def finished_outputting_blob():
+            self.last_blob_outputted += 1
+            final_blob_num = self.download_manager.final_blob_num()
+            if final_blob_num is not None and final_blob_num == self.last_blob_outputted:
+                self._finished_outputting()
+                self.outputting_d.callback(True)
+                self.outputting_d = None
+            else:
+                reactor.callLater(0, self._output_loop)
+
+        if current_blob_num in blobs and blobs[current_blob_num].is_validated():
+            logging.info("Outputting blob %s", str(current_blob_num))
+            self.provided_blob_nums.append(current_blob_num)
+            d = self.download_manager.handle_blob(current_blob_num)
+            d.addCallback(lambda _: finished_outputting_blob())
+            d.addCallback(lambda _: self._finished_with_blob(current_blob_num))
+        elif blobs and max(blobs.iterkeys()) > self.last_blob_outputted + self.max_before_skip_ahead - 1:
+            self.last_blob_outputted += 1
+            logging.info("Skipping blob number %s due to knowing about blob number %s",
+                         str(self.last_blob_outputted), str(max(blobs.iterkeys())))
+            self._finished_with_blob(current_blob_num)
+            reactor.callLater(0, self._output_loop)
+        else:
+            self.outputting_d.callback(True)
+            self.outputting_d = None

lbrynet/lbrylive/server/LiveBlobInfoQueryHandler.py

@@ -0,0 +1,180 @@
+import logging
+from twisted.internet import defer
+from zope.interface import implements
+from lbrynet.interfaces import IQueryHandlerFactory, IQueryHandler
+
+
+class CryptBlobInfoQueryHandlerFactory(object):
+    implements(IQueryHandlerFactory)
+
+    def __init__(self, stream_info_manager, wallet, payment_rate_manager):
+        self.stream_info_manager = stream_info_manager
+        self.wallet = wallet
+        self.payment_rate_manager = payment_rate_manager
+
+    ######### IQueryHandlerFactory #########
+
+    def build_query_handler(self):
+        q_h = CryptBlobInfoQueryHandler(self.stream_info_manager, self.wallet, self.payment_rate_manager)
+        return q_h
+
+    def get_primary_query_identifier(self):
+        return 'further_blobs'
+
+    def get_description(self):
+        return ("Stream Blob Information - blob hashes that are associated with streams,"
+                " and the blobs' associated metadata")
+
+
+class CryptBlobInfoQueryHandler(object):
+    implements(IQueryHandler)
+
+    def __init__(self, stream_info_manager, wallet, payment_rate_manager):
+        self.stream_info_manager = stream_info_manager
+        self.wallet = wallet
+        self.payment_rate_manager = payment_rate_manager
+        self.query_identifiers = ['blob_info_payment_rate', 'further_blobs']
+        self.blob_info_payment_rate = None
+        self.peer = None
+
+    ######### IQueryHandler #########
+
+    def register_with_request_handler(self, request_handler, peer):
+        self.peer = peer
+        request_handler.register_query_handler(self, self.query_identifiers)
+
+    def handle_queries(self, queries):
+        response = {}
+
+        if self.query_identifiers[0] in queries:
+            if not self.handle_blob_info_payment_rate(queries[self.query_identifiers[0]]):
+                return defer.succeed({'blob_info_payment_rate': 'RATE_TOO_LOW'})
+            else:
+                response['blob_info_payment_rate'] = "RATE_ACCEPTED"
+
+        if self.query_identifiers[1] in queries:
+            further_blobs_request = queries[self.query_identifiers[1]]
+            logging.debug("Received the client's request for additional blob information")
+
+            if self.blob_info_payment_rate is None:
+                response['further_blobs'] = {'error': 'RATE_UNSET'}
+                return defer.succeed(response)
+
+            def count_and_charge(blob_infos):
+                if len(blob_infos) != 0:
+                    logging.info("Responding with %s infos", str(len(blob_infos)))
+                    expected_payment = 1.0 * len(blob_infos) * self.blob_info_payment_rate / 1000.0
+                    self.wallet.add_expected_payment(self.peer, expected_payment)
+                    self.peer.update_stats('uploaded_crypt_blob_infos', len(blob_infos))
+                return blob_infos
+
+            def set_field(further_blobs):
+                response['further_blobs'] = {'blob_infos': further_blobs}
+                return response
+
+            def get_further_blobs(stream_hash):
+                if stream_hash is None:
+                    response['further_blobs'] = {'error': 'REFERENCE_HASH_UNKNOWN'}
+                    return defer.succeed(response)
+                start = further_blobs_request.get("start")
+                end = further_blobs_request.get("end")
+                count = further_blobs_request.get("count")
+                if count is not None:
+                    try:
+                        count = int(count)
+                    except ValueError:
+                        response['further_blobs'] = {'error': 'COUNT_NON_INTEGER'}
+                        return defer.succeed(response)
+
+                if len([x for x in [start, end, count] if x is not None]) < 2:
+                    response['further_blobs'] = {'error': 'TOO_FEW_PARAMETERS'}
+                    return defer.succeed(response)
+
+                inner_d = self.get_further_blobs(stream_hash, start, end, count)
+
+                inner_d.addCallback(count_and_charge)
+                inner_d.addCallback(self.format_blob_infos)
+                inner_d.addCallback(set_field)
+                return inner_d
+
+            if 'reference' in further_blobs_request:
+                d = self.get_stream_hash_from_reference(further_blobs_request['reference'])
+                d.addCallback(get_further_blobs)
+                return d
+            else:
+                response['further_blobs'] = {'error': 'NO_REFERENCE_SENT'}
+                return defer.succeed(response)
+        else:
+            return defer.succeed({})
+
+    ######### internal #########
+
+    def handle_blob_info_payment_rate(self, requested_payment_rate):
+        if not self.payment_rate_manager.accept_rate_live_blob_info(self.peer, requested_payment_rate):
+            return False
+        else:
+            self.blob_info_payment_rate = requested_payment_rate
+            return True
+
+    def format_blob_infos(self, blobs):
+        blob_infos = []
+        for blob_hash, blob_num, revision, iv, length, signature in blobs:
+            blob_info = {}
+            if length != 0:
+                blob_info['blob_hash'] = blob_hash
+            blob_info['blob_num'] = blob_num
+            blob_info['revision'] = revision
+            blob_info['iv'] = iv
+            blob_info['length'] = length
+            blob_info['signature'] = signature
+            blob_infos.append(blob_info)
+        return blob_infos
+
+    def get_stream_hash_from_reference(self, reference):
+        d = self.stream_info_manager.check_if_stream_exists(reference)
+
+        def check_if_stream_found(result):
+            if result is True:
+                return reference
+            else:
+                return self.stream_info_manager.get_stream_of_blob(reference)
+
+        d.addCallback(check_if_stream_found)
+        return d
+
+    def get_further_blobs(self, stream_hash, start, end, count):
+        ds = []
+        if start is not None and start != "beginning":
+            ds.append(self.stream_info_manager.get_stream_of_blob(start))
+        if end is not None and end != 'end':
+            ds.append(self.stream_info_manager.get_stream_of_blob(end))
+        dl = defer.DeferredList(ds, fireOnOneErrback=True)
+
+        def ensure_streams_match(results):
+            for success, stream_of_blob in results:
+                if stream_of_blob != stream_hash:
+                    raise ValueError("Blob does not match stream")
+            return True
+
+        def get_blob_infos():
+            reverse = False
+            count_to_use = count
+            if start is None:
+                reverse = True
+            elif end is not None and count_to_use is not None and count_to_use < 0:
+                reverse = True
+            if count_to_use is not None and count_to_use < 0:
+                count_to_use *= -1
+            if start == "beginning" or start is None:
+                s = None
+            else:
+                s = start
+            if end == "end" or end is None:
+                e = None
+            else:
+                e = end
+            return self.stream_info_manager.get_blobs_for_stream(stream_hash, s, e, count_to_use, reverse)
+
+        dl.addCallback(ensure_streams_match)
+        dl.addCallback(lambda _: get_blob_infos())
+        return dl

lbrynet/lbrynet_console/ConsoleControl.py

@@ -0,0 +1,60 @@
+from twisted.protocols import basic
+from twisted.internet import defer
+
+
+class ConsoleControl(basic.LineReceiver):
+    from os import linesep as delimiter
+
+    def __init__(self, control_handlers):
+        self.control_handlers = {}
+        self.categories = {}
+        categories = set([category for category, handler in control_handlers])
+        prompt_number = 0
+        for category in categories:
+            self.categories[prompt_number] = category
+            for handler in [handler for cat, handler in control_handlers if cat == category]:
+                self.control_handlers[prompt_number] = handler
+                prompt_number += 1
+        self.current_handler = None
+
+    def connectionMade(self):
+        self.show_prompt()
+
+    def lineReceived(self, line):
+
+        def show_response(response):
+            if response is not None:
+                self.sendLine(response)
+
+        def show_error(err):
+            self.sendLine(err.getTraceback())
+
+        if self.current_handler is None:
+            try:
+                num = int(line)
+            except ValueError:
+                num = None
+            if num in self.control_handlers:
+                self.current_handler = self.control_handlers[num].get_handler()
+                line = None
+        if self.current_handler is not None:
+            try:
+                r = self.current_handler.handle_line(line)
+                done, ds = r[0], [d for d in r[1:] if d is not None]
+            except Exception as e:
+                done = True
+                ds = [defer.fail(e)]
+            if done is True:
+                self.current_handler = None
+            map(lambda d: d.addCallbacks(show_response, show_error), ds)
+        if self.current_handler is None:
+            self.show_prompt()
+
+    def show_prompt(self):
+        self.sendLine("Options:")
+        for num, handler in self.control_handlers.iteritems():
+            if num in self.categories:
+                self.sendLine("")
+                self.sendLine(self.categories[num])
+                self.sendLine("")
+            self.sendLine("[" + str(num) + "] " + handler.get_prompt_description())

lbrynet/lbrynet_console/LBRYConsole.py

@@ -0,0 +1,407 @@
+import logging
+from lbrynet.core.Session import LBRYSession
+import os.path
+import argparse
+from yapsy.PluginManager import PluginManager
+from twisted.internet import defer, threads, stdio, task
+from lbrynet.lbrynet_console.ConsoleControl import ConsoleControl
+from lbrynet.lbrynet_console.LBRYSettings import LBRYSettings
+from lbrynet.lbryfilemanager.LBRYFileManager import LBRYFileManager
+from lbrynet.conf import MIN_BLOB_DATA_PAYMENT_RATE  # , MIN_BLOB_INFO_PAYMENT_RATE
+from lbrynet.core.utils import generate_id
+from lbrynet.core.StreamDescriptor import StreamDescriptorIdentifier
+from lbrynet.core.PaymentRateManager import PaymentRateManager
+from lbrynet.core.server.BlobAvailabilityHandler import BlobAvailabilityHandlerFactory
+from lbrynet.core.server.BlobRequestHandler import BlobRequestHandlerFactory
+from lbrynet.core.server.ServerProtocol import ServerProtocolFactory
+from lbrynet.core.PTCWallet import PTCWallet
+from lbrynet.lbryfile.client.LBRYFileDownloader import LBRYFileOpenerFactory
+from lbrynet.lbryfile.StreamDescriptor import LBRYFileStreamType
+from lbrynet.lbryfile.LBRYFileMetadataManager import DBLBRYFileMetadataManager, TempLBRYFileMetadataManager
+#from lbrynet.lbrylive.PaymentRateManager import LiveStreamPaymentRateManager
+from lbrynet.lbrynet_console.ControlHandlers import ApplicationStatusFactory, GetWalletBalancesFactory, ShutDownFactory
+from lbrynet.lbrynet_console.ControlHandlers import LBRYFileStatusFactory, DeleteLBRYFileChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import ToggleLBRYFileRunningChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import ModifyApplicationDefaultsFactory
+from lbrynet.lbrynet_console.ControlHandlers import CreateLBRYFileFactory, PublishStreamDescriptorChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import ShowPublishedSDHashesChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import CreatePlainStreamDescriptorChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import ShowLBRYFileStreamHashChooserFactory, AddStreamFromHashFactory
+from lbrynet.lbrynet_console.ControlHandlers import AddStreamFromSDFactory, AddStreamFromLBRYcrdNameFactory
+from lbrynet.lbrynet_console.ControlHandlers import ClaimNameFactory
+from lbrynet.lbrynet_console.ControlHandlers import ShowServerStatusFactory, ModifyServerSettingsFactory
+from lbrynet.lbrynet_console.ControlHandlers import ModifyLBRYFileOptionsChooserFactory
+from lbrynet.lbrynet_console.ControlHandlers import PeerStatsAndSettingsChooserFactory
+from lbrynet.core.LBRYcrdWallet import LBRYcrdWallet
+
+
+class LBRYConsole():
+    """A class which can upload and download file streams to and from the network"""
+    def __init__(self, peer_port, dht_node_port, known_dht_nodes, control_class, wallet_type, lbrycrd_rpc_port,
+                 use_upnp, conf_dir, data_dir):
+        """
+        @param peer_port: the network port on which to listen for peers
+
+        @param dht_node_port: the network port on which to listen for dht node requests
+
+        @param known_dht_nodes: a list of (ip_address, dht_port) which will be used to join the DHT network
+        """
+        self.peer_port = peer_port
+        self.dht_node_port = dht_node_port
+        self.known_dht_nodes = known_dht_nodes
+        self.wallet_type = wallet_type
+        self.wallet_rpc_port = lbrycrd_rpc_port
+        self.use_upnp = use_upnp
+        self.lbry_server_port = None
+        self.control_class = control_class
+        self.session = None
+        self.lbry_file_metadata_manager = None
+        self.lbry_file_manager = None
+        self.conf_dir = conf_dir
+        self.data_dir = data_dir
+        self.plugin_manager = PluginManager()
+        self.plugin_manager.setPluginPlaces([
+            os.path.join(self.conf_dir, "plugins"),
+            os.path.join(os.path.dirname(__file__), "plugins"),
+        ])
+        self.control_handlers = []
+        self.query_handlers = {}
+
+        self.settings = LBRYSettings(self.conf_dir)
+        self.blob_request_payment_rate_manager = None
+        self.lbryid = None
+        self.sd_identifier = StreamDescriptorIdentifier()
+
+    def start(self):
+        """Initialize the session and restore everything to its saved state"""
+        d = threads.deferToThread(self._create_directory)
+        d.addCallback(lambda _: self._get_settings())
+        d.addCallback(lambda _: self._get_session())
+        d.addCallback(lambda _: self._setup_lbry_file_manager())
+        d.addCallback(lambda _: self._setup_lbry_file_opener())
+        d.addCallback(lambda _: self._setup_control_handlers())
+        d.addCallback(lambda _: self._setup_query_handlers())
+        d.addCallback(lambda _: self._load_plugins())
+        d.addCallback(lambda _: self._setup_server())
+        d.addCallback(lambda _: self._start_controller())
+        return d
+
+    def shut_down(self):
+        """Stop the session, all currently running streams, and stop the server"""
+        d = self.session.shut_down()
+        d.addCallback(lambda _: self._shut_down())
+        return d
+
+    def add_control_handlers(self, control_handlers):
+        for control_handler in control_handlers:
+            self.control_handlers.append(control_handler)
+
+    def add_query_handlers(self, query_handlers):
+
+        def _set_query_handlers(statuses):
+            from future_builtins import zip
+            for handler, (success, status) in zip(query_handlers, statuses):
+                if success is True:
+                    self.query_handlers[handler] = status
+
+        ds = []
+        for handler in query_handlers:
+            ds.append(self.settings.get_query_handler_status(handler.get_primary_query_identifier()))
+        dl = defer.DeferredList(ds)
+        dl.addCallback(_set_query_handlers)
+        return dl
+
+    def _create_directory(self):
+        if not os.path.exists(self.conf_dir):
+            os.makedirs(self.conf_dir)
+            logging.debug("Created the configuration directory: %s", str(self.conf_dir))
+        if not os.path.exists(self.data_dir):
+            os.makedirs(self.data_dir)
+            logging.debug("Created the data directory: %s", str(self.data_dir))
+
+    def _get_settings(self):
+        d = self.settings.start()
+        d.addCallback(lambda _: self.settings.get_lbryid())
+        d.addCallback(self.set_lbryid)
+        return d
+
+    def set_lbryid(self, lbryid):
+        if lbryid is None:
+            return self._make_lbryid()
+        else:
+            self.lbryid = lbryid
+
+    def _make_lbryid(self):
+        self.lbryid = generate_id()
+        d = self.settings.save_lbryid(self.lbryid)
+        return d
+
+    def _get_session(self):
+        d = self.settings.get_default_data_payment_rate()
+
+        def create_session(default_data_payment_rate):
+            if default_data_payment_rate is None:
+                default_data_payment_rate = MIN_BLOB_DATA_PAYMENT_RATE
+            if self.wallet_type == "lbrycrd":
+                wallet = LBRYcrdWallet("rpcuser", "rpcpassword", "127.0.0.1", self.wallet_rpc_port)
+            else:
+                wallet = PTCWallet(self.conf_dir)
+            self.session = LBRYSession(default_data_payment_rate, db_dir=self.conf_dir, lbryid=self.lbryid,
+                                       blob_dir=self.data_dir, dht_node_port=self.dht_node_port,
+                                       known_dht_nodes=self.known_dht_nodes, peer_port=self.peer_port,
+                                       use_upnp=self.use_upnp, wallet=wallet)
+
+        d.addCallback(create_session)
+
+        d.addCallback(lambda _: self.session.setup())
+
+        return d
+
+    def _setup_lbry_file_manager(self):
+        self.lbry_file_metadata_manager = DBLBRYFileMetadataManager(self.conf_dir)
+        d = self.lbry_file_metadata_manager.setup()
+
+        def set_lbry_file_manager():
+            self.lbry_file_manager = LBRYFileManager(self.session, self.lbry_file_metadata_manager, self.sd_identifier)
+            return self.lbry_file_manager.setup()
+
+        d.addCallback(lambda _: set_lbry_file_manager())
+
+        return d
+
+    def _setup_lbry_file_opener(self):
+        stream_info_manager = TempLBRYFileMetadataManager()
+        downloader_factory = LBRYFileOpenerFactory(self.session.peer_finder, self.session.rate_limiter,
+                                                   self.session.blob_manager, stream_info_manager,
+                                                   self.session.wallet)
+        self.sd_identifier.add_stream_downloader_factory(LBRYFileStreamType, downloader_factory)
+        return defer.succeed(True)
+
+    def _setup_control_handlers(self):
+        handlers = [
+            ('General',
+             ApplicationStatusFactory(self.session.rate_limiter, self.session.dht_node)),
+            ('General',
+             GetWalletBalancesFactory(self.session.wallet)),
+            ('General',
+             ModifyApplicationDefaultsFactory(self)),
+            ('General',
+             ShutDownFactory(self)),
+            ('General',
+             PeerStatsAndSettingsChooserFactory(self.session.peer_manager)),
+            ('lbryfile',
+             LBRYFileStatusFactory(self.lbry_file_manager)),
+            ('Stream Downloading',
+             AddStreamFromSDFactory(self.sd_identifier, self.session.base_payment_rate_manager)),
+            ('lbryfile',
+             DeleteLBRYFileChooserFactory(self.lbry_file_metadata_manager, self.session.blob_manager,
+                                          self.lbry_file_manager)),
+            ('lbryfile',
+             ToggleLBRYFileRunningChooserFactory(self.lbry_file_manager)),
+            ('lbryfile',
+             CreateLBRYFileFactory(self.session, self.lbry_file_manager)),
+            ('lbryfile',
+             PublishStreamDescriptorChooserFactory(self.lbry_file_metadata_manager,
+                                                   self.session.blob_manager,
+                                                   self.lbry_file_manager)),
+            ('lbryfile',
+             ShowPublishedSDHashesChooserFactory(self.lbry_file_metadata_manager,
+                                                 self.lbry_file_manager)),
+            ('lbryfile',
+             CreatePlainStreamDescriptorChooserFactory(self.lbry_file_manager)),
+            ('lbryfile',
+             ShowLBRYFileStreamHashChooserFactory(self.lbry_file_manager)),
+            ('lbryfile',
+             ModifyLBRYFileOptionsChooserFactory(self.lbry_file_manager)),
+            ('Stream Downloading',
+             AddStreamFromHashFactory(self.sd_identifier, self.session))
+        ]
+        self.add_control_handlers(handlers)
+        if self.wallet_type == 'lbrycrd':
+            lbrycrd_handlers = [
+                ('Stream Downloading',
+                 AddStreamFromLBRYcrdNameFactory(self.sd_identifier, self.session,
+                                                 self.session.wallet)),
+                ('General',
+                 ClaimNameFactory(self.session.wallet)),
+            ]
+            self.add_control_handlers(lbrycrd_handlers)
+        if self.peer_port is not None:
+            server_handlers = [
+                ('Server',
+                 ShowServerStatusFactory(self)),
+                ('Server',
+                 ModifyServerSettingsFactory(self)),
+            ]
+            self.add_control_handlers(server_handlers)
+
+    def _setup_query_handlers(self):
+        handlers = [
+            #CryptBlobInfoQueryHandlerFactory(self.lbry_file_metadata_manager, self.session.wallet,
+            #                                 self._server_payment_rate_manager),
+            BlobAvailabilityHandlerFactory(self.session.blob_manager),
+            #BlobRequestHandlerFactory(self.session.blob_manager, self.session.wallet,
+            #                          self._server_payment_rate_manager),
+            self.session.wallet.get_wallet_info_query_handler_factory(),
+        ]
+
+        def get_blob_request_handler_factory(rate):
+            self.blob_request_payment_rate_manager = PaymentRateManager(
+                self.session.base_payment_rate_manager, rate
+            )
+            handlers.append(BlobRequestHandlerFactory(self.session.blob_manager, self.session.wallet,
+                                                      self.blob_request_payment_rate_manager))
+
+        d1 = self.settings.get_server_data_payment_rate()
+        d1.addCallback(get_blob_request_handler_factory)
+
+        dl = defer.DeferredList([d1])
+        dl.addCallback(lambda _: self.add_query_handlers(handlers))
+        return dl
+
+    def _load_plugins(self):
+        d = threads.deferToThread(self.plugin_manager.collectPlugins)
+
+        def setup_plugins():
+            ds = []
+            for plugin in self.plugin_manager.getAllPlugins():
+                ds.append(plugin.plugin_object.setup(self))
+            return defer.DeferredList(ds)
+
+        d.addCallback(lambda _: setup_plugins())
+        return d
+
+    def _setup_server(self):
+
+        def restore_running_status(running):
+            if running is True:
+                return self.start_server()
+            return defer.succeed(True)
+
+        dl = self.settings.get_server_running_status()
+        dl.addCallback(restore_running_status)
+        return dl
+
+    def start_server(self):
+
+        if self.peer_port is not None:
+
+            server_factory = ServerProtocolFactory(self.session.rate_limiter,
+                                                   self.query_handlers,
+                                                   self.session.peer_manager)
+            from twisted.internet import reactor
+            self.lbry_server_port = reactor.listenTCP(self.peer_port, server_factory)
+        return defer.succeed(True)
+
+    def stop_server(self):
+        if self.lbry_server_port is not None:
+            self.lbry_server_port, p = None, self.lbry_server_port
+            return defer.maybeDeferred(p.stopListening)
+        else:
+            return defer.succeed(True)
+
+    def _start_controller(self):
+        self.control_class(self.control_handlers)
+        return defer.succeed(True)
+
+    def _shut_down(self):
+        self.plugin_manager = None
+        d1 = self.lbry_file_metadata_manager.stop()
+        d1.addCallback(lambda _: self.lbry_file_manager.stop())
+        d2 = self.stop_server()
+        dl = defer.DeferredList([d1, d2])
+        return dl
+
+
+class StdIOControl():
+    def __init__(self, control_handlers):
+        stdio.StandardIO(ConsoleControl(control_handlers))
+
+
+def launch_lbry_console():
+
+    from twisted.internet import reactor
+
+    parser = argparse.ArgumentParser(description="Launch a lbrynet console")
+    parser.add_argument("--no_listen_peer",
+                        help="Don't listen for incoming data connections.",
+                        action="store_true")
+    parser.add_argument("--peer_port",
+                        help="The port on which the console will listen for incoming data connections.",
+                        type=int, default=3333)
+    parser.add_argument("--no_listen_dht",
+                        help="Don't listen for incoming DHT connections.",
+                        action="store_true")
+    parser.add_argument("--dht_node_port",
+                        help="The port on which the console will listen for DHT connections.",
+                        type=int, default=4444)
+    parser.add_argument("--wallet_type",
+                        help="Either 'lbrycrd' or 'ptc'.",
+                        type=str, default="lbrycrd")
+    parser.add_argument("--lbrycrd_wallet_rpc_port",
+                        help="The rpc port on which the LBRYcrd wallet is listening",
+                        type=int, default=8332)
+    parser.add_argument("--no_dht_bootstrap",
+                        help="Don't try to connect to the DHT",
+                        action="store_true")
+    parser.add_argument("--dht_bootstrap_host",
+                        help="The hostname of a known DHT node, to be used to bootstrap into the DHT. "
+                             "Must be used with --dht_bootstrap_port",
+                        type=str, default='104.236.42.182')
+    parser.add_argument("--dht_bootstrap_port",
+                        help="The port of a known DHT node, to be used to bootstrap into the DHT. Must "
+                             "be used with --dht_bootstrap_host",
+                        type=int, default=4000)
+    parser.add_argument("--use_upnp",
+                        help="Try to use UPnP to enable incoming connections through the firewall",
+                        action="store_true")
+    parser.add_argument("--conf_dir",
+                        help=("The full path to the directory in which to store configuration "
+                              "options and user added plugins. Default: ~/.lbrynet"),
+                        type=str)
+    parser.add_argument("--data_dir",
+                        help=("The full path to the directory in which to store data chunks "
+                              "downloaded from lbrynet. Default: <conf_dir>/blobfiles"),
+                        type=str)
+
+    args = parser.parse_args()
+
+    if args.no_dht_bootstrap:
+        bootstrap_nodes = []
+    else:
+        bootstrap_nodes = [(args.dht_bootstrap_host, args.dht_bootstrap_port)]
+
+    if args.no_listen_peer:
+        peer_port = None
+    else:
+        peer_port = args.peer_port
+
+    if args.no_listen_dht:
+        dht_node_port = None
+    else:
+        dht_node_port = args.dht_node_port
+
+    if not args.conf_dir:
+        conf_dir = os.path.join(os.path.expanduser("~"), ".lbrynet")
+    else:
+        conf_dir = args.conf_dir
+    if not os.path.exists(conf_dir):
+        os.mkdir(conf_dir)
+    if not args.data_dir:
+        data_dir = os.path.join(conf_dir, "blobfiles")
+    else:
+        data_dir = args.data_dir
+    if not os.path.exists(data_dir):
+        os.mkdir(data_dir)
+
+    log_format = "(%(asctime)s)[%(filename)s:%(lineno)s] %(funcName)s(): %(message)s"
+    logging.basicConfig(level=logging.DEBUG, filename=os.path.join(conf_dir, "console.log"),
+                        format=log_format)
+
+    console = LBRYConsole(peer_port, dht_node_port, bootstrap_nodes, StdIOControl, wallet_type=args.wallet_type,
+                          lbrycrd_rpc_port=args.lbrycrd_wallet_rpc_port, use_upnp=args.use_upnp,
+                          conf_dir=conf_dir, data_dir=data_dir)
+
+    d = task.deferLater(reactor, 0, console.start)
+    reactor.addSystemEventTrigger('before', 'shutdown', console.shut_down)
+    reactor.run()

lbrynet/lbrynet_console/LBRYPlugin.py

@@ -0,0 +1,10 @@
+from yapsy.IPlugin import IPlugin
+
+
+class LBRYPlugin(IPlugin):
+
+    def __init__(self):
+        IPlugin.__init__(self)
+
+    def setup(self, lbry_console):
+        raise NotImplementedError

lbrynet/lbrynet_console/LBRYSettings.py

@@ -0,0 +1,116 @@
+import binascii
+import json
+import leveldb
+import logging
+import os
+from twisted.internet import threads, defer
+
+
+class LBRYSettings(object):
+    def __init__(self, db_dir):
+        self.db_dir = db_dir
+        self.db = None
+
+    def start(self):
+        return threads.deferToThread(self._open_db)
+
+    def stop(self):
+        self.db = None
+        return defer.succeed(True)
+
+    def _open_db(self):
+        logging.debug("Opening %s as the settings database", str(os.path.join(self.db_dir, "settings.db")))
+        self.db = leveldb.LevelDB(os.path.join(self.db_dir, "settings.db"))
+
+    def save_lbryid(self, lbryid):
+
+        def save_lbryid():
+            self.db.Put("lbryid", binascii.hexlify(lbryid), sync=True)
+
+        return threads.deferToThread(save_lbryid)
+
+    def get_lbryid(self):
+
+        def get_lbryid():
+            try:
+                return binascii.unhexlify(self.db.Get("lbryid"))
+            except KeyError:
+                return None
+
+        return threads.deferToThread(get_lbryid)
+
+    def get_server_running_status(self):
+
+        def get_status():
+            try:
+                return json.loads(self.db.Get("server_running"))
+            except KeyError:
+                return True
+
+        return threads.deferToThread(get_status)
+
+    def save_server_running_status(self, running):
+
+        def save_status():
+            self.db.Put("server_running", json.dumps(running), sync=True)
+
+        return threads.deferToThread(save_status)
+
+    def get_default_data_payment_rate(self):
+        return self._get_payment_rate("default_data_payment_rate")
+
+    def save_default_data_payment_rate(self, rate):
+        return self._save_payment_rate("default_data_payment_rate", rate)
+
+    def get_server_data_payment_rate(self):
+        return self._get_payment_rate("server_data_payment_rate")
+
+    def save_server_data_payment_rate(self, rate):
+        return self._save_payment_rate("server_data_payment_rate", rate)
+
+    def get_server_crypt_info_payment_rate(self):
+        return self._get_payment_rate("server_crypt_info_payment_rate")
+
+    def save_server_crypt_info_payment_rate(self, rate):
+        return self._save_payment_rate("server_crypt_info_payment_rate", rate)
+
+    def _get_payment_rate(self, rate_type):
+
+        def get_rate():
+            try:
+                return json.loads(self.db.Get(rate_type))
+            except KeyError:
+                return None
+
+        return threads.deferToThread(get_rate)
+
+    def _save_payment_rate(self, rate_type, rate):
+
+        def save_rate():
+            if rate is not None:
+                self.db.Put(rate_type, json.dumps(rate), sync=True)
+            else:
+                self.db.Delete(rate_type, sync=True)
+
+        return threads.deferToThread(save_rate)
+
+    def get_query_handler_status(self, query_identifier):
+
+        def get_status():
+            try:
+                return json.loads(self.db.Get(json.dumps(('q_h', query_identifier))))
+            except KeyError:
+                return True
+
+        return threads.deferToThread(get_status)
+
+    def enable_query_handler(self, query_identifier):
+        return self._set_query_handler_status(query_identifier, True)
+
+    def disable_query_handler(self, query_identifier):
+        return self._set_query_handler_status(query_identifier, False)
+
+    def _set_query_handler_status(self, query_identifier, status):
+        def set_status():
+            self.db.Put(json.dumps(('q_h', query_identifier)), json.dumps(status), sync=True)
+        return threads.deferToThread(set_status)

lbrynet/lbrynet_console/__init__.py

@@ -0,0 +1,8 @@
+"""
+A plugin-enabled console application for interacting with the LBRY network called lbrynet-console.
+
+lbrynet-console can be used to download and upload LBRY Files and includes plugins for streaming
+LBRY Files to an external application and to download unknown chunks of data for the purpose of
+re-uploading them. It gives the user some control over how much will be paid for data and
+metadata and also what types of queries from clients.
+"""

lbrynet/lbrynet_console/interfaces.py

@@ -0,0 +1,14 @@
+from zope.interface import Interface
+
+
+class IControlHandlerFactory(Interface):
+    def get_prompt_description(self):
+        pass
+
+    def get_handler(self):
+        pass
+
+
+class IControlHandler(Interface):
+    def handle_line(self, line):
+        pass

lbrynet/lbrynet_console/plugins/BlindRepeater/BlindBlobHandler.py

@@ -0,0 +1,15 @@
+from zope.interface import implements
+from lbrynet.interfaces import IBlobHandler
+from twisted.internet import defer
+
+
+class BlindBlobHandler(object):
+    implements(IBlobHandler)
+
+    def __init__(self):
+        pass
+
+    ######### IBlobHandler #########
+
+    def handle_blob(self, blob, blob_info):
+        return defer.succeed(True)

lbrynet/lbrynet_console/plugins/BlindRepeater/BlindInfoManager.py

@@ -0,0 +1,62 @@
+from twisted.internet import threads, defer
+from ValuableBlobInfo import ValuableBlobInfo
+from db_keys import BLOB_INFO_TYPE
+import json
+import leveldb
+
+
+class BlindInfoManager(object):
+
+    def __init__(self, db, peer_manager):
+        self.db = db
+        self.peer_manager = peer_manager
+
+    def setup(self):
+        return defer.succeed(True)
+
+    def stop(self):
+        self.db = None
+        return defer.succeed(True)
+
+    def get_all_blob_infos(self):
+        d = threads.deferToThread(self._get_all_blob_infos)
+
+        def make_blob_infos(blob_data):
+            blob_infos = []
+            for blob in blob_data:
+                blob_hash, length, reference, peer_host, peer_port, peer_score = blob
+                peer = self.peer_manager.get_peer(peer_host, peer_port)
+                blob_info = ValuableBlobInfo(blob_hash, length, reference, peer, peer_score)
+                blob_infos.append(blob_info)
+            return blob_infos
+        d.addCallback(make_blob_infos)
+        return d
+
+    def save_blob_infos(self, blob_infos):
+        blobs = []
+        for blob_info in blob_infos:
+            blob_hash = blob_info.blob_hash
+            length = blob_info.length
+            reference = blob_info.reference
+            peer_host = blob_info.peer.host
+            peer_port = blob_info.peer.port
+            peer_score = blob_info.peer_score
+            blobs.append((blob_hash, length, reference, peer_host, peer_port, peer_score))
+        return threads.deferToThread(self._save_blob_infos, blobs)
+
+    def _get_all_blob_infos(self):
+        blob_infos = []
+        for key, blob_info in self.db.RangeIter():
+            key_type, blob_hash = json.loads(key)
+            if key_type == BLOB_INFO_TYPE:
+                blob_infos.append([blob_hash] + json.loads(blob_info))
+        return blob_infos
+
+    def _save_blob_infos(self, blobs):
+        batch = leveldb.WriteBatch()
+        for blob in blobs:
+            try:
+                self.db.Get(json.dumps((BLOB_INFO_TYPE, blob[0])))
+            except KeyError:
+                batch.Put(json.dumps((BLOB_INFO_TYPE, blob[0])), json.dumps(blob[1:]))
+        self.db.Write(batch, sync=True)