LBRYCommunity/hub
1
0
Fork 0
Code Issues 35 Pull requests 6 Projects Releases Packages Wiki Activity

Update README.md

Browse source
This commit is contained in:
Jack Robison 2022-05-18 16:31:52 -04:00 committed by GitHub
parent 9e9c778edd
commit 36c1471dcf
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 26 additions and 26 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

52
README.md
View file

@ -1,33 +1,33 @@
## Scribe ## LBRY Hub
Scribe is a python library for building services that use the processed data from the [LBRY blockchain](https://github.com/lbryio/lbrycrd) in an ongoing manner. Scribe contains a set of three core executable services that are used together: This repo provides a python library, `hub`, for building services that use the processed data from the [LBRY blockchain](https://github.com/lbryio/lbrycrd) in an ongoing manner. Hub contains a set of three core executable services that are used together:
* `scribe` ([scribe.blockchain.service](https://github.com/lbryio/scribe/tree/master/scribe/blockchain/service.py)) - maintains a [rocksdb](https://github.com/lbryio/lbry-rocksdb) database containing the LBRY blockchain. * `scribe` ([hub.scribe.service](https://github.com/lbryio/lbry-hub/tree/master/hub/service.py)) - maintains a [rocksdb](https://github.com/lbryio/lbry-rocksdb) database containing the LBRY blockchain.
* `scribe-hub` ([scribe.hub.service](https://github.com/lbryio/scribe/tree/master/scribe/hub/service.py)) - an electrum server for thin-wallet clients (such as [lbry-sdk](https://github.com/lbryio/lbry-sdk)), provides an api for clients to use thin simple-payment-verification (spv) wallets and to resolve and search claims published to the LBRY blockchain. * `herald` ([hub.herald.service](https://github.com/lbryio/lbry-hub/tree/master/hub/herald/service.py)) - an electrum server for thin-wallet clients (such as [lbry-sdk](https://github.com/lbryio/lbry-sdk)), provides an api for clients to use thin simple-payment-verification (spv) wallets and to resolve and search claims published to the LBRY blockchain.
* `scribe-elastic-sync` ([scribe.elasticsearch.service](https://github.com/lbryio/scribe/tree/master/scribe/elasticsearch/service.py)) - a utility to maintain an elasticsearch database of metadata for claims in the LBRY blockchain * `scribe-elastic-sync` ([hub.elastic_sync.service](https://github.com/lbryio/lbry-hub/tree/master/hub/elastic_sync/service.py)) - a utility to maintain an elasticsearch database of metadata for claims in the LBRY blockchain
Features and overview of scribe as a python library: Features and overview of `hub` as a python library:
* Uses Python 3.7-3.9 (3.10 probably works but hasn't yet been tested) * Uses Python 3.7-3.9 (3.10 probably works but hasn't yet been tested)
* An interface developers may implement in order to build their own applications able to receive up-to-date blockchain data in an ongoing manner ([scribe.service.BlockchainReaderService](https://github.com/lbryio/scribe/tree/master/scribe/service.py)) * An interface developers may implement in order to build their own applications able to receive up-to-date blockchain data in an ongoing manner ([hub.service.BlockchainReaderService](https://github.com/lbryio/lbry-hub/tree/master/hub/service.py))
* Protobuf schema for encoding and decoding metadata stored on the blockchain ([scribe.schema](https://github.com/lbryio/scribe/tree/master/scribe/schema)) * Protobuf schema for encoding and decoding metadata stored on the blockchain ([hub.schema](https://github.com/lbryio/lbry-hub/tree/master/hub/schema))
* [Rocksdb 6.25.3](https://github.com/lbryio/lbry-rocksdb/) based database containing the blockchain data ([scribe.db](https://github.com/lbryio/scribe/tree/master/scribe/db)) * [Rocksdb 6.25.3](https://github.com/lbryio/lbry-rocksdb/) based database containing the blockchain data ([hub.db](https://github.com/lbryio/lbry-hub/tree/master/hub/db))
* [A community driven performant trending algorithm](https://raw.githubusercontent.com/lbryio/scribe/master/scribe/elasticsearch/trending%20algorithm.pdf) for searching claims ([code](https://github.com/lbryio/scribe/blob/master/scribe/elasticsearch/fast_ar_trending.py)) * [A community driven performant trending algorithm](https://raw.githubusercontent.com/lbryio/lbry-hub/master/docs/trending%20algorithm.pdf) for searching claims ([code](https://github.com/lbryio/lbry-hub/blob/master/hub/elastic_sync/fast_ar_trending.py))
## Installation ## Installation
Scribe may be run from source, a binary, or a docker image. Scribe may be run from source, a binary, or a docker image.
Our [releases page](https://github.com/lbryio/scribe/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS and Debian-based Linux. Our [releases page](https://github.com/lbryio/lbry-hub/releases) contains pre-built binaries of the latest release, pre-releases, and past releases for macOS and Debian-based Linux.
Prebuilt [docker images](https://hub.docker.com/r/lbry/scribe/latest-release) are also available. Prebuilt [docker images](https://hub.docker.com/r/lbry/lbry-hub/latest-release) are also available.
### Prebuilt docker image ### Prebuilt docker image
`docker pull lbry/scribe:latest-release` `docker pull lbry/lbry-hub:latest-release`
### Build your own docker image ### Build your own docker image
``` ```
git clone https://github.com/lbryio/scribe.git git clone https://github.com/lbryio/lbry-hub.git
cd scribe cd lbry-hub
docker build -f ./docker/Dockerfile.scribe -t lbry/scribe:development . docker build -t lbry/lbry-hub:development .
``` ```
### Install from source ### Install from source
@ -36,22 +36,22 @@ Scribe has been tested with python 3.7-3.9. Higher versions probably work but ha
1. clone the scribe repo 1. clone the scribe repo
``` ```
git clone https://github.com/lbryio/scribe.git git clone https://github.com/lbryio/lbry-hub.git
cd scribe cd lbry-hub
``` ```
2. make a virtual env 2. make a virtual env
``` ```
python3.9 -m venv scribe-venv python3.9 -m venv lbry-hub-venv
``` ```
3. from the virtual env, install scribe 3. from the virtual env, install scribe
``` ```
source scribe-venv/bin/activate source lbry-hub-venv/bin/activate
pip install -e . pip install -e .
``` ```
That completes the installation, now you should have the commands `scribe`, `scribe-elastic-sync` and `scribe-hub` That completes the installation, now you should have the commands `scribe`, `scribe-elastic-sync` and `herald`
These can also optionally be run with `python -m scribe.blockchain`, `python -m scribe.elasticsearch`, and `python -m scribe.hub` These can also optionally be run with `python -m hub.scribe`, `python -m hub.elastic_sync`, and `python -m hub.herald`
## Usage ## Usage
@ -76,13 +76,13 @@ If you have the resources to run all of the services on one machine (at least 30
For various reasons it may be desirable to block or filtering content from claim search and resolve results, [here](https://github.com/lbryio/scribe/blob/master/blocking.md) are instructions for how to configure and use this feature as well as information about the recommended defaults. For various reasons it may be desirable to block or filtering content from claim search and resolve results, [here](https://github.com/lbryio/scribe/blob/master/blocking.md) are instructions for how to configure and use this feature as well as information about the recommended defaults.
#### Common options across `scribe`, `scribe-hub`, and `scribe-elastic-sync`: #### Common options across `scribe`, `herald`, and `scribe-elastic-sync`:
- `--db_dir` (required) Path of the directory containing lbry-rocksdb, set from the environment with `DB_DIRECTORY` - `--db_dir` (required) Path of the directory containing lbry-rocksdb, set from the environment with `DB_DIRECTORY`
- `--daemon_url` (required for `scribe` and `scribe-hub`) URL for rpc from lbrycrd or lbcd<rpcuser>:<rpcpassword>@<lbrycrd rpc ip><lbrycrd rpc port>. - `--daemon_url` (required for `scribe` and `herald`) URL for rpc from lbrycrd or lbcd<rpcuser>:<rpcpassword>@<lbrycrd rpc ip><lbrycrd rpc port>.
- `--reorg_limit` Max reorg depth, defaults to 200, set from the environment with `REORG_LIMIT`. - `--reorg_limit` Max reorg depth, defaults to 200, set from the environment with `REORG_LIMIT`.
- `--chain` With blockchain to use - either `mainnet`, `testnet`, or `regtest` - set from the environment with `NET` - `--chain` With blockchain to use - either `mainnet`, `testnet`, or `regtest` - set from the environment with `NET`
- `--max_query_workers` Size of the thread pool, set from the environment with `MAX_QUERY_WORKERS` - `--max_query_workers` Size of the thread pool, set from the environment with `MAX_QUERY_WORKERS`
- `--cache_all_tx_hashes` If this flag is set, all tx hashes will be stored in memory. For `scribe`, this speeds up the rate it can apply blocks as well as process mempool. For `scribe-hub`, this will speed up syncing address histories. This setting will use 10+g of memory. It can be set from the environment with `CACHE_ALL_TX_HASHES=Yes` - `--cache_all_tx_hashes` If this flag is set, all tx hashes will be stored in memory. For `scribe`, this speeds up the rate it can apply blocks as well as process mempool. For `herald`, this will speed up syncing address histories. This setting will use 10+g of memory. It can be set from the environment with `CACHE_ALL_TX_HASHES=Yes`
- `--cache_all_claim_txos` If this flag is set, all claim txos will be indexed in memory. Set from the environment with `CACHE_ALL_CLAIM_TXOS=Yes` - `--cache_all_claim_txos` If this flag is set, all claim txos will be indexed in memory. Set from the environment with `CACHE_ALL_CLAIM_TXOS=Yes`
- `--prometheus_port` If provided this port will be used to provide prometheus metrics, set from the environment with `PROMETHEUS_PORT` - `--prometheus_port` If provided this port will be used to provide prometheus metrics, set from the environment with `PROMETHEUS_PORT`
@ -94,7 +94,7 @@ For various reasons it may be desirable to block or filtering content from claim
#### Options for `scribe-elastic-sync` #### Options for `scribe-elastic-sync`
- `--reindex` If this flag is set drop and rebuild the elasticsearch index. - `--reindex` If this flag is set drop and rebuild the elasticsearch index.
#### Options for `scribe-hub` #### Options for `herald`
- `--host` Interface for server to listen on, use 0.0.0.0 to listen on the external interface. Can be set from the environment with `HOST` - `--host` Interface for server to listen on, use 0.0.0.0 to listen on the external interface. Can be set from the environment with `HOST`
- `--tcp_port` Electrum TCP port to listen on for hub server. Can be set from the environment with `TCP_PORT` - `--tcp_port` Electrum TCP port to listen on for hub server. Can be set from the environment with `TCP_PORT`
- `--udp_port` UDP port to listen on for hub server. Can be set from the environment with `UDP_PORT` - `--udp_port` UDP port to listen on for hub server. Can be set from the environment with `UDP_PORT`