| .github/workflows | ||
| db | ||
| docker | ||
| internal | ||
| meta | ||
| protobuf | ||
| scripts | ||
| server | ||
| testdata | ||
| .gitignore | ||
| action.yml | ||
| diagram.png | ||
| docker-compose-hub-server.yml | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| main.go | ||
| readme.md | ||
| requirements.txt | ||
| signal.go | ||
| signalsigterm.go | ||
| version.txt | ||
LBRY Herald
herald.go is a not yet feature complete go rewrite of the existing implementation in python. A herald server provides back-end services to LBRY clients. Services include
- URL resolution
- search
- hub federation and discovery
This project will eventually subsume and replace the herald and the lighthouse search provider.
Installation
No install instructions yet. See Contributing below.
Usage
Prerequisite: run python block processor and search plugin
Follow the instructions here.
Run this hub
./herald serve
# run with remote services disabled so it can run completely solo
./herald serve --disable-rocksdb-refresh --disable-load-peers --disable-resolve --disable-es --disable-blocking-and-filtering
Search for stuff
./herald search text goes here
Contributing
Contributions to this project are welcome, encouraged, and compensated. Details here.
Dev Dependencies
Install Go 1.18+
- Ubuntu:
sudo snap install go - OSX:
brew install go - Windows https://golang.org/doc/install
Download protoc from https://github.com/protocolbuffers/protobuf/releases and make sure it is
executable and in your path.
Install Go plugin for protoc and python:
go get google.golang.org/protobuf/cmd/protoc-gen-go google.golang.org/grpc/cmd/protoc-gen-go-grpc
pip install grpcio grpcio-tools github3.py
Lastly the hub needs protobuf version 3.17.1, it may work with newer version but this is what it's built with, on ubuntu systems you'll have to install this from source see the GitHub actions in .github/workflows for an example of this.
Install rocksdb and dependencies your CGO flags, on ubuntu. We use v6.29.5 for feature and statis build support.
sudo apt-get install libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev libzstd-dev liblz4-dev
wget https://github.com/facebook/rocksdb/archive/refs/tags/v6.29.5.tar.gz
tar xfzv rocksdb-6.29.5.tar.gz
cd rocksdb-6.29.5
make static_lib
sudo make install
export CGO_CFLAGS="-I/usr/local/lib"
export CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd -lbz2"
https://github.com/protocolbuffers/protobuf/releases/download/v3.17.1/protobuf-all-3.17.1.tar.gz
If you can run ./protobuf/build.sh without errors, you have go and protoc installed correctly.
On Linux you probably need to instead the open file limits
ulimit -n 1000000
sysctl -w fs.file-max=1000000
and /etc/security/limits.conf or /etc/sysctl.conf change:
fs.file-max = 1000000
Finally, run the block processor as described under Usage.
Running from Source
Run ./dev.sh to start the hub. The script will restart the hub as you make changes to *.go files.
To search, use go run . search text goes here.
Windows
reflex doesn't work on windows, so you'll need to run go run . serve and restart manually as you make changes.
License
This project is MIT licensed. For the full license, see LICENSE.
Security
We take security seriously. Please contact security@lbry.com regarding any security issues. Our PGP key is here if you need it.
Contact
The primary contact for this project is @lyoshenka (grin@lbry.com).