2022-07-11 19:31:27 +02:00
# LBRY Herald
2021-03-18 21:12:48 +01:00
2022-08-02 17:53:23 +02:00
herald.go is a not yet feature complete go rewrite of the [existing implementation in python ](https://github.com/lbryio/hub/tree/master/hub/herald ). A herald server provides back-end services to LBRY clients. Services include
2021-03-18 21:12:48 +01:00
- URL resolution
- search
- hub federation and discovery
2021-03-19 20:10:32 +01:00
This project will eventually subsume and replace the
2022-08-02 17:53:23 +02:00
[herald ](https://github.com/lbryio/hub/blob/master/docs/docker_examples/hub-compose.yml#L38 )
2021-03-18 21:12:48 +01:00
and the [lighthouse ](https://github.com/lbryio/lighthouse ) search provider.
2022-08-10 17:05:39 +02:00
2021-03-19 20:10:32 +01:00
2021-03-18 21:12:48 +01:00
## Installation
2021-04-21 22:19:05 +02:00
No install instructions yet. See Contributing below.
2021-03-18 21:12:48 +01:00
## Usage
2021-04-21 22:19:05 +02:00
### Prerequisite: run python block processor and search plugin
2021-04-19 21:25:34 +02:00
Follow the instructions [here ](https://lbry.tech/resources/wallet-server ).
### Run this hub
```bash
2022-07-11 19:31:27 +02:00
./herald serve
```
```bash
# run with remote services disabled so it can run completely solo
./herald serve --disable-rocksdb-refresh --disable-load-peers --disable-resolve --disable-es --disable-blocking-and-filtering
2021-04-19 21:25:34 +02:00
```
### Search for stuff
```bash
2022-07-11 19:31:27 +02:00
./herald search text goes here
2021-04-19 21:25:34 +02:00
```
2021-03-18 21:12:48 +01:00
## Contributing
2021-04-21 22:19:05 +02:00
Contributions to this project are welcome, encouraged, and compensated. Details [here ](https://lbry.tech/contribute ).
2021-03-18 21:12:48 +01:00
2021-03-18 22:14:56 +01:00
### Dev Dependencies
2022-07-11 19:31:27 +02:00
Install Go 1.18+
2021-03-18 22:14:56 +01:00
2022-07-11 19:31:27 +02:00
- Ubuntu: `sudo snap install go`
2021-03-18 22:14:56 +01:00
- OSX: `brew install go`
2021-05-13 18:02:03 +02:00
- Windows https://golang.org/doc/install
2021-03-18 22:14:56 +01:00
2021-03-19 20:10:32 +01:00
Download `protoc` from https://github.com/protocolbuffers/protobuf/releases and make sure it is
2021-03-18 22:14:56 +01:00
executable and in your path.
2021-07-06 02:20:38 +02:00
Install Go plugin for protoc and python:
2021-03-18 22:14:56 +01:00
```
go get google.golang.org/protobuf/cmd/protoc-gen-go google.golang.org/grpc/cmd/protoc-gen-go-grpc
2021-07-06 02:20:38 +02:00
pip install grpcio grpcio-tools github3.py
```
Lastly the hub needs protobuf version 3.17.1, it may work with newer version but this is what it's built with, on ubuntu systems you'll have to install this from source see the GitHub actions in `.github/workflows` for an example of this.
2022-07-11 19:31:27 +02:00
Install rocksdb and dependencies your CGO flags, on ubuntu. We use v6.29.5 for feature and statis build support.
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
```
sudo apt-get install libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev libzstd-dev liblz4-dev
2022-07-11 19:31:27 +02:00
wget https://github.com/facebook/rocksdb/archive/refs/tags/v6.29.5.tar.gz
tar xfzv rocksdb-6.29.5.tar.gz
cd rocksdb-6.29.5
make static_lib
sudo make install
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
```
2021-07-06 02:20:38 +02:00
```
https://github.com/protocolbuffers/protobuf/releases/download/v3.17.1/protobuf-all-3.17.1.tar.gz
2021-03-18 22:14:56 +01:00
```
2021-04-21 22:19:05 +02:00
If you can run `./protobuf/build.sh` without errors, you have `go` and `protoc` installed correctly.
Finally, run the block processor as described under Usage.
### Running from Source
Run `./dev.sh` to start the hub. The script will restart the hub as you make changes to `*.go` files.
To search, use `go run . search text goes here` .
2021-03-18 22:14:56 +01:00
2021-05-13 18:02:03 +02:00
#### Windows
reflex doesn't work on windows, so you'll need to run `go run . serve` and restart manually as you make changes.
2021-03-18 21:12:48 +01:00
## License
This project is MIT licensed. For the full license, see [LICENSE ](LICENSE ).
## Security
2021-05-04 00:04:03 +02:00
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our PGP key is here ](https://lbry.com/faq/pgp-key ) if you need it.
2021-03-18 21:12:48 +01:00
## Contact
2021-03-19 20:10:32 +01:00
The primary contact for this project is [@lyoshenka ](https://github.com/lyoshenka ) ([grin@lbry.com](mailto:grin@lbry.com)).
