Commit graph

42 commits

Author SHA1 Message Date
Jonathan Moody 8fb3db8136
Add subscribe/unsubscribe RPCs. Add session, sessionManager, and serve JSON RPC (without HTTP). (#66)
* Move and rename BlockchainCodec, BlockchainCodecRequest.
These are not specifically "blockchain", rather they are
specific to how gorilla/rpc works.

* Move claimtrie-related service/handlers to jsonrpc_claimtrie.go.

* Pull out decode logic into named func newBlockHeaderElectrum().

* Rename BlockchainService -> BlockchainBlockService.

* Drop http.Request arg from handlers, and use RegisterTCPService().

* Implement GetStatus() to pull data from HashXStatus table.

* Make the service objects independent, so we don't have inheritance.

* Add core session/subscription logic (session.go).
Implement subsribe/unsubscribe handlers.

* Support both pure JSON and JSON-over-HTTP services.
Forward NotifierChan messages to sessionManager.

* Only assign default port (50001) if neither --json-rpc-port nor
--json-rpc-http-port are specified.

* Handle failures with goto instead of break. Update error logging.

* Add --max-sessions, --session-timeout args. Enforce max sessions.

* Changes to make session.go testable. Conn created with Pipe()
used in testing has no unique Addr.

* Add tests for headers, headers.subscribe, address.subscribe.

* HashXStatus, HashXMempoolStatus not populated by default. Fix GetStatus().

* Use time.Ticker object to drive management activity.
2022-10-04 17:05:06 +03:00
Jonathan Moody f5b8f2ce0d Add --chain=X argument, and use it to determine chain when DB is empty. 2022-09-07 14:32:34 -05:00
Jonathan Moody 50f7e91ead Infer chain (mainnet, testnet3, regtest) based on DBStateValue.
Correct typo DDVersion -> DBVersion. Misc logging improvements.
2022-09-07 14:32:34 -05:00
Jonathan Moody fe18c70bf7 Add some blockchain RPC handlers and database fetching routines. 2022-09-07 14:32:20 -05:00
Jeffrey Picard 9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jeffrey Picard b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jeffrey Picard b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00
Jeffrey Picard 28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard 2a1d6fa7d4 Updates based on code review 2021-12-06 11:32:45 -05:00
Jeffrey Picard 2e52c1639c Refactor and fixes related to PR comments. 2021-12-06 11:32:45 -05:00
Jeffrey Picard 283686ecac Fix startup flags 2021-12-06 11:32:45 -05:00
Jeffrey Picard 72ea236d86 Run gofmt 2021-12-06 11:31:51 -05:00
Jeffrey Picard 355eab682c changes based on comments 2021-12-06 11:26:29 -05:00
Jeffrey Picard 395e1db489 UDPServer / ip address resolution
Got the UDPServer ping/pong protocol working internally, only tested
against other running go hub servers. Should in theory work with
python server and clients, but still need to test that.

Also switched to serving udp on the same port as grpc, and taking that
into account when pinging other hubs with udp.

Unit test for udp ip address lookup.
2021-12-06 11:26:29 -05:00
Jeffrey Picard 15614c6a5b Remove unused code 2021-11-05 11:11:26 -04:00
Jeffrey Picard c61c8db92a Some cleanup based on review and use map + sync.RWMutex instead of
sync.Map
2021-11-04 20:57:23 -04:00
Jeffrey Picard 914d2bfc61 Rework federation to be synchronous, and add tests 2021-10-30 01:33:55 -04:00
Jeffrey Picard a67f9b43d1 Remove unused streaming endpoints 2021-10-29 17:27:27 -04:00
Jeffrey Picard 5387aeeebe Most of federation is written, need to finish udp and test
Cleanup, more reorg, more arguments, started adding tests
Comments and another test
Simplify writing of peers and add unit test
2021-10-29 16:56:36 -04:00
Jeffrey Picard 45e9817ced Cleanup metrics
Consolidated error metrics into a vector counter with a label,
removed other uneeded counters.
2021-10-29 16:56:35 -04:00
Jeffrey Picard a27b6e730d Record metrics
There are a couple metrics that try to mirror the ones from python,
many of the python ones don't apply, and vice-versa.
2021-10-29 16:56:35 -04:00
Jeffrey Picard a6d47e662a Don't swallow error, cachettl and refresh delta as params 2021-10-04 18:04:26 -04:00
Jeffrey Picard 0d59480f3c bug fixes and cache purging 2021-10-02 22:49:49 -04:00
Jeffrey Picard 36b4a3cdd9 implement caching 2021-10-02 17:56:12 -04:00
Jeffrey Picard d99f993c21 Fix versioning 2021-09-24 16:24:22 -04:00
Jeffrey Picard 4f7d3b20c8 cleanup and formatting 2021-09-21 14:02:09 -04:00
Jeffrey Picard fdbd7ec509 proto fixes and cleanup 2021-09-18 13:36:49 -04:00
Jeffrey Picard 81c3de3bfd Merge remote-tracking branch 'origin/master' into prometheus-and-versioning 2021-09-18 13:19:47 -04:00
Jeffrey Picard 9c89907bb8 versioning and metrics 2021-09-18 12:06:56 -04:00
Ivan Hranat 64870c54bc linitng 2021-08-24 11:45:30 +03:00
Victor Shyba 8a2a98726d initialize the server when making it instead of during request 2021-08-13 19:16:30 -03:00
Victor Shyba b4782ce6ac add --debug to show queries and error details 2021-08-13 18:36:43 -03:00
Victor Shyba d2d17bee3b specify index, remove --dev 2021-08-11 00:39:37 -03:00
Jeffrey Picard 3f64655a26 Updates from review feedback 2021-06-18 00:01:47 -04:00
Jeffrey Picard 4280edc465 cleanup and updates 2021-06-09 20:04:06 -04:00
Jeffrey Picard 944496b076 Add dockerfile and docker-compose with es 2021-06-04 12:38:17 -04:00
Jeffrey Picard b557bf8237 Implementing url resolving server side 2021-06-04 01:56:50 -04:00
Jeffrey Picard 8a995557f7 More fixes / supported fields 2021-05-31 22:19:10 -04:00
Jeffrey Picard 0b57dceabd Add source and normalization, remove authentication 2021-05-30 21:34:57 -04:00
Jeffrey Picard 092495b2de Handle name param and change flag parsing. 2021-05-24 23:28:43 -04:00
Jeffrey Picard 4ca5f168d9 Added port, username and password options to client and server. 2021-05-24 20:09:28 -04:00
Alex Grintsvayg 3ca27f3740
very basic elastic search works 2021-04-19 15:28:57 -04:00
Renamed from server.go (Browse further)