Commit graph

109 commits

Author SHA1 Message Date
Jonathan Moody
d2193e980a
blockchain.transaction.broadcast implementation (#80)
* Generate secondary hashXNotification(s) on every headerNotification.

* Attempt LBCD connection with rpc.Client.

* Optional --daemon-url.

* Correct HashXStatusKey field. Should be HASHX_LEN.

* Connect to lbcd using lbcd/rpcclient library.

* Handle deprecation of node.js 12 actions.

* Add --daemon-ca-path argument and establish HTTPS connection if specified.

* Remove dead code. Tighten definition of TransactionBroadcastReq.

* Correct default daemon URL.
2022-12-16 13:54:19 -06:00
Jeffrey Picard
711c4b4b7b
WIP: Json rpc federation, search/getclaimbyid, and shutdown (#76)
* Cleanup shutdown and peers subscribe

this has errors currently, need to figure out data race

* fixed data race finally....

* getclaimbyid and search

* hook jsonrpc peers subcribe into current federation

* cleanup some peers stuff

* remove commented code

* tie into session manager in jsonrpc peers subscribe

* use int for port args

* cleanup and fix a bunch of compiler warnings

* use logrus everywhere

* cleanup test
2022-12-07 11:01:36 -05:00
Jonathan Moody
317cdf7129
WIP: blockchain.transaction.yyy JSON RPC implementations (#78)
* Partial blockchain.transaction.yyy RPC implementations.

* Register RPC service object.

* Move session manager start/stop to a better place.

* Attempt to fill in the details of transaction.get_batch,
including merkle path.

* Correct interpretation of DBStateValue Genesis hash.

* Convert Args.Port to int and validate. Run UDP ping server on JSONRPCPort too.

* Add BlockHeader to HeightHash notification.

* Limit session-based JSON RPC service to IPv4.
Client not ready for IPv6.

* Adapt to new HeightHash struct.

* Fine tune JSON RPC handlers and types to match lbry-sdk expectations.
Implement UnmarshalJSON()/MarshalJSON() for several types.

* Add more special handling of DBStateValue.Genesis hash.

* Set IncludeStop=false generally to avoid returning extra rows.
Other misc fixes.
2022-12-06 16:14:28 -05:00
Jonathan Moody
e070e8a51e
JSON RPC compatibility workarounds to support lbry-sdk (#75)
* Fix log spam (Already shutting down...)

* Workaround to allow lbry-sdk to call server.version and server.features.
Incoming/outgoing JSON is patched using yet another codec (jsonPatchingCodec).
Add more logging of raw/patched JSON.

* Elaborate comment on jsonPatchingCodec.
2022-10-29 18:42:24 +03:00
Jeffrey Picard
cd7f20a461
Server endpoints goroutine refactor (#69)
* server.xxx endpoints

Additional server endpoints in jsonrpc and also some refactoring

* server.banner

* more endpoints

* use lbry.go stop pattern

* set genesis hash properly

* updates and refactors

* remove shutdowncalled and itmut

* remove OpenIterators

* remove shutdown and done channels from db and use stop group

* currently broken, incorporated stop groups into the session manager

* set the rest of the default args for tests

* add default json rpc http port and cleanup

* tests for new jsonrpc endpoints

* cleanup and add manage goroutine to stopper pattern

* cleanup

* NewDebug

* asdf

* fix
2022-10-25 08:48:13 +03:00
Jeffrey Picard
537b8c7ddd
integration testing scripts (#64)
* integration testing scripts

some scripts for integration testing and a docker file for an action.
Still need to figure out how to properly run a more realistic version
in ci.

* update

* changes

* db shutdown racecondition fix

* changes per pr

* changes per code review

* fix testing

* add shutdowncalled bool to db
2022-10-04 20:25:44 +03:00
Jonathan Moody
8fb3db8136
Add subscribe/unsubscribe RPCs. Add session, sessionManager, and serve JSON RPC (without HTTP). (#66)
* Move and rename BlockchainCodec, BlockchainCodecRequest.
These are not specifically "blockchain", rather they are
specific to how gorilla/rpc works.

* Move claimtrie-related service/handlers to jsonrpc_claimtrie.go.

* Pull out decode logic into named func newBlockHeaderElectrum().

* Rename BlockchainService -> BlockchainBlockService.

* Drop http.Request arg from handlers, and use RegisterTCPService().

* Implement GetStatus() to pull data from HashXStatus table.

* Make the service objects independent, so we don't have inheritance.

* Add core session/subscription logic (session.go).
Implement subsribe/unsubscribe handlers.

* Support both pure JSON and JSON-over-HTTP services.
Forward NotifierChan messages to sessionManager.

* Only assign default port (50001) if neither --json-rpc-port nor
--json-rpc-http-port are specified.

* Handle failures with goto instead of break. Update error logging.

* Add --max-sessions, --session-timeout args. Enforce max sessions.

* Changes to make session.go testable. Conn created with Pipe()
used in testing has no unique Addr.

* Add tests for headers, headers.subscribe, address.subscribe.

* HashXStatus, HashXMempoolStatus not populated by default. Fix GetStatus().

* Use time.Ticker object to drive management activity.
2022-10-04 17:05:06 +03:00
Jonathan Moody
86a287ec69 Skip TestUDPPing if production server is down. 2022-09-13 15:35:54 -05:00
Jonathan Moody
891f63fb5c Update jsonrpc_blockchain_tests.go for new sample data. 2022-09-13 15:33:24 -05:00
Jonathan Moody
789974227f Add sample data from test_variety_of_transactions_and_longish_history.
Rework tests to use the sample data.
2022-09-13 15:05:11 -05:00
Jonathan Moody
71e79c553e One more RPC (get_server_height), and update comment
to include full RPC name.
2022-09-08 13:17:52 -05:00
Jonathan Moody
b298454727 Fix RPC handler registration and BlockGetChunkResp name. 2022-09-08 11:50:06 -05:00
Jonathan Moody
8c8871b4d2 Register blockchain.* handlers in jsonrpc_service.go. 2022-09-07 15:01:47 -05:00
Jonathan Moody
20e32437e9 Rename blockchain.go -> jsonrpc_blockchain.go. 2022-09-07 14:32:34 -05:00
Jonathan Moody
d0d6145f9d Refactor blockchain.go handlers to be compatible with gorilla/rpc.
Add speculative BlockchainCodecRequest which might handle rewriting
incoming method names.
2022-09-07 14:32:34 -05:00
Jonathan Moody
90afae7cd5 Add scripthash variants of RPC handlers. 2022-09-07 14:32:34 -05:00
Jonathan Moody
f5b8f2ce0d Add --chain=X argument, and use it to determine chain when DB is empty. 2022-09-07 14:32:34 -05:00
Jonathan Moody
321bcf6420 Fix encoding of TX and Block hashes in response. 2022-09-07 14:32:34 -05:00
Jonathan Moody
293a3f685e Fix some DB logic and add tests (blockchain_test.go). 2022-09-07 14:32:34 -05:00
Jonathan Moody
f55a5ed777 Continuing blockchain RPC handler work. Add JSON tags.
Fetch Height using TxCounts.
2022-09-07 14:32:34 -05:00
Jonathan Moody
50f7e91ead Infer chain (mainnet, testnet3, regtest) based on DBStateValue.
Correct typo DDVersion -> DBVersion. Misc logging improvements.
2022-09-07 14:32:34 -05:00
Jonathan Moody
fe18c70bf7 Add some blockchain RPC handlers and database fetching routines. 2022-09-07 14:32:20 -05:00
Jeffrey Picard
9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jeffrey Picard
b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jonathan Moody
49e5d7b8c1
Fix release_time query to match python hub. (#41)
* Revise error handling and cleanup in GetProdDB(). Avoids panic when DB not found.

* Fix release_time query to match python hub. Claims with unset release_time match the query.
2022-08-03 19:19:25 +03:00
Jeffrey Picard
b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00
Jeffrey Picard
28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard
d51f5ca064 Fix behavior when the index isn't there.
One of the tests expects an empty list returned, we're currently
returning an error.
2022-01-05 10:07:15 -05:00
Victor Shyba
8e99bc56d0
add sd_hash prefix search (#30)
* add sd_hash to search request

* search sd_hash by prefix
2021-12-17 09:37:32 -05:00
Jeffrey Picard
2a1d6fa7d4 Updates based on code review 2021-12-06 11:32:45 -05:00
Jeffrey Picard
2e52c1639c Refactor and fixes related to PR comments. 2021-12-06 11:32:45 -05:00
Jeffrey Picard
1c1d288654 Set UDP flags correctly. 2021-12-06 11:32:45 -05:00
Jeffrey Picard
283686ecac Fix startup flags 2021-12-06 11:32:45 -05:00
Jeffrey Picard
ca0e8562f3 Add disable-federation flag, default false 2021-12-06 11:32:45 -05:00
Jeffrey Picard
72ea236d86 Run gofmt 2021-12-06 11:31:51 -05:00
Jeffrey Picard
0a1ba43d66 Full SPVPong parsing and tests against prod server. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
4e6b47c2a3 Test UDPPing country parsing. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
ee5fcaef14 Added test for getting IP with udp from prod servers. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
159f4b941b Add claim protobuf definition for UDP country encoding. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
355eab682c changes based on comments 2021-12-06 11:26:29 -05:00
Jeffrey Picard
395e1db489 UDPServer / ip address resolution
Got the UDPServer ping/pong protocol working internally, only tested
against other running go hub servers. Should in theory work with
python server and clients, but still need to test that.

Also switched to serving udp on the same port as grpc, and taking that
into account when pinging other hubs with udp.

Unit test for udp ip address lookup.
2021-12-06 11:26:29 -05:00
Jeffrey Picard
d511b08736 Switch RangeField to int32 in the protobuf, as none of those fields use
strings.
2021-12-03 19:39:40 -05:00
Jeffrey Picard
2537a84b86 Update RangeField
Allow repeated range fields per term (for i.e. 0 < x < 100)
2021-12-03 13:02:06 -05:00
Jeffrey Picard
15614c6a5b Remove unused code 2021-11-05 11:11:26 -04:00
Jeffrey Picard
c61c8db92a Some cleanup based on review and use map + sync.RWMutex instead of
sync.Map
2021-11-04 20:57:23 -04:00
Jeffrey Picard
2ee8d2c3cc Add test to build 2021-10-30 08:44:52 -04:00
Jeffrey Picard
914d2bfc61 Rework federation to be synchronous, and add tests 2021-10-30 01:33:55 -04:00
Jeffrey Picard
a67f9b43d1 Remove unused streaming endpoints 2021-10-29 17:27:27 -04:00
Jeffrey Picard
5387aeeebe Most of federation is written, need to finish udp and test
Cleanup, more reorg, more arguments, started adding tests
Comments and another test
Simplify writing of peers and add unit test
2021-10-29 16:56:36 -04:00
Jeffrey Picard
45e9817ced Cleanup metrics
Consolidated error metrics into a vector counter with a label,
removed other uneeded counters.
2021-10-29 16:56:35 -04:00