Commit graph

196 commits

Author SHA1 Message Date
Jonathan Moody d2193e980a
blockchain.transaction.broadcast implementation (#80)
* Generate secondary hashXNotification(s) on every headerNotification.

* Attempt LBCD connection with rpc.Client.

* Optional --daemon-url.

* Correct HashXStatusKey field. Should be HASHX_LEN.

* Connect to lbcd using lbcd/rpcclient library.

* Handle deprecation of node.js 12 actions.

* Add --daemon-ca-path argument and establish HTTPS connection if specified.

* Remove dead code. Tighten definition of TransactionBroadcastReq.

* Correct default daemon URL.
2022-12-16 13:54:19 -06:00
Jeffrey Picard 711c4b4b7b
WIP: Json rpc federation, search/getclaimbyid, and shutdown (#76)
* Cleanup shutdown and peers subscribe

this has errors currently, need to figure out data race

* fixed data race finally....

* getclaimbyid and search

* hook jsonrpc peers subcribe into current federation

* cleanup some peers stuff

* remove commented code

* tie into session manager in jsonrpc peers subscribe

* use int for port args

* cleanup and fix a bunch of compiler warnings

* use logrus everywhere

* cleanup test
2022-12-07 11:01:36 -05:00
Jonathan Moody 317cdf7129
WIP: blockchain.transaction.yyy JSON RPC implementations (#78)
* Partial blockchain.transaction.yyy RPC implementations.

* Register RPC service object.

* Move session manager start/stop to a better place.

* Attempt to fill in the details of transaction.get_batch,
including merkle path.

* Correct interpretation of DBStateValue Genesis hash.

* Convert Args.Port to int and validate. Run UDP ping server on JSONRPCPort too.

* Add BlockHeader to HeightHash notification.

* Limit session-based JSON RPC service to IPv4.
Client not ready for IPv6.

* Adapt to new HeightHash struct.

* Fine tune JSON RPC handlers and types to match lbry-sdk expectations.
Implement UnmarshalJSON()/MarshalJSON() for several types.

* Add more special handling of DBStateValue.Genesis hash.

* Set IncludeStop=false generally to avoid returning extra rows.
Other misc fixes.
2022-12-06 16:14:28 -05:00
Jonathan Moody e070e8a51e
JSON RPC compatibility workarounds to support lbry-sdk (#75)
* Fix log spam (Already shutting down...)

* Workaround to allow lbry-sdk to call server.version and server.features.
Incoming/outgoing JSON is patched using yet another codec (jsonPatchingCodec).
Add more logging of raw/patched JSON.

* Elaborate comment on jsonPatchingCodec.
2022-10-29 18:42:24 +03:00
Jeffrey Picard cd7f20a461
Server endpoints goroutine refactor (#69)
* server.xxx endpoints

Additional server endpoints in jsonrpc and also some refactoring

* server.banner

* more endpoints

* use lbry.go stop pattern

* set genesis hash properly

* updates and refactors

* remove shutdowncalled and itmut

* remove OpenIterators

* remove shutdown and done channels from db and use stop group

* currently broken, incorporated stop groups into the session manager

* set the rest of the default args for tests

* add default json rpc http port and cleanup

* tests for new jsonrpc endpoints

* cleanup and add manage goroutine to stopper pattern

* cleanup

* NewDebug

* asdf

* fix
2022-10-25 08:48:13 +03:00
Jeffrey Picard b85556499f v0.2022.10.05.1 2022-10-05 06:25:42 +03:00
Jeffrey Picard 8eb7841600
little fixes for debugging and shutdown (#67) 2022-10-05 06:24:42 +03:00
Jeffrey Picard 6d4b9b5e37 v0.2022.10.04.1 2022-10-04 20:52:47 +03:00
Jeffrey Picard 537b8c7ddd
integration testing scripts (#64)
* integration testing scripts

some scripts for integration testing and a docker file for an action.
Still need to figure out how to properly run a more realistic version
in ci.

* update

* changes

* db shutdown racecondition fix

* changes per pr

* changes per code review

* fix testing

* add shutdowncalled bool to db
2022-10-04 20:25:44 +03:00
Jonathan Moody 8fb3db8136
Add subscribe/unsubscribe RPCs. Add session, sessionManager, and serve JSON RPC (without HTTP). (#66)
* Move and rename BlockchainCodec, BlockchainCodecRequest.
These are not specifically "blockchain", rather they are
specific to how gorilla/rpc works.

* Move claimtrie-related service/handlers to jsonrpc_claimtrie.go.

* Pull out decode logic into named func newBlockHeaderElectrum().

* Rename BlockchainService -> BlockchainBlockService.

* Drop http.Request arg from handlers, and use RegisterTCPService().

* Implement GetStatus() to pull data from HashXStatus table.

* Make the service objects independent, so we don't have inheritance.

* Add core session/subscription logic (session.go).
Implement subsribe/unsubscribe handlers.

* Support both pure JSON and JSON-over-HTTP services.
Forward NotifierChan messages to sessionManager.

* Only assign default port (50001) if neither --json-rpc-port nor
--json-rpc-http-port are specified.

* Handle failures with goto instead of break. Update error logging.

* Add --max-sessions, --session-timeout args. Enforce max sessions.

* Changes to make session.go testable. Conn created with Pipe()
used in testing has no unique Addr.

* Add tests for headers, headers.subscribe, address.subscribe.

* HashXStatus, HashXMempoolStatus not populated by default. Fix GetStatus().

* Use time.Ticker object to drive management activity.
2022-10-04 17:05:06 +03:00
Jonathan Moody 979d0d16b6
Adjust EffectiveAmountValue to include ActivatedSupportSum. (#61)
Make use of this in GetEffectiveAmount() tests.
2022-09-16 18:03:15 +03:00
Jonathan Moody 7d24ff82bf
Merge pull request #55 from moodyjon/blockchain_rpc1
Add some blockchain RPC handlers and DB fetching routines
2022-09-14 10:23:34 -05:00
Jonathan Moody 86a287ec69 Skip TestUDPPing if production server is down. 2022-09-13 15:35:54 -05:00
Jonathan Moody 891f63fb5c Update jsonrpc_blockchain_tests.go for new sample data. 2022-09-13 15:33:24 -05:00
Jonathan Moody 789974227f Add sample data from test_variety_of_transactions_and_longish_history.
Rework tests to use the sample data.
2022-09-13 15:05:11 -05:00
Jonathan Moody 71e79c553e One more RPC (get_server_height), and update comment
to include full RPC name.
2022-09-08 13:17:52 -05:00
Jonathan Moody b298454727 Fix RPC handler registration and BlockGetChunkResp name. 2022-09-08 11:50:06 -05:00
Jonathan Moody 8c8871b4d2 Register blockchain.* handlers in jsonrpc_service.go. 2022-09-07 15:01:47 -05:00
Jonathan Moody 20e32437e9 Rename blockchain.go -> jsonrpc_blockchain.go. 2022-09-07 14:32:34 -05:00
Jonathan Moody d0d6145f9d Refactor blockchain.go handlers to be compatible with gorilla/rpc.
Add speculative BlockchainCodecRequest which might handle rewriting
incoming method names.
2022-09-07 14:32:34 -05:00
Jonathan Moody 90afae7cd5 Add scripthash variants of RPC handlers. 2022-09-07 14:32:34 -05:00
Jonathan Moody f5b8f2ce0d Add --chain=X argument, and use it to determine chain when DB is empty. 2022-09-07 14:32:34 -05:00
Jonathan Moody 321bcf6420 Fix encoding of TX and Block hashes in response. 2022-09-07 14:32:34 -05:00
Jonathan Moody 293a3f685e Fix some DB logic and add tests (blockchain_test.go). 2022-09-07 14:32:34 -05:00
Jonathan Moody f55a5ed777 Continuing blockchain RPC handler work. Add JSON tags.
Fetch Height using TxCounts.
2022-09-07 14:32:34 -05:00
Jonathan Moody 50f7e91ead Infer chain (mainnet, testnet3, regtest) based on DBStateValue.
Correct typo DDVersion -> DBVersion. Misc logging improvements.
2022-09-07 14:32:34 -05:00
Jonathan Moody fe18c70bf7 Add some blockchain RPC handlers and database fetching routines. 2022-09-07 14:32:20 -05:00
Jeffrey Picard 9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jonathan Moody 09fd939b60
Merge pull request #60 from moodyjon/slicebacked_type_param
Update go.mod, go.sum for use of constraints (x/exp).
2022-09-07 07:22:01 -05:00
Jonathan Moody dc9b4ada2a Update go.mod, go.sum for use of constraints (x/exp). 2022-09-07 07:13:51 -05:00
Jonathan Moody c38134b645
Merge pull request #56 from moodyjon/slicebacked_type_param
Add element type param T to SlicedBacked[T]. Require T satisfy
2022-09-07 05:39:34 -05:00
Jonathan Moody d025ea1616
Add "on: pull_request" to worflow. (#59) 2022-09-06 22:13:13 +03:00
Jonathan Moody 5b690ff2ff
Merge pull request #58 from moodyjon/hashx_history_fix
Fix struct annotation for HashXHistoryValue. TxNums now little-endian.
2022-09-06 13:03:07 -05:00
Jonathan Moody aa16207aa5 Fix struct annotation for HashXHistoryValue. TxNums now little-endian. 2022-09-06 12:57:45 -05:00
Jonathan Moody 78b9a625eb
Merge pull request #54 from moodyjon/hashx_history_fix
Payload of HashXHistoryValue should be an array of uint32 representing "txnums"
2022-09-06 12:40:06 -05:00
Jonathan Moody e46ac7c913 HashXHistoryValue TxNums are unique in that they are little-endian
(at least when written by Python scribe on ARM64 Mac or x86).
2022-09-01 13:01:16 -05:00
Jonathan Moody 8ac89195db Add element type param T to SlicedBacked[T]. Require T satisfy
constraints.Ordered to make BisectRight() statically type-safe.
2022-08-30 16:24:43 -05:00
Jonathan Moody 4e11433325 Payload of HashXHistoryValue should be an array of uint32 representing "txnums". 2022-08-26 10:18:01 -04:00
Jonathan Moody 9d9c73f97f
Add RepostedCount, EffectiveAmount prefix rows (#51)
* Rename prefix EffectiveAmount -> BidOrder.

* Add RepostedCount, EffectiveAmount prefix rows. Add testdata.

* Update db_get.go helpers to use EffectiveAmount, RepostedCount
tables. Update tests.
2022-08-26 16:24:39 +03:00
Jeffrey Picard cbdcc5faeb v0.2022.08.16.1 2022-08-16 14:56:19 +03:00
Jeffrey Picard 3a53f46114
Updates for build (#50)
* Updates for build

* go 1.18.1 in dockerfile

* use go 1.18.5

* trying this ...

* asdf
2022-08-16 14:52:26 +03:00
Jonathan Moody 071aa2a7ad
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.

* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.

* Simplify type-specific partial packing, and simplify dispatch for pack key/value.

* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.

* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.

* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).

* Undo rename. TouchedOrDeleted -> ClaimDiff.

* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).

* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.

* Undo accidental bump of rocksdb version.

* Add .vscode dir to gitignore.

* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.

* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.

* Remove .DS_Store, and ignore in future.

* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.

* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).

* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.

* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 08:45:41 +03:00
Jeffrey Picard b018217899
fix release script (#48) 2022-08-10 21:06:56 +03:00
Jack Robison 13479794ed
Update readme.md 2022-08-10 11:05:39 -04:00
Jeffrey Picard ab2a39ac6c v0.2022.08.09.1 2022-08-09 17:19:53 +03:00
Jonathan Moody ffa426b899
Bump to grpc v1.46.0. (#45) 2022-08-09 14:44:00 +03:00
Jeffrey Picard b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jonathan Moody 49e5d7b8c1
Fix release_time query to match python hub. (#41)
* Revise error handling and cleanup in GetProdDB(). Avoids panic when DB not found.

* Fix release_time query to match python hub. Claims with unset release_time match the query.
2022-08-03 19:19:25 +03:00
Jack Robison 8b4b27bdb9
Update readme.md 2022-08-02 11:53:23 -04:00
Jeffrey Picard b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00