Commit graph

192 commits

Author SHA1 Message Date
Jeffrey Picard
cd7f20a461
Server endpoints goroutine refactor (#69)
* server.xxx endpoints

Additional server endpoints in jsonrpc and also some refactoring

* server.banner

* more endpoints

* use lbry.go stop pattern

* set genesis hash properly

* updates and refactors

* remove shutdowncalled and itmut

* remove OpenIterators

* remove shutdown and done channels from db and use stop group

* currently broken, incorporated stop groups into the session manager

* set the rest of the default args for tests

* add default json rpc http port and cleanup

* tests for new jsonrpc endpoints

* cleanup and add manage goroutine to stopper pattern

* cleanup

* NewDebug

* asdf

* fix
2022-10-25 08:48:13 +03:00
Jeffrey Picard
b85556499f v0.2022.10.05.1 2022-10-05 06:25:42 +03:00
Jeffrey Picard
8eb7841600
little fixes for debugging and shutdown (#67) 2022-10-05 06:24:42 +03:00
Jeffrey Picard
6d4b9b5e37 v0.2022.10.04.1 2022-10-04 20:52:47 +03:00
Jeffrey Picard
537b8c7ddd
integration testing scripts (#64)
* integration testing scripts

some scripts for integration testing and a docker file for an action.
Still need to figure out how to properly run a more realistic version
in ci.

* update

* changes

* db shutdown racecondition fix

* changes per pr

* changes per code review

* fix testing

* add shutdowncalled bool to db
2022-10-04 20:25:44 +03:00
Jonathan Moody
8fb3db8136
Add subscribe/unsubscribe RPCs. Add session, sessionManager, and serve JSON RPC (without HTTP). (#66)
* Move and rename BlockchainCodec, BlockchainCodecRequest.
These are not specifically "blockchain", rather they are
specific to how gorilla/rpc works.

* Move claimtrie-related service/handlers to jsonrpc_claimtrie.go.

* Pull out decode logic into named func newBlockHeaderElectrum().

* Rename BlockchainService -> BlockchainBlockService.

* Drop http.Request arg from handlers, and use RegisterTCPService().

* Implement GetStatus() to pull data from HashXStatus table.

* Make the service objects independent, so we don't have inheritance.

* Add core session/subscription logic (session.go).
Implement subsribe/unsubscribe handlers.

* Support both pure JSON and JSON-over-HTTP services.
Forward NotifierChan messages to sessionManager.

* Only assign default port (50001) if neither --json-rpc-port nor
--json-rpc-http-port are specified.

* Handle failures with goto instead of break. Update error logging.

* Add --max-sessions, --session-timeout args. Enforce max sessions.

* Changes to make session.go testable. Conn created with Pipe()
used in testing has no unique Addr.

* Add tests for headers, headers.subscribe, address.subscribe.

* HashXStatus, HashXMempoolStatus not populated by default. Fix GetStatus().

* Use time.Ticker object to drive management activity.
2022-10-04 17:05:06 +03:00
Jonathan Moody
979d0d16b6
Adjust EffectiveAmountValue to include ActivatedSupportSum. (#61)
Make use of this in GetEffectiveAmount() tests.
2022-09-16 18:03:15 +03:00
Jonathan Moody
7d24ff82bf
Merge pull request #55 from moodyjon/blockchain_rpc1
Add some blockchain RPC handlers and DB fetching routines
2022-09-14 10:23:34 -05:00
Jonathan Moody
86a287ec69 Skip TestUDPPing if production server is down. 2022-09-13 15:35:54 -05:00
Jonathan Moody
891f63fb5c Update jsonrpc_blockchain_tests.go for new sample data. 2022-09-13 15:33:24 -05:00
Jonathan Moody
789974227f Add sample data from test_variety_of_transactions_and_longish_history.
Rework tests to use the sample data.
2022-09-13 15:05:11 -05:00
Jonathan Moody
71e79c553e One more RPC (get_server_height), and update comment
to include full RPC name.
2022-09-08 13:17:52 -05:00
Jonathan Moody
b298454727 Fix RPC handler registration and BlockGetChunkResp name. 2022-09-08 11:50:06 -05:00
Jonathan Moody
8c8871b4d2 Register blockchain.* handlers in jsonrpc_service.go. 2022-09-07 15:01:47 -05:00
Jonathan Moody
20e32437e9 Rename blockchain.go -> jsonrpc_blockchain.go. 2022-09-07 14:32:34 -05:00
Jonathan Moody
d0d6145f9d Refactor blockchain.go handlers to be compatible with gorilla/rpc.
Add speculative BlockchainCodecRequest which might handle rewriting
incoming method names.
2022-09-07 14:32:34 -05:00
Jonathan Moody
90afae7cd5 Add scripthash variants of RPC handlers. 2022-09-07 14:32:34 -05:00
Jonathan Moody
f5b8f2ce0d Add --chain=X argument, and use it to determine chain when DB is empty. 2022-09-07 14:32:34 -05:00
Jonathan Moody
321bcf6420 Fix encoding of TX and Block hashes in response. 2022-09-07 14:32:34 -05:00
Jonathan Moody
293a3f685e Fix some DB logic and add tests (blockchain_test.go). 2022-09-07 14:32:34 -05:00
Jonathan Moody
f55a5ed777 Continuing blockchain RPC handler work. Add JSON tags.
Fetch Height using TxCounts.
2022-09-07 14:32:34 -05:00
Jonathan Moody
50f7e91ead Infer chain (mainnet, testnet3, regtest) based on DBStateValue.
Correct typo DDVersion -> DBVersion. Misc logging improvements.
2022-09-07 14:32:34 -05:00
Jonathan Moody
fe18c70bf7 Add some blockchain RPC handlers and database fetching routines. 2022-09-07 14:32:20 -05:00
Jeffrey Picard
9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jonathan Moody
09fd939b60
Merge pull request #60 from moodyjon/slicebacked_type_param
Update go.mod, go.sum for use of constraints (x/exp).
2022-09-07 07:22:01 -05:00
Jonathan Moody
dc9b4ada2a Update go.mod, go.sum for use of constraints (x/exp). 2022-09-07 07:13:51 -05:00
Jonathan Moody
c38134b645
Merge pull request #56 from moodyjon/slicebacked_type_param
Add element type param T to SlicedBacked[T]. Require T satisfy
2022-09-07 05:39:34 -05:00
Jonathan Moody
d025ea1616
Add "on: pull_request" to worflow. (#59) 2022-09-06 22:13:13 +03:00
Jonathan Moody
5b690ff2ff
Merge pull request #58 from moodyjon/hashx_history_fix
Fix struct annotation for HashXHistoryValue. TxNums now little-endian.
2022-09-06 13:03:07 -05:00
Jonathan Moody
aa16207aa5 Fix struct annotation for HashXHistoryValue. TxNums now little-endian. 2022-09-06 12:57:45 -05:00
Jonathan Moody
78b9a625eb
Merge pull request #54 from moodyjon/hashx_history_fix
Payload of HashXHistoryValue should be an array of uint32 representing "txnums"
2022-09-06 12:40:06 -05:00
Jonathan Moody
e46ac7c913 HashXHistoryValue TxNums are unique in that they are little-endian
(at least when written by Python scribe on ARM64 Mac or x86).
2022-09-01 13:01:16 -05:00
Jonathan Moody
8ac89195db Add element type param T to SlicedBacked[T]. Require T satisfy
constraints.Ordered to make BisectRight() statically type-safe.
2022-08-30 16:24:43 -05:00
Jonathan Moody
4e11433325 Payload of HashXHistoryValue should be an array of uint32 representing "txnums". 2022-08-26 10:18:01 -04:00
Jonathan Moody
9d9c73f97f
Add RepostedCount, EffectiveAmount prefix rows (#51)
* Rename prefix EffectiveAmount -> BidOrder.

* Add RepostedCount, EffectiveAmount prefix rows. Add testdata.

* Update db_get.go helpers to use EffectiveAmount, RepostedCount
tables. Update tests.
2022-08-26 16:24:39 +03:00
Jeffrey Picard
cbdcc5faeb v0.2022.08.16.1 2022-08-16 14:56:19 +03:00
Jeffrey Picard
3a53f46114
Updates for build (#50)
* Updates for build

* go 1.18.1 in dockerfile

* use go 1.18.5

* trying this ...

* asdf
2022-08-16 14:52:26 +03:00
Jonathan Moody
071aa2a7ad
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.

* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.

* Simplify type-specific partial packing, and simplify dispatch for pack key/value.

* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.

* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.

* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).

* Undo rename. TouchedOrDeleted -> ClaimDiff.

* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).

* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.

* Undo accidental bump of rocksdb version.

* Add .vscode dir to gitignore.

* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.

* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.

* Remove .DS_Store, and ignore in future.

* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.

* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).

* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.

* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 08:45:41 +03:00
Jeffrey Picard
b018217899
fix release script (#48) 2022-08-10 21:06:56 +03:00
Jack Robison
13479794ed
Update readme.md 2022-08-10 11:05:39 -04:00
Jeffrey Picard
ab2a39ac6c v0.2022.08.09.1 2022-08-09 17:19:53 +03:00
Jonathan Moody
ffa426b899
Bump to grpc v1.46.0. (#45) 2022-08-09 14:44:00 +03:00
Jeffrey Picard
b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jonathan Moody
49e5d7b8c1
Fix release_time query to match python hub. (#41)
* Revise error handling and cleanup in GetProdDB(). Avoids panic when DB not found.

* Fix release_time query to match python hub. Claims with unset release_time match the query.
2022-08-03 19:19:25 +03:00
Jack Robison
8b4b27bdb9
Update readme.md 2022-08-02 11:53:23 -04:00
Jeffrey Picard
b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00
Jeffrey Picard
7fcb4a750d remove old workflow 2022-04-29 22:30:11 +00:00
Jeffrey Picard
3dfe5a5bc3 fix workflows 2022-04-29 17:05:15 +00:00
Jeffrey Picard
28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard
ff0694def1 v0.2022.01.21.1 2022-01-21 12:27:09 -05:00