Commit graph

29 commits

Author SHA1 Message Date
Jonathan Moody
d2193e980a
blockchain.transaction.broadcast implementation (#80)
* Generate secondary hashXNotification(s) on every headerNotification.

* Attempt LBCD connection with rpc.Client.

* Optional --daemon-url.

* Correct HashXStatusKey field. Should be HASHX_LEN.

* Connect to lbcd using lbcd/rpcclient library.

* Handle deprecation of node.js 12 actions.

* Add --daemon-ca-path argument and establish HTTPS connection if specified.

* Remove dead code. Tighten definition of TransactionBroadcastReq.

* Correct default daemon URL.
2022-12-16 13:54:19 -06:00
Jeffrey Picard
cd7f20a461
Server endpoints goroutine refactor (#69)
* server.xxx endpoints

Additional server endpoints in jsonrpc and also some refactoring

* server.banner

* more endpoints

* use lbry.go stop pattern

* set genesis hash properly

* updates and refactors

* remove shutdowncalled and itmut

* remove OpenIterators

* remove shutdown and done channels from db and use stop group

* currently broken, incorporated stop groups into the session manager

* set the rest of the default args for tests

* add default json rpc http port and cleanup

* tests for new jsonrpc endpoints

* cleanup and add manage goroutine to stopper pattern

* cleanup

* NewDebug

* asdf

* fix
2022-10-25 08:48:13 +03:00
Jonathan Moody
fe18c70bf7 Add some blockchain RPC handlers and database fetching routines. 2022-09-07 14:32:20 -05:00
Jeffrey Picard
9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jonathan Moody
dc9b4ada2a Update go.mod, go.sum for use of constraints (x/exp). 2022-09-07 07:13:51 -05:00
Jeffrey Picard
3a53f46114
Updates for build (#50)
* Updates for build

* go 1.18.1 in dockerfile

* use go 1.18.5

* trying this ...

* asdf
2022-08-16 14:52:26 +03:00
Jonathan Moody
071aa2a7ad
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.

* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.

* Simplify type-specific partial packing, and simplify dispatch for pack key/value.

* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.

* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.

* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).

* Undo rename. TouchedOrDeleted -> ClaimDiff.

* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).

* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.

* Undo accidental bump of rocksdb version.

* Add .vscode dir to gitignore.

* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.

* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.

* Remove .DS_Store, and ignore in future.

* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.

* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).

* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.

* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 08:45:41 +03:00
Jonathan Moody
ffa426b899
Bump to grpc v1.46.0. (#45) 2022-08-09 14:44:00 +03:00
Jeffrey Picard
b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jeffrey Picard
b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00
Jeffrey Picard
28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard
2537a84b86 Update RangeField
Allow repeated range fields per term (for i.e. 0 < x < 100)
2021-12-03 13:02:06 -05:00
Jeffrey Picard
914d2bfc61 Rework federation to be synchronous, and add tests 2021-10-30 01:33:55 -04:00
Jeffrey Picard
5387aeeebe Most of federation is written, need to finish udp and test
Cleanup, more reorg, more arguments, started adding tests
Comments and another test
Simplify writing of peers and add unit test
2021-10-29 16:56:36 -04:00
Jeffrey Picard
a27b6e730d Record metrics
There are a couple metrics that try to mirror the ones from python,
many of the python ones don't apply, and vice-versa.
2021-10-29 16:56:35 -04:00
Jeffrey Picard
0d59480f3c bug fixes and cache purging 2021-10-02 22:49:49 -04:00
Jeffrey Picard
36b4a3cdd9 implement caching 2021-10-02 17:56:12 -04:00
Jeffrey Picard
fdbd7ec509 proto fixes and cleanup 2021-09-18 13:36:49 -04:00
Jeffrey Picard
9c89907bb8 versioning and metrics 2021-09-18 12:06:56 -04:00
Jeffrey Picard
9498257c69 Use util functions from lbry.go 2021-06-28 13:54:49 -04:00
Jeffrey Picard
d6af17df23 Implement searchHead, still need to implement deleteDuplicates 2021-06-07 16:57:41 -04:00
Jeffrey Picard
d7906e9a1c Bug fixes, and use Bool/Int32 values 2021-05-31 14:53:08 -04:00
Jeffrey Picard
0b57dceabd Add source and normalization, remove authentication 2021-05-30 21:34:57 -04:00
Jeffrey Picard
4ca5f168d9 Added port, username and password options to client and server. 2021-05-24 20:09:28 -04:00
Jeffrey Picard
a671683fe8 Add support for more fields 2021-05-15 23:13:14 -04:00
Jeffrey Picard
2b6f5c4aed Start adding support for more query types 2021-05-13 16:06:19 -04:00
Alex Grintsvayg
3ca27f3740
very basic elastic search works 2021-04-19 15:28:57 -04:00
Alex Grintsvayg
a5f235c4f5
we're doing grpc 2021-03-18 17:24:48 -04:00
Alex Grintsvayg
04ea7b5f0e
it begins 2021-03-18 16:12:48 -04:00