Commit graph

169 commits

Author SHA1 Message Date
Jeffrey Picard
9403d84a83
WIP: Resolve json rpc (#57)
* jsonrpc

* update readme for open file limits

* add CGO flags to readme

* remove uneeded logging

* don't start jsonrpc server in unit tests

* cleanup and add args for json rpc

* correct rpc default port

* remove unused test_rpc.sh script

Co-authored-by: Ubuntu <ubuntu@ns5010184.ip-15-235-15.net>
2022-09-07 21:36:07 +03:00
Jonathan Moody
09fd939b60
Merge pull request #60 from moodyjon/slicebacked_type_param
Update go.mod, go.sum for use of constraints (x/exp).
2022-09-07 07:22:01 -05:00
Jonathan Moody
dc9b4ada2a Update go.mod, go.sum for use of constraints (x/exp). 2022-09-07 07:13:51 -05:00
Jonathan Moody
c38134b645
Merge pull request #56 from moodyjon/slicebacked_type_param
Add element type param T to SlicedBacked[T]. Require T satisfy
2022-09-07 05:39:34 -05:00
Jonathan Moody
d025ea1616
Add "on: pull_request" to worflow. (#59) 2022-09-06 22:13:13 +03:00
Jonathan Moody
5b690ff2ff
Merge pull request #58 from moodyjon/hashx_history_fix
Fix struct annotation for HashXHistoryValue. TxNums now little-endian.
2022-09-06 13:03:07 -05:00
Jonathan Moody
aa16207aa5 Fix struct annotation for HashXHistoryValue. TxNums now little-endian. 2022-09-06 12:57:45 -05:00
Jonathan Moody
78b9a625eb
Merge pull request #54 from moodyjon/hashx_history_fix
Payload of HashXHistoryValue should be an array of uint32 representing "txnums"
2022-09-06 12:40:06 -05:00
Jonathan Moody
e46ac7c913 HashXHistoryValue TxNums are unique in that they are little-endian
(at least when written by Python scribe on ARM64 Mac or x86).
2022-09-01 13:01:16 -05:00
Jonathan Moody
8ac89195db Add element type param T to SlicedBacked[T]. Require T satisfy
constraints.Ordered to make BisectRight() statically type-safe.
2022-08-30 16:24:43 -05:00
Jonathan Moody
4e11433325 Payload of HashXHistoryValue should be an array of uint32 representing "txnums". 2022-08-26 10:18:01 -04:00
Jonathan Moody
9d9c73f97f
Add RepostedCount, EffectiveAmount prefix rows (#51)
* Rename prefix EffectiveAmount -> BidOrder.

* Add RepostedCount, EffectiveAmount prefix rows. Add testdata.

* Update db_get.go helpers to use EffectiveAmount, RepostedCount
tables. Update tests.
2022-08-26 16:24:39 +03:00
Jeffrey Picard
cbdcc5faeb v0.2022.08.16.1 2022-08-16 14:56:19 +03:00
Jeffrey Picard
3a53f46114
Updates for build (#50)
* Updates for build

* go 1.18.1 in dockerfile

* use go 1.18.5

* trying this ...

* asdf
2022-08-16 14:52:26 +03:00
Jonathan Moody
071aa2a7ad
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.

* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.

* Simplify type-specific partial packing, and simplify dispatch for pack key/value.

* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.

* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.

* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).

* Undo rename. TouchedOrDeleted -> ClaimDiff.

* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).

* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.

* Undo accidental bump of rocksdb version.

* Add .vscode dir to gitignore.

* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.

* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.

* Remove .DS_Store, and ignore in future.

* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.

* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).

* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.

* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 08:45:41 +03:00
Jeffrey Picard
b018217899
fix release script (#48) 2022-08-10 21:06:56 +03:00
Jack Robison
13479794ed
Update readme.md 2022-08-10 11:05:39 -04:00
Jeffrey Picard
ab2a39ac6c v0.2022.08.09.1 2022-08-09 17:19:53 +03:00
Jonathan Moody
ffa426b899
Bump to grpc v1.46.0. (#45) 2022-08-09 14:44:00 +03:00
Jeffrey Picard
b9f7d595bd
Herald.go (#47)
* switch herald to herald.go

* update ci/cd stuff

* fix issues with binary name

* we're no longer building dynamically, so turn CGO back on, and fix names

* update package names in proto files
2022-08-09 14:43:01 +03:00
Jonathan Moody
49e5d7b8c1
Fix release_time query to match python hub. (#41)
* Revise error handling and cleanup in GetProdDB(). Avoids panic when DB not found.

* Fix release_time query to match python hub. Claims with unset release_time match the query.
2022-08-03 19:19:25 +03:00
Jack Robison
8b4b27bdb9
Update readme.md 2022-08-02 11:53:23 -04:00
Jeffrey Picard
b2dd70bc7c
Jeffreypicard/feature/move to herald (#37)
* changes for repo move

* Go back to protobuf 1.27.1, update go to 1.18

* update readme for new rocksdb version
2022-07-11 20:31:27 +03:00
Jeffrey Picard
7fcb4a750d remove old workflow 2022-04-29 22:30:11 +00:00
Jeffrey Picard
3dfe5a5bc3 fix workflows 2022-04-29 17:05:15 +00:00
Jeffrey Picard
28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard
ff0694def1 v0.2022.01.21.1 2022-01-21 12:27:09 -05:00
Alex Grin
78f35cc96c
Merge pull request #31 from lbryio/bugfix/x/jeffreypicard/return-empty-list-on-no-index
Fix behavior when the index isn't there.
2022-01-10 11:30:26 -05:00
Jeffrey Picard
d51f5ca064 Fix behavior when the index isn't there.
One of the tests expects an empty list returned, we're currently
returning an error.
2022-01-05 10:07:15 -05:00
Jeffrey Picard
9e2a39e938 v0.2021.12.18.1 2021-12-18 11:14:37 -05:00
Victor Shyba
8e99bc56d0
add sd_hash prefix search (#30)
* add sd_hash to search request

* search sd_hash by prefix
2021-12-17 09:37:32 -05:00
Jeffrey Picard
447e931f18 v0.2021.12.06.1 2021-12-06 17:42:34 -05:00
Jeffrey Picard
516b95d96b
Merge pull request #24 from lbryio/feature/18/jeffreypicard/udp-endpoint
UDPServer / ip address resolution
2021-12-06 11:57:47 -05:00
Jeffrey Picard
2a1d6fa7d4 Updates based on code review 2021-12-06 11:32:45 -05:00
Jeffrey Picard
2e52c1639c Refactor and fixes related to PR comments. 2021-12-06 11:32:45 -05:00
Jeffrey Picard
1c1d288654 Set UDP flags correctly. 2021-12-06 11:32:45 -05:00
Jeffrey Picard
283686ecac Fix startup flags 2021-12-06 11:32:45 -05:00
Jeffrey Picard
ca0e8562f3 Add disable-federation flag, default false 2021-12-06 11:32:45 -05:00
Jeffrey Picard
72ea236d86 Run gofmt 2021-12-06 11:31:51 -05:00
Jeffrey Picard
0a1ba43d66 Full SPVPong parsing and tests against prod server. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
4e6b47c2a3 Test UDPPing country parsing. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
ee5fcaef14 Added test for getting IP with udp from prod servers. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
159f4b941b Add claim protobuf definition for UDP country encoding. 2021-12-06 11:26:29 -05:00
Jeffrey Picard
355eab682c changes based on comments 2021-12-06 11:26:29 -05:00
Jeffrey Picard
395e1db489 UDPServer / ip address resolution
Got the UDPServer ping/pong protocol working internally, only tested
against other running go hub servers. Should in theory work with
python server and clients, but still need to test that.

Also switched to serving udp on the same port as grpc, and taking that
into account when pinging other hubs with udp.

Unit test for udp ip address lookup.
2021-12-06 11:26:29 -05:00
Jeffrey Picard
602292281c
Merge pull request #28 from lbryio/feature/jeffreypicard/update-range-query
Update RangeField
2021-12-06 11:24:47 -05:00
Jeffrey Picard
d511b08736 Switch RangeField to int32 in the protobuf, as none of those fields use
strings.
2021-12-03 19:39:40 -05:00
Jeffrey Picard
2537a84b86 Update RangeField
Allow repeated range fields per term (for i.e. 0 < x < 100)
2021-12-03 13:02:06 -05:00
Jeffrey Picard
67b3dce492 v0.2021.11.05.1 2021-11-05 11:16:40 -04:00
Jeffrey Picard
284f825d22
Merge pull request #20 from lbryio/feature/6/jeffreypicard/hub-federation
Most of federation is written, need to finish udp and test
2021-11-05 11:14:05 -04:00