Commit graph

195 commits

Author SHA1 Message Date
Jeffrey Picard 3dfe5a5bc3 fix workflows 2022-04-29 17:05:15 +00:00
Jeffrey Picard 28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00
Jeffrey Picard ff0694def1 v0.2022.01.21.1 2022-01-21 12:27:09 -05:00
Alex Grin 78f35cc96c
Merge pull request #31 from lbryio/bugfix/x/jeffreypicard/return-empty-list-on-no-index
Fix behavior when the index isn't there.
2022-01-10 11:30:26 -05:00
Jeffrey Picard d51f5ca064 Fix behavior when the index isn't there.
One of the tests expects an empty list returned, we're currently
returning an error.
2022-01-05 10:07:15 -05:00
Jeffrey Picard 9e2a39e938 v0.2021.12.18.1 2021-12-18 11:14:37 -05:00
Victor Shyba 8e99bc56d0
add sd_hash prefix search (#30)
* add sd_hash to search request

* search sd_hash by prefix
2021-12-17 09:37:32 -05:00
Jeffrey Picard 447e931f18 v0.2021.12.06.1 2021-12-06 17:42:34 -05:00
Jeffrey Picard 516b95d96b
Merge pull request #24 from lbryio/feature/18/jeffreypicard/udp-endpoint
UDPServer / ip address resolution
2021-12-06 11:57:47 -05:00
Jeffrey Picard 2a1d6fa7d4 Updates based on code review 2021-12-06 11:32:45 -05:00
Jeffrey Picard 2e52c1639c Refactor and fixes related to PR comments. 2021-12-06 11:32:45 -05:00
Jeffrey Picard 1c1d288654 Set UDP flags correctly. 2021-12-06 11:32:45 -05:00
Jeffrey Picard 283686ecac Fix startup flags 2021-12-06 11:32:45 -05:00
Jeffrey Picard ca0e8562f3 Add disable-federation flag, default false 2021-12-06 11:32:45 -05:00
Jeffrey Picard 72ea236d86 Run gofmt 2021-12-06 11:31:51 -05:00
Jeffrey Picard 0a1ba43d66 Full SPVPong parsing and tests against prod server. 2021-12-06 11:26:29 -05:00
Jeffrey Picard 4e6b47c2a3 Test UDPPing country parsing. 2021-12-06 11:26:29 -05:00
Jeffrey Picard ee5fcaef14 Added test for getting IP with udp from prod servers. 2021-12-06 11:26:29 -05:00
Jeffrey Picard 159f4b941b Add claim protobuf definition for UDP country encoding. 2021-12-06 11:26:29 -05:00
Jeffrey Picard 355eab682c changes based on comments 2021-12-06 11:26:29 -05:00
Jeffrey Picard 395e1db489 UDPServer / ip address resolution
Got the UDPServer ping/pong protocol working internally, only tested
against other running go hub servers. Should in theory work with
python server and clients, but still need to test that.

Also switched to serving udp on the same port as grpc, and taking that
into account when pinging other hubs with udp.

Unit test for udp ip address lookup.
2021-12-06 11:26:29 -05:00
Jeffrey Picard 602292281c
Merge pull request #28 from lbryio/feature/jeffreypicard/update-range-query
Update RangeField
2021-12-06 11:24:47 -05:00
Jeffrey Picard d511b08736 Switch RangeField to int32 in the protobuf, as none of those fields use
strings.
2021-12-03 19:39:40 -05:00
Jeffrey Picard 2537a84b86 Update RangeField
Allow repeated range fields per term (for i.e. 0 < x < 100)
2021-12-03 13:02:06 -05:00
Jeffrey Picard 67b3dce492 v0.2021.11.05.1 2021-11-05 11:16:40 -04:00
Jeffrey Picard 284f825d22
Merge pull request #20 from lbryio/feature/6/jeffreypicard/hub-federation
Most of federation is written, need to finish udp and test
2021-11-05 11:14:05 -04:00
Jeffrey Picard 15614c6a5b Remove unused code 2021-11-05 11:11:26 -04:00
Jeffrey Picard c61c8db92a Some cleanup based on review and use map + sync.RWMutex instead of
sync.Map
2021-11-04 20:57:23 -04:00
Jeffrey Picard 2ee8d2c3cc Add test to build 2021-10-30 08:44:52 -04:00
Jeffrey Picard 914d2bfc61 Rework federation to be synchronous, and add tests 2021-10-30 01:33:55 -04:00
Jeffrey Picard a67f9b43d1 Remove unused streaming endpoints 2021-10-29 17:27:27 -04:00
Jeffrey Picard 58a44bfce3 Merge branch 'master' of github.com:lbryio/hub into feature/6/jeffreypicard/hub-federation 2021-10-29 17:05:26 -04:00
Jeffrey Picard 5387aeeebe Most of federation is written, need to finish udp and test
Cleanup, more reorg, more arguments, started adding tests
Comments and another test
Simplify writing of peers and add unit test
2021-10-29 16:56:36 -04:00
Jeffrey Picard 45e9817ced Cleanup metrics
Consolidated error metrics into a vector counter with a label,
removed other uneeded counters.
2021-10-29 16:56:35 -04:00
Jeffrey Picard a27b6e730d Record metrics
There are a couple metrics that try to mirror the ones from python,
many of the python ones don't apply, and vice-versa.
2021-10-29 16:56:35 -04:00
Jeffrey Picard 02dbea4775 v0.2021.10.25.1 2021-10-25 11:51:09 -04:00
Jeffrey Picard e12c26fe20
Merge pull request #17 from lbryio/jeffreypicard/8/add-prometheus-metrics
Record metrics
2021-10-21 08:32:43 -04:00
Jeffrey Picard f0369e12e7 Cleanup metrics
Consolidated error metrics into a vector counter with a label,
removed other uneeded counters.
2021-10-21 02:52:03 -04:00
Jeffrey Picard d69af2db8a v0.2021.10.16.1 2021-10-16 10:08:16 -04:00
Jeffrey Picard a9357d3ba5 v0.2021.10.16.1 2021-10-16 10:07:48 -04:00
Jeffrey Picard 8084a15598 v0.2021.10.16.1 2021-10-16 09:47:15 -04:00
Jeffrey Picard e043ca89c9 Record metrics
There are a couple metrics that try to mirror the ones from python,
many of the python ones don't apply, and vice-versa.
2021-10-15 17:26:37 -04:00
Jeffrey Picard 0d51cbfde4
Merge pull request #15 from lbryio/query-caching
Query caching
2021-10-04 18:05:43 -04:00
Jeffrey Picard a6d47e662a Don't swallow error, cachettl and refresh delta as params 2021-10-04 18:04:26 -04:00
Jeffrey Picard 220a42984e put jack's changes back 2021-10-04 12:44:55 -04:00
Jeffrey Picard a0de5164ba cleanup 2021-10-02 23:13:47 -04:00
Jeffrey Picard 0d59480f3c bug fixes and cache purging 2021-10-02 22:49:49 -04:00
Jeffrey Picard 36b4a3cdd9 implement caching 2021-10-02 17:56:12 -04:00
Jeffrey Picard 076adcca39 Initial caching 2021-09-30 15:03:00 -04:00
Jeffrey Picard 4839f4aa8a
Merge pull request #14 from lbryio/prometheus-and-versioning
Prometheus and versioning
2021-09-28 08:34:36 -04:00