Commit graph

2 commits

Author SHA1 Message Date
Jonathan Moody
071aa2a7ad
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.

* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.

* Simplify type-specific partial packing, and simplify dispatch for pack key/value.

* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.

* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.

* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).

* Undo rename. TouchedOrDeleted -> ClaimDiff.

* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).

* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.

* Undo accidental bump of rocksdb version.

* Add .vscode dir to gitignore.

* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.

* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.

* Remove .DS_Store, and ignore in future.

* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.

* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).

* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.

* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 08:45:41 +03:00
Jeffrey Picard
28a299efa8
rocksdb (#29)
* Initial rocksdb commit

Basic reading from rocksdb works

* Try github action thing

* try local dockerfile

* asdf

* qwer

* asdf

* Try adding test db with git-lfs

* update action

* cleanup

* Don't hardcode stop on read

* Progress of reading rocksdb

* fixes and arg test

* asdf

* Fix rocksdb iterator and tests

* update script

* asdf

* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.

* asdf

* asdf

* asdf

* Implementation, testing, and cleanup.

Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.

* more tests, prefixes and small refactor

* Another prefix

* EffectiveAmount

* ActiveAmount

* ActivatedClaimAndSupport

* PendingActivation

* ClaimTakeover

* ClaimExpiration

* SupportToClaim

* ClaimToSupport

* Fix bug with variable length keys

* ChannelToClaim

* ClaimToChannel

* ClaimShortID

* TXOToClaim

* ClaimToTXO

* BlockHeader

* BlockHash

* Undo

* HashXHistory

* Tx and big refactor

* rest the the keys

* Refactor and starting to add resolve

* asdf

* Refactor tests and add column families

* changes

* more work on implementing resolve

* code cleanup, function tests

* small code refactoring

* start building pieces of the test data set for full resolve.

* Export constant, add test

* another test

* TestGetTxHash

* more tests

* more tests

* More tests

* Refactor db functions into three files

* added slice backed stack, need to fix tests

* fix some issues with test suite

* some cleanup and adding arguments and db load / refresh to server command

* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy

* logrus, protobuf updates, resolve grpc endpoint

* don't run integration test with unit tests

* signal handling and cleanup functions

* signal handling code files

* Unit tests for db stack

* reorganize bisect function so we lock it properly

* fix txcounts loading

* cleanup some logic around iterators and fix a bug where I was running two detect changes threads

* add some metrics

* cleanup

* blocking and filtering implemented

* add params for blocking and filtering channels and streams

* updates and fixes for integration tests

* use newer version of lbry.go when possible

* Add height endpoint and move string functions internal

* remove gitattributes, unused

* some cleanup

* more cleanup / refactor. almost ready for another review

* More cleanup

* use chainhash.Hash types from lbcd where appropriate

* update github action to go-1.17.8

* update go version needed

* trying to fix these builds

* cleanup

* trying to fix memory leak

* fix memory leak (iterator never finished so cleanup didn't run)

* changes per code review

* remove lbry.go v2

* rename sort.go search.go

* fix test
2022-04-29 11:04:01 -04:00