rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
package prefixes
|
|
|
|
|
|
|
|
// The prefixes package contains the key/value structures for the rocksdb prefix
|
|
|
|
// store and related functions.
|
|
|
|
// Each key/value prefix pair has a pack and unpack function for serializing / deserializing
|
|
|
|
// it into bytes for storage in rocksdb. They also have a pack partial function which allows
|
|
|
|
// for serialization of a set number of fields, this is used i.e. for keys with a hash, tx_num
|
|
|
|
// and height where there multiple be multiple with the same hash but different tx_num and height.
|
|
|
|
|
|
|
|
import (
|
|
|
|
"bytes"
|
|
|
|
"encoding/binary"
|
|
|
|
"encoding/hex"
|
|
|
|
"fmt"
|
|
|
|
"log"
|
|
|
|
"reflect"
|
|
|
|
"sort"
|
|
|
|
"strings"
|
|
|
|
|
2022-08-09 13:43:01 +02:00
|
|
|
"github.com/lbryio/herald.go/internal"
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
"github.com/lbryio/lbcd/chaincfg/chainhash"
|
|
|
|
)
|
|
|
|
|
|
|
|
const (
|
|
|
|
ClaimToSupport = 'K'
|
|
|
|
SupportToClaim = 'L'
|
|
|
|
|
|
|
|
ClaimToTXO = 'E'
|
|
|
|
TXOToClaim = 'G'
|
|
|
|
|
|
|
|
ClaimToChannel = 'I'
|
|
|
|
ChannelToClaim = 'J'
|
|
|
|
|
|
|
|
ClaimShortIdPrefix = 'F'
|
2022-08-26 15:24:39 +02:00
|
|
|
BidOrder = 'D'
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
ClaimExpiration = 'O'
|
|
|
|
|
|
|
|
ClaimTakeover = 'P'
|
|
|
|
PendingActivation = 'Q'
|
|
|
|
ActivatedClaimAndSupport = 'R'
|
|
|
|
ActiveAmount = 'S'
|
|
|
|
|
|
|
|
Repost = 'V'
|
|
|
|
RepostedClaim = 'W'
|
|
|
|
|
|
|
|
Undo = 'M'
|
|
|
|
ClaimDiff = 'Y'
|
|
|
|
|
|
|
|
Tx = 'B'
|
|
|
|
BlockHash = 'C'
|
|
|
|
Header = 'H'
|
|
|
|
TxNum = 'N'
|
|
|
|
TxCount = 'T'
|
|
|
|
TxHash = 'X'
|
|
|
|
UTXO = 'u'
|
|
|
|
HashXUTXO = 'h'
|
|
|
|
HashXHistory = 'x'
|
|
|
|
DBState = 's'
|
|
|
|
ChannelCount = 'Z'
|
|
|
|
SupportAmount = 'a'
|
|
|
|
BlockTXs = 'b'
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
TrendingNotifications = 'c'
|
|
|
|
MempoolTx = 'd'
|
|
|
|
TouchedHashX = 'e'
|
|
|
|
HashXStatus = 'f'
|
|
|
|
HashXMempoolStatus = 'g'
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
EffectiveAmount = 'i'
|
|
|
|
RepostedCount = 'j'
|
|
|
|
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
ActivateClaimTXOType = 1
|
|
|
|
ActivatedSupportTXOType = 2
|
|
|
|
|
|
|
|
OnesCompTwiddle64 uint64 = 0xffffffffffffffff
|
|
|
|
OnesCompTwiddle32 uint32 = 0xffffffff
|
|
|
|
)
|
|
|
|
|
|
|
|
// GetPrefixes returns an array of all the byte prefix constants.
|
|
|
|
func GetPrefixes() [][]byte {
|
|
|
|
return [][]byte{
|
|
|
|
{ClaimToSupport},
|
|
|
|
{SupportToClaim},
|
|
|
|
{ClaimToTXO},
|
|
|
|
{TXOToClaim},
|
|
|
|
{ClaimToChannel},
|
|
|
|
{ChannelToClaim},
|
|
|
|
{ClaimShortIdPrefix},
|
2022-08-26 15:24:39 +02:00
|
|
|
{BidOrder},
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
{ClaimExpiration},
|
|
|
|
{ClaimTakeover},
|
|
|
|
{PendingActivation},
|
|
|
|
{ActivatedClaimAndSupport},
|
|
|
|
{ActiveAmount},
|
|
|
|
{Repost},
|
|
|
|
{RepostedClaim},
|
|
|
|
{Undo},
|
|
|
|
{ClaimDiff},
|
|
|
|
{Tx},
|
|
|
|
{BlockHash},
|
|
|
|
{Header},
|
|
|
|
{TxNum},
|
|
|
|
{TxCount},
|
|
|
|
{TxHash},
|
|
|
|
{UTXO},
|
|
|
|
{HashXUTXO},
|
|
|
|
{HashXHistory},
|
|
|
|
{DBState},
|
|
|
|
{ChannelCount},
|
|
|
|
{SupportAmount},
|
|
|
|
{BlockTXs},
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
{TrendingNotifications},
|
|
|
|
{MempoolTx},
|
|
|
|
{TouchedHashX},
|
|
|
|
{HashXStatus},
|
|
|
|
{HashXMempoolStatus},
|
2022-08-26 15:24:39 +02:00
|
|
|
{EffectiveAmount},
|
|
|
|
{RepostedCount},
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// PrefixRowKV is a generic key/value pair for a prefix.
|
|
|
|
type PrefixRowKV struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Key BaseKey
|
|
|
|
Value BaseValue
|
|
|
|
RawKey []byte
|
|
|
|
RawValue []byte
|
|
|
|
}
|
|
|
|
|
|
|
|
type BaseKey interface {
|
|
|
|
NumFields() int
|
|
|
|
PartialPack(fields int) []byte
|
|
|
|
PackKey() []byte
|
|
|
|
}
|
|
|
|
|
|
|
|
type BaseValue interface {
|
|
|
|
PackValue() []byte
|
|
|
|
}
|
|
|
|
|
|
|
|
type KeyUnpacker interface {
|
|
|
|
UnpackKey(buf []byte)
|
|
|
|
}
|
|
|
|
|
|
|
|
type ValueUnpacker interface {
|
|
|
|
UnpackValue(buf []byte)
|
|
|
|
}
|
|
|
|
|
|
|
|
type LengthEncodedName struct {
|
|
|
|
NameLen uint16 `struct:"sizeof=Name"`
|
|
|
|
Name string `json:"name"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewLengthEncodedName(s string) LengthEncodedName {
|
|
|
|
return LengthEncodedName{
|
|
|
|
NameLen: uint16(len(s)),
|
|
|
|
Name: s,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type LengthEncodedNormalizedName struct {
|
|
|
|
NormalizedNameLen uint16 `struct:"sizeof=NormalizedName"`
|
|
|
|
NormalizedName string `json:"normalized_name"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewLengthEncodedNormalizedName(s string) LengthEncodedNormalizedName {
|
|
|
|
return LengthEncodedNormalizedName{
|
|
|
|
NormalizedNameLen: uint16(len(s)),
|
|
|
|
NormalizedName: s,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type LengthEncodedPartialClaimId struct {
|
|
|
|
PartialClaimIdLen uint8 `struct:"sizeof=PartialClaimId"`
|
|
|
|
PartialClaimId string `json:"partial_claim_id"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewLengthEncodedPartialClaimId(s string) LengthEncodedPartialClaimId {
|
|
|
|
return LengthEncodedPartialClaimId{
|
|
|
|
PartialClaimIdLen: uint8(len(s)),
|
|
|
|
PartialClaimId: s,
|
|
|
|
}
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type DBStateKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type DBStateValue struct {
|
|
|
|
Genesis *chainhash.Hash
|
|
|
|
Height uint32
|
|
|
|
TxCount uint32
|
|
|
|
Tip *chainhash.Hash
|
|
|
|
UtxoFlushCount uint32
|
|
|
|
WallTime uint32
|
|
|
|
FirstSync bool
|
|
|
|
DDVersion uint8
|
|
|
|
HistFlushCount int32
|
|
|
|
CompFlushCount int32
|
|
|
|
CompCursor int32
|
|
|
|
EsSyncHeight uint32
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewDBStateValue() *DBStateValue {
|
|
|
|
return &DBStateValue{
|
|
|
|
Genesis: new(chainhash.Hash),
|
|
|
|
Height: 0,
|
|
|
|
TxCount: 0,
|
|
|
|
Tip: new(chainhash.Hash),
|
|
|
|
UtxoFlushCount: 0,
|
|
|
|
WallTime: 0,
|
|
|
|
FirstSync: true,
|
|
|
|
DDVersion: 0,
|
|
|
|
HistFlushCount: 0,
|
|
|
|
CompFlushCount: -1,
|
|
|
|
CompCursor: -1,
|
|
|
|
EsSyncHeight: 0,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewDBStateKey() *DBStateKey {
|
|
|
|
return &DBStateKey{
|
|
|
|
Prefix: []byte{DBState},
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *DBStateKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
n := prefixLen
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *DBStateValue) PackValue() []byte {
|
|
|
|
// b'>32sLL32sLLBBlllL'
|
|
|
|
n := 32 + 4 + 4 + 32 + 4 + 4 + 1 + 1 + 4 + 4 + 4 + 4
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.Genesis[:32])
|
|
|
|
binary.BigEndian.PutUint32(value[32:], v.Height)
|
|
|
|
binary.BigEndian.PutUint32(value[32+4:], v.TxCount)
|
|
|
|
copy(value[32+4+4:], v.Tip[:32])
|
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32:], v.UtxoFlushCount)
|
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32+4:], v.WallTime)
|
|
|
|
var bitSetVar uint8
|
|
|
|
if v.FirstSync {
|
|
|
|
bitSetVar = 1
|
|
|
|
}
|
|
|
|
value[32+4+4+32+4+4] = bitSetVar
|
|
|
|
value[32+4+4+32+4+4+1] = v.DDVersion
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
|
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32+4+4+1+1:], uint32(v.HistFlushCount))
|
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32+4+4+1+1+4:], uint32(v.CompFlushCount))
|
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32+4+4+1+1+4+4:], uint32(v.CompCursor))
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
binary.BigEndian.PutUint32(value[32+4+4+32+4+4+1+1+4+4+4:], v.EsSyncHeight)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *DBStateKey) NumFields() int {
|
|
|
|
return 0
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *DBStateKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func DBStateKeyUnpack(key []byte) *DBStateKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &DBStateKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func DBStateValueUnpack(value []byte) *DBStateValue {
|
|
|
|
genesis := (*chainhash.Hash)(value[:32])
|
|
|
|
tip := (*chainhash.Hash)(value[32+4+4 : 32+4+4+32])
|
|
|
|
x := &DBStateValue{
|
|
|
|
Genesis: genesis,
|
|
|
|
Height: binary.BigEndian.Uint32(value[32:]),
|
|
|
|
TxCount: binary.BigEndian.Uint32(value[32+4:]),
|
|
|
|
Tip: tip,
|
|
|
|
UtxoFlushCount: binary.BigEndian.Uint32(value[32+4+4+32:]),
|
|
|
|
WallTime: binary.BigEndian.Uint32(value[32+4+4+32+4:]),
|
|
|
|
FirstSync: value[32+4+4+32+4+4] == 1,
|
|
|
|
DDVersion: value[32+4+4+32+4+4+1],
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
HistFlushCount: int32(binary.BigEndian.Uint32(value[32+4+4+32+4+4+1+1:])),
|
|
|
|
CompFlushCount: int32(binary.BigEndian.Uint32(value[32+4+4+32+4+4+1+1+4:])),
|
|
|
|
CompCursor: int32(binary.BigEndian.Uint32(value[32+4+4+32+4+4+1+1+4+4:])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
EsSyncHeight: binary.BigEndian.Uint32(value[32+4+4+32+4+4+1+1+4+4+4:]),
|
|
|
|
}
|
|
|
|
return x
|
|
|
|
}
|
|
|
|
|
|
|
|
type UndoKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint64 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type UndoValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Data []byte `struct-while:"!_eof" json:"data"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *UndoKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 8
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint64(key[prefixLen:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *UndoValue) PackValue() []byte {
|
|
|
|
len := len(v.Data)
|
|
|
|
value := make([]byte, len)
|
|
|
|
copy(value, v.Data)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *UndoKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *UndoKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 8
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint64(key[prefixLen:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func UndoKeyUnpack(key []byte) *UndoKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &UndoKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint64(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func UndoValueUnpack(value []byte) *UndoValue {
|
|
|
|
return &UndoValue{
|
|
|
|
Data: value,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type UTXOKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
HashX []byte `struct:"[11]byte" json:"hashx"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Nout uint16 `json:"nout"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type UTXOValue struct {
|
|
|
|
Amount uint64 `json:"amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXUTXOKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ShortTXHash []byte `struct:"[4]byte" json:"short_tx_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Nout uint16 `json:"nout"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXUTXOValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
HashX []byte `struct:"[11]byte" json:"hashx"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
//
|
|
|
|
// HashXUTXOKey / HashXUTXOValue
|
|
|
|
//
|
|
|
|
|
|
|
|
func (k *HashXUTXOKey) String() string {
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(short_tx_hash=%s, tx_num=%d, nout=%d)",
|
|
|
|
reflect.TypeOf(k),
|
|
|
|
hex.EncodeToString(k.ShortTXHash),
|
|
|
|
k.TxNum,
|
|
|
|
k.Nout,
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *HashXUTXOValue) String() string {
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(hashX=%s)",
|
|
|
|
reflect.TypeOf(v),
|
|
|
|
hex.EncodeToString(v.HashX),
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *HashXUTXOKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>4sLH'
|
|
|
|
n := prefixLen + 4 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ShortTXHash)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+4:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+8:], k.Nout)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *HashXUTXOValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 11)
|
|
|
|
copy(value, v.HashX)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *HashXUTXOKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// HashXUTXOKeyPackPartial packs a variable number of fields into a byte
|
|
|
|
// array
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *HashXUTXOKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ShortTXHash)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+4:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+8:], k.Nout)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func HashXUTXOKeyUnpack(key []byte) *HashXUTXOKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &HashXUTXOKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ShortTXHash: key[prefixLen : prefixLen+4],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+4:]),
|
|
|
|
Nout: binary.BigEndian.Uint16(key[prefixLen+8:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func HashXUTXOValueUnpack(value []byte) *HashXUTXOValue {
|
|
|
|
return &HashXUTXOValue{
|
|
|
|
HashX: value[:11],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXHistoryKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
HashX []byte `struct:"[11]byte" json:"hashx"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXHistoryValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
HashXes []uint16 `struct-while:"!_eof" json:"hashxes"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *HashXHistoryKey) String() string {
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(hashx=%s, height=%d)",
|
|
|
|
reflect.TypeOf(k),
|
|
|
|
hex.EncodeToString(k.HashX),
|
|
|
|
k.Height,
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *HashXHistoryKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>11sL'
|
|
|
|
n := prefixLen + 11 + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.HashX)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+11:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *HashXHistoryValue) PackValue() []byte {
|
|
|
|
n := len(v.HashXes)
|
|
|
|
value := make([]byte, n*2)
|
|
|
|
for i, x := range v.HashXes {
|
|
|
|
binary.BigEndian.PutUint16(value[i*2:], x)
|
|
|
|
}
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *HashXHistoryKey) NumFields() int {
|
|
|
|
return 2
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// HashXHistoryKeyPackPartial packs a variable number of fields into a byte
|
|
|
|
// array
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *HashXHistoryKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 2 {
|
|
|
|
fields = 2
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 11
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.HashX[:11])
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+11:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func HashXHistoryKeyUnpack(key []byte) *HashXHistoryKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &HashXHistoryKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
HashX: key[prefixLen : prefixLen+11],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen+11:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func HashXHistoryValueUnpack(value []byte) *HashXHistoryValue {
|
|
|
|
n := len(value) / 2
|
|
|
|
hashxes := make([]uint16, n)
|
|
|
|
for i := 0; i < n; i++ {
|
|
|
|
hashxes[i] = binary.BigEndian.Uint16(value[i*2:])
|
|
|
|
}
|
|
|
|
return &HashXHistoryValue{
|
|
|
|
HashXes: hashxes,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockHashKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockHashValue struct {
|
|
|
|
BlockHash *chainhash.Hash `json:"block_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewBlockHashKey(height uint32) *BlockHashKey {
|
|
|
|
return &BlockHashKey{
|
|
|
|
Prefix: []byte{BlockHash},
|
|
|
|
Height: height,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *BlockHashKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *BlockHashValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 32)
|
|
|
|
copy(value, v.BlockHash[:32])
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *BlockHashKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *BlockHashKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockHashKeyUnpack(key []byte) *BlockHashKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &BlockHashKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockHashValueUnpack(value []byte) *BlockHashValue {
|
|
|
|
hash := (*chainhash.Hash)(value)
|
|
|
|
return &BlockHashValue{
|
|
|
|
BlockHash: hash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockTxsKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockTxsValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
TxHashes []*chainhash.Hash `struct-while:"!_eof" json:"tx_hashes"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *BlockTxsKey) NewBlockTxsKey(height uint32) *BlockTxsKey {
|
|
|
|
return &BlockTxsKey{
|
|
|
|
Prefix: []byte{BlockTXs},
|
|
|
|
Height: height,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *BlockTxsKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *BlockTxsValue) PackValue() []byte {
|
|
|
|
numHashes := len(v.TxHashes)
|
|
|
|
n := numHashes * 32
|
|
|
|
value := make([]byte, n)
|
|
|
|
|
|
|
|
for i, tx := range v.TxHashes {
|
|
|
|
if len(tx) != 32 {
|
|
|
|
log.Println("Warning, txhash not 32 bytes", tx)
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
copy(value[i*32:], tx[:])
|
|
|
|
}
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *BlockTxsKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *BlockTxsKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockTxsKeyUnpack(key []byte) *BlockTxsKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &BlockTxsKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockTxsValueUnpack(value []byte) *BlockTxsValue {
|
|
|
|
numHashes := len(value) / 32
|
|
|
|
txs := make([]*chainhash.Hash, numHashes)
|
|
|
|
for i := 0; i < numHashes; i++ {
|
|
|
|
txs[i] = (*chainhash.Hash)(value[i*32 : (i+1)*32])
|
|
|
|
}
|
|
|
|
return &BlockTxsValue{
|
|
|
|
TxHashes: txs,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxCountKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxCountValue struct {
|
|
|
|
TxCount uint32 `json:"tx_count"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewTxCountKey(height uint32) *TxCountKey {
|
|
|
|
return &TxCountKey{
|
|
|
|
Prefix: []byte{TxCount},
|
|
|
|
Height: height,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TxCountKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TxCountValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 4)
|
|
|
|
binary.BigEndian.PutUint32(value, v.TxCount)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TxCountKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TxCountKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxCountKeyUnpack(key []byte) *TxCountKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TxCountKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxCountValueUnpack(value []byte) *TxCountValue {
|
|
|
|
return &TxCountValue{
|
|
|
|
TxCount: binary.BigEndian.Uint32(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxHashKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxHashValue struct {
|
|
|
|
TxHash *chainhash.Hash `json:"tx_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewTxHashKey(txNum uint32) *TxHashKey {
|
|
|
|
return &TxHashKey{
|
|
|
|
Prefix: []byte{TxHash},
|
|
|
|
TxNum: txNum,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TxHashKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TxHashValue) PackValue() []byte {
|
|
|
|
n := len(v.TxHash)
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.TxHash[:n])
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TxHashKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TxHashKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxHashKeyUnpack(key []byte) *TxHashKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TxHashKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxHashValueUnpack(value []byte) *TxHashValue {
|
|
|
|
return &TxHashValue{
|
|
|
|
TxHash: (*chainhash.Hash)(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxNumKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxHash *chainhash.Hash `json:"tx_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxNumValue struct {
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TxNumKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 32
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.TxHash[:32])
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TxNumValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 4)
|
|
|
|
binary.BigEndian.PutUint32(value, v.TxNum)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TxNumKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TxNumKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 32
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.TxHash[:32])
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxNumKeyUnpack(key []byte) *TxNumKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TxNumKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxHash: (*chainhash.Hash)(key[prefixLen : prefixLen+32]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxNumValueUnpack(value []byte) *TxNumValue {
|
|
|
|
return &TxNumValue{
|
|
|
|
TxNum: binary.BigEndian.Uint32(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type TxKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
TxHash *chainhash.Hash `struct:"*[32]byte" json:"tx_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type TxValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
RawTx []byte `struct-while:"!_eof" json:"raw_tx"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TxKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 32
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.TxHash[:32])
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TxValue) PackValue() []byte {
|
|
|
|
value := make([]byte, len(v.RawTx))
|
|
|
|
copy(value, v.RawTx)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TxKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TxKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 32
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.TxHash[:32])
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxKeyUnpack(key []byte) *TxKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TxKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxHash: (*chainhash.Hash)(key[prefixLen : prefixLen+32]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TxValueUnpack(value []byte) *TxValue {
|
|
|
|
return &TxValue{
|
|
|
|
RawTx: value,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockHeaderKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type BlockHeaderValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Header []byte `struct:"[112]byte" json:"header"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *BlockHeaderValue) Equals(v *BlockHeaderValue) bool {
|
|
|
|
return bytes.Equal(k.Header, v.Header)
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewHeaderKey(height uint32) *BlockHeaderKey {
|
|
|
|
return &BlockHeaderKey{
|
|
|
|
Prefix: []byte{Header},
|
|
|
|
Height: height,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *BlockHeaderKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *BlockHeaderValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 112)
|
|
|
|
copy(value, v.Header)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *BlockHeaderKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *BlockHeaderKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockHeaderKeyUnpack(key []byte) *BlockHeaderKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &BlockHeaderKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func BlockHeaderValueUnpack(value []byte) *BlockHeaderValue {
|
|
|
|
return &BlockHeaderValue{
|
|
|
|
Header: value[:112],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToTXOKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToTXOValue struct {
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
RootTxNum uint32 `json:"root_tx_num"`
|
|
|
|
RootPosition uint16 `json:"root_position"`
|
|
|
|
Amount uint64 `json:"amount"`
|
|
|
|
ChannelSignatureIsValid bool `json:"channel_signature_is_valid"`
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
LengthEncodedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewClaimToTXOKey(claimHash []byte) *ClaimToTXOKey {
|
|
|
|
return &ClaimToTXOKey{
|
|
|
|
Prefix: []byte{ClaimToTXO},
|
|
|
|
ClaimHash: claimHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimToTXOValue) NormalizedName() string {
|
|
|
|
//TODO implement? Might not need to do anything.
|
|
|
|
return internal.NormalizeName(v.Name)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimToTXOKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20s'
|
|
|
|
n := prefixLen + 20
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimToTXOValue) PackValue() []byte {
|
|
|
|
nameLen := len(v.Name)
|
|
|
|
n := 4 + 2 + 4 + 2 + 8 + 1 + 2 + nameLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
binary.BigEndian.PutUint32(value, v.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(value[4:], v.Position)
|
|
|
|
binary.BigEndian.PutUint32(value[6:], v.RootTxNum)
|
|
|
|
binary.BigEndian.PutUint16(value[10:], v.RootPosition)
|
|
|
|
binary.BigEndian.PutUint64(value[12:], v.Amount)
|
|
|
|
var bitSetVar uint8
|
|
|
|
if v.ChannelSignatureIsValid {
|
|
|
|
bitSetVar = 1
|
|
|
|
}
|
|
|
|
value[20] = bitSetVar
|
|
|
|
binary.BigEndian.PutUint16(value[21:], uint16(nameLen))
|
|
|
|
copy(value[23:], []byte(v.Name[:nameLen]))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimToTXOKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimToTXOKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToTXOKeyUnpack(key []byte) *ClaimToTXOKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ClaimToTXOKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToTXOValueUnpack(value []byte) *ClaimToTXOValue {
|
|
|
|
nameLen := binary.BigEndian.Uint16(value[21:])
|
|
|
|
return &ClaimToTXOValue{
|
|
|
|
TxNum: binary.BigEndian.Uint32(value),
|
|
|
|
Position: binary.BigEndian.Uint16(value[4:]),
|
|
|
|
RootTxNum: binary.BigEndian.Uint32(value[6:]),
|
|
|
|
RootPosition: binary.BigEndian.Uint16(value[10:]),
|
|
|
|
Amount: binary.BigEndian.Uint64(value[12:]),
|
|
|
|
ChannelSignatureIsValid: value[20] == 1,
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
LengthEncodedName: NewLengthEncodedName(string(value[23 : 23+nameLen])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type TXOToClaimKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TXOToClaimValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
LengthEncodedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewTXOToClaimKey(txNum uint32, position uint16) *TXOToClaimKey {
|
|
|
|
return &TXOToClaimKey{
|
|
|
|
Prefix: []byte{TXOToClaim},
|
|
|
|
TxNum: txNum,
|
|
|
|
Position: position,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TXOToClaimKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>LH'
|
|
|
|
n := prefixLen + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+4:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TXOToClaimValue) PackValue() []byte {
|
|
|
|
nameLen := len(v.Name)
|
|
|
|
n := 20 + 2 + nameLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint16(value[20:], uint16(nameLen))
|
|
|
|
copy(value[22:], []byte(v.Name))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TXOToClaimKey) NumFields() int {
|
|
|
|
return 2
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TXOToClaimKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 2 {
|
|
|
|
fields = 2
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+4:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TXOToClaimKeyUnpack(key []byte) *TXOToClaimKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TXOToClaimKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+4:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TXOToClaimValueUnpack(value []byte) *TXOToClaimValue {
|
|
|
|
nameLen := binary.BigEndian.Uint16(value[20:])
|
|
|
|
return &TXOToClaimValue{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash: value[:20],
|
|
|
|
LengthEncodedName: NewLengthEncodedName(string(value[22 : 22+nameLen])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimShortIDKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
|
|
|
LengthEncodedPartialClaimId // fields PartialClaimIdLen, PartialClaimId
|
|
|
|
RootTxNum uint32 `json:"root_tx_num"`
|
|
|
|
RootPosition uint16 `json:"root_position"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimShortIDValue struct {
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewClaimShortIDKey(normalizedName, partialClaimId string) *ClaimShortIDKey {
|
|
|
|
return &ClaimShortIDKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: []byte{ClaimShortIdPrefix},
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(normalizedName),
|
|
|
|
LengthEncodedPartialClaimId: NewLengthEncodedPartialClaimId(partialClaimId),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimShortIDKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
partialClaimLen := len(k.PartialClaimId)
|
|
|
|
log.Printf("nameLen: %d, partialClaimLen: %d\n", nameLen, partialClaimLen)
|
|
|
|
n := prefixLen + 2 + nameLen + 1 + partialClaimLen + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName[:nameLen]))
|
|
|
|
key[prefixLen+2+nameLen] = uint8(partialClaimLen)
|
|
|
|
copy(key[prefixLen+2+nameLen+1:], []byte(k.PartialClaimId[:partialClaimLen]))
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+2+nameLen+1+partialClaimLen:], k.RootTxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+2+nameLen+1+partialClaimLen+4:], k.RootPosition)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimShortIDValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 6)
|
|
|
|
binary.BigEndian.PutUint32(value, v.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(value[4:], v.Position)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimShortIDKey) NumFields() int {
|
|
|
|
return 4
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimShortIDKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 4 {
|
|
|
|
fields = 4
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
partialClaimLen := len(k.PartialClaimId)
|
|
|
|
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 2 + nameLen
|
|
|
|
case 2:
|
|
|
|
n += 1 + partialClaimLen
|
|
|
|
case 3:
|
|
|
|
n += 4
|
|
|
|
case 4:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName))
|
|
|
|
case 2:
|
|
|
|
key[prefixLen+2+nameLen] = uint8(partialClaimLen)
|
|
|
|
copy(key[prefixLen+2+nameLen+1:], []byte(k.PartialClaimId))
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+2+nameLen+1+partialClaimLen:], k.RootTxNum)
|
|
|
|
case 4:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+2+nameLen+1+partialClaimLen+4:], k.RootPosition)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimShortIDKeyUnpack(key []byte) *ClaimShortIDKey {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := int(binary.BigEndian.Uint16(key[prefixLen:]))
|
|
|
|
partialClaimLen := int(uint8(key[prefixLen+2+nameLen]))
|
|
|
|
return &ClaimShortIDKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(key[prefixLen+2 : prefixLen+2+nameLen])),
|
|
|
|
LengthEncodedPartialClaimId: NewLengthEncodedPartialClaimId(string(key[prefixLen+2+nameLen+1 : prefixLen+2+nameLen+1+partialClaimLen])),
|
|
|
|
RootTxNum: binary.BigEndian.Uint32(key[prefixLen+2+nameLen+1+partialClaimLen:]),
|
|
|
|
RootPosition: binary.BigEndian.Uint16(key[prefixLen+2+nameLen+1+partialClaimLen+4:]),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimShortIDValueUnpack(value []byte) *ClaimShortIDValue {
|
|
|
|
return &ClaimShortIDValue{
|
|
|
|
TxNum: binary.BigEndian.Uint32(value),
|
|
|
|
Position: binary.BigEndian.Uint16(value[4:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToChannelKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToChannelValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
SigningHash []byte `struct:"[20]byte" json:"signing_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewClaimToChannelKey(claimHash []byte, txNum uint32, position uint16) *ClaimToChannelKey {
|
|
|
|
return &ClaimToChannelKey{
|
|
|
|
Prefix: []byte{ClaimToChannel},
|
|
|
|
ClaimHash: claimHash,
|
|
|
|
TxNum: txNum,
|
|
|
|
Position: position,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimToChannelKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sLH'
|
|
|
|
n := prefixLen + 20 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimToChannelValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 20)
|
|
|
|
copy(value, v.SigningHash[:20])
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimToChannelKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimToChannelKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToChannelKeyUnpack(key []byte) *ClaimToChannelKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ClaimToChannelKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+20:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+24:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToChannelValueUnpack(value []byte) *ClaimToChannelValue {
|
|
|
|
return &ClaimToChannelValue{
|
|
|
|
SigningHash: value[:20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ChannelToClaimKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
SigningHash []byte `struct:"[20]byte" json:"signing_hash"`
|
|
|
|
LengthEncodedName // fields NameLen, Name
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type ChannelToClaimValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewChannelToClaimKey(channelHash []byte, normalizedName string) *ChannelToClaimKey {
|
|
|
|
return &ChannelToClaimKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: []byte{ChannelToClaim},
|
|
|
|
SigningHash: channelHash,
|
|
|
|
LengthEncodedName: NewLengthEncodedName(normalizedName),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewChannelToClaimKeyWHash(channelHash []byte) *ChannelToClaimKey {
|
|
|
|
return &ChannelToClaimKey{
|
|
|
|
Prefix: []byte{ChannelToClaim},
|
|
|
|
SigningHash: channelHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ChannelToClaimKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := len(k.Name)
|
|
|
|
n := prefixLen + 20 + 2 + nameLen + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.SigningHash[:20])
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+20:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+22:], []byte(k.Name[:nameLen]))
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+22+nameLen:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+22+nameLen+4:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ChannelToClaimValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 20)
|
|
|
|
copy(value, v.ClaimHash[:20])
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ChannelToClaimKey) NumFields() int {
|
|
|
|
return 4
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ChannelToClaimKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 4 {
|
|
|
|
fields = 4
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
nameLen := len(k.Name)
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
case 2:
|
|
|
|
n += 2 + nameLen
|
|
|
|
case 3:
|
|
|
|
n += 4
|
|
|
|
case 4:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.SigningHash[:20])
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+20:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+22:], []byte(k.Name))
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+22+nameLen:], k.TxNum)
|
|
|
|
case 4:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+22+nameLen+4:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ChannelToClaimKeyUnpack(key []byte) *ChannelToClaimKey {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := int(binary.BigEndian.Uint16(key[prefixLen+20:]))
|
|
|
|
return &ChannelToClaimKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
SigningHash: key[prefixLen : prefixLen+20],
|
|
|
|
LengthEncodedName: NewLengthEncodedName(string(key[prefixLen+22 : prefixLen+22+nameLen])),
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+22+nameLen:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+22+nameLen+4:]),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ChannelToClaimValueUnpack(value []byte) *ChannelToClaimValue {
|
|
|
|
return &ChannelToClaimValue{
|
|
|
|
ClaimHash: value[:20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ChannelCountKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ChannelHash []byte `struct:"[20]byte" json:"channel_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type ChannelCountValue struct {
|
|
|
|
Count uint32 `json:"count"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewChannelCountKey(channelHash []byte) *ChannelCountKey {
|
|
|
|
return &ChannelCountKey{
|
|
|
|
Prefix: []byte{ChannelCount},
|
|
|
|
ChannelHash: channelHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ChannelCountKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sLH'
|
|
|
|
n := prefixLen + 20
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ChannelHash[:20])
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ChannelCountValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 4)
|
|
|
|
binary.BigEndian.PutUint32(value, v.Count)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ChannelCountKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ChannelCountKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ChannelHash)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ChannelCountKeyUnpack(key []byte) *ChannelCountKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ChannelCountKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ChannelHash: key[prefixLen : prefixLen+20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ChannelCountValueUnpack(value []byte) *ChannelCountValue {
|
|
|
|
return &ChannelCountValue{
|
|
|
|
Count: binary.BigEndian.Uint32(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type SupportAmountKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type SupportAmountValue struct {
|
|
|
|
Amount uint64 `json:"amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewSupportAmountKey(claimHash []byte) *SupportAmountKey {
|
|
|
|
return &SupportAmountKey{
|
|
|
|
Prefix: []byte{SupportAmount},
|
|
|
|
ClaimHash: claimHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *SupportAmountKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sLH'
|
|
|
|
n := prefixLen + 20
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *SupportAmountValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 8)
|
|
|
|
binary.BigEndian.PutUint64(value, v.Amount)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *SupportAmountKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *SupportAmountKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func SupportAmountKeyUnpack(key []byte) *SupportAmountKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &SupportAmountKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func SupportAmountValueUnpack(value []byte) *SupportAmountValue {
|
|
|
|
return &SupportAmountValue{
|
|
|
|
Amount: binary.BigEndian.Uint64(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToSupportKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimToSupportValue struct {
|
|
|
|
Amount uint64 `json:"amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimToSupportKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sLH'
|
|
|
|
n := prefixLen + 20 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimToSupportValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 8)
|
|
|
|
binary.BigEndian.PutUint64(value, v.Amount)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimToSupportKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimToSupportKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToSupportKeyUnpack(key []byte) *ClaimToSupportKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ClaimToSupportKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+20:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+24:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimToSupportValueUnpack(value []byte) *ClaimToSupportValue {
|
|
|
|
return &ClaimToSupportValue{
|
|
|
|
Amount: binary.BigEndian.Uint64(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type SupportToClaimKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type SupportToClaimValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *SupportToClaimKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>LH'
|
|
|
|
n := prefixLen + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+4:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *SupportToClaimValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 20)
|
|
|
|
copy(value, v.ClaimHash)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *SupportToClaimKey) NumFields() int {
|
|
|
|
return 2
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *SupportToClaimKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 2 {
|
|
|
|
fields = 2
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.TxNum)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+4:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func SupportToClaimKeyUnpack(key []byte) *SupportToClaimKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &SupportToClaimKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+4:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func SupportToClaimValueUnpack(value []byte) *SupportToClaimValue {
|
|
|
|
return &SupportToClaimValue{
|
|
|
|
ClaimHash: value[:20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimExpirationKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Expiration uint32 `json:"expiration"`
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimExpirationValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimExpirationKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>LLH'
|
|
|
|
n := prefixLen + 4 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Expiration)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+4:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+8:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimExpirationValue) PackValue() []byte {
|
|
|
|
nameLen := len(v.NormalizedName)
|
|
|
|
n := 20 + 2 + nameLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.ClaimHash)
|
|
|
|
binary.BigEndian.PutUint16(value[20:], uint16(nameLen))
|
|
|
|
copy(value[22:], []byte(v.NormalizedName))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimExpirationKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimExpirationKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Expiration)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+4:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+8:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimExpirationKeyUnpack(key []byte) *ClaimExpirationKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ClaimExpirationKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Expiration: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+4:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+8:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimExpirationValueUnpack(value []byte) *ClaimExpirationValue {
|
|
|
|
nameLen := binary.BigEndian.Uint16(value[20:])
|
|
|
|
return &ClaimExpirationValue{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash: value[:20],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(value[22 : 22+nameLen])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimTakeoverKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type ClaimTakeoverValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewClaimTakeoverKey(normalizedName string) *ClaimTakeoverKey {
|
|
|
|
return &ClaimTakeoverKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: []byte{ClaimTakeover},
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(normalizedName),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimTakeoverValue) String() string {
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(claim_hash=%s, height=%d)",
|
|
|
|
reflect.TypeOf(v),
|
|
|
|
hex.EncodeToString(v.ClaimHash),
|
|
|
|
v.Height,
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ClaimTakeoverKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
n := prefixLen + 2 + nameLen
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName))
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ClaimTakeoverValue) PackValue() []byte {
|
|
|
|
// b'>20sL'
|
|
|
|
value := make([]byte, 24)
|
|
|
|
copy(value, v.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint32(value[20:], uint32(v.Height))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ClaimTakeoverKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ClaimTakeoverKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 2 + nameLen
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimTakeoverKeyUnpack(key []byte) *ClaimTakeoverKey {
|
|
|
|
prefixLen := 1
|
|
|
|
nameLen := binary.BigEndian.Uint16(key[prefixLen:])
|
|
|
|
return &ClaimTakeoverKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(key[prefixLen+2 : prefixLen+2+int(nameLen)])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ClaimTakeoverValueUnpack(value []byte) *ClaimTakeoverValue {
|
|
|
|
return &ClaimTakeoverValue{
|
|
|
|
ClaimHash: value[:20],
|
|
|
|
Height: binary.BigEndian.Uint32(value[20:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type PendingActivationKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
TxoType uint8 `json:"txo_type"`
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *PendingActivationKey) IsSupport() bool {
|
|
|
|
return k.TxoType == ActivatedSupportTXOType
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *PendingActivationKey) IsClaim() bool {
|
|
|
|
return k.TxoType == ActivateClaimTXOType
|
|
|
|
}
|
|
|
|
|
|
|
|
type PendingActivationValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (k *PendingActivationKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>LBLH'
|
|
|
|
n := prefixLen + 4 + 1 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
key[prefixLen+4] = k.TxoType
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+5:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+9:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *PendingActivationValue) PackValue() []byte {
|
|
|
|
nameLen := len(v.NormalizedName)
|
|
|
|
n := 20 + 2 + nameLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint16(value[20:], uint16(nameLen))
|
|
|
|
copy(value[22:], []byte(v.NormalizedName))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *PendingActivationKey) NumFields() int {
|
|
|
|
return 4
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *PendingActivationKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 4 {
|
|
|
|
fields = 4
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>4sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
case 2:
|
|
|
|
n += 1
|
|
|
|
case 3:
|
|
|
|
n += 4
|
|
|
|
case 4:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], k.Height)
|
|
|
|
case 2:
|
|
|
|
key[prefixLen+4] = k.TxoType
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+5:], k.TxNum)
|
|
|
|
case 4:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+9:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func PendingActivationKeyUnpack(key []byte) *PendingActivationKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &PendingActivationKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: binary.BigEndian.Uint32(key[prefixLen:]),
|
|
|
|
TxoType: key[prefixLen+4],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+5:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+9:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func PendingActivationValueUnpack(value []byte) *PendingActivationValue {
|
|
|
|
nameLen := binary.BigEndian.Uint16(value[20:])
|
|
|
|
return &PendingActivationValue{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash: value[:20],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(value[22 : 22+nameLen])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ActivationKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxoType uint8 `json:"txo_type"`
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type ActivationValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Height uint32 `json:"height"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewActivationKey(txoType uint8, txNum uint32, position uint16) *ActivationKey {
|
|
|
|
return &ActivationKey{
|
|
|
|
Prefix: []byte{ActivatedClaimAndSupport},
|
|
|
|
TxoType: txoType,
|
|
|
|
TxNum: txNum,
|
|
|
|
Position: position,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ActivationKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>BLH'
|
|
|
|
n := prefixLen + 1 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], []byte{k.TxoType})
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+1:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+5:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ActivationValue) PackValue() []byte {
|
|
|
|
nameLen := len(v.NormalizedName)
|
|
|
|
n := 4 + 20 + 2 + nameLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
binary.BigEndian.PutUint32(value, v.Height)
|
|
|
|
copy(value[4:], v.ClaimHash[:20])
|
|
|
|
binary.BigEndian.PutUint16(value[24:], uint16(nameLen))
|
|
|
|
copy(value[26:], []byte(v.NormalizedName))
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ActivationKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ActivationKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 1
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
key[prefixLen] = k.TxoType
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+1:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+5:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ActivationKeyUnpack(key []byte) *ActivationKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ActivationKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
TxoType: key[prefixLen],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+1:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+5:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ActivationValueUnpack(value []byte) *ActivationValue {
|
|
|
|
nameLen := binary.BigEndian.Uint16(value[24:])
|
|
|
|
return &ActivationValue{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Height: binary.BigEndian.Uint32(value),
|
|
|
|
ClaimHash: value[4 : 20+4],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(value[26 : 26+nameLen])),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type ActiveAmountKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxoType uint8 `json:"txo_type"`
|
|
|
|
ActivationHeight uint32 `json:"activation_height"`
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type ActiveAmountValue struct {
|
|
|
|
Amount uint64 `json:"amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewActiveAmountKey(claimHash []byte, txoType uint8, activationHeight uint32) *ActiveAmountKey {
|
|
|
|
return &ActiveAmountKey{
|
|
|
|
Prefix: []byte{ActiveAmount},
|
|
|
|
ClaimHash: claimHash,
|
|
|
|
TxoType: txoType,
|
|
|
|
ActivationHeight: activationHeight,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *ActiveAmountKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sBLLH'
|
|
|
|
n := prefixLen + 20 + 1 + 4 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash[:20])
|
|
|
|
copy(key[prefixLen+20:], []byte{k.TxoType})
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20+1:], k.ActivationHeight)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20+1+4:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+20+1+4+4:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *ActiveAmountValue) PackValue() []byte {
|
|
|
|
// b'>Q'
|
|
|
|
value := make([]byte, 8)
|
|
|
|
binary.BigEndian.PutUint64(value, v.Amount)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *ActiveAmountKey) NumFields() int {
|
|
|
|
return 5
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *ActiveAmountKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 5 {
|
|
|
|
fields = 5
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
case 2:
|
|
|
|
n += 1
|
|
|
|
case 3:
|
|
|
|
n += 4
|
|
|
|
case 4:
|
|
|
|
n += 4
|
|
|
|
case 5:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash)
|
|
|
|
case 2:
|
|
|
|
copy(key[prefixLen+20:], []byte{k.TxoType})
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20+1:], k.ActivationHeight)
|
|
|
|
case 4:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20+1+4:], k.TxNum)
|
|
|
|
case 5:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+20+1+4+4:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func ActiveAmountKeyUnpack(key []byte) *ActiveAmountKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &ActiveAmountKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
TxoType: uint8(key[prefixLen+20 : prefixLen+20+1][0]),
|
|
|
|
ActivationHeight: binary.BigEndian.Uint32(key[prefixLen+20+1:]),
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+20+1+4:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+20+1+4+4:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func ActiveAmountValueUnpack(value []byte) *ActiveAmountValue {
|
|
|
|
return &ActiveAmountValue{
|
|
|
|
Amount: binary.BigEndian.Uint64(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
type OnesComplementEffectiveAmount uint64
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
type BidOrderKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
LengthEncodedNormalizedName // fields NormalizedNameLen, NormalizedName
|
|
|
|
EffectiveAmount OnesComplementEffectiveAmount `json:"effective_amount"`
|
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
type BidOrderValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func NewBidOrderKey(normalizedName string) *BidOrderKey {
|
|
|
|
return &BidOrderKey{
|
|
|
|
Prefix: []byte{BidOrder},
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(normalizedName),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func (k *BidOrderKey) PackKey() []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
prefixLen := 1
|
|
|
|
// 2 byte length field, plus number of bytes in name
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
nameLenLen := 2 + nameLen
|
|
|
|
// b'>QLH'
|
|
|
|
n := prefixLen + nameLenLen + 8 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName))
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
binary.BigEndian.PutUint64(key[prefixLen+nameLenLen:], OnesCompTwiddle64-uint64(k.EffectiveAmount))
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+nameLenLen+8:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+nameLenLen+8+4:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func (v *BidOrderValue) PackValue() []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// b'>20s'
|
|
|
|
value := make([]byte, 20)
|
|
|
|
copy(value, v.ClaimHash[:20])
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func (kv *BidOrderKey) NumFields() int {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return 4
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func (k *BidOrderKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
nameLen := len(k.NormalizedName)
|
|
|
|
nameLenLen := 2 + nameLen
|
|
|
|
if fields > 4 {
|
|
|
|
fields = 4
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 2 + nameLen
|
|
|
|
case 2:
|
|
|
|
n += 8
|
|
|
|
case 3:
|
|
|
|
n += 4
|
|
|
|
case 4:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen:], uint16(nameLen))
|
|
|
|
copy(key[prefixLen+2:], []byte(k.NormalizedName))
|
|
|
|
case 2:
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
binary.BigEndian.PutUint64(key[prefixLen+nameLenLen:], OnesCompTwiddle64-uint64(k.EffectiveAmount))
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+nameLenLen+8:], k.TxNum)
|
|
|
|
case 4:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+nameLenLen+8+4:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func BidOrderKeyUnpack(key []byte) *BidOrderKey {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
prefixLen := 1
|
|
|
|
nameLen := binary.BigEndian.Uint16(key[prefixLen:])
|
2022-08-26 15:24:39 +02:00
|
|
|
return &BidOrderKey{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
LengthEncodedNormalizedName: NewLengthEncodedNormalizedName(string(key[prefixLen+2 : prefixLen+2+int(nameLen)])),
|
|
|
|
EffectiveAmount: OnesComplementEffectiveAmount(OnesCompTwiddle64 - binary.BigEndian.Uint64(key[prefixLen+2+int(nameLen):])),
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+2+int(nameLen)+8:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+2+int(nameLen)+8+4:]),
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
func BidOrderValueUnpack(value []byte) *BidOrderValue {
|
|
|
|
return &BidOrderValue{
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
ClaimHash: value[:20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type RepostKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
type RepostValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
RepostedClaimHash []byte `struct:"[20]byte" json:"reposted_claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewRepostKey(claimHash []byte) *RepostKey {
|
|
|
|
return &RepostKey{
|
|
|
|
Prefix: []byte{Repost},
|
|
|
|
ClaimHash: claimHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *RepostKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20s'
|
|
|
|
n := prefixLen + 20
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.ClaimHash)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *RepostValue) PackValue() []byte {
|
|
|
|
// FIXME: Is there a limit to this length?
|
|
|
|
n := len(v.RepostedClaimHash)
|
|
|
|
value := make([]byte, n)
|
|
|
|
copy(value, v.RepostedClaimHash)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *RepostKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *RepostKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.ClaimHash)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func RepostKeyUnpack(key []byte) *RepostKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &RepostKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
ClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func RepostValueUnpack(value []byte) *RepostValue {
|
|
|
|
return &RepostValue{
|
|
|
|
RepostedClaimHash: value[:],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type RepostedKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
RepostedClaimHash []byte `struct:"[20]byte" json:"reposted_claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
TxNum uint32 `json:"tx_num"`
|
|
|
|
Position uint16 `json:"position"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type RepostedValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func NewRepostedKey(claimHash []byte) *RepostedKey {
|
|
|
|
return &RepostedKey{
|
|
|
|
Prefix: []byte{RepostedClaim},
|
|
|
|
RepostedClaimHash: claimHash,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *RepostedKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>20sLH'
|
|
|
|
n := prefixLen + 20 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.RepostedClaimHash)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *RepostedValue) PackValue() []byte {
|
|
|
|
// b'>20s'
|
|
|
|
value := make([]byte, 20)
|
|
|
|
copy(value, v.ClaimHash)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *RepostedKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *RepostedKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 20
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.RepostedClaimHash)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+20:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+24:], k.Position)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func RepostedKeyUnpack(key []byte) *RepostedKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &RepostedKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
RepostedClaimHash: key[prefixLen : prefixLen+20],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+20:]),
|
|
|
|
Position: binary.BigEndian.Uint16(key[prefixLen+24:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func RepostedValueUnpack(value []byte) *RepostedValue {
|
|
|
|
return &RepostedValue{
|
|
|
|
ClaimHash: value[:20],
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
type RepostedCountKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type RepostedCountValue struct {
|
|
|
|
RepostedCount uint32 `json:"reposted_count"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountKey) NumFields() int {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>20s'
|
|
|
|
n := len(kv.Prefix) + 20
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Prefix[:1])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
offset += copy(buf[offset:], kv.ClaimHash[:20])
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>20s'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.ClaimHash = buf[offset : offset+20]
|
|
|
|
offset += 20
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountValue) PackValue() []byte {
|
|
|
|
// b'>L'
|
|
|
|
n := 4
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
binary.BigEndian.PutUint32(buf[offset:], kv.RepostedCount)
|
|
|
|
offset += 4
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *RepostedCountValue) UnpackValue(buf []byte) {
|
|
|
|
// b'>L'
|
|
|
|
offset := 0
|
|
|
|
kv.RepostedCount = binary.BigEndian.Uint32(buf[offset:])
|
|
|
|
offset += 4
|
|
|
|
}
|
|
|
|
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
type TouchedOrDeletedClaimKey struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
Height int32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TouchedOrDeletedClaimValue struct {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
TouchedClaimsLen uint32 `struct:"sizeof=TouchedClaims"`
|
|
|
|
DeletedClaimsLen uint32 `struct:"sizeof=DeletedClaims"`
|
|
|
|
TouchedClaims [][]byte `struct:"[][20]byte" json:"touched_claims"`
|
|
|
|
DeletedClaims [][]byte `struct:"[][20]byte" json:"deleted_claims"`
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TouchedOrDeletedClaimValue) String() string {
|
|
|
|
touchedSB := strings.Builder{}
|
|
|
|
touchedLen := len(v.TouchedClaims)
|
|
|
|
for i, claim := range v.TouchedClaims {
|
|
|
|
touchedSB.WriteString(hex.EncodeToString(claim))
|
|
|
|
if i < touchedLen-1 {
|
|
|
|
touchedSB.WriteString(",")
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
deletedSB := strings.Builder{}
|
|
|
|
deletedLen := len(v.DeletedClaims)
|
|
|
|
for i, claim := range v.DeletedClaims {
|
|
|
|
deletedSB.WriteString(hex.EncodeToString(claim))
|
|
|
|
if i < deletedLen-1 {
|
|
|
|
deletedSB.WriteString(",")
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(touched_claims=%s, deleted_claims=%s)",
|
|
|
|
reflect.TypeOf(v),
|
|
|
|
touchedSB.String(),
|
|
|
|
deletedSB.String(),
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *TouchedOrDeletedClaimKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>L'
|
|
|
|
n := prefixLen + 4
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], uint32(k.Height))
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *TouchedOrDeletedClaimValue) PackValue() []byte {
|
|
|
|
var touchedLen, deletedLen uint32 = 0, 0
|
|
|
|
if v.TouchedClaims != nil {
|
|
|
|
for _, claim := range v.TouchedClaims {
|
|
|
|
if len(claim) != 20 {
|
|
|
|
log.Println("TouchedOrDeletedClaimValue: claim not length 20?!?")
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
touchedLen = uint32(len(v.TouchedClaims))
|
|
|
|
}
|
|
|
|
if v.DeletedClaims != nil {
|
|
|
|
for _, claim := range v.DeletedClaims {
|
|
|
|
if len(claim) != 20 {
|
|
|
|
log.Println("TouchedOrDeletedClaimValue: claim not length 20?!?")
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
deletedLen = uint32(len(v.DeletedClaims))
|
|
|
|
}
|
|
|
|
n := 4 + 4 + 20*touchedLen + 20*deletedLen
|
|
|
|
value := make([]byte, n)
|
|
|
|
binary.BigEndian.PutUint32(value, touchedLen)
|
|
|
|
binary.BigEndian.PutUint32(value[4:], deletedLen)
|
|
|
|
// These are sorted for consistency with the Python implementation
|
|
|
|
sort.Slice(v.TouchedClaims, func(i, j int) bool { return bytes.Compare(v.TouchedClaims[i], v.TouchedClaims[j]) < 0 })
|
|
|
|
sort.Slice(v.DeletedClaims, func(i, j int) bool { return bytes.Compare(v.DeletedClaims[i], v.DeletedClaims[j]) < 0 })
|
|
|
|
|
|
|
|
var i = 8
|
|
|
|
for j := 0; j < int(touchedLen); j++ {
|
|
|
|
copy(value[i:], v.TouchedClaims[j])
|
|
|
|
i += 20
|
|
|
|
}
|
|
|
|
for j := 0; j < int(deletedLen); j++ {
|
|
|
|
copy(value[i:], v.DeletedClaims[j])
|
|
|
|
i += 20
|
|
|
|
}
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *TouchedOrDeletedClaimKey) NumFields() int {
|
|
|
|
return 1
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *TouchedOrDeletedClaimKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 1 {
|
|
|
|
fields = 1
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 4
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen:], uint32(k.Height))
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func TouchedOrDeletedClaimKeyUnpack(key []byte) *TouchedOrDeletedClaimKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &TouchedOrDeletedClaimKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
Height: int32(binary.BigEndian.Uint32(key[prefixLen:])),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func TouchedOrDeletedClaimValueUnpack(value []byte) *TouchedOrDeletedClaimValue {
|
|
|
|
touchedLen := binary.BigEndian.Uint32(value)
|
|
|
|
deletedLen := binary.BigEndian.Uint32(value[4:])
|
|
|
|
touchedClaims := make([][]byte, touchedLen)
|
|
|
|
deletedClaims := make([][]byte, deletedLen)
|
|
|
|
var j = 8
|
|
|
|
for i := 0; i < int(touchedLen); i++ {
|
|
|
|
touchedClaims[i] = value[j : j+20]
|
|
|
|
j += 20
|
|
|
|
}
|
|
|
|
for i := 0; i < int(deletedLen); i++ {
|
|
|
|
deletedClaims[i] = value[j : j+20]
|
|
|
|
j += 20
|
|
|
|
}
|
|
|
|
return &TouchedOrDeletedClaimValue{
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
TouchedClaimsLen: touchedLen,
|
|
|
|
DeletedClaimsLen: deletedLen,
|
|
|
|
TouchedClaims: touchedClaims,
|
|
|
|
DeletedClaims: deletedClaims,
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *UTXOKey) String() string {
|
|
|
|
return fmt.Sprintf(
|
|
|
|
"%s(hashX=%s, tx_num=%d, nout=%d)",
|
|
|
|
reflect.TypeOf(k),
|
|
|
|
hex.EncodeToString(k.HashX),
|
|
|
|
k.TxNum,
|
|
|
|
k.Nout,
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *UTXOKey) PackKey() []byte {
|
|
|
|
prefixLen := 1
|
|
|
|
// b'>11sLH'
|
|
|
|
n := prefixLen + 11 + 4 + 2
|
|
|
|
key := make([]byte, n)
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
copy(key[prefixLen:], k.HashX)
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+11:], k.TxNum)
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+15:], k.Nout)
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func (k *UTXOValue) PackValue() []byte {
|
|
|
|
value := make([]byte, 8)
|
|
|
|
binary.BigEndian.PutUint64(value, k.Amount)
|
|
|
|
|
|
|
|
return value
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (kv *UTXOKey) NumFields() int {
|
|
|
|
return 3
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// UTXOKeyPackPartial packs a variable number of fields for a UTXOKey into
|
|
|
|
// a byte array.
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func (k *UTXOKey) PartialPack(fields int) []byte {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
// Limit fields between 0 and number of fields, we always at least need
|
|
|
|
// the prefix, and we never need to iterate past the number of fields.
|
|
|
|
if fields > 3 {
|
|
|
|
fields = 3
|
|
|
|
}
|
|
|
|
if fields < 0 {
|
|
|
|
fields = 0
|
|
|
|
}
|
|
|
|
|
|
|
|
// b'>11sLH'
|
|
|
|
prefixLen := 1
|
|
|
|
var n = prefixLen
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 1:
|
|
|
|
n += 11
|
|
|
|
case 2:
|
|
|
|
n += 4
|
|
|
|
case 3:
|
|
|
|
n += 2
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
key := make([]byte, n)
|
|
|
|
|
|
|
|
for i := 0; i <= fields; i++ {
|
|
|
|
switch i {
|
|
|
|
case 0:
|
|
|
|
copy(key, k.Prefix)
|
|
|
|
case 1:
|
|
|
|
copy(key[prefixLen:], k.HashX)
|
|
|
|
case 2:
|
|
|
|
binary.BigEndian.PutUint32(key[prefixLen+11:], k.TxNum)
|
|
|
|
case 3:
|
|
|
|
binary.BigEndian.PutUint16(key[prefixLen+15:], k.Nout)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return key
|
|
|
|
}
|
|
|
|
|
|
|
|
func UTXOKeyUnpack(key []byte) *UTXOKey {
|
|
|
|
prefixLen := 1
|
|
|
|
return &UTXOKey{
|
|
|
|
Prefix: key[:prefixLen],
|
|
|
|
HashX: key[prefixLen : prefixLen+11],
|
|
|
|
TxNum: binary.BigEndian.Uint32(key[prefixLen+11:]),
|
|
|
|
Nout: binary.BigEndian.Uint16(key[prefixLen+15:]),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func UTXOValueUnpack(value []byte) *UTXOValue {
|
|
|
|
return &UTXOValue{
|
|
|
|
Amount: binary.BigEndian.Uint64(value),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
type TrendingNotificationKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
Height uint32 `json:"height"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TrendingNotificationValue struct {
|
|
|
|
PreviousAmount uint64 `json:"previous_amount"`
|
|
|
|
NewAmount uint64 `json:"new_amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationKey) NumFields() int {
|
|
|
|
return 2
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>L20s'
|
|
|
|
n := len(kv.Prefix) + 4 + 20
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf, kv.Prefix[offset:])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
binary.BigEndian.PutUint32(buf[offset:], kv.Height)
|
|
|
|
offset += 4
|
|
|
|
if fields -= 1; fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
offset += copy(buf[offset:], kv.ClaimHash[:20])
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>L20s'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.Height = binary.BigEndian.Uint32(buf[offset:])
|
|
|
|
offset += 4
|
|
|
|
kv.ClaimHash = buf[offset : offset+20]
|
|
|
|
offset += 20
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationValue) PackValue() []byte {
|
|
|
|
// b'>QQ'
|
|
|
|
n := 8 + 8
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
binary.BigEndian.PutUint64(buf[offset:], kv.PreviousAmount)
|
|
|
|
offset += 8
|
|
|
|
binary.BigEndian.PutUint64(buf[offset:], kv.NewAmount)
|
|
|
|
offset += 8
|
|
|
|
return buf
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TrendingNotificationValue) UnpackValue(buf []byte) {
|
|
|
|
// b'>QQ'
|
|
|
|
offset := 0
|
|
|
|
kv.PreviousAmount = binary.BigEndian.Uint64(buf[offset:])
|
|
|
|
offset += 8
|
|
|
|
kv.NewAmount = binary.BigEndian.Uint64(buf[offset:])
|
|
|
|
offset += 8
|
|
|
|
}
|
|
|
|
|
|
|
|
type MempoolTxKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
TxHash []byte `struct:"[32]byte" json:"tx_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type MempoolTxValue struct {
|
|
|
|
RawTx []byte `struct-while:"!_eof" json:"raw_tx"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxKey) NumFields() int {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>32s'
|
|
|
|
n := len(kv.Prefix) + 32
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Prefix[:1])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
offset += copy(buf[offset:], kv.TxHash[:32])
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>32s'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.TxHash = buf[offset : offset+32]
|
|
|
|
offset += 32
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxValue) PackValue() []byte {
|
|
|
|
// variable length bytes
|
|
|
|
n := len(kv.RawTx)
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf, kv.RawTx)
|
|
|
|
return buf
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *MempoolTxValue) UnpackValue(buf []byte) {
|
|
|
|
// variable length bytes
|
|
|
|
offset := 0
|
|
|
|
kv.RawTx = buf[:]
|
|
|
|
offset += len(buf)
|
|
|
|
}
|
|
|
|
|
|
|
|
type TouchedHashXKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
Height uint32 `json:"height"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type TouchedHashXValue struct {
|
|
|
|
TouchedHashXs [][]byte `struct:"[][11]byte" struct-while:"!_eof" json:"touched_hashXs"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXKey) NumFields() int {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>L'
|
|
|
|
n := len(kv.Prefix) + 4
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Prefix[:1])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
binary.BigEndian.PutUint32(buf[offset:], kv.Height)
|
|
|
|
offset += 4
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>L'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.Height = binary.BigEndian.Uint32(buf[offset:])
|
|
|
|
offset += 4
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXValue) PackValue() []byte {
|
|
|
|
// variable length bytes
|
|
|
|
n := len(kv.TouchedHashXs) * 11
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
for i := range kv.TouchedHashXs {
|
|
|
|
offset += copy(buf[offset:], kv.TouchedHashXs[i][:11])
|
|
|
|
}
|
|
|
|
return buf
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *TouchedHashXValue) UnpackValue(buf []byte) {
|
|
|
|
// variable length bytes
|
|
|
|
n := len(buf)
|
|
|
|
kv.TouchedHashXs = make([][]byte, n/11)
|
|
|
|
for i, offset := 0, 0; offset+11 <= n; i, offset = i+1, offset+11 {
|
|
|
|
kv.TouchedHashXs[i] = buf[offset : offset+11]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXStatusKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
HashX []byte `struct:"[20]byte" json:"hashX"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXStatusValue struct {
|
|
|
|
Status []byte `struct:"[32]byte" json:"status"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusKey) NumFields() int {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>20s'
|
|
|
|
n := len(kv.Prefix) + 20
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Prefix[:1])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
offset += copy(buf[offset:], kv.HashX[:20])
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>20s'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.HashX = buf[offset : offset+20]
|
|
|
|
offset += 20
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusValue) PackValue() []byte {
|
|
|
|
// b'32s'
|
|
|
|
n := 32
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Status[:32])
|
|
|
|
return buf
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *HashXStatusValue) UnpackValue(buf []byte) {
|
|
|
|
// b'32s'
|
|
|
|
offset := 0
|
|
|
|
kv.Status = buf[offset : offset+32]
|
|
|
|
offset += 32
|
|
|
|
}
|
|
|
|
|
|
|
|
type HashXMempoolStatusKey = HashXStatusKey
|
|
|
|
type HashXMempoolStatusValue = HashXStatusValue
|
|
|
|
|
2022-08-26 15:24:39 +02:00
|
|
|
type EffectiveAmountKey struct {
|
|
|
|
Prefix []byte `struct:"[1]byte" json:"prefix"`
|
|
|
|
ClaimHash []byte `struct:"[20]byte" json:"claim_hash"`
|
|
|
|
}
|
|
|
|
|
|
|
|
type EffectiveAmountValue struct {
|
|
|
|
EffectiveAmount uint64 `json:"effective_amount"`
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountKey) NumFields() int {
|
|
|
|
return 1
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountKey) PartialPack(fields int) []byte {
|
|
|
|
// b'>20s'
|
|
|
|
n := len(kv.Prefix) + 20
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
offset += copy(buf[offset:], kv.Prefix[:1])
|
|
|
|
if fields <= 0 {
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
offset += copy(buf[offset:], kv.ClaimHash[:20])
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountKey) PackKey() []byte {
|
|
|
|
return kv.PartialPack(kv.NumFields())
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountKey) UnpackKey(buf []byte) {
|
|
|
|
// b'>20s'
|
|
|
|
offset := 0
|
|
|
|
kv.Prefix = buf[offset : offset+1]
|
|
|
|
offset += 1
|
|
|
|
kv.ClaimHash = buf[offset : offset+20]
|
|
|
|
offset += 20
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountValue) PackValue() []byte {
|
|
|
|
// b'>Q'
|
|
|
|
n := 8
|
|
|
|
buf := make([]byte, n)
|
|
|
|
offset := 0
|
|
|
|
binary.BigEndian.PutUint64(buf[offset:], kv.EffectiveAmount)
|
|
|
|
offset += 8
|
|
|
|
return buf[:offset]
|
|
|
|
}
|
|
|
|
|
|
|
|
func (kv *EffectiveAmountValue) UnpackValue(buf []byte) {
|
|
|
|
// b'>Q'
|
|
|
|
offset := 0
|
|
|
|
kv.EffectiveAmount = binary.BigEndian.Uint64(buf[offset:])
|
|
|
|
offset += 8
|
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func UnpackGenericKey(key []byte) (BaseKey, error) {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
if len(key) == 0 {
|
|
|
|
return nil, fmt.Errorf("key length zero")
|
|
|
|
}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
// Look up the prefix metadata, and use the registered function(s)
|
|
|
|
// to create and unpack key of appropriate type.
|
|
|
|
t, ok := prefixRegistry[key[0]]
|
|
|
|
if !ok {
|
|
|
|
return nil, fmt.Errorf("unpack key function for %v not implemented", key[0])
|
|
|
|
}
|
|
|
|
if t.newKeyUnpack != nil {
|
|
|
|
// Type provides <Type>KeyUnpack() function.
|
|
|
|
return t.newKeyUnpack(key).(BaseKey), nil
|
|
|
|
}
|
|
|
|
if t.newKey != nil {
|
|
|
|
// Type provides a new<Type> function.
|
|
|
|
k := t.newKey()
|
|
|
|
unpacker, ok := k.(KeyUnpacker)
|
|
|
|
if ok {
|
|
|
|
unpacker.UnpackKey(key)
|
|
|
|
return unpacker.(BaseKey), nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return nil, fmt.Errorf("unpack key function for %v not implemented", key[0])
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func UnpackGenericValue(key, value []byte) (BaseValue, error) {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
if len(key) == 0 {
|
|
|
|
return nil, fmt.Errorf("key length zero")
|
|
|
|
}
|
|
|
|
if len(value) == 0 {
|
|
|
|
return nil, fmt.Errorf("value length zero")
|
|
|
|
}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
// Look up the prefix metadata, and use the registered function(s)
|
|
|
|
// to create and unpack value of appropriate type.
|
|
|
|
t, ok := prefixRegistry[key[0]]
|
|
|
|
if !ok {
|
|
|
|
return nil, fmt.Errorf("unpack value function for %v not implemented", key[0])
|
|
|
|
}
|
|
|
|
if t.newValueUnpack != nil {
|
|
|
|
// Type provides <Type>ValueUnpack() function.
|
|
|
|
return t.newValueUnpack(value).(BaseValue), nil
|
|
|
|
}
|
|
|
|
if t.newValue != nil {
|
|
|
|
// Type provides a new<Type> function.
|
|
|
|
k := t.newValue()
|
|
|
|
unpacker, ok := k.(ValueUnpacker)
|
|
|
|
if ok {
|
|
|
|
unpacker.UnpackValue(value)
|
|
|
|
return unpacker.(BaseValue), nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return nil, fmt.Errorf("unpack key function for %v not implemented", key[0])
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func PackPartialGenericKey(key BaseKey, fields int) ([]byte, error) {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
if key == nil {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return nil, fmt.Errorf("key is nil")
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return key.PartialPack(fields), nil
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func PackGenericKey(key BaseKey) ([]byte, error) {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
if key == nil {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return nil, fmt.Errorf("key is nil")
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return key.PackKey(), nil
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
func PackGenericValue(value BaseValue) ([]byte, error) {
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
if value == nil {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
return nil, fmt.Errorf("value is nil")
|
|
|
|
}
|
|
|
|
return value.PackValue(), nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// Metadata associated with each prefix/table. Currently used to
|
|
|
|
// implement generic unpacking.
|
|
|
|
|
|
|
|
type prefixMeta struct {
|
|
|
|
newKey func() interface{}
|
|
|
|
newValue func() interface{}
|
|
|
|
newKeyUnpack func([]byte) interface{}
|
|
|
|
newValueUnpack func([]byte) interface{}
|
|
|
|
API *SerializationAPI
|
|
|
|
}
|
|
|
|
|
|
|
|
var prefixRegistry = map[byte]prefixMeta{
|
|
|
|
ClaimToSupport: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimToSupportKey{Prefix: []byte{ClaimToSupport}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimToSupportValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToSupportKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToSupportValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
SupportToClaim: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &SupportToClaimKey{Prefix: []byte{SupportToClaim}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &SupportToClaimValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return SupportToClaimKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return SupportToClaimValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
ClaimToTXO: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimToTXOKey{Prefix: []byte{ClaimToTXO}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimToTXOValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToTXOKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToTXOValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
TXOToClaim: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TXOToClaimKey{Prefix: []byte{TXOToClaim}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TXOToClaimValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TXOToClaimKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TXOToClaimValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
ClaimToChannel: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimToChannelKey{Prefix: []byte{ClaimToChannel}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimToChannelValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToChannelKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimToChannelValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
ChannelToClaim: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ChannelToClaimKey{Prefix: []byte{ChannelToClaim}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ChannelToClaimValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ChannelToClaimKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ChannelToClaimValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
ClaimShortIdPrefix: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimShortIDKey{Prefix: []byte{ClaimShortIdPrefix}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimShortIDValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimShortIDKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimShortIDValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
2022-08-26 15:24:39 +02:00
|
|
|
BidOrder: {
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
newKey: func() interface{} {
|
2022-08-26 15:24:39 +02:00
|
|
|
return &BidOrderKey{Prefix: []byte{BidOrder}}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
2022-08-26 15:24:39 +02:00
|
|
|
return &BidOrderValue{}
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
2022-08-26 15:24:39 +02:00
|
|
|
return BidOrderKeyUnpack(buf)
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
2022-08-26 15:24:39 +02:00
|
|
|
return BidOrderValueUnpack(buf)
|
Catchup to python-herald schema. Plus lots of refactoring. (#49)
* Make prefixes_test.go more resilient against garbage left
by a prior crash. Also correct error logging.
* Don't do the ones' complement thing with DBStateValue fields
HistFlushCount, CompFlushCount, CompCursor. Python-herald
doesn't do it, and it presents one more irregular case for
(un)marshalling fields.
* Simplify type-specific partial packing, and simplify dispatch for pack key/value.
* Add struct field annotations and refactor to prepare for
use of "restruct" generic packing/unpacking.
* Add dynamic pack/unpack based on "restruct" module.
Dispatch normal pack/unpack through tableRegistry[] map
instead of switch.
* Add 5 new prefixes/tables (TrendingNotifications..HashXMempoolStatus).
* Undo rename. TouchedOrDeleted -> ClaimDiff.
* Fixup callers of eliminated partial pack functions. Have them use key.PartialPack(n).
* Add pluggable SerializationAPI. Use it in prefixes_test.
Populate PrefixRowKV.RawKey,RawValue when appropriate.
* Undo accidental bump of rocksdb version.
* Add .vscode dir to gitignore.
* Fix ClaimToChannelValue annotation. Implement BlockTxsValue workaround
as I can't find the right annotation to get it marshalled/unmarshalled.
* Strengthen partial packing verification. Fix bugs
in UnpackKey/UnpackValue for new types.
* Remove .DS_Store, and ignore in future.
* Fix MempoolTxKey, TouchedHashXValue. Remove some unneeded struct tags.
* Generate test data and complete the tests for the new tables.
Add Fuzz tests for TouchedHashXKey, TouchedHashXValue with
happy path test data (only).
* Move tableRegistry to prefixes.go and rename it prefixRegistry.
Other minor fixes, comments.
* Add test that runs through GetPrefixes() contents, and verifies
they are registered in prefixRegistry.
2022-08-16 07:45:41 +02:00
|
|
|
},
|
|
|
|
},
|
|
|
|
ClaimExpiration: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimExpirationKey{Prefix: []byte{ClaimExpiration}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimExpirationValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimExpirationKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimExpirationValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
ClaimTakeover: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ClaimTakeoverKey{Prefix: []byte{ClaimTakeover}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ClaimTakeoverValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimTakeoverKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ClaimTakeoverValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
PendingActivation: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &PendingActivationKey{Prefix: []byte{PendingActivation}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &PendingActivationValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return PendingActivationKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return PendingActivationValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
ActivatedClaimAndSupport: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ActivationKey{Prefix: []byte{ActivatedClaimAndSupport}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ActivationValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ActivationKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ActivationValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
ActiveAmount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ActiveAmountKey{Prefix: []byte{ActiveAmount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ActiveAmountValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ActiveAmountKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ActiveAmountValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
Repost: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &RepostKey{Prefix: []byte{Repost}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &RepostValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return RepostKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return RepostValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
RepostedClaim: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &RepostedKey{Prefix: []byte{RepostedClaim}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &RepostedValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return RepostedKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return RepostedValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
Undo: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &UndoKey{Prefix: []byte{Undo}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &UndoValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return UndoKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return UndoValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
ClaimDiff: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TouchedOrDeletedClaimKey{Prefix: []byte{ClaimDiff}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TouchedOrDeletedClaimValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TouchedOrDeletedClaimKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TouchedOrDeletedClaimValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
Tx: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TxKey{Prefix: []byte{Tx}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TxValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
BlockHash: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &BlockHashKey{Prefix: []byte{BlockHash}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &BlockHashValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockHashKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockHashValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
Header: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &BlockHeaderKey{Prefix: []byte{Header}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &BlockHeaderValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockHeaderKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockHeaderValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
TxNum: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TxNumKey{Prefix: []byte{TxNum}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TxNumValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxNumKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxNumValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
TxCount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TxCountKey{Prefix: []byte{TxCount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TxCountValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxCountKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxCountValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
TxHash: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TxHashKey{Prefix: []byte{TxHash}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TxHashValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxHashKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return TxHashValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
UTXO: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &UTXOKey{Prefix: []byte{UTXO}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &UTXOValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return UTXOKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return UTXOValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
HashXUTXO: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &HashXUTXOKey{Prefix: []byte{HashXUTXO}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &HashXUTXOValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return HashXUTXOKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return HashXUTXOValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
HashXHistory: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &HashXHistoryKey{Prefix: []byte{HashXHistory}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &HashXHistoryValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return HashXHistoryKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return HashXHistoryValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
DBState: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &DBStateKey{Prefix: []byte{DBState}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &DBStateValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return DBStateKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return DBStateValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
ChannelCount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &ChannelCountKey{Prefix: []byte{ChannelCount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &ChannelCountValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return ChannelCountKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return ChannelCountValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
SupportAmount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &SupportAmountKey{Prefix: []byte{SupportAmount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &SupportAmountValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return SupportAmountKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return SupportAmountValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
BlockTXs: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &BlockTxsKey{Prefix: []byte{BlockTXs}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &BlockTxsValue{}
|
|
|
|
},
|
|
|
|
newKeyUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockTxsKeyUnpack(buf)
|
|
|
|
},
|
|
|
|
newValueUnpack: func(buf []byte) interface{} {
|
|
|
|
return BlockTxsValueUnpack(buf)
|
|
|
|
},
|
|
|
|
},
|
|
|
|
|
|
|
|
TrendingNotifications: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TrendingNotificationKey{Prefix: []byte{TrendingNotifications}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TrendingNotificationValue{}
|
|
|
|
},
|
|
|
|
},
|
|
|
|
MempoolTx: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &MempoolTxKey{Prefix: []byte{MempoolTx}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &MempoolTxValue{}
|
|
|
|
},
|
|
|
|
},
|
|
|
|
TouchedHashX: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &TouchedHashXKey{Prefix: []byte{TouchedHashX}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &TouchedHashXValue{}
|
|
|
|
},
|
|
|
|
},
|
|
|
|
HashXStatus: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &HashXStatusKey{Prefix: []byte{HashXStatus}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &HashXStatusValue{}
|
|
|
|
},
|
|
|
|
},
|
|
|
|
HashXMempoolStatus: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &HashXMempoolStatusKey{Prefix: []byte{HashXMempoolStatus}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &HashXMempoolStatusValue{}
|
|
|
|
},
|
|
|
|
},
|
2022-08-26 15:24:39 +02:00
|
|
|
RepostedCount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &RepostedCountKey{Prefix: []byte{RepostedCount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &RepostedCountValue{}
|
|
|
|
},
|
|
|
|
},
|
|
|
|
EffectiveAmount: {
|
|
|
|
newKey: func() interface{} {
|
|
|
|
return &EffectiveAmountKey{Prefix: []byte{EffectiveAmount}}
|
|
|
|
},
|
|
|
|
newValue: func() interface{} {
|
|
|
|
return &EffectiveAmountValue{}
|
|
|
|
},
|
|
|
|
},
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|