rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
package server
|
|
|
|
|
|
|
|
import (
|
|
|
|
"encoding/binary"
|
|
|
|
"net"
|
|
|
|
|
2022-08-09 13:43:01 +02:00
|
|
|
"github.com/lbryio/herald.go/internal"
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
"github.com/sirupsen/logrus"
|
|
|
|
)
|
|
|
|
|
|
|
|
const NotifierResponseLength = 40
|
|
|
|
|
|
|
|
// AddHeightSub adds a new height subscriber
|
|
|
|
func (s *Server) AddHeightSub(addr net.Addr, conn net.Conn) {
|
|
|
|
s.HeightSubsMut.Lock()
|
|
|
|
defer s.HeightSubsMut.Unlock()
|
|
|
|
s.HeightSubs[addr] = conn
|
|
|
|
}
|
|
|
|
|
|
|
|
// DoNotify sends a notification to all height subscribers
|
|
|
|
func (s *Server) DoNotify(heightHash *internal.HeightHash) error {
|
|
|
|
buff := make([]byte, NotifierResponseLength)
|
|
|
|
toDelete := make([]net.Addr, 0)
|
|
|
|
|
|
|
|
s.HeightSubsMut.RLock()
|
|
|
|
for addr, conn := range s.HeightSubs {
|
|
|
|
// struct.pack(b'>Q32s', height, block_hash)
|
|
|
|
binary.BigEndian.PutUint64(buff, heightHash.Height)
|
|
|
|
copy(buff[8:], heightHash.BlockHash[:32])
|
|
|
|
logrus.Tracef("notifying %s", addr)
|
|
|
|
n, err := conn.Write(buff)
|
|
|
|
if err != nil {
|
|
|
|
logrus.Warn(err)
|
|
|
|
toDelete = append(toDelete, addr)
|
|
|
|
}
|
|
|
|
if n != NotifierResponseLength {
|
|
|
|
logrus.Warn("not all bytes written")
|
|
|
|
}
|
|
|
|
}
|
|
|
|
s.HeightSubsMut.RUnlock()
|
|
|
|
|
|
|
|
if len(toDelete) > 0 {
|
|
|
|
s.HeightSubsMut.Lock()
|
|
|
|
for _, v := range toDelete {
|
|
|
|
delete(s.HeightSubs, v)
|
|
|
|
}
|
|
|
|
s.HeightSubsMut.Unlock()
|
|
|
|
}
|
|
|
|
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// RunNotifier Runs the notfying action forever
|
|
|
|
func (s *Server) RunNotifier() error {
|
Add subscribe/unsubscribe RPCs. Add session, sessionManager, and serve JSON RPC (without HTTP). (#66)
* Move and rename BlockchainCodec, BlockchainCodecRequest.
These are not specifically "blockchain", rather they are
specific to how gorilla/rpc works.
* Move claimtrie-related service/handlers to jsonrpc_claimtrie.go.
* Pull out decode logic into named func newBlockHeaderElectrum().
* Rename BlockchainService -> BlockchainBlockService.
* Drop http.Request arg from handlers, and use RegisterTCPService().
* Implement GetStatus() to pull data from HashXStatus table.
* Make the service objects independent, so we don't have inheritance.
* Add core session/subscription logic (session.go).
Implement subsribe/unsubscribe handlers.
* Support both pure JSON and JSON-over-HTTP services.
Forward NotifierChan messages to sessionManager.
* Only assign default port (50001) if neither --json-rpc-port nor
--json-rpc-http-port are specified.
* Handle failures with goto instead of break. Update error logging.
* Add --max-sessions, --session-timeout args. Enforce max sessions.
* Changes to make session.go testable. Conn created with Pipe()
used in testing has no unique Addr.
* Add tests for headers, headers.subscribe, address.subscribe.
* HashXStatus, HashXMempoolStatus not populated by default. Fix GetStatus().
* Use time.Ticker object to drive management activity.
2022-10-04 16:05:06 +02:00
|
|
|
for notification := range s.NotifierChan {
|
|
|
|
switch notification.(type) {
|
|
|
|
case internal.HeightHash:
|
|
|
|
heightHash, _ := notification.(internal.HeightHash)
|
|
|
|
s.DoNotify(&heightHash)
|
|
|
|
}
|
|
|
|
s.sessionManager.doNotify(notification)
|
rocksdb (#29)
* Initial rocksdb commit
Basic reading from rocksdb works
* Try github action thing
* try local dockerfile
* asdf
* qwer
* asdf
* Try adding test db with git-lfs
* update action
* cleanup
* Don't hardcode stop on read
* Progress of reading rocksdb
* fixes and arg test
* asdf
* Fix rocksdb iterator and tests
* update script
* asdf
* Better iterator. Need to implement a lot of keys next, and tests, maybe
tests needed.
* asdf
* asdf
* asdf
* Implementation, testing, and cleanup.
Implemented more prefixes. Figured out a good test that should work for
all prefixes. Removed binary databases so we can just store human
readable csv files.
* more tests, prefixes and small refactor
* Another prefix
* EffectiveAmount
* ActiveAmount
* ActivatedClaimAndSupport
* PendingActivation
* ClaimTakeover
* ClaimExpiration
* SupportToClaim
* ClaimToSupport
* Fix bug with variable length keys
* ChannelToClaim
* ClaimToChannel
* ClaimShortID
* TXOToClaim
* ClaimToTXO
* BlockHeader
* BlockHash
* Undo
* HashXHistory
* Tx and big refactor
* rest the the keys
* Refactor and starting to add resolve
* asdf
* Refactor tests and add column families
* changes
* more work on implementing resolve
* code cleanup, function tests
* small code refactoring
* start building pieces of the test data set for full resolve.
* Export constant, add test
* another test
* TestGetTxHash
* more tests
* more tests
* More tests
* Refactor db functions into three files
* added slice backed stack, need to fix tests
* fix some issues with test suite
* some cleanup and adding arguments and db load / refresh to server command
* fix some bugs, start using logrus for leveled logging, upgrade to go 1.17, run go mod tidy
* logrus, protobuf updates, resolve grpc endpoint
* don't run integration test with unit tests
* signal handling and cleanup functions
* signal handling code files
* Unit tests for db stack
* reorganize bisect function so we lock it properly
* fix txcounts loading
* cleanup some logic around iterators and fix a bug where I was running two detect changes threads
* add some metrics
* cleanup
* blocking and filtering implemented
* add params for blocking and filtering channels and streams
* updates and fixes for integration tests
* use newer version of lbry.go when possible
* Add height endpoint and move string functions internal
* remove gitattributes, unused
* some cleanup
* more cleanup / refactor. almost ready for another review
* More cleanup
* use chainhash.Hash types from lbcd where appropriate
* update github action to go-1.17.8
* update go version needed
* trying to fix these builds
* cleanup
* trying to fix memory leak
* fix memory leak (iterator never finished so cleanup didn't run)
* changes per code review
* remove lbry.go v2
* rename sort.go search.go
* fix test
2022-04-29 17:04:01 +02:00
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// NotifierServer implements the TCP protocol for height/blockheader notifications
|
|
|
|
func (s *Server) NotifierServer() error {
|
|
|
|
address := ":" + s.Args.NotifierPort
|
|
|
|
addr, err := net.ResolveTCPAddr("tcp", address)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
listen, err := net.ListenTCP("tcp", addr)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
defer listen.Close()
|
|
|
|
|
|
|
|
for {
|
|
|
|
|
|
|
|
logrus.Info("Waiting for connection")
|
|
|
|
conn, err := listen.Accept()
|
|
|
|
if err != nil {
|
|
|
|
logrus.Warn(err)
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
|
|
|
|
addr := conn.RemoteAddr()
|
|
|
|
|
|
|
|
logrus.Println(addr)
|
|
|
|
|
|
|
|
// _, err = conn.Write([]byte(addr.String()))
|
|
|
|
// if err != nil {
|
|
|
|
// logrus.Warn(err)
|
|
|
|
// continue
|
|
|
|
// }
|
|
|
|
|
|
|
|
go s.AddHeightSub(addr, conn)
|
|
|
|
}
|
|
|
|
}
|