lbry-sdk not making progress during initial address sync when connecting to herald.go #77

New issue

Open

opened 2022-11-19 00:43:17 +01:00 by moodyjon · 2 comments

moodyjon commented

2022-11-19 00:43:17 +01:00

(Migrated from github.com)

This data was gathered with some local changes plus my published branch herald.go/blockchain_tx_rpc1:

failure.txt

It's not completing, even though some messages are emitted like these:

2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - subscribed to 15/15 addresses on 127.0.0.1:50002
2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - finished subscribing to 15 addresses on 127.0.0.1:50002

It just keeps spinning in some loop alternating between address.get_history and transaction.get_batch. It's also only mentioning a couple of specific transactions -- one at height=0 and one at height=1

I will continue investigating this, but maybe @eukreign, @jackrobison, or @shyba could point me in the right direction.

Herald.go is not generating address notifications and header notifications at this time AFAICT. That's one difference in behavior between herald.go and python herald that may be relevant.

This data was gathered with some local changes plus my published branch herald.go/blockchain_tx_rpc1: [failure.txt](https://github.com/lbryio/herald.go/files/10045483/failure.txt) It's not completing, even though some messages are emitted like these: ``` 2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - subscribed to 15/15 addresses on 127.0.0.1:50002 2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - finished subscribing to 15 addresses on 127.0.0.1:50002 ``` It just keeps spinning in some loop alternating between `address.get_history` and `transaction.get_batch`. It's also only mentioning a couple of specific transactions -- one at height=0 and one at height=1 I will continue investigating this, but maybe @eukreign, @jackrobison, or @shyba could point me in the right direction. Herald.go is not generating address notifications and header notifications at this time AFAICT. That's one difference in behavior between herald.go and python herald that may be relevant.

moodyjon commented

2022-11-20 18:45:12 +01:00

(Migrated from github.com)

Lbry-sdk <-> Herald RPCs sent & responses:
Baseline: a.txt
Herald.go: b.txt

It's starting to diverge at the point of blockchain.address.subcribe (Python hub returns None a lot of the time) and later at blockchain.address.get_history (Herald.go returns 2 different transactions at height 0 and height 1, Python returns same transaction at height -1 and later height 201).

Lbry-sdk <-> Herald RPCs sent & responses: Baseline: [a.txt](https://github.com/lbryio/herald.go/files/10050302/a.txt) Herald.go: [b.txt](https://github.com/lbryio/herald.go/files/10050303/b.txt) It's starting to diverge at the point of `blockchain.address.subcribe` (Python hub returns None a lot of the time) and later at `blockchain.address.get_history` (Herald.go returns 2 different transactions at height 0 and height 1, Python returns same transaction at height -1 and later height 201).

moodyjon commented

2022-12-06 17:42:01 +01:00

(Migrated from github.com)

Found a problem with address.get_history. The interpretation of IncludeStop iteration option was confusing me. Hence I was finding an extra record following the desired hashX when no matching record existed.

Secondly, I have narrowed down where it's stuck. Test setup sequence is now at the stage (send_to_address_and_wait) where mempool TXs must generate notifications and be reportable. The existing mechanism detectChanges() is only monitoring height in DBStateValue. But there can't be a change in height until send_to_address_and_wait moves on to generate blocks.

Found a problem with `address.get_history`. The interpretation of `IncludeStop` iteration option was confusing me. Hence I was finding an extra record **following** the desired `hashX` when no matching record existed. Secondly, I have narrowed down where it's stuck. Test setup sequence is now at the stage (`send_to_address_and_wait`) where mempool TXs must generate notifications and be reportable. The existing mechanism `detectChanges()` is only monitoring height in `DBStateValue`. But there can't be a change in height until `send_to_address_and_wait` moves on to generate blocks.