LBRYCommunity/hub
1
0
Fork 0
Code Issues 35 Pull requests 6 Projects Releases Packages Wiki Activity

scribe full sync failed at block 213495: db op tries to delete with incorrect existing value DELETE activated_claim_and_support #111

New issue
Closed
opened 2022-11-11 21:24:53 +01:00 by moodyjon · 0 comments
moodyjon commented 2022-11-11 21:24:53 +01:00 (Migrated from github.com)
Copy link

Looks like the problems with scribe full sync are not over. ☹️ It seemed a lot faster though. Took only 1-2 hours to reach this point. 😆

I have the latest hub checked out: dcd4d7a7a8

python3 -m hub.scribe --db_dir ./scribe_db/ --daemon_url https:/<user>:<pass>@localhost

2022-11-11 12:23:36,039 INFO hub.service.BlockchainProcessorService:234: advanced to 213483 (took 0.021s) | +2    txs (total txs: 2060407  ) | ---3/+++3 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,099 INFO hub.service.BlockchainProcessorService:234: advanced to 213484 (took 0.060s) | +51   txs (total txs: 2060458  ) | -101/+160 utxos | ---1/+++4 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,122 INFO hub.service.BlockchainProcessorService:234: advanced to 213485 (took 0.023s) | +4    txs (total txs: 2060462  ) | ---5/+++7 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,169 INFO hub.service.BlockchainProcessorService:234: advanced to 213486 (took 0.047s) | +11   txs (total txs: 2060473  ) | --17/++38 utxos | ---0/+++2 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,194 INFO hub.service.BlockchainProcessorService:234: advanced to 213487 (took 0.025s) | +5    txs (total txs: 2060478  ) | ---6/+++9 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,225 INFO hub.service.BlockchainProcessorService:234: advanced to 213488 (took 0.031s) | +9    txs (total txs: 2060487  ) | --13/++41 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,248 INFO hub.service.BlockchainProcessorService:234: advanced to 213489 (took 0.024s) | +3    txs (total txs: 2060490  ) | ---4/+++5 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,269 INFO hub.service.BlockchainProcessorService:234: advanced to 213490 (took 0.020s) | +1    txs (total txs: 2060491  ) | ---1/+++1 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,290 INFO hub.service.BlockchainProcessorService:234: advanced to 213491 (took 0.021s) | +1    txs (total txs: 2060492  ) | ---1/+++1 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,357 INFO hub.service.BlockchainProcessorService:234: advanced to 213492 (took 0.067s) | +67   txs (total txs: 2060559  ) | --79/+175 utxos | ---1/+++3 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,391 INFO hub.service.BlockchainProcessorService:234: advanced to 213493 (took 0.033s) | +14   txs (total txs: 2060573  ) | --26/++73 utxos | ---0/+++1 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,422 INFO hub.service.BlockchainProcessorService:234: advanced to 213494 (took 0.031s) | +15   txs (total txs: 2060588  ) | --15/++29 utxos | ---0/+++2 claims | ---0/+++0 supports | abandoned: 0    /    0
2022-11-11 12:23:36,442 ERROR hub.service.BlockchainProcessorService:242: advance blocks failed
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks
    txo_count, time.perf_counter() - start
  File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock
    async def run_in_thread():
  File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked
    async def run_in_thread(self, func, *args):
  File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block
    raise ChainError(
  File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit
    else:
  File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash
    def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]):
  File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops
  File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops
    need_put.append(op)
hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi)
vs
b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten"
2022-11-11 12:23:36,445 ERROR hub.service.BlockchainProcessorService:2080: error while processing txs: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi)
vs
b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten"
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 2076, in process_blocks_and_mempool_forever
  File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks
    txo_count, time.perf_counter() - start
  File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock
    async def run_in_thread():
  File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked
    async def run_in_thread(self, func, *args):
  File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block
    raise ChainError(
  File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit
    else:
  File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash
    def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]):
  File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops
  File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops
    need_put.append(op)
hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi)
vs
b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten"
2022-11-11 12:23:36,446 ERROR hub.service.BlockchainProcessorService:2088: fatal error in block processor loop: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi)
vs
b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten"
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 2081, in process_blocks_and_mempool_forever
  File "/Users/swdev1/hub/hub/scribe/service.py", line 2076, in process_blocks_and_mempool_forever
  File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks
    txo_count, time.perf_counter() - start
  File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock
    async def run_in_thread():
  File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked
    async def run_in_thread(self, func, *args):
  File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block
    raise ChainError(
  File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit
    else:
  File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash
    def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]):
  File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops
  File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops
    need_put.append(op)
hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi)
vs
b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten"
Looks like the problems with scribe full sync are not over. ☹️ It seemed a lot faster though. Took only 1-2 hours to reach this point. 😆 I have the latest hub checked out: dcd4d7a7a8de3ab41d9fb859b910c3b962eaa2fe `python3 -m hub.scribe --db_dir ./scribe_db/ --daemon_url https:/<user>:<pass>@localhost` ``` 2022-11-11 12:23:36,039 INFO hub.service.BlockchainProcessorService:234: advanced to 213483 (took 0.021s) | +2 txs (total txs: 2060407 ) | ---3/+++3 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,099 INFO hub.service.BlockchainProcessorService:234: advanced to 213484 (took 0.060s) | +51 txs (total txs: 2060458 ) | -101/+160 utxos | ---1/+++4 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,122 INFO hub.service.BlockchainProcessorService:234: advanced to 213485 (took 0.023s) | +4 txs (total txs: 2060462 ) | ---5/+++7 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,169 INFO hub.service.BlockchainProcessorService:234: advanced to 213486 (took 0.047s) | +11 txs (total txs: 2060473 ) | --17/++38 utxos | ---0/+++2 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,194 INFO hub.service.BlockchainProcessorService:234: advanced to 213487 (took 0.025s) | +5 txs (total txs: 2060478 ) | ---6/+++9 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,225 INFO hub.service.BlockchainProcessorService:234: advanced to 213488 (took 0.031s) | +9 txs (total txs: 2060487 ) | --13/++41 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,248 INFO hub.service.BlockchainProcessorService:234: advanced to 213489 (took 0.024s) | +3 txs (total txs: 2060490 ) | ---4/+++5 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,269 INFO hub.service.BlockchainProcessorService:234: advanced to 213490 (took 0.020s) | +1 txs (total txs: 2060491 ) | ---1/+++1 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,290 INFO hub.service.BlockchainProcessorService:234: advanced to 213491 (took 0.021s) | +1 txs (total txs: 2060492 ) | ---1/+++1 utxos | ---0/+++0 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,357 INFO hub.service.BlockchainProcessorService:234: advanced to 213492 (took 0.067s) | +67 txs (total txs: 2060559 ) | --79/+175 utxos | ---1/+++3 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,391 INFO hub.service.BlockchainProcessorService:234: advanced to 213493 (took 0.033s) | +14 txs (total txs: 2060573 ) | --26/++73 utxos | ---0/+++1 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,422 INFO hub.service.BlockchainProcessorService:234: advanced to 213494 (took 0.031s) | +15 txs (total txs: 2060588 ) | --15/++29 utxos | ---0/+++2 claims | ---0/+++0 supports | abandoned: 0 / 0 2022-11-11 12:23:36,442 ERROR hub.service.BlockchainProcessorService:242: advance blocks failed Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks txo_count, time.perf_counter() - start File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock async def run_in_thread(): File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked async def run_in_thread(self, func, *args): File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block raise ChainError( File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit else: File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]): File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops need_put.append(op) hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi) vs b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten" 2022-11-11 12:23:36,445 ERROR hub.service.BlockchainProcessorService:2080: error while processing txs: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi) vs b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten" Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 2076, in process_blocks_and_mempool_forever File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks txo_count, time.perf_counter() - start File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock async def run_in_thread(): File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked async def run_in_thread(self, func, *args): File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block raise ChainError( File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit else: File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]): File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops need_put.append(op) hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi) vs b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten" 2022-11-11 12:23:36,446 ERROR hub.service.BlockchainProcessorService:2088: fatal error in block processor loop: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi) vs b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten" Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 2081, in process_blocks_and_mempool_forever File "/Users/swdev1/hub/hub/scribe/service.py", line 2076, in process_blocks_and_mempool_forever File "/Users/swdev1/hub/hub/scribe/service.py", line 229, in check_and_advance_blocks txo_count, time.perf_counter() - start File "/Users/swdev1/hub/hub/scribe/service.py", line 156, in run_in_thread_with_lock async def run_in_thread(): File "/Users/swdev1/hub/hub/scribe/service.py", line 155, in run_in_thread_locked async def run_in_thread(self, func, *args): File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1759, in advance_block raise ChainError( File "/Users/swdev1/hub/hub/db/interface.py", line 220, in unsafe_commit else: File "/Users/swdev1/hub/hub/db/interface.py", line 292, in apply_stash def multi_delete(self, items: typing.List[typing.Tuple[bytes, bytes]]): File "/Users/swdev1/hub/hub/db/revertable.py", line 175, in validate_and_apply_stashed_ops File "/Users/swdev1/hub/hub/db/revertable.py", line 160, in validate_and_apply_stashed_ops need_put.append(op) hub.db.revertable.OpStackIntegrity: db op tries to delete with incorrect existing value DELETE activated_claim_and_support: ActivationKey(txo_type=2, tx_num=2059795, position=1) | ActivationValue(height=213469, claim_hash=8e04c56d27e6767f65a3e00158d866764972ff2d, normalized_name=holi) vs b"\x00\x03A\xdd\x8e\x04\xc5m'\xe6v\x7fe\xa3\xe0\x01X\xd8fvIr\xff-\x00\x03ten" ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
dependabot/pip/aiohttp-3.8.5
dependabot/pip/grpcio-1.53.0
moodyjon/ipv6
ipv6
moodyjon/limited_repost
add-channel-sponsor
es-failover
moodyjon/grpcio_v1.46.0
go-hub
No results found.
Labels
Clear labels   
area: database

   
area: documentation

Improvements or additions to documentation

  
area: elasticsearch

   
area: herald

   
area: packaging

   
area: scribe

   
consider soon

Discuss this issue at the next planning meeting, then remove this label

  
critical

   
dependencies

Pull requests that update a dependency file

  
good first issue

Good for newcomers

  
hacktoberfest

Welcome to Hacktoberfest

  
help wanted

Extra attention is needed

  
improvement

   
needs: repro

Needs a way to reproduce

  
new feature

   
priority: blocker

Issue is blocking release, do ASAP

  
priority: high

Work needs to be moved into sprint ASAP

  
priority: low

Work should be done but can stay in the backlog for now

  
priority: medium

Work needs to be done within 2-3 sprints

  
type: bug

Existing functionality is wrong or broken

  
type: bug-fix

   
type: discussion

A conversation. No specific changes requested

  
type: feature request

   
type: improvement

Existing (or partially existing) functionality needs to be changed

  
type: new feature

New functionality that does not exist yet

  
type: refactor

Minimal user-visible changes, but significant internal work

  
type: task

Either work that's not related to the code, or a small chore that does not fit into other categories

No labels
area: database
area: documentation
area: elasticsearch
area: herald
area: packaging
area: scribe
consider soon
critical
dependencies
good first issue
hacktoberfest
help wanted
improvement
needs: repro
new feature
priority: blocker
priority: high
priority: low
priority: medium
type: bug
type: bug-fix
type: discussion
type: feature request
type: improvement
type: new feature
type: refactor
type: task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/hub#111
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?