LBRYCommunity/hub
1
0
Fork 0
Code Issues 35 Pull requests 6 Projects Releases Packages Wiki Activity

scribe full sync error: error while processing txs: int too large to convert (effective_amount table) #97

New issue
Closed
opened 2022-09-20 22:11:38 +02:00 by moodyjon · 5 comments
moodyjon commented 2022-09-20 22:11:38 +02:00 (Migrated from github.com)
Copy link

I was doing a scribe full sync and saw something fail at block 1056660:

(venv) swdev1@Jonathans-Mac-mini hub % python3 -m hub.scribe --db_dir=./scribe_db --daemon_url="https://yRwidFQYAibqCpzFfOSbxaB7K4=:o4kIdKWu3X2nPVxR26N+ZUXW8Yw=@127.0.0.1:9245" --daemon_ca_path="~/Library/Application Support/Lbcd/rpc.cert"
2022-09-20 15:03:15,955 INFO hub.scribe.daemon:80: daemon #1 at 127.0.0.1:9245/ (current)
2022-09-20 15:03:15,959 INFO hub.service.BlockchainProcessorService:81: software version: 1.0.0
2022-09-20 15:03:15,959 INFO hub.service.BlockchainProcessorService:83: reorg limit is 200 blocks
2022-09-20 15:03:38,131 INFO hub.db.db:1025: opened db for writing: lbry-rocksdb (./scribe_db/lbry-rocksdb)
2022-09-20 15:03:38,132 INFO hub.db.db:1035: DB version: 11
2022-09-20 15:03:38,132 INFO hub.db.db:1036: coin: LBRY
2022-09-20 15:03:38,132 INFO hub.db.db:1037: network: mainnet
2022-09-20 15:03:38,133 INFO hub.db.db:1038: height: 1,056,659
2022-09-20 15:03:38,133 INFO hub.db.db:1039: tip: f04f87c18e63f9e9919e964bbc746b44e3e98fca2fc4b8d48fd65382f7e2fb26
2022-09-20 15:03:38,133 INFO hub.db.db:1040: tx count: 63,218,189
2022-09-20 15:03:38,133 INFO hub.db.db:1043: using address status index: False
2022-09-20 15:03:38,133 INFO hub.db.db:1053: flush count: 1,056,660
2022-09-20 15:03:38,133 INFO hub.service.BlockchainProcessorService:86: initializing caches
2022-09-20 15:03:39,128 INFO hub.db.db:1157: populating header merkle cache...
2022-09-20 15:03:40,027 INFO hub.db.db:1162: header merkle cache populated in 0.9s
2022-09-20 15:03:40,027 INFO hub.service.BlockchainProcessorService:89: opened db at block 1056659
2022-09-20 15:03:40,054 INFO hub.scribe.prefetcher:82: catching up to daemon height 1,229,193 (172,534 blocks behind)
2022-09-20 15:03:50,213 ERROR hub.service.BlockchainProcessorService:238: advance blocks failed
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks
    txo_count = await self.run_in_thread_with_lock(self.advance_block, block)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked
    return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args)
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block
    self._get_cumulative_update_ops(height)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops
    self.db.prefix_db.effective_amount.stage_multi_put(
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp>
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value
    return super().pack_value(effective_amount, support_sum)
  File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value
    return cls.value_struct.pack(*args)
struct.error: int too large to convert
2022-09-20 15:03:50,215 ERROR hub.service.BlockchainProcessorService:1804: error while processing txs: int too large to convert
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1800, in process_blocks_and_mempool_forever
    await self.check_and_advance_blocks(blocks)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks
    txo_count = await self.run_in_thread_with_lock(self.advance_block, block)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked
    return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args)
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block
    self._get_cumulative_update_ops(height)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops
    self.db.prefix_db.effective_amount.stage_multi_put(
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp>
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value
    return super().pack_value(effective_amount, support_sum)
  File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value
    return cls.value_struct.pack(*args)
struct.error: int too large to convert
2022-09-20 15:03:50,215 ERROR hub.service.BlockchainProcessorService:1812: fatal error in block processor loop: int too large to convert
Traceback (most recent call last):
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1805, in process_blocks_and_mempool_forever
    raise err
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1800, in process_blocks_and_mempool_forever
    await self.check_and_advance_blocks(blocks)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks
    txo_count = await self.run_in_thread_with_lock(self.advance_block, block)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked
    return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args)
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block
    self._get_cumulative_update_ops(height)
  File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops
    self.db.prefix_db.effective_amount.stage_multi_put(
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp>
    self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items])
  File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value
    return super().pack_value(effective_amount, support_sum)
  File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value
    return cls.value_struct.pack(*args)
struct.error: int too large to convert

I had shut scribe down previously and have a snapshot at 1030239. Will see if it reproduces....

I was doing a scribe full sync and saw something fail at block 1056660: ``` (venv) swdev1@Jonathans-Mac-mini hub % python3 -m hub.scribe --db_dir=./scribe_db --daemon_url="https://yRwidFQYAibqCpzFfOSbxaB7K4=:o4kIdKWu3X2nPVxR26N+ZUXW8Yw=@127.0.0.1:9245" --daemon_ca_path="~/Library/Application Support/Lbcd/rpc.cert" 2022-09-20 15:03:15,955 INFO hub.scribe.daemon:80: daemon #1 at 127.0.0.1:9245/ (current) 2022-09-20 15:03:15,959 INFO hub.service.BlockchainProcessorService:81: software version: 1.0.0 2022-09-20 15:03:15,959 INFO hub.service.BlockchainProcessorService:83: reorg limit is 200 blocks 2022-09-20 15:03:38,131 INFO hub.db.db:1025: opened db for writing: lbry-rocksdb (./scribe_db/lbry-rocksdb) 2022-09-20 15:03:38,132 INFO hub.db.db:1035: DB version: 11 2022-09-20 15:03:38,132 INFO hub.db.db:1036: coin: LBRY 2022-09-20 15:03:38,132 INFO hub.db.db:1037: network: mainnet 2022-09-20 15:03:38,133 INFO hub.db.db:1038: height: 1,056,659 2022-09-20 15:03:38,133 INFO hub.db.db:1039: tip: f04f87c18e63f9e9919e964bbc746b44e3e98fca2fc4b8d48fd65382f7e2fb26 2022-09-20 15:03:38,133 INFO hub.db.db:1040: tx count: 63,218,189 2022-09-20 15:03:38,133 INFO hub.db.db:1043: using address status index: False 2022-09-20 15:03:38,133 INFO hub.db.db:1053: flush count: 1,056,660 2022-09-20 15:03:38,133 INFO hub.service.BlockchainProcessorService:86: initializing caches 2022-09-20 15:03:39,128 INFO hub.db.db:1157: populating header merkle cache... 2022-09-20 15:03:40,027 INFO hub.db.db:1162: header merkle cache populated in 0.9s 2022-09-20 15:03:40,027 INFO hub.service.BlockchainProcessorService:89: opened db at block 1056659 2022-09-20 15:03:40,054 INFO hub.scribe.prefetcher:82: catching up to daemon height 1,229,193 (172,534 blocks behind) 2022-09-20 15:03:50,213 ERROR hub.service.BlockchainProcessorService:238: advance blocks failed Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks txo_count = await self.run_in_thread_with_lock(self.advance_block, block) File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock return await asyncio.shield(run_in_thread_locked()) File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args) File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block self._get_cumulative_update_ops(height) File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops self.db.prefix_db.effective_amount.stage_multi_put( File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp> self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value return super().pack_value(effective_amount, support_sum) File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value return cls.value_struct.pack(*args) struct.error: int too large to convert 2022-09-20 15:03:50,215 ERROR hub.service.BlockchainProcessorService:1804: error while processing txs: int too large to convert Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 1800, in process_blocks_and_mempool_forever await self.check_and_advance_blocks(blocks) File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks txo_count = await self.run_in_thread_with_lock(self.advance_block, block) File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock return await asyncio.shield(run_in_thread_locked()) File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args) File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block self._get_cumulative_update_ops(height) File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops self.db.prefix_db.effective_amount.stage_multi_put( File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp> self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value return super().pack_value(effective_amount, support_sum) File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value return cls.value_struct.pack(*args) struct.error: int too large to convert 2022-09-20 15:03:50,215 ERROR hub.service.BlockchainProcessorService:1812: fatal error in block processor loop: int too large to convert Traceback (most recent call last): File "/Users/swdev1/hub/hub/scribe/service.py", line 1805, in process_blocks_and_mempool_forever raise err File "/Users/swdev1/hub/hub/scribe/service.py", line 1800, in process_blocks_and_mempool_forever await self.check_and_advance_blocks(blocks) File "/Users/swdev1/hub/hub/scribe/service.py", line 226, in check_and_advance_blocks txo_count = await self.run_in_thread_with_lock(self.advance_block, block) File "/Users/swdev1/hub/hub/scribe/service.py", line 153, in run_in_thread_with_lock return await asyncio.shield(run_in_thread_locked()) File "/Users/swdev1/hub/hub/scribe/service.py", line 152, in run_in_thread_locked return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args) File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/Users/swdev1/hub/hub/scribe/service.py", line 1477, in advance_block self._get_cumulative_update_ops(height) File "/Users/swdev1/hub/hub/scribe/service.py", line 1336, in _get_cumulative_update_ops self.db.prefix_db.effective_amount.stage_multi_put( File "/Users/swdev1/hub/hub/db/interface.py", line 122, in stage_multi_put self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/interface.py", line 122, in <listcomp> self._op_stack.multi_put([RevertablePut(self.pack_key(*k), self.pack_value(*v)) for k, v in items]) File "/Users/swdev1/hub/hub/db/prefixes.py", line 1802, in pack_value return super().pack_value(effective_amount, support_sum) File "/Users/swdev1/hub/hub/db/interface.py", line 152, in pack_value return cls.value_struct.pack(*args) struct.error: int too large to convert ``` I had shut scribe down previously and have a snapshot at 1030239. Will see if it reproduces....
moodyjon commented 2022-09-20 22:22:41 +02:00 (Migrated from github.com)
Copy link

This was in my developer workspace, I was synced to master + some logging changes (I think). I don't think I had any changes on top that would interfere, but will try to check on that to confirm.

This was in my developer workspace, I was synced to master + some logging changes (I think). I don't think I had any changes on top that would interfere, but will try to check on that to confirm.
moodyjon commented 2022-09-21 01:28:21 +02:00 (Migrated from github.com)
Copy link

Logging requested:
scribe.txt

This covers only the transition 1056659 -> 1056660.

Logging requested: [scribe.txt](https://github.com/lbryio/hub/files/9611821/scribe.txt) This covers only the transition 1056659 -> 1056660.
moodyjon commented 2022-09-21 17:03:39 +02:00 (Migrated from github.com)
Copy link

New logging:
scribe2.txt

New logging: [scribe2.txt](https://github.com/lbryio/hub/files/9617896/scribe2.txt)
moodyjon commented 2022-09-21 18:06:38 +02:00 (Migrated from github.com)
Copy link

Uncommented print()s in _get_takeover_ops():
scribe3.txt

Uncommented print()s in _get_takeover_ops(): [scribe3.txt](https://github.com/lbryio/hub/files/9618438/scribe3.txt)
moodyjon commented 2022-09-21 18:36:00 +02:00 (Migrated from github.com)
Copy link
            print(f"delayed_activate_ops current_effective_amount_values: {current_effective_amount_values}")
            print(f"claim: {name} {claim_hash.hex()} delay: {delay}")

scribe4.txt

``` print(f"delayed_activate_ops current_effective_amount_values: {current_effective_amount_values}") print(f"claim: {name} {claim_hash.hex()} delay: {delay}") ``` [scribe4.txt](https://github.com/lbryio/hub/files/9618654/scribe4.txt)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
dependabot/pip/aiohttp-3.8.5
dependabot/pip/grpcio-1.53.0
moodyjon/ipv6
ipv6
moodyjon/limited_repost
add-channel-sponsor
es-failover
moodyjon/grpcio_v1.46.0
go-hub
No results found.
Labels
Clear labels   
area: database

   
area: documentation

Improvements or additions to documentation

  
area: elasticsearch

   
area: herald

   
area: packaging

   
area: scribe

   
consider soon

Discuss this issue at the next planning meeting, then remove this label

  
critical

   
dependencies

Pull requests that update a dependency file

  
good first issue

Good for newcomers

  
hacktoberfest

Welcome to Hacktoberfest

  
help wanted

Extra attention is needed

  
improvement

   
needs: repro

Needs a way to reproduce

  
new feature

   
priority: blocker

Issue is blocking release, do ASAP

  
priority: high

Work needs to be moved into sprint ASAP

  
priority: low

Work should be done but can stay in the backlog for now

  
priority: medium

Work needs to be done within 2-3 sprints

  
type: bug

Existing functionality is wrong or broken

  
type: bug-fix

   
type: discussion

A conversation. No specific changes requested

  
type: feature request

   
type: improvement

Existing (or partially existing) functionality needs to be changed

  
type: new feature

New functionality that does not exist yet

  
type: refactor

Minimal user-visible changes, but significant internal work

  
type: task

Either work that's not related to the code, or a small chore that does not fit into other categories

No labels
area: database
area: documentation
area: elasticsearch
area: herald
area: packaging
area: scribe
consider soon
critical
dependencies
good first issue
hacktoberfest
help wanted
improvement
needs: repro
new feature
priority: blocker
priority: high
priority: low
priority: medium
type: bug
type: bug-fix
type: discussion
type: feature request
type: improvement
type: new feature
type: refactor
type: task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/hub#97
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?