`scribe-hub` update loop crashes #2

New issue

Closed

opened 2022-03-21 15:44:52 +01:00 by jackrobison · 2 comments

jackrobison commented

2022-03-21 15:44:52 +01:00

(Migrated from github.com)

There is an uncaught exception in the update loop for scribe-hub which causes it to stop polling for updates. This causes it to appear stuck on a block and stop sending notifications, even though scribe maybe continue writing more blocks.

jackrobison commented

2022-03-23 01:21:02 +01:00

(Migrated from github.com)

future: <Task finished name='Task-6' coro=<BaseBlockchainReader.refresh_blocks_forever() done, defined at /home/lbry/scribe/scribe/reader/interface.py:127> exception=RocksIOError(b'IO error: No such file or directory: While open a file for random read: /home/lbry/rocksdb-volume/lbry-rocksdb/006188.ldb: No such file or directory')>
Traceback (most recent call last):
  File "/home/lbry/scribe/scribe/reader/interface.py", line 131, in refresh_blocks_forever
    await self.poll_for_changes()
  File "/home/lbry/scribe/scribe/reader/hub_server.py", line 51, in poll_for_changes
    await super().poll_for_changes()
  File "/home/lbry/scribe/scribe/reader/interface.py", line 125, in poll_for_changes
    await asyncio.get_event_loop().run_in_executor(self._executor, self._detect_changes)
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lbry/scribe/scribe/reader/hub_server.py", line 48, in _detect_changes
    self.mempool_notifications.update(self.mempool.refresh())
  File "/home/lbry/scribe/scribe/hub/mempool.py", line 56, in refresh
    new_mempool = {k.tx_hash: v.raw_tx for k, v in prefix_db.mempool_tx.iterate()}  # TODO: make this more efficient
  File "/home/lbry/scribe/scribe/hub/mempool.py", line 56, in <dictcomp>
    new_mempool = {k.tx_hash: v.raw_tx for k, v in prefix_db.mempool_tx.iterate()}  # TODO: make this more efficient
  File "/home/lbry/scribe/scribe/db/interface.py", line 67, in iterate
    it = self._db.iterator(
  File "rocksdb/_rocksdb.pyx", line 2024, in rocksdb._rocksdb.DB.iterator
  File "rocksdb/_rocksdb.pyx", line 2531, in rocksdb._rocksdb.BaseIterator.seek
  File "rocksdb/_rocksdb.pyx", line 2535, in rocksdb._rocksdb.BaseIterator.seek
  File "rocksdb/_rocksdb.pyx", line 94, in rocksdb._rocksdb.check_status
rocksdb.errors.RocksIOError: b'IO error: No such file or directory: While open a file for random read: /home/lbry/rocksdb-volume/lbry-rocksdb/006188.ldb: No such file or directory'```

```Task exception was never retrieved future: <Task finished name='Task-6' coro=<BaseBlockchainReader.refresh_blocks_forever() done, defined at /home/lbry/scribe/scribe/reader/interface.py:127> exception=RocksIOError(b'IO error: No such file or directory: While open a file for random read: /home/lbry/rocksdb-volume/lbry-rocksdb/006188.ldb: No such file or directory')> Traceback (most recent call last): File "/home/lbry/scribe/scribe/reader/interface.py", line 131, in refresh_blocks_forever await self.poll_for_changes() File "/home/lbry/scribe/scribe/reader/hub_server.py", line 51, in poll_for_changes await super().poll_for_changes() File "/home/lbry/scribe/scribe/reader/interface.py", line 125, in poll_for_changes await asyncio.get_event_loop().run_in_executor(self._executor, self._detect_changes) File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/lbry/scribe/scribe/reader/hub_server.py", line 48, in _detect_changes self.mempool_notifications.update(self.mempool.refresh()) File "/home/lbry/scribe/scribe/hub/mempool.py", line 56, in refresh new_mempool = {k.tx_hash: v.raw_tx for k, v in prefix_db.mempool_tx.iterate()} # TODO: make this more efficient File "/home/lbry/scribe/scribe/hub/mempool.py", line 56, in <dictcomp> new_mempool = {k.tx_hash: v.raw_tx for k, v in prefix_db.mempool_tx.iterate()} # TODO: make this more efficient File "/home/lbry/scribe/scribe/db/interface.py", line 67, in iterate it = self._db.iterator( File "rocksdb/_rocksdb.pyx", line 2024, in rocksdb._rocksdb.DB.iterator File "rocksdb/_rocksdb.pyx", line 2531, in rocksdb._rocksdb.BaseIterator.seek File "rocksdb/_rocksdb.pyx", line 2535, in rocksdb._rocksdb.BaseIterator.seek File "rocksdb/_rocksdb.pyx", line 94, in rocksdb._rocksdb.check_status rocksdb.errors.RocksIOError: b'IO error: No such file or directory: While open a file for random read: /home/lbry/rocksdb-volume/lbry-rocksdb/006188.ldb: No such file or directory'```

jackrobison commented

2022-03-24 18:37:19 +01:00

(Migrated from github.com)

32b8afe154 (diff-ea1202b26351451db031fe1ae0261c2a130e8e909ce04335a9a5a4d69d6db187R61)

https://github.com/lbryio/scribe/commit/32b8afe154a4a0ca562a451fe1eebc831e280161#diff-ea1202b26351451db031fe1ae0261c2a130e8e909ce04335a9a5a4d69d6db187R61

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/hub#2

No description provided.

scribe-hub update loop crashes #2

`scribe-hub` update loop crashes #2