Improve memory usage during initial ES sync #57

2022-06-28T02:33:03+02:00

jackrobison commented

2022-06-28 02:33:03 +02:00

(Migrated from github.com)

updates ES sync to use batched claim fetching shared by resolve

shyba (Migrated from github.com) approved these changes

2022-06-29 21:52:50 +02:00

shyba (Migrated from github.com) left a comment

I don't see anything critical. Added some suggestions (that can be done in other PRs too), but otherwise looks good.

hub/elastic_sync/db.py

					
				@ -150,0 +91,4 @@

				                    reposted_fee_amount = 0

				                else:

				                    reposted_fee_amount = int(max(reposted_metadata.stream.fee.amount or 0, 0) * 1000)

				                    if reposted_fee_amount >= 9223372036854775807:

shyba (Migrated from github.com) commented

2022-06-29 21:16:58 +02:00

suggestion: if this is a dict of {tx_hash: nout} you may be able to avoid the two transformations on get_claim_metadatas, reusing the same data structure

hub/elastic_sync/db.py

					
				@ -150,0 +116,4 @@

				                reposted_claim_hash) or self.blocked_channels.get(claim_hash) or self.blocked_channels.get(

				                reposted_claim_hash) or self.blocked_channels.get(claim.channel_hash)

				            filtered_hash = self.filtered_streams.get(claim_hash) or self.filtered_streams.get(

				                reposted_claim_hash) or self.filtered_channels.get(claim_hash) or self.filtered_channels.get(

shyba (Migrated from github.com) commented

2022-06-29 21:20:40 +02:00

probably not in this PR, but would be nice to extract those huge numbers to something with meaningful names

hub/elastic_sync/db.py

shyba (Migrated from github.com) commented

2022-06-29 21:48:37 +02:00

maybe aggregate and log once?

jackrobison (Migrated from github.com) reviewed

2022-06-30 17:20:52 +02:00

hub/elastic_sync/db.py

					
				@ -150,0 +91,4 @@

				                    reposted_fee_amount = 0

				                else:

				                    reposted_fee_amount = int(max(reposted_metadata.stream.fee.amount or 0, 0) * 1000)

				                    if reposted_fee_amount >= 9223372036854775807:

jackrobison (Migrated from github.com) commented

2022-06-30 17:20:52 +02:00

It has to also handle the case where you're requesting multiple outputs from the same tx

Sign in to join this conversation.

No reviewers

shyba

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/hub#57

Rows
Columns