Bring herald/db/prefixes.go up to date with new tables #46

New issue

Closed

opened 2022-08-03 18:16:23 +02:00 by moodyjon · 3 comments

moodyjon commented

2022-08-03 18:16:23 +02:00

(Migrated from github.com)

Latest tables in (python) hub:

4845b92248/hub/db/common.py (L50)

    channel_count = b'Z'
    support_amount = b'a'
    block_tx = b'b'
    trending_notifications = b'c'
    mempool_tx = b'd'
    touched_hashX = b'e'
    hashX_status = b'f'
    hashX_mempool_status = b'g'

Latest tables in herald.go:

8b4b27bdb9/db/prefixes/prefixes.go (L61)

	ChannelCount  = 'Z'
	SupportAmount = 'a'
	BlockTXs      = 'b'

Latest tables in (python) hub: https://github.com/lbryio/hub/blob/4845b9224842e6800808aa1860330b34929b06ae/hub/db/common.py#L50 ``` channel_count = b'Z' support_amount = b'a' block_tx = b'b' trending_notifications = b'c' mempool_tx = b'd' touched_hashX = b'e' hashX_status = b'f' hashX_mempool_status = b'g' ``` Latest tables in herald.go: https://github.com/lbryio/herald.go/blob/8b4b27bdb95ed899a055a9c0ebd8f10e7ca4b5e6/db/prefixes/prefixes.go#L61 ``` ChannelCount = 'Z' SupportAmount = 'a' BlockTXs = 'b' ```

moodyjon commented

2022-08-04 16:17:00 +02:00

(Migrated from github.com)

https://pkg.go.dev/github.com/lunixbochs/struc
https://pkg.go.dev/github.com/go-restruct/restruct

Performance benchmark from 7 years ago indicated it was 10x slower than manually written code.
https://github.com/lunixbochs/struc#benchmark

After writing some Pack/Unpack() code for one of the new tables, I'm searching for something that can automate this. Using reflect to make the Pack/Unpack() generic is not a new idea. Found these packages that do it: https://pkg.go.dev/github.com/lunixbochs/struc https://pkg.go.dev/github.com/go-restruct/restruct Performance benchmark from 7 years ago indicated it was 10x slower than manually written code. https://github.com/lunixbochs/struc#benchmark

jeffreypicard commented

2022-08-04 18:34:11 +02:00

(Migrated from github.com)

After writing some Pack/Unpack() code for one of the new tables, I'm searching for something that can automate this. Using reflect to make the Pack/Unpack() generic is not a new idea. Found these packages that do it:

https://pkg.go.dev/github.com/lunixbochs/struc https://pkg.go.dev/github.com/go-restruct/restruct

Performance benchmark from 7 years ago indicated it was 10x slower than manually written code. https://github.com/lunixbochs/struc#benchmark

The plan is to use generics from go 1.18 for this https://go.dev/blog/why-generics. If we can get performance improvements out of this tangentially to that, it sounds good. We should talk about this in the call.

> After writing some Pack/Unpack() code for one of the new tables, I'm searching for something that can automate this. Using reflect to make the Pack/Unpack() generic is not a new idea. Found these packages that do it: > > https://pkg.go.dev/github.com/lunixbochs/struc https://pkg.go.dev/github.com/go-restruct/restruct > > Performance benchmark from 7 years ago indicated it was 10x slower than manually written code. https://github.com/lunixbochs/struc#benchmark The plan is to use generics from go 1.18 for this https://go.dev/blog/why-generics. If we can get performance improvements out of this tangentially to that, it sounds good. We should talk about this in the call.

moodyjon commented

2022-08-05 22:33:55 +02:00

(Migrated from github.com)

I have successfully tagged 2 existing structs (EffectiveAmountKey/Value) structs and ran Pack()/Unpack() based on https://pkg.go.dev/github.com/go-restruct/restruct and reflection.

I also set up a testing framework where the different Pack/Unpack() implementations can be tested against each other. For example, manual Pack() with "restruct" Unpack(), and vice versa.

TODO: Need to gather some test data for the new tables and finish the manually-implemented Pack/Unpack() for them.

I have successfully tagged 2 existing structs (EffectiveAmountKey/Value) structs and ran Pack()/Unpack() based on https://pkg.go.dev/github.com/go-restruct/restruct and reflection. I also set up a testing framework where the different Pack/Unpack() implementations can be tested against each other. For example, manual Pack() with "restruct" Unpack(), and vice versa. TODO: Need to gather some test data for the new tables and finish the manually-implemented Pack/Unpack() for them.