Add treapNode pool. Reduce cloneTreapNode() allocations.

moodyjon commented

2022-06-03 17:54:34 +02:00

(Migrated from github.com)

Unbundled from PR https://github.com/lbryio/lbcd/pull/43

See that for usage/testing.

Based on work by @BrannonKing in mem_pressure_try_2 and reduce_memory_pressure.

I am not entirely comfortable with the changes. It's my best attempt to combine original Immutable treap semantics with the way that ffldb/transaction actually wants to use it. I was very successful in reducing temporary allocations, but I'm still not sure it's safe. It could also be fragile if treap is used in different ways in the future.

Unbundled from PR https://github.com/lbryio/lbcd/pull/43 See that for usage/testing. Based on work by @BrannonKing in mem_pressure_try_2 and reduce_memory_pressure. I am not entirely comfortable with the changes. It's my best attempt to combine original Immutable treap semantics with the way that ffldb/transaction actually wants to use it. I was very successful in reducing temporary allocations, but I'm still not sure it's safe. It could also be fragile if treap is used in different ways in the future.

roylee17 commented

2022-07-07 06:30:09 +02:00

(Migrated from github.com)

It seems the changes doesn't translate to the performance in terms of sync time, and memory pressure.
I rebased this changes onto v10.22.102 and did a full sync compare to it. Both took around 12 hours for sync up to 1070K blocks with 5.1 GB memory usage. So I'm inclining to leave this out.

Note:

Currently we, include your contributions, have made a lot efforts to improve performance, and make the user/developer experience much better. And I suspect further tuning would have to involve more deeper and intrusive changes with marginal gain.

With limited bandwidth, we're prioritizing functional parity/compatibility with lbrycrd, and help users migrate to lbcd. Check the RPC Availability page, which tracks the full list of bitcoin-core v22.0.0 RPCs. Those with open issues are reported by users that are missing or have different attributes from lbrycrd.

If interested, we can discuss more on discord ping me @roylee17 or join #lbcd channel.

It seems the changes doesn't translate to the performance in terms of sync time, and memory pressure. I rebased this changes onto v10.22.102 and did a full sync compare to it. Both took around 12 hours for sync up to 1070K blocks with 5.1 GB memory usage. So I'm inclining to leave this out. Note: Currently we, include your contributions, have made a lot efforts to improve performance, and make the user/developer experience much better. And I suspect further tuning would have to involve more deeper and intrusive changes with marginal gain. With limited bandwidth, we're prioritizing functional parity/compatibility with **lbrycrd**, and help users migrate to **lbcd**. Check the [RPC Availability](https://github.com/lbryio/lbcd/wiki/RPC-availability ) page, which tracks the full list of bitcoin-core v22.0.0 RPCs. Those with open issues are reported by users that are missing or have different attributes from **lbrycrd**. If interested, we can discuss more on discord ping me @roylee17 or join #lbcd channel.

moodyjon commented

2022-07-08 18:57:50 +02:00

(Migrated from github.com)

I'm OK with closing this. It seems likely that network latency/throughput is the controlling factor. Blocks are processed serially during network sync, using a fraction of one CPU. As long as there is some CPU headroom (e.g. second CPU core), the garbage collector is able to run in a second thread without much effect on network sync time.

moodyjon commented

2022-07-08 19:02:33 +02:00

(Migrated from github.com)

If interested, we can discuss more on discord ping me @roylee17 or join #lbcd channel.

I don't see an #lbcd on the discord server. All I see is a generic #developers. Is there a second server?

> If interested, we can discuss more on discord ping me @roylee17 or join #lbcd channel. I don't see an #lbcd on the discord server. All I see is a generic #developers. Is there a second server?

roylee17 commented

2022-07-08 19:05:22 +02:00

(Migrated from github.com)

Myself is not very familiar with discord yet. But here's the link, see if that works.

https://discord.gg/KRh5Gdjh

Myself is not very familiar with discord yet. But here's the link, see if that works. https://discord.gg/KRh5Gdjh

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Add treapNode pool. Reduce cloneTreapNode() allocations. #47

Pull request closed