lbrycrdd should have a logroll if its printing console output to logs.

Leopere commented

2018-10-06 20:36:48 +02:00

(Migrated from github.com)

Since it prints every line about every bit of indexing that it does it should be able to roll the log file a bit so that you don't end up with a full blockchain sized log file.

👍 1

BrannonKing commented

2018-10-08 19:17:02 +02:00

(Migrated from github.com)

I think we should also audit what is currently being written to debug.log.

kaykurokawa commented

2018-10-13 21:51:41 +02:00

(Migrated from github.com)

I removed the biggest culprit here, this would dump about 200mb worth of stuff on every claimtrie flush:
7eca4f00a7

Did an audit of the other logs, they don't cause too much of a problem in terms of the volume of logs. However their are a bunch of unnecessary logging that can be omitted.

These definitely should be removed, as they serve no useful purpose other than debugging for devs:

https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L608
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L614
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1598
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1629
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1945
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2111
https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2384

Below could be removed they do show some useful claim information (for example, a new claim gets created), but not sure if that's something that should go into the logs.

I removed the biggest culprit here, this would dump about 200mb worth of stuff on every claimtrie flush: https://github.com/lbryio/lbrycrd/commit/7eca4f00a7a6d24d8386dab733e136940ef457f2 Did an audit of the other logs, they don't cause too much of a problem in terms of the volume of logs. However their are a bunch of unnecessary logging that can be omitted. These definitely should be removed, as they serve no useful purpose other than debugging for devs: https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L608 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L614 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1598 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1629 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1945 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2111 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2384 Below could be removed they do show some useful claim information (for example, a new claim gets created), but not sure if that's something that should go into the logs. https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L58 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L65 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1592 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1611 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L1697 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2001 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2020 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2098 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2105 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2163 https://github.com/lbryio/lbrycrd/blob/v0.12.2.2/src/claimtrie.cpp#L2215

BrannonKing commented

2018-10-14 04:59:35 +02:00

(Migrated from github.com)

Your assessment looks good. I think we should also remove those 7 you suggest. I think we should keep a log entry for each user addition/subtraction, so I think most on the bottom list should stay. It looks like we may be logging added claims twice, though -- both add and insert?

bvbfan commented

2018-10-14 07:43:34 +02:00

(Migrated from github.com)

I'm worried of this line too -> 80940714a1 (diff-1f647c4b5527c3d3e86cfb735380b479R60)

I'm worried of this line too -> https://github.com/lbryio/lbrycrd/pull/159/commits/80940714a1015dd8dcc22de8d53ae99bff65328f#diff-1f647c4b5527c3d3e86cfb735380b479R60

BrannonKing commented

2018-10-15 17:46:03 +02:00

(Migrated from github.com)

@bvbfan , I do have that line removed in my normalization work.

kaykurokawa commented

2018-10-15 22:12:41 +02:00

(Migrated from github.com)

The fixes in this release https://github.com/lbryio/lbrycrd/releases/tag/v0.12.2.3 will remove much of the unnecessary logging. lbrycrdd has a function that will automatically shrink large log files upon startup, so we will not implement a log roll/rotation.

Users that want a log roll/rotation should implement their own (the automatic shrinking of log files can be disabled through option -shrinkdebugfile )

The fixes in this release https://github.com/lbryio/lbrycrd/releases/tag/v0.12.2.3 will remove much of the unnecessary logging. lbrycrdd has a function that will automatically shrink large log files upon startup, so we will not implement a log roll/rotation. Users that want a log roll/rotation should implement their own (the automatic shrinking of log files can be disabled through option -shrinkdebugfile )

BrannonKing commented

2019-09-06 22:26:52 +02:00

(Migrated from github.com)

I propose that we start a new file with each run, and that we backup the old file. Hence, I should always have this run and the previous one available. That's not quite a "roll" of logs, in that it won't fix the maximum size while the daemon is running, but I think it would help.

gahag commented

2019-10-21 01:43:19 +02:00

(Migrated from github.com)

@BrannonKing can I work on this to implement what you described in your last comment?

BrannonKing commented

2019-10-21 15:35:26 +02:00

(Migrated from github.com)

@gahag , yes you can work on this.

👍 1

gahag commented

2019-10-22 00:31:29 +02:00

(Migrated from github.com)

I've explored the code, and I'm thinking of implementing in BCLog::Logger::OpenDebugLog. The code would move the debug file if it exists to a .old file. This way, each file would correspond to one execution of the entire application. Another option would be to implement such code in the default constructor for BCLog::Logger. What do you guys think?

I've explored the code, and I'm thinking of implementing in `BCLog::Logger::OpenDebugLog`. The code would move the debug file if it exists to a `.old` file. This way, each file would correspond to one execution of the entire application. Another option would be to implement such code in the default constructor for `BCLog::Logger`. What do you guys think?

BrannonKing commented

2019-10-22 15:24:10 +02:00

(Migrated from github.com)

Look at the top of AppInitMain(); the "shrink" code there would also need to change.

Look at the top of `AppInitMain()`; the "shrink" code there would also need to change.

gahag commented

2019-10-22 20:04:21 +02:00

(Migrated from github.com)

Sure. What are we expecting to be the new behavior of the -shrinkdebugfile flag? Should it be removed, or should it shrink the old log file?

BrannonKing commented

2019-10-22 20:57:02 +02:00

(Migrated from github.com)

I think the flag should be removed.

👍 1

Leopere commented

2019-10-24 20:13:02 +02:00

(Migrated from github.com)

This looks like it should've been closed by #344 I don't have time to really validate this functionality but if people see this PR didn't hit the notes that this thread evolved to hit then please feel free to re-open it.

lbrycrdd should have a logroll if its printing console output to logs. #211