lbcd/log.go

356 lines
9.3 KiB
Go
Raw Normal View History

// Copyright (c) 2013-2014 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package main
import (
"fmt"
2014-07-02 15:50:08 +02:00
"os"
"strings"
2014-07-02 15:50:08 +02:00
"time"
"github.com/btcsuite/btcd/addrmgr"
"github.com/btcsuite/btcd/blockchain"
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
database "github.com/btcsuite/btcd/database2"
peer: Refactor peer code into its own package. This commit introduces package peer which contains peer related features refactored from peer.go. The following is an overview of the features the package provides: - Provides a basic concurrent safe bitcoin peer for handling bitcoin communications via the peer-to-peer protocol - Full duplex reading and writing of bitcoin protocol messages - Automatic handling of the initial handshake process including protocol version negotiation - Automatic periodic keep-alive pinging and pong responses - Asynchronous message queueing of outbound messages with optional channel for notification when the message is actually sent - Inventory message batching and send trickling with known inventory detection and avoidance - Ability to wait for shutdown/disconnect - Flexible peer configuration - Caller is responsible for creating outgoing connections and listening for incoming connections so they have flexibility to establish connections as they see fit (proxies, etc.) - User agent name and version - Bitcoin network - Service support signalling (full nodes, bloom filters, etc.) - Maximum supported protocol version - Ability to register callbacks for handling bitcoin protocol messages - Proper handling of bloom filter related commands when the caller does not specify the related flag to signal support - Disconnects the peer when the protocol version is high enough - Does not invoke the related callbacks for older protocol versions - Snapshottable peer statistics such as the total number of bytes read and written, the remote address, user agent, and negotiated protocol version - Helper functions for pushing addresses, getblocks, getheaders, and reject messages - These could all be sent manually via the standard message output function, but the helpers provide additional nice functionality such as duplicate filtering and address randomization - Full documentation with example usage - Test coverage In addition to the addition of the new package, btcd has been refactored to make use of the new package by extending the basic peer it provides to work with the blockmanager and server to act as a full node. The following is a broad overview of the changes to integrate the package: - The server is responsible for all connection management including persistent peers and banning - Callbacks for all messages that are required to implement a full node are registered - Logic necessary to serve data and behave as a full node is now in the callback registered with the peer Finally, the following peer-related things have been improved as a part of this refactor: - Don't log or send reject message due to peer disconnects - Remove trace logs that aren't particularly helpful - Finish an old TODO to switch the queue WaitGroup over to a channel - Improve various comments and fix some code consistency cases - Improve a few logging bits - Implement a most-recently-used nonce tracking for detecting self connections and generate a unique nonce for each peer
2015-10-02 08:03:20 +02:00
"github.com/btcsuite/btcd/peer"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btclog"
"github.com/btcsuite/seelog"
)
const (
// maxRejectReasonLen is the maximum length of a sanitized reject reason
// that will be logged.
maxRejectReasonLen = 250
)
// Loggers per subsytem. Note that backendLog is a seelog logger that all of
// the subsystem loggers route their messages to. When adding new subsystems,
// add a reference here, to the subsystemLoggers map, and the useLogger
// function.
var (
backendLog = seelog.Disabled
adxrLog = btclog.Disabled
amgrLog = btclog.Disabled
bcdbLog = btclog.Disabled
bmgrLog = btclog.Disabled
btcdLog = btclog.Disabled
chanLog = btclog.Disabled
discLog = btclog.Disabled
minrLog = btclog.Disabled
peerLog = btclog.Disabled
rpcsLog = btclog.Disabled
scrpLog = btclog.Disabled
srvrLog = btclog.Disabled
txmpLog = btclog.Disabled
)
// subsystemLoggers maps each subsystem identifier to its associated logger.
var subsystemLoggers = map[string]btclog.Logger{
"ADXR": adxrLog,
"AMGR": amgrLog,
"BCDB": bcdbLog,
"BMGR": bmgrLog,
"BTCD": btcdLog,
"CHAN": chanLog,
"DISC": discLog,
"MINR": minrLog,
"PEER": peerLog,
"RPCS": rpcsLog,
"SCRP": scrpLog,
"SRVR": srvrLog,
"TXMP": txmpLog,
}
// logClosure is used to provide a closure over expensive logging operations
// so don't have to be performed when the logging level doesn't warrant it.
type logClosure func() string
// String invokes the underlying function and returns the result.
func (c logClosure) String() string {
return c()
}
// newLogClosure returns a new closure over a function that returns a string
// which itself provides a Stringer interface so that it can be used with the
// logging system.
func newLogClosure(c func() string) logClosure {
return logClosure(c)
}
// useLogger updates the logger references for subsystemID to logger. Invalid
// subsystems are ignored.
func useLogger(subsystemID string, logger btclog.Logger) {
if _, ok := subsystemLoggers[subsystemID]; !ok {
return
}
subsystemLoggers[subsystemID] = logger
switch subsystemID {
case "ADXR":
adxrLog = logger
case "AMGR":
amgrLog = logger
addrmgr.UseLogger(logger)
case "BCDB":
bcdbLog = logger
database.UseLogger(logger)
case "BMGR":
bmgrLog = logger
case "BTCD":
btcdLog = logger
case "CHAN":
chanLog = logger
blockchain.UseLogger(logger)
case "DISC":
discLog = logger
case "MINR":
minrLog = logger
case "PEER":
peerLog = logger
peer: Refactor peer code into its own package. This commit introduces package peer which contains peer related features refactored from peer.go. The following is an overview of the features the package provides: - Provides a basic concurrent safe bitcoin peer for handling bitcoin communications via the peer-to-peer protocol - Full duplex reading and writing of bitcoin protocol messages - Automatic handling of the initial handshake process including protocol version negotiation - Automatic periodic keep-alive pinging and pong responses - Asynchronous message queueing of outbound messages with optional channel for notification when the message is actually sent - Inventory message batching and send trickling with known inventory detection and avoidance - Ability to wait for shutdown/disconnect - Flexible peer configuration - Caller is responsible for creating outgoing connections and listening for incoming connections so they have flexibility to establish connections as they see fit (proxies, etc.) - User agent name and version - Bitcoin network - Service support signalling (full nodes, bloom filters, etc.) - Maximum supported protocol version - Ability to register callbacks for handling bitcoin protocol messages - Proper handling of bloom filter related commands when the caller does not specify the related flag to signal support - Disconnects the peer when the protocol version is high enough - Does not invoke the related callbacks for older protocol versions - Snapshottable peer statistics such as the total number of bytes read and written, the remote address, user agent, and negotiated protocol version - Helper functions for pushing addresses, getblocks, getheaders, and reject messages - These could all be sent manually via the standard message output function, but the helpers provide additional nice functionality such as duplicate filtering and address randomization - Full documentation with example usage - Test coverage In addition to the addition of the new package, btcd has been refactored to make use of the new package by extending the basic peer it provides to work with the blockmanager and server to act as a full node. The following is a broad overview of the changes to integrate the package: - The server is responsible for all connection management including persistent peers and banning - Callbacks for all messages that are required to implement a full node are registered - Logic necessary to serve data and behave as a full node is now in the callback registered with the peer Finally, the following peer-related things have been improved as a part of this refactor: - Don't log or send reject message due to peer disconnects - Remove trace logs that aren't particularly helpful - Finish an old TODO to switch the queue WaitGroup over to a channel - Improve various comments and fix some code consistency cases - Improve a few logging bits - Implement a most-recently-used nonce tracking for detecting self connections and generate a unique nonce for each peer
2015-10-02 08:03:20 +02:00
peer.UseLogger(logger)
case "RPCS":
rpcsLog = logger
case "SCRP":
scrpLog = logger
txscript.UseLogger(logger)
case "SRVR":
srvrLog = logger
case "TXMP":
txmpLog = logger
}
}
// initSeelogLogger initializes a new seelog logger that is used as the backend
// for all logging subsytems.
func initSeelogLogger(logFile string) {
config := `
<seelog type="adaptive" mininterval="2000000" maxinterval="100000000"
critmsgcount="500" minlevel="trace">
<outputs formatid="all">
<console />
<rollingfile type="size" filename="%s" maxsize="10485760" maxrolls="3" />
</outputs>
<formats>
<format id="all" format="%%Time %%Date [%%LEV] %%Msg%%n" />
</formats>
</seelog>`
config = fmt.Sprintf(config, logFile)
logger, err := seelog.LoggerFromConfigAsString(config)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to create logger: %v", err)
os.Exit(1)
}
backendLog = logger
}
// setLogLevel sets the logging level for provided subsystem. Invalid
// subsystems are ignored. Uninitialized subsystems are dynamically created as
// needed.
func setLogLevel(subsystemID string, logLevel string) {
// Ignore invalid subsystems.
logger, ok := subsystemLoggers[subsystemID]
if !ok {
return
}
// Default to info if the log level is invalid.
level, ok := btclog.LogLevelFromString(logLevel)
if !ok {
level = btclog.InfoLvl
}
// Create new logger for the subsystem if needed.
if logger == btclog.Disabled {
logger = btclog.NewSubsystemLogger(backendLog, subsystemID+": ")
useLogger(subsystemID, logger)
}
logger.SetLevel(level)
}
// setLogLevels sets the log level for all subsystem loggers to the passed
// level. It also dynamically creates the subsystem loggers as needed, so it
// can be used to initialize the logging system.
func setLogLevels(logLevel string) {
// Configure all sub-systems with the new logging level. Dynamically
// create loggers as needed.
for subsystemID := range subsystemLoggers {
setLogLevel(subsystemID, logLevel)
}
}
// directionString is a helper function that returns a string that represents
// the direction of a connection (inbound or outbound).
func directionString(inbound bool) string {
if inbound {
return "inbound"
}
return "outbound"
}
// formatLockTime returns a transaction lock time as a human-readable string.
func formatLockTime(lockTime uint32) string {
// The lock time field of a transaction is either a block height at
// which the transaction is finalized or a timestamp depending on if the
// value is before the txscript.LockTimeThreshold. When it is under the
// threshold it is a block height.
if lockTime < txscript.LockTimeThreshold {
return fmt.Sprintf("height %d", lockTime)
}
return time.Unix(int64(lockTime), 0).String()
}
2014-09-08 21:19:47 +02:00
// invSummary returns an inventory message as a human-readable string.
func invSummary(invList []*wire.InvVect) string {
// No inventory.
invLen := len(invList)
if invLen == 0 {
return "empty"
}
// One inventory item.
if invLen == 1 {
iv := invList[0]
switch iv.Type {
case wire.InvTypeError:
return fmt.Sprintf("error %s", iv.Hash)
case wire.InvTypeBlock:
return fmt.Sprintf("block %s", iv.Hash)
case wire.InvTypeTx:
return fmt.Sprintf("tx %s", iv.Hash)
}
return fmt.Sprintf("unknown (%d) %s", uint32(iv.Type), iv.Hash)
}
// More than one inv item.
return fmt.Sprintf("size %d", invLen)
}
// locatorSummary returns a block locator as a human-readable string.
func locatorSummary(locator []*wire.ShaHash, stopHash *wire.ShaHash) string {
if len(locator) > 0 {
return fmt.Sprintf("locator %s, stop %s", locator[0], stopHash)
}
return fmt.Sprintf("no locator, stop %s", stopHash)
}
// sanitizeString strips any characters which are even remotely dangerous, such
// as html control characters, from the passed string. It also limits it to
// the passed maximum size, which can be 0 for unlimited. When the string is
// limited, it will also add "..." to the string to indicate it was truncated.
func sanitizeString(str string, maxLength uint) string {
const safeChars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXY" +
"Z01234567890 .,;_/:?@"
// Strip any characters not in the safeChars string removed.
str = strings.Map(func(r rune) rune {
if strings.IndexRune(safeChars, r) >= 0 {
return r
}
return -1
}, str)
// Limit the string to the max allowed length.
if maxLength > 0 && uint(len(str)) > maxLength {
str = str[:maxLength]
str = str + "..."
}
return str
}
// messageSummary returns a human-readable string which summarizes a message.
// Not all messages have or need a summary. This is used for debug logging.
func messageSummary(msg wire.Message) string {
switch msg := msg.(type) {
case *wire.MsgVersion:
return fmt.Sprintf("agent %s, pver %d, block %d",
msg.UserAgent, msg.ProtocolVersion, msg.LastBlock)
case *wire.MsgVerAck:
// No summary.
case *wire.MsgGetAddr:
// No summary.
case *wire.MsgAddr:
return fmt.Sprintf("%d addr", len(msg.AddrList))
case *wire.MsgPing:
// No summary - perhaps add nonce.
case *wire.MsgPong:
// No summary - perhaps add nonce.
case *wire.MsgAlert:
// No summary.
case *wire.MsgMemPool:
// No summary.
case *wire.MsgTx:
return fmt.Sprintf("hash %s, %d inputs, %d outputs, lock %s",
msg.TxSha(), len(msg.TxIn), len(msg.TxOut),
formatLockTime(msg.LockTime))
case *wire.MsgBlock:
header := &msg.Header
return fmt.Sprintf("hash %s, ver %d, %d tx, %s", msg.BlockSha(),
2014-01-19 04:10:36 +01:00
header.Version, len(msg.Transactions), header.Timestamp)
case *wire.MsgInv:
return invSummary(msg.InvList)
case *wire.MsgNotFound:
return invSummary(msg.InvList)
case *wire.MsgGetData:
return invSummary(msg.InvList)
case *wire.MsgGetBlocks:
return locatorSummary(msg.BlockLocatorHashes, &msg.HashStop)
case *wire.MsgGetHeaders:
return locatorSummary(msg.BlockLocatorHashes, &msg.HashStop)
case *wire.MsgHeaders:
return fmt.Sprintf("num %d", len(msg.Headers))
case *wire.MsgReject:
// Ensure the variable length strings don't contain any
// characters which are even remotely dangerous such as HTML
// control characters, etc. Also limit them to sane length for
// logging.
rejCommand := sanitizeString(msg.Cmd, wire.CommandSize)
rejReason := sanitizeString(msg.Reason, maxRejectReasonLen)
summary := fmt.Sprintf("cmd %v, code %v, reason %v", rejCommand,
msg.Code, rejReason)
if rejCommand == wire.CmdBlock || rejCommand == wire.CmdTx {
summary += fmt.Sprintf(", hash %v", msg.Hash)
}
return summary
}
// No summary for other messages.
return ""
}