We used to have a trickle node, a node which was chosen in each iteration of
the send loop that was privileged and allowed to send out queued up non-time
critical messages. Since the removal of the fixed sleeps in the network code,
this resulted in fast and attackable treatment of such broadcasts.
This pull request changes the 3 remaining trickle use cases by random delays:
* Local address broadcast (while also removing the the wiping of the seen filter)
* Address relay
* Inv relay (for transactions; blocks are always relayed immediately)
The code is based on older commits by Patrick Strateman.
- Avoids string typos (by making the compiler check)
- Makes it easier to grep for handling/generation of a certain message type
- Refer directly to documentation by following the symbol in IDE
- Move list of valid message types to protocol.cpp:
protocol.cpp is a more appropriate place for this, and having
the array there makes it easier to keep things consistent.
aa4b0c2 When not filtering blocks, getdata sends more in one test (Pieter Wuille)
d41e44c Actually only use filterInventoryKnown with MSG_TX inventory messages. (Gregory Maxwell)
b6a0da4 Only use filterInventoryKnown with MSG_TX inventory messages. (Patick Strateman)
6b84935 Rename setInventoryKnown filterInventoryKnown (Patick Strateman)
e206724 Remove mruset as it is no longer used. (Gregory Maxwell)
ec73ef3 Replace setInventoryKnown with a rolling bloom filter. (Gregory Maxwell)
ebb25f4 Limit setAskFor and retire requested entries only when a getdata returns. (Gregory Maxwell)
5029698 prevent peer flooding request queue for an inv (kazcw)
Mruset setInventoryKnown was reduced to a remarkably small 1000
entries as a side effect of sendbuffer size reductions in 2012.
This removes setInventoryKnown filtering from merkleBlock responses
because false positives there are especially unattractive and
also because I'm not sure if there aren't race conditions around
the relay pool that would cause some transactions there to
be suppressed. (Also, ProcessGetData was accessing
setInventoryKnown without taking the required lock.)
The setAskFor duplicate elimination was too eager and removed entries
when we still had no getdata response, allowing the peer to keep
INVing and not responding.
mapAlreadyAskedFor does not keep track of which peer has a request queued for a
particular tx. As a result, a peer can blind a node to a tx indefinitely by
sending many invs for the same tx, and then never replying to getdatas for it.
Each inv received will be placed 2 minutes farther back in mapAlreadyAskedFor,
so a short message containing 10 invs would render that tx unavailable for 20
minutes.
This is fixed by disallowing a peer from having more than one entry for a
particular inv in mapAlreadyAskedFor at a time.
- Force AUTHCOOKIE size to be 32 bytes: This provides protection against
an attack where a process pretends to be Tor and uses the cookie
authentication method to nab arbitrary files such as the
wallet
- torcontrol logging
- fix cookie auth
- add HASHEDPASSWORD auth, fix fd leak when fwrite() fails
- better error reporting when cookie file is not ok
- better init/shutdown flow
- stop advertizing service when disconnected from tor control port
- COOKIE->SAFECOOKIE auth
Starting with Tor version 0.2.7.1 it is possible, through Tor's control socket
API, to create and destroy 'ephemeral' hidden services programmatically.
https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service
This means that if Tor is running (and proper authorization is available),
bitcoin automatically creates a hidden service to listen on, without user
manual configuration. This will positively affect the number of available
.onion nodes.
- When the node is started, connect to Tor through control socket
- Send `ADD_ONION` command
- First time:
- Make it create a hidden service key
- Save the key in the data directory for later usage
- Make it redirect port 8333 to the local port 8333 (or whatever port we're listening on).
- Keep control socket connection open for as long node is running. The hidden service will
(by default) automatically go away when the connection is closed.
* -maxuploadtarget can be set in MiB
* if <limit> - ( time-left-in-24h-cycle / 600 * MAX_BLOCK_SIZE ) has reach, stop serve blocks older than one week and filtered blocks
* no action if limit has reached, no guarantee that the target will not be surpassed
* add outbound limit informations to rpc getnettotals
Nagle appears to be a significant contributor to latency now that the static
sleeps are gone. Most of our messages are relatively large compared to
IP + TCP so I do not expect this to create enormous overhead.
This may also reduce traffic burstyness somewhat.
- to match the peers.dat handling also supply a debug.log entry for how
many entries were loaded from banlist.dat and how long it took
- add a GUI init message for loading the banlist (same as with peers.dat)
- move the same message for peers.dat upwards in the code, to be able to
reuse the timing variable nStart and also just log, if our read from
peers.dat didn't fail
- only start working on/with banlist data, if reading in the banlist from
disk didn't fail
- as CNode::setBannedIsDirty is false (default) when reading fails, we
don't need to explicitly set it to false to prevent writing
banlist.dat in that case either
`nMinPingUsecTime` was left uninitialized in CNode.
The correct initialization for a minimum-until-now is int64_t's max value, so initialize it to that.
Thanks @MarcoFalke for noticing.
027de94 Use network group instead of CNetAddr in final pass to select node to disconnect (Patrick Strateman)
000c18a Fix comment (Patrick Strateman)
fed3094 Acquire cs_vNodes before changing refrence counts (Patrick Strateman)
69ee1aa CNodeRef copy constructor and assignment operator (Patrick Strateman)
dc81dd0 Return false early if vEvictionCandidates is empty (Patrick Strateman)
17f3533 Better support for nodes with non-standard nMaxConnections (Patrick Strateman)
1317cd1 RAII wrapper for CNode* (Patrick Strateman)
df23937 Add comments to AttemptToEvictConnection (Patrick Strateman)
a8f6e45 Remove redundant whiteconnections option (Patrick Strateman)
b105ba3 Prefer to disconnect peers in favor of whitelisted peers (Patrick Strateman)
2c70153 AttemptToEvictConnection (Patrick Strateman)
4bac601 Record nMinPingUsecTime (Patrick Strateman)
ae037b7 Refactor: Move failure conditions to the top of AcceptConnection (Patrick Strateman)
1ef4817 Refactor: Bail early in AcceptConnection (Patrick Strateman)
541a1dd Refactor: AcceptConnection (Patrick Strateman)
When running the rpc tests in Wine, nodes often fail to listen on localhost
due to a stale socket from a previous run. This aligns the behavior with other
platforms.
7b79cbd limit total length of user agent comments (Pavol Rusnak)
557f8ea implement uacomment config parameter which can add comments to user agent as per BIP-0014 (Pavol Rusnak)
While CBloomFilter is usually used with an explicitly set nTweak,
CRollingBloomFilter is only used internally. Requiring every caller to
set nTweak is error-prone and redundant; better to have the class handle
that for you with a high-quality randomness source.
Additionally when clearing the filter it makes sense to change nTweak as
well to recover from a bad setting, e.g. due to insufficient randomness
at initialization, so the clear() method is replaced by a reset() method
that sets a new, random, nTweak value.
a794284 locking: add a quick example of GUARDED_BY (Cory Fields)
2b890dd locking: fix a few small issues uncovered by -Wthread-safety (Cory Fields)
cd27bba locking: teach Clang's -Wthread-safety to cope with our scoped lock macros (Cory Fields)
- rpcwallet: No need to lock twice here
- openssl: Clang doesn't understand selective lock/unlock here. Ignore it.
- CNode: Fix a legitimate (though very unlikely) locking bug.
This sets aside a number of connection slots for whitelisted peers,
useful for ensuring your local users and miners can always get in,
even if your limit on inbound connections has already been reached.
Use a probabilistic bloom filter to keep track of which addresses
we think we have given our peers, instead of a list.
This uses much less memory, at the cost of sometimes failing to
relay an address to a peer-- worst case if the bloom filter happens
to be as full as it gets, 1-in-1,000.
Measured memory usage of a full mruset setAddrKnown: 650Kbytes
Constant memory usage of CRollingBloomFilter addrKnown: 37Kbytes.
This will also help heap fragmentation, because the 37K of storage
is allocated when a CNode is created (when a connection to a peer
is established) and then there is no per-item-remembered memory
allocation.
I plan on testing by restarting a full node with an empty peers.dat,
running a while with -debug=addrman and -debug=net, and making sure
that the 'addr' message traffic out is reasonable.
(suggestions for better tests welcome)
1d21ba2 Scale up addrman (Pieter Wuille)
c6a63ce Always use a 50% chance to choose between tried and new entries (Pieter Wuille)
f68ba3f Do not bias outgoing connections towards fresh addresses (Pieter Wuille)
a8ff7c6 Simplify hashing code (Pieter Wuille)
e6b343d Make addrman's bucket placement deterministic. (Pieter Wuille)
b23add5 Switch addrman key from vector to uint256 (Pieter Wuille)
This change was suggested as Countermeasure 2 in
Eclipse Attacks on Bitcoin’s Peer-to-Peer Network, Ethan Heilman,
Alison Kendler, Aviv Zohar, Sharon Goldberg. ePrint Archive Report
2015/263. March 2015.
Normally bitcoin core does not display any network originated strings without
sanitizing or hex encoding. This wasn't done for strcommand in many places.
This could be used to play havoc with a terminal displaying the logs,
especially with printtoconsole in use.
Thanks to Evil-Knievel for reporting this issue.
856e862 namespace: drop most boost namespaces and a few header cleanups (Cory Fields)
9b1ab86 namespace: drop boost::assign altogether here (Cory Fields)
a324199 namespace: remove boost namespace pollution (Cory Fields)
This avoids connecting to them again too soon in ThreadOpenConnections.
Make an exception for connection failures to the proxy as these
shouldn't affect the status of specific nodes.
This is a simplified re-do of closed pull #3088.
This patch eliminates the privacy and reliability problematic use
of centralized web services for discovering the node's addresses
for advertisement.
The Bitcoin protocol already allows your peers to tell you what
IP they think you have, but this data isn't trustworthy since
they could lie. So the challenge is using it without creating a
DOS vector.
To accomplish this we adopt an approach similar to the one used
by P2Pool: If we're announcing and don't have a better address
discovered (e.g. via UPNP) or configured we just announce to
each peer the address that peer told us. Since peers could
already replace, forge, or drop our address messages this cannot
create a new vulnerability... but if even one of our peers is
giving us a good address we'll eventually make a useful
advertisement.
We also may randomly use the peer-provided address for the
daily rebroadcast even if we otherwise have a seemingly routable
address, just in case we've been misconfigured (e.g. by UPNP).
To avoid privacy problems, we only do these things if discovery
is enabled.
Many changes:
* Do not use 'getblocks', but 'getheaders', and use it to build a headers tree.
* Blocks are fetched in parallel from all available outbound peers, using a
limited moving window. When one peer stalls the movement of the window, it is
disconnected.
* No more orphan blocks. At all. We only ever request a block for which we have
verified the headers, and store it to disk immediately. This means that a
disk-fill attack would require PoW.
* Require protocol version 31800 for every peer (released in december 2010).
* No more syncnode (we sync from everyone we can, though limited to 1 during
initial *headers* sync).
* Introduce some extra named constants, comments and asserts.
Avoids that SOCKS5 negotiation will hold up the shutdown process.
- Sockets can stay in non-blocking mode, no need to switch it on/off
anymore
- Adds a timeout (20 seconds) on SOCK5 negotiation. This should be
enough for even Tor to get a connection to a hidden service, and
avoids blocking the opencon thread indefinitely on a hanging proxy.
Fixes#2954.
Split up util.cpp/h into:
- string utilities (hex, base32, base64): no internal dependencies, no dependency on boost (apart from foreach)
- money utilities (parsesmoney, formatmoney)
- time utilities (gettime*, sleep, format date):
- and the rest (logging, argument parsing, config file parsing)
The latter is basically the environment and OS handling,
and is stripped of all utility functions, so we may want to
rename it to something else than util.cpp/h for clarity (Matt suggested
osinterface).
Breaks dependency of sha256.cpp on all the things pulled in by util.
aa82795 Add detailed network info to getnetworkinfo RPC (Wladimir J. van der Laan)
075cf49 Add GetNetworkName function (Wladimir J. van der Laan)
c91a947 Add IsReachable(net) function (Wladimir J. van der Laan)
60dc8e4 Allow -onlynet=onion to be used (Wladimir J. van der Laan)
- the call to CloseSocket() is placed after the WSAGetLastError(), because
a CloseSocket() can trigger an error also, which we don't want for the
logging in this two cases
Simpler alternative to #4348.
The current setup with closesocket() is strange. It poses
as a compatibility wrapper but adds functionality.
Rename it and make it a documented utility function in netbase.
Code movement only, zero effect on the functionality.
4eedf4f make RandAddSeed() use OPENSSL_cleanse() (Philip Kaufmann)
6354935 move rand functions from util to new random.h/.cpp (Philip Kaufmann)
001a53d add GetRandBytes() as wrapper for RAND_bytes() (Philip Kaufmann)
This adds a -whitelist option to specify subnet ranges from which peers
that connect are whitelisted. In addition, there is a -whitebind option
which works like -bind, except peers connecting to it are also
whitelisted (allowing a separate listen port for trusted connections).
Being whitelisted has two effects (for now):
* They are immune to DoS disconnection/banning.
* Transactions they broadcast (which are valid) are always relayed,
even if they were already in the mempool. This means that a node
can function as a gateway for a local network, and that rebroadcasts
from the local network will work as expected.
Whitelisting replaces the magic exemption localhost had for DoS
disconnection (local addresses are still never banned, though), which
implied hidden service connects (from a localhost Tor node) were
incorrectly immune to DoS disconnection as well. This old
behaviour is removed for that reason, but can be restored using
-whitelist=127.0.0.1 or -whitelist=::1 can be specified. -whitebind
is safer to use in case non-trusted localhost connections are expected
(like hidden services).
- add a small wrapper in util around RAND_bytes() and replace with
GetRandBytes() in the code to log errors from calling RAND_bytes()
- remove OpenSSL header rand.h where no longer needed
- SO_NOSIGPIPE isn't available on WIN32 so merge the 2 non-WIN32 blocks
- use predefined names from header for IPV6_PROTECTION_LEVEL and
PROTECTION_LEVEL_UNRESTRICTED
bitcoin-config.h moved, but the old file is likely to still exist when
reconfiguring or switching branches. This would've caused files to not rebuild
correctly, and other strange problems.
Make the path explicit so that the old one cannot be found.
Core libs use config/bitcoin-config.h.
Libs (like crypto) which don't want access to bitcoin's headers continue
to use -Iconfig and #include bitcoin-config.h.
- remove an unneded else in ConnectNode()
- make 0 a double and change to 0.0 in ConnectNode()
- rename strDest to pszDest in OpenNetworkConnection()
- remove an unneded call to our REF() macro in BindListenPort()
- small style cleanups and removal of unneeded new-lines
... instead of after 30 minutes of no sending, for latency measurement
and keep-alive. Also, disconnect if no reply arrives within 20 minutes,
instead of 90 of inactivity (for peers supporting the 'pong' message).
NodeSyncScore() should find the node which we recv data most recently, so put a negative sign to pnode->nLastRecv is indeed wrong.
Also change the return value type to int64_t.
Signed-off-by: Huang Le <4tarhl@gmail.com>
Log the name of the error as well as the error code if a network problem
happens. This makes network troubleshooting more convenient.
Use thread-safe strerror_r and the WIN32 equivalent FormatMessage.
Adds two new info query commands that take over information from
hodge-podge `getinfo`.
Also some new information is added:
- `getblockchaininfo`
- `chain`: (string) current chain (main, testnet3, regtest)
- `verificationprogress: (numeric) estimated verification progress
- `chainwork`
- `getnetworkinfo`
- `localaddresses`: (array) local addresses, from mapLocalHost (fixes#1734)
The year is 2014. All supported operating systems have IPv6 support,
most certainly at build time (this doesn't mean that IPv6 is configured,
of course).
If noone is exercising the functionality to disable it, that means it
doesn't get tested, and IMO it's better to get rid of it.
(it's also not used consistently in RPC/boost and Net code...)
When we are over our outbound limit ThreadSocketHandler would try to
keep the connection if the peer was addnoded.
This didn't actually work for two reasons: It didn't actually run
the accept code due to mistaken code flow, and because we have a
limited number of outbound semaphores it couldn't actually use the
connection.
Instead it leaked the socket, which might have caused issue #4034.
This patch just takes out the non-functioning white-listing for now.
When bitcoind can't bind, bitcoin server (or Bitcoin Core Daemon) is probably already running. Add the missing word "server". Bitcoin itself is definitely running ;-)
Add _(...) so the string can be localized.
I apologize for such trivial changes, learning github interface.
Amend to d5f1e72. It turns out that BerkelyDB was including inttypes.h
indirectly, so we cannot fix this with just macros.
Trivial commit: apply the following script to all .cpp and .h files:
# Middle
sed -i 's/"PRIx64"/x/g' "$1"
sed -i 's/"PRIu64"/u/g' "$1"
sed -i 's/"PRId64"/d/g' "$1"
# Initial
sed -i 's/PRIx64"/"x/g' "$1"
sed -i 's/PRIu64"/"u/g' "$1"
sed -i 's/PRId64"/"d/g' "$1"
# Trailing
sed -i 's/"PRIx64/x"/g' "$1"
sed -i 's/"PRIu64/u"/g' "$1"
sed -i 's/"PRId64/d"/g' "$1"
After this commit, `git grep` for PRI.64 should turn up nothing except
the defines in util.h.
contrib/devtools/fix-copyright-headers.py script to be able to perform this maintenance task with ease during the rest of the year, every year. Modifications to contrib/devtools/README.md to document what fix-copyright-headers.py does.
After the tinyformat switch sprintf() family functions support passing
actual std::string objects.
Remove unnecessary c_str calls (236 of them) in logging and formatting.
This was a leftover from the times in which
peers.dat depended in BDB.
Other functions in db.cpp still depend on BerkelyDB,
to be able to compile without BDB this (small)
functionality needs to be moved to another file.
Use misc methods of avoiding unnecesary header includes.
Replace int typedefs with int##_t from stdint.h.
Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h.
Normalize QT_VERSION ifs where possible.
Resolve some indirect dependencies as direct ones.
Remove extern declarations from .cpp files.
The existing CNode::addrLocal member is revealed to the user,
as an address string, similar to the existing "addr" field.
Instead of showing garbage or empty string,
it simply will not appear in the output if local address not known yet.
New RPC "ping" command to request ping.
Implemented "pong" message handler.
New "pingtime" field in getpeerinfo, to provide results to user.
New "pingwait" field, to show pings still in flight, to better see newly lagging peers.
- fixes src\net.cpp:1601: Error:invalid conversion from 'void*' to
'const char*' [-fpermissive] in a setsockopt() call on Win32 that was
found by using MinGW 4.8.1 compiler suite
The new class is accessed via the Params() method and holds
most things that vary between main, test and regtest networks.
The regtest mode has two purposes, one is to run the
bitcoind/bitcoinj comparison tool which compares two separate
implementations of the Bitcoin protocol looking for divergence.
The other is that when run, you get a local node which can mine
a single block instantly, which is highly convenient for testing
apps during development as there's no need to wait 10 minutes for
a block on the testnet.
- removes our NewThread() function an replaces remaining calls with
boost::thread with our TraceThread template
- remove ExitThread() function
- fix THREAD_PRIORITY_ABOVE_NORMAL for non Windows OSes
Added explicit include of main.h in init.cpp, changed include of init.h to include of main.h in net.cpp.
Added function registration for net.cpp in init.cpp's network initialization.
Removed protocol.cpp's dependency on main.h.
TODO: Remove main.h include in net.cpp.
Instead of killing a connection when the receive buffer overflows,
just temporarily halt receiving before that happens. Also, no
matter what, always allow at least one full message in the receive
buffer (otherwise blocks larger than the configured buffer size
would pause indefinitely).
This introduces the concept of the 'sync node', which is the one we
asked for missing blocks. In case the sync node goes away, a new one
will be selected.
For now, the heuristic is very simple, but it can easily be extended
later to add better policies.
It seems there were two mechanisms for assessing whether a CNode
was still in use: a refcount and a release timestamp. The latter
seems to have been there for a long time, as a safety mechanism.
However, this timer also keeps CNode objects alive for far longer
than necessary after disconnects, potentially opening up a DoS
window.
This commit removes the timestamp-based mechanism, and replaces
it with an assert(nRefCount >= 0), to verify that the refcounting
is indeed correctly working.
Two reasons for this change:
1. Need to always use boost::thread's sleep, even on Windows, so the
sleeps can be interrupted (prior code used Windows' built-in Sleep).
2. I always forgot what units the old Sleep took.
Create a boost::thread_group object at the qt/bitcoind main-loop level
that will hold pointers to all the main-loop threads.
This will replace the vnThreadsRunning[] array.
For testing, ported the BitcoinMiner threads to use its
own boost::thread_group.
This will result in re-requesting invs if we are under heavy inv
load, however as long as we get no more than 16,000 invs in two
minutes, this should have no effect on runtime behavior.
* Change CNode::vRecvMsg to be a deque instead of a vector (less copying)
* Make sure to acquire cs_vRecvMsg in CNode::CloseSocketDisconnect (as it
may be called without that lock).
1) "optimistic write": Push each message to kernel socket buffer immediately.
2) If there is write data at select time, that implies send() blocked
during optimistic write. Drain write queue, before receiving
any more messages.
This avoids needlessly queueing received data, if the remote peer
is not themselves receiving data.
Result: write buffer (and thus memory usage) is kept small, DoS
potential is slightly lower, and TCP flow control signalling is
properly utilized.
The kernel will queue data into the socket buffer, then signal the
remote peer to stop sending data, until we resume reading again.
Replaces CNode::vRecv buffer with a vector of CNetMessage's. This simplifies
ProcessMessages() and eliminates several redundant data copies.
Overview:
* socket thread now parses incoming message datastream into
header/data components, as encapsulated by CNetMessage
* socket thread adds each CNetMessage to a vector inside CNode
* message thread (ProcessMessages) iterates through CNode's CNetMessage vector
Message parsing is made more strict:
* Socket is disconnected, if message larger than MAX_SIZE
or if CMessageHeader deserialization fails (latter is impossible?).
Previously, code would simply eat garbage data all day long.
* Socket is disconnected, if we fail to find pchMessageStart.
We do not search through garbage, to find pchMessageStart. Each
message must begin precisely after the last message ends.
ProcessMessages() always processes a complete message, and is more efficient:
* buffer is always precisely sized, using CDataStream::resize(),
rather than progressively sized in 64k chunks. More efficient
for large messages like "block".
* whole-buffer memory copy eliminated (vRecv -> vMsg)
* other buffer-shifting memory copies eliminated (vRecv.insert, vRecv.erase)
Note that the default value for fRelayTxes is false, meaning we
now no longer relay tx inv messages before receiving the remote
peer's version message.
