LBRYCommunity/lbrycrd
1
0
Fork 0
Code Issues 29 Pull requests 1 Projects Releases 32 Packages Wiki Activity
lbrycrd/src/test/serialize_tests.cpp

369 lines
14 KiB
C++
Raw Normal View History

Increment MIT Licence copyright header year on files modified in 2016 Edited via: $ contrib/devtools/copyright_header.py update .
2016-12-31 19:01:21 +01:00
// Copyright (c) 2012-2016 The Bitcoin Core developers
Remove references to X11 licence
2014-12-13 05:09:33 +01:00
// Distributed under the MIT software license, see the accompanying
Add licenses for tests and test data - Add license headers to source files (years based on commit dates) in `src/test` as well as `qa` - Add `README.md` to `src/test/data` specifying MIT license Fixes #3848
2014-03-18 10:11:00 +01:00
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include "serialize.h"
boost: split stream classes out of serialize.h serialization now has no dependencies.
2014-10-22 21:08:30 +02:00
#include "streams.h"
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
#include "hash.h"
tests: add a BasicTestingSetup and apply to all tests Make sure that chainparams and logging is properly initialized. Doing this for every test may be overkill, but this initialization is so simple that that does not matter. This should fix the travis issues.
2015-03-12 09:34:42 +01:00
#include "test/test_bitcoin.h"
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include <stdint.h>
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include <boost/test/unit_test.hpp>
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
tests: add a BasicTestingSetup and apply to all tests Make sure that chainparams and logging is properly initialized. Doing this for every test may be overkill, but this initialization is so simple that that does not matter. This should fix the travis issues.
2015-03-12 09:34:42 +01:00
BOOST_FIXTURE_TEST_SUITE(serialize_tests, BasicTestingSetup)
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
class CSerializeMethodsTestSingle
{
protected:
int intval;
bool boolval;
std::string stringval;
const char* charstrval;
Make CTransaction actually immutable
2016-11-12 01:23:17 +01:00
CTransactionRef txval;
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
public:
CSerializeMethodsTestSingle() = default;
Make CTransaction actually immutable
2016-11-12 01:23:17 +01:00
CSerializeMethodsTestSingle(int intvalin, bool boolvalin, std::string stringvalin, const char* charstrvalin, CTransaction txvalin) : intval(intvalin), boolval(boolvalin), stringval(std::move(stringvalin)), charstrval(charstrvalin), txval(MakeTransactionRef(txvalin)){}
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
ADD_SERIALIZE_METHODS;
template <typename Stream, typename Operation>
Get rid of nType and nVersion Remove the nType and nVersion as parameters to all serialization methods and functions. There is only one place where it's read and has an impact (in CAddress), and even there it does not impact any of the recursively invoked serializers. Instead, the few places that need nType or nVersion are changed to read it directly from the stream object, through GetType() and GetVersion() methods which are added to all stream classes.
2016-10-29 01:29:17 +02:00
inline void SerializationOp(Stream& s, Operation ser_action) {
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
READWRITE(intval);
READWRITE(boolval);
READWRITE(stringval);
READWRITE(FLATDATA(charstrval));
READWRITE(txval);
}
bool operator==(const CSerializeMethodsTestSingle& rhs)
{
return intval == rhs.intval && \
boolval == rhs.boolval && \
stringval == rhs.stringval && \
strcmp(charstrval, rhs.charstrval) == 0 && \
Make CTransaction actually immutable
2016-11-12 01:23:17 +01:00
*txval == *rhs.txval;
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
}
};
class CSerializeMethodsTestMany : public CSerializeMethodsTestSingle
{
public:
using CSerializeMethodsTestSingle::CSerializeMethodsTestSingle;
ADD_SERIALIZE_METHODS;
template <typename Stream, typename Operation>
Get rid of nType and nVersion Remove the nType and nVersion as parameters to all serialization methods and functions. There is only one place where it's read and has an impact (in CAddress), and even there it does not impact any of the recursively invoked serializers. Instead, the few places that need nType or nVersion are changed to read it directly from the stream object, through GetType() and GetVersion() methods which are added to all stream classes.
2016-10-29 01:29:17 +02:00
inline void SerializationOp(Stream& s, Operation ser_action) {
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
READWRITEMANY(intval, boolval, stringval, FLATDATA(charstrval), txval);
}
};
src/serialize.h: base serialization level endianness neutrality Serialization type-safety and endianness compatibility.
2014-12-19 11:41:50 +01:00
BOOST_AUTO_TEST_CASE(sizes)
{
BOOST_CHECK_EQUAL(sizeof(char), GetSerializeSize(char(0), 0));
BOOST_CHECK_EQUAL(sizeof(int8_t), GetSerializeSize(int8_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(uint8_t), GetSerializeSize(uint8_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(int16_t), GetSerializeSize(int16_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(uint16_t), GetSerializeSize(uint16_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(int32_t), GetSerializeSize(int32_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(uint32_t), GetSerializeSize(uint32_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(int64_t), GetSerializeSize(int64_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(uint64_t), GetSerializeSize(uint64_t(0), 0));
BOOST_CHECK_EQUAL(sizeof(float), GetSerializeSize(float(0), 0));
BOOST_CHECK_EQUAL(sizeof(double), GetSerializeSize(double(0), 0));
// Bool is serialized as char
BOOST_CHECK_EQUAL(sizeof(char), GetSerializeSize(bool(0), 0));
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
// Sanity-check GetSerializeSize and c++ type matching
BOOST_CHECK_EQUAL(GetSerializeSize(char(0), 0), 1);
BOOST_CHECK_EQUAL(GetSerializeSize(int8_t(0), 0), 1);
BOOST_CHECK_EQUAL(GetSerializeSize(uint8_t(0), 0), 1);
BOOST_CHECK_EQUAL(GetSerializeSize(int16_t(0), 0), 2);
BOOST_CHECK_EQUAL(GetSerializeSize(uint16_t(0), 0), 2);
BOOST_CHECK_EQUAL(GetSerializeSize(int32_t(0), 0), 4);
BOOST_CHECK_EQUAL(GetSerializeSize(uint32_t(0), 0), 4);
BOOST_CHECK_EQUAL(GetSerializeSize(int64_t(0), 0), 8);
BOOST_CHECK_EQUAL(GetSerializeSize(uint64_t(0), 0), 8);
BOOST_CHECK_EQUAL(GetSerializeSize(float(0), 0), 4);
BOOST_CHECK_EQUAL(GetSerializeSize(double(0), 0), 8);
BOOST_CHECK_EQUAL(GetSerializeSize(bool(0), 0), 1);
src/serialize.h: base serialization level endianness neutrality Serialization type-safety and endianness compatibility.
2014-12-19 11:41:50 +01:00
}
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
BOOST_AUTO_TEST_CASE(floats_conversion)
{
[trivial] Fix typos in comments
2017-01-18 16:15:37 +01:00
// Choose values that map unambiguously to binary floating point to avoid
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
// rounding issues at the compiler side.
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x00000000), 0.0F);
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x3f000000), 0.5F);
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x3f800000), 1.0F);
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x40000000), 2.0F);
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x40800000), 4.0F);
BOOST_CHECK_EQUAL(ser_uint32_to_float(0x44444444), 785.066650390625F);
BOOST_CHECK_EQUAL(ser_float_to_uint32(0.0F), 0x00000000);
BOOST_CHECK_EQUAL(ser_float_to_uint32(0.5F), 0x3f000000);
BOOST_CHECK_EQUAL(ser_float_to_uint32(1.0F), 0x3f800000);
BOOST_CHECK_EQUAL(ser_float_to_uint32(2.0F), 0x40000000);
BOOST_CHECK_EQUAL(ser_float_to_uint32(4.0F), 0x40800000);
BOOST_CHECK_EQUAL(ser_float_to_uint32(785.066650390625F), 0x44444444);
}
BOOST_AUTO_TEST_CASE(doubles_conversion)
{
[trivial] Fix typos in comments
2017-01-18 16:15:37 +01:00
// Choose values that map unambiguously to binary floating point to avoid
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
// rounding issues at the compiler side.
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x0000000000000000ULL), 0.0);
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x3fe0000000000000ULL), 0.5);
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x3ff0000000000000ULL), 1.0);
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x4000000000000000ULL), 2.0);
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x4010000000000000ULL), 4.0);
BOOST_CHECK_EQUAL(ser_uint64_to_double(0x4088888880000000ULL), 785.066650390625);
BOOST_CHECK_EQUAL(ser_double_to_uint64(0.0), 0x0000000000000000ULL);
BOOST_CHECK_EQUAL(ser_double_to_uint64(0.5), 0x3fe0000000000000ULL);
BOOST_CHECK_EQUAL(ser_double_to_uint64(1.0), 0x3ff0000000000000ULL);
BOOST_CHECK_EQUAL(ser_double_to_uint64(2.0), 0x4000000000000000ULL);
BOOST_CHECK_EQUAL(ser_double_to_uint64(4.0), 0x4010000000000000ULL);
BOOST_CHECK_EQUAL(ser_double_to_uint64(785.066650390625), 0x4088888880000000ULL);
}
/*
Python code to generate the below hashes:
def reversed_hex(x):
return binascii.hexlify(''.join(reversed(x)))
def dsha256(x):
return hashlib.sha256(hashlib.sha256(x).digest()).digest()
reversed_hex(dsha256(''.join(struct.pack('<f', x) for x in range(0,1000)))) == '8e8b4cf3e4df8b332057e3e23af42ebc663b61e0495d5e7e32d85099d7f3fe0c'
reversed_hex(dsha256(''.join(struct.pack('<d', x) for x in range(0,1000)))) == '43d0c82591953c4eafe114590d392676a01585d25b25d433557f0d7878b23f96'
*/
src/serialize.h: base serialization level endianness neutrality Serialization type-safety and endianness compatibility.
2014-12-19 11:41:50 +01:00
BOOST_AUTO_TEST_CASE(floats)
{
Add serialize float/double tests
2015-01-14 11:07:13 +01:00
CDataStream ss(SER_DISK, 0);
// encode
for (int i = 0; i < 1000; i++) {
ss << float(i);
}
BOOST_CHECK(Hash(ss.begin(), ss.end()) == uint256S("8e8b4cf3e4df8b332057e3e23af42ebc663b61e0495d5e7e32d85099d7f3fe0c"));
// decode
for (int i = 0; i < 1000; i++) {
float j;
ss >> j;
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
}
BOOST_AUTO_TEST_CASE(doubles)
{
CDataStream ss(SER_DISK, 0);
// encode
for (int i = 0; i < 1000; i++) {
ss << double(i);
}
BOOST_CHECK(Hash(ss.begin(), ss.end()) == uint256S("43d0c82591953c4eafe114590d392676a01585d25b25d433557f0d7878b23f96"));
// decode
for (int i = 0; i < 1000; i++) {
double j;
ss >> j;
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
src/serialize.h: base serialization level endianness neutrality Serialization type-safety and endianness compatibility.
2014-12-19 11:41:50 +01:00
}
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
BOOST_AUTO_TEST_CASE(varints)
{
// encode
CDataStream ss(SER_DISK, 0);
CDataStream::size_type size = 0;
for (int i = 0; i < 100000; i++) {
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
for (uint64_t i = 0; i < 100000000000ULL; i += 999999937) {
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
// decode
for (int i = 0; i < 100000; i++) {
Fix signed/unsigned comparison warnings
2013-03-07 18:38:25 +01:00
int j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
for (uint64_t i = 0; i < 100000000000ULL; i += 999999937) {
uint64_t j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
}
tests: add varints_bitpatterns test The current tests for varint only check that serialization-deserialization is a roundtrip. That is a useful test, but it is also good to check for some exact bit patterns, to prevent a code change that changes the serialization format from going undetected. As the varint functions are templated, also check with different types.
2016-04-09 06:52:43 +02:00
BOOST_AUTO_TEST_CASE(varints_bitpatterns)
{
CDataStream ss(SER_DISK, 0);
ss << VARINT(0); BOOST_CHECK_EQUAL(HexStr(ss), "00"); ss.clear();
ss << VARINT(0x7f); BOOST_CHECK_EQUAL(HexStr(ss), "7f"); ss.clear();
ss << VARINT((int8_t)0x7f); BOOST_CHECK_EQUAL(HexStr(ss), "7f"); ss.clear();
ss << VARINT(0x80); BOOST_CHECK_EQUAL(HexStr(ss), "8000"); ss.clear();
ss << VARINT((uint8_t)0x80); BOOST_CHECK_EQUAL(HexStr(ss), "8000"); ss.clear();
ss << VARINT(0x1234); BOOST_CHECK_EQUAL(HexStr(ss), "a334"); ss.clear();
ss << VARINT((int16_t)0x1234); BOOST_CHECK_EQUAL(HexStr(ss), "a334"); ss.clear();
ss << VARINT(0xffff); BOOST_CHECK_EQUAL(HexStr(ss), "82fe7f"); ss.clear();
ss << VARINT((uint16_t)0xffff); BOOST_CHECK_EQUAL(HexStr(ss), "82fe7f"); ss.clear();
ss << VARINT(0x123456); BOOST_CHECK_EQUAL(HexStr(ss), "c7e756"); ss.clear();
ss << VARINT((int32_t)0x123456); BOOST_CHECK_EQUAL(HexStr(ss), "c7e756"); ss.clear();
ss << VARINT(0x80123456U); BOOST_CHECK_EQUAL(HexStr(ss), "86ffc7e756"); ss.clear();
ss << VARINT((uint32_t)0x80123456U); BOOST_CHECK_EQUAL(HexStr(ss), "86ffc7e756"); ss.clear();
ss << VARINT(0xffffffff); BOOST_CHECK_EQUAL(HexStr(ss), "8efefefe7f"); ss.clear();
ss << VARINT(0x7fffffffffffffffLL); BOOST_CHECK_EQUAL(HexStr(ss), "fefefefefefefefe7f"); ss.clear();
ss << VARINT(0xffffffffffffffffULL); BOOST_CHECK_EQUAL(HexStr(ss), "80fefefefefefefefe7f"); ss.clear();
}
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
BOOST_AUTO_TEST_CASE(compactsize)
{
CDataStream ss(SER_DISK, 0);
Refactoring: Removed using namespace <xxx> from bench/ and test/ source files.
2016-12-05 08:03:53 +01:00
std::vector<char>::size_type i, j;
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
for (i = 1; i <= MAX_SIZE; i *= 2)
{
WriteCompactSize(ss, i-1);
WriteCompactSize(ss, i);
}
for (i = 1; i <= MAX_SIZE; i *= 2)
{
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE((i-1) == j, "decoded:" << j << " expected:" << (i-1));
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
}
static bool isCanonicalException(const std::ios_base::failure& ex)
{
Improve unit test code not to compare with explanatory messages for each platform. Instead, use have an exception object to check if the string returned by what() on the raised exception matches the string returned by what() on the expected exception instance. This way, we do not need to list all different possible explanatory strings for different platforms in the test code, and make it simple. (The idea is by Cory Fields.)
2013-12-16 01:26:04 +01:00
std::ios_base::failure expectedException("non-canonical ReadCompactSize()");
Fix unit test error on OSX 10.9 using Apple LLVM v5.0. Before the fix, there were 6 errors such as : serialize_tests.cpp:77: error in "noncanonical": incorrect exception std::ios_base::failure is caught It turns out that ex.what() returns following string instead of "non-canonical ReadCompactSize()" "non-canonical ReadCompactSize(): unspecified iostream_category error" After the fix, unit test passed. The test ran using Apple LLVM v5.0 on OSX 10.9 and the unit test error happened because of different error messages by different compilers. g++ --version on my development environment. ``` Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix ```
2013-12-15 15:25:41 +01:00
Improve unit test code not to compare with explanatory messages for each platform. Instead, use have an exception object to check if the string returned by what() on the raised exception matches the string returned by what() on the expected exception instance. This way, we do not need to list all different possible explanatory strings for different platforms in the test code, and make it simple. (The idea is by Cory Fields.)
2013-12-16 01:26:04 +01:00
// The string returned by what() can be different for different platforms.
// Instead of directly comparing the ex.what() with an expected string,
// create an instance of exception to see if ex.what() matches
// the expected explanatory string returned by the exception instance.
return strcmp(expectedException.what(), ex.what()) == 0;
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
}
Fix unit test error on OSX 10.9 using Apple LLVM v5.0. Before the fix, there were 6 errors such as : serialize_tests.cpp:77: error in "noncanonical": incorrect exception std::ios_base::failure is caught It turns out that ex.what() returns following string instead of "non-canonical ReadCompactSize()" "non-canonical ReadCompactSize(): unspecified iostream_category error" After the fix, unit test passed. The test ran using Apple LLVM v5.0 on OSX 10.9 and the unit test error happened because of different error messages by different compilers. g++ --version on my development environment. ``` Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix ```
2013-12-15 15:25:41 +01:00
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
BOOST_AUTO_TEST_CASE(noncanonical)
{
// Write some non-canonical CompactSize encodings, and
// make sure an exception is thrown when read back.
CDataStream ss(SER_DISK, 0);
Refactoring: Removed using namespace <xxx> from bench/ and test/ source files.
2016-12-05 08:03:53 +01:00
std::vector<char>::size_type n;
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
// zero encoded with three bytes:
ss.write("\xfd\x00\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfc encoded with three bytes:
ss.write("\xfd\xfc\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfd encoded with three bytes is OK:
ss.write("\xfd\xfd\x00", 3);
n = ReadCompactSize(ss);
BOOST_CHECK(n == 0xfd);
// zero encoded with five bytes:
ss.write("\xfe\x00\x00\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xffff encoded with five bytes:
ss.write("\xfe\xff\xff\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// zero encoded with nine bytes:
ss.write("\xff\x00\x00\x00\x00\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
// 0x01ffffff encoded with nine bytes:
ss.write("\xff\xff\xff\xff\x01\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
}
Bug fix: CDataStream::GetAndClear() when nReadPos > 0 Changed CDataStream::GetAndClear() to use the most obvious get get and clear instead of a tricky swap(). Added a unit test for CDataStream insert/erase/GetAndClear. Note: GetAndClear() is not performance critical, it is used only by the send-a-message-to-the-network code. Bug was not noticed before now because the send-a-message code never erased from the stream.
2013-10-29 02:16:27 +01:00
BOOST_AUTO_TEST_CASE(insert_delete)
{
// Test inserting/deleting bytes.
CDataStream ss(SER_DISK, 0);
BOOST_CHECK_EQUAL(ss.size(), 0);
ss.write("\x00\x01\x02\xff", 4);
BOOST_CHECK_EQUAL(ss.size(), 4);
char c = (char)11;
// Inserting at beginning/end/middle:
ss.insert(ss.begin(), c);
BOOST_CHECK_EQUAL(ss.size(), 5);
BOOST_CHECK_EQUAL(ss[0], c);
BOOST_CHECK_EQUAL(ss[1], 0);
ss.insert(ss.end(), c);
BOOST_CHECK_EQUAL(ss.size(), 6);
BOOST_CHECK_EQUAL(ss[4], (char)0xff);
BOOST_CHECK_EQUAL(ss[5], c);
ss.insert(ss.begin()+2, c);
BOOST_CHECK_EQUAL(ss.size(), 7);
BOOST_CHECK_EQUAL(ss[2], c);
// Delete at beginning/end/middle
ss.erase(ss.begin());
BOOST_CHECK_EQUAL(ss.size(), 6);
BOOST_CHECK_EQUAL(ss[0], 0);
ss.erase(ss.begin()+ss.size()-1);
BOOST_CHECK_EQUAL(ss.size(), 5);
BOOST_CHECK_EQUAL(ss[4], (char)0xff);
ss.erase(ss.begin()+1);
BOOST_CHECK_EQUAL(ss.size(), 4);
BOOST_CHECK_EQUAL(ss[0], 0);
BOOST_CHECK_EQUAL(ss[1], 1);
BOOST_CHECK_EQUAL(ss[2], 2);
BOOST_CHECK_EQUAL(ss[3], (char)0xff);
// Make sure GetAndClear does the right thing:
CSerializeData d;
ss.GetAndClear(d);
BOOST_CHECK_EQUAL(ss.size(), 0);
}
serialization: teach serializers variadics Also add a variadic CDataStream ctor for ease-of-use.
2016-09-12 20:38:01 +02:00
BOOST_AUTO_TEST_CASE(class_methods)
{
int intval(100);
bool boolval(true);
std::string stringval("testing");
const char* charstrval("testing charstr");
CMutableTransaction txval;
CSerializeMethodsTestSingle methodtest1(intval, boolval, stringval, charstrval, txval);
CSerializeMethodsTestMany methodtest2(intval, boolval, stringval, charstrval, txval);
CSerializeMethodsTestSingle methodtest3;
CSerializeMethodsTestMany methodtest4;
CDataStream ss(SER_DISK, PROTOCOL_VERSION);
BOOST_CHECK(methodtest1 == methodtest2);
ss << methodtest1;
ss >> methodtest4;
ss << methodtest2;
ss >> methodtest3;
BOOST_CHECK(methodtest1 == methodtest2);
BOOST_CHECK(methodtest2 == methodtest3);
BOOST_CHECK(methodtest3 == methodtest4);
CDataStream ss2(SER_DISK, PROTOCOL_VERSION, intval, boolval, stringval, FLATDATA(charstrval), txval);
ss2 >> methodtest3;
BOOST_CHECK(methodtest3 == methodtest4);
}
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
BOOST_AUTO_TEST_SUITE_END()
Reference in a new issue Copy permalink