Merge #15277: contrib: Enable building in Guix containers
751549b52acontrib: guix: Additional clarifications re: substitutes (Carl Dong)cd3e947f50contrib: guix: Various improvements. (Carl Dong)8dff3e48a9contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)3e80ec3ea9contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](72bd4ab867/contrib/gitian-descriptors/gitian-linux.yml (L10))), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](d6e700e40f/contrib/gitian-descriptors/gitian-win-signer.yml (L14)) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK751549b52alaanwj: ACK751549b52aTree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
This commit is contained in:
Wladimir J. van der Laan
commit
3453cf26db
5 changed files with 634 additions and 1 deletions
229
contrib/guix/README.md
Normal file
229
contrib/guix/README.md
Normal file
|
|
@ -0,0 +1,229 @@
|
||||||
|
# Bootstrappable Bitcoin Core Builds
|
||||||
|
|
||||||
|
This directory contains the files necessary to perform bootstrappable Bitcoin
|
||||||
|
Core builds.
|
||||||
|
|
||||||
|
[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
|
||||||
|
to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
|
||||||
|
downloads.
|
||||||
|
|
||||||
|
We achieve bootstrappability by using Guix as a functional package manager.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
Conservatively, a x86_64 machine with:
|
||||||
|
|
||||||
|
- 2 or more logical cores
|
||||||
|
- 4GB of free disk space on the partition that /gnu/store will reside in
|
||||||
|
- 24GB of free disk space on the partition that the Bitcoin Core git repository
|
||||||
|
resides in
|
||||||
|
|
||||||
|
> Note: these requirements are slightly less onerous than those of Gitian builds
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
### Installing Guix
|
||||||
|
|
||||||
|
If you're just testing this out, you can use the
|
||||||
|
[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
|
||||||
|
your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
|
||||||
|
If you don't want this behaviour, refer to the [next
|
||||||
|
section](#choosing-your-security-model).
|
||||||
|
|
||||||
|
Otherwise, follow the [Guix installation guide][guix/bin-install].
|
||||||
|
|
||||||
|
> Note: For those who like to keep their filesystems clean, Guix is designed to
|
||||||
|
> be very standalone and _will not_ conflict with your system's package
|
||||||
|
> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
|
||||||
|
> `~/.config/guix`.
|
||||||
|
|
||||||
|
### Choosing your security model
|
||||||
|
|
||||||
|
Guix allows us to achieve better binary security by using our CPU time to build
|
||||||
|
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
|
||||||
|
this: users can decide whether or not to bootstrap and to use substitutes.
|
||||||
|
|
||||||
|
After installation, you may want to consider [adding substitute
|
||||||
|
servers](#speeding-up-builds-with-substitute-servers) to speed up your build if
|
||||||
|
that fits your security model (say, if you're just testing that this works).
|
||||||
|
This is skippable if you're using the [Dockerfile][fanquake/guix-docker].
|
||||||
|
|
||||||
|
If you prefer not to use any substitutes, make sure to set
|
||||||
|
`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build
|
||||||
|
will take a while, but the resulting packages will be cached for future builds.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes'
|
||||||
|
```
|
||||||
|
|
||||||
|
Likewise, to perform a bootstrapped build (takes even longer):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using the right Guix
|
||||||
|
|
||||||
|
Once Guix is installed, deploy our patched version into your current Guix
|
||||||
|
profile. The changes there are slowly being upstreamed.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
guix pull --url=https://github.com/dongcarl/guix.git \
|
||||||
|
--commit=82c77e52b8b46e0a3aad2cb12307c2e30547deec \
|
||||||
|
--max-jobs=4 # change accordingly
|
||||||
|
```
|
||||||
|
|
||||||
|
Make sure that you are using your current profile. (You are prompted to do this
|
||||||
|
at the end of the `guix pull`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
> Note: There is ongoing work to eliminate this entire section using Guix
|
||||||
|
> [inferiors][guix/inferiors] and [channels][guix/channels].
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### As a Development Environment
|
||||||
|
|
||||||
|
For a Bitcoin Core depends development environment, simply invoke
|
||||||
|
|
||||||
|
```sh
|
||||||
|
guix environment --manifest=contrib/guix/manifest.scm
|
||||||
|
```
|
||||||
|
|
||||||
|
And you'll land back in your shell with all the build dependencies required for
|
||||||
|
a `depends` build injected into your environment.
|
||||||
|
|
||||||
|
### As a Tool for Deterministic Builds
|
||||||
|
|
||||||
|
From the top of a clean Bitcoin Core repository:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./contrib/guix/guix-build.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
After the build finishes successfully (check the status code please), compare
|
||||||
|
hashes:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Recognized environment variables
|
||||||
|
|
||||||
|
* _**HOSTS**_
|
||||||
|
|
||||||
|
Override the space-separated list of platform triples for which to perform a
|
||||||
|
bootstrappable build. _(defaults to "i686-linux-gnu x86\_64-linux-gnu
|
||||||
|
arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_
|
||||||
|
|
||||||
|
> Windows and OS X platform triplet support are WIP.
|
||||||
|
|
||||||
|
* _**SOURCES_PATH**_
|
||||||
|
|
||||||
|
Set the depends tree download cache for sources. This is passed through to the
|
||||||
|
depends tree. Setting this to the same directory across multiple builds of the
|
||||||
|
depends tree can eliminate unnecessary redownloading of package sources.
|
||||||
|
|
||||||
|
* _**MAX_JOBS**_
|
||||||
|
|
||||||
|
Override the maximum number of jobs to run simultaneously, you might want to
|
||||||
|
do so on a memory-limited machine. This may be passed to `make` as in `make
|
||||||
|
--jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
|
||||||
|
value of `nproc` outside the container)_
|
||||||
|
|
||||||
|
* _**SOURCE_DATE_EPOCH**_
|
||||||
|
|
||||||
|
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
|
||||||
|
the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
|
||||||
|
to the output of `$(git log --format=%at -1)`)_
|
||||||
|
|
||||||
|
* _**V**_
|
||||||
|
|
||||||
|
If non-empty, will pass `V=1` to all `make` invocations, making `make` output
|
||||||
|
verbose.
|
||||||
|
|
||||||
|
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
|
||||||
|
|
||||||
|
Additional flags to be passed to `guix environment`. For a fully-bootstrapped
|
||||||
|
build, set this to `--bootstrap --no-substitutes` (refer to the [security
|
||||||
|
model section](#choosing-your-security-model) for more details). Note that a
|
||||||
|
fully-bootstrapped build will take quite a long time on the first run.
|
||||||
|
|
||||||
|
## Tips and Tricks
|
||||||
|
|
||||||
|
### Speeding up builds with substitute servers
|
||||||
|
|
||||||
|
_This whole section is automatically done in the convenience
|
||||||
|
[Dockerfiles][fanquake/guix-docker]_
|
||||||
|
|
||||||
|
For those who are used to life in the fast _(and trustful)_ lane, you can use
|
||||||
|
[substitute servers][guix/substitutes] to enable binary downloads of packages.
|
||||||
|
|
||||||
|
> For those who only want to use substitutes from the official Guix build farm
|
||||||
|
> and have authorized the build farm's signing key during Guix's installation,
|
||||||
|
> you don't need to do anything.
|
||||||
|
|
||||||
|
#### Authorize the signing keys
|
||||||
|
|
||||||
|
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
|
||||||
|
|
||||||
|
```
|
||||||
|
guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
|
||||||
|
```
|
||||||
|
|
||||||
|
For dongcarl's substitute server at https://guix.carldong.io, run as root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Use the substitute servers
|
||||||
|
|
||||||
|
The official Guix build farm at https://ci.guix.gnu.org is automatically used
|
||||||
|
unless the `--no-substitutes` flag is supplied.
|
||||||
|
|
||||||
|
This can be overridden for all `guix` invocations by passing the
|
||||||
|
`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
|
||||||
|
overridden on a call-by-call basis by passing the same `--substitute-urls`
|
||||||
|
option to client tools such at `guix environment`.
|
||||||
|
|
||||||
|
To use dongcarl's substitute server for Bitcoin Core builds after having
|
||||||
|
[authorized his signing key](#authorize-the-signing-keys):
|
||||||
|
|
||||||
|
```
|
||||||
|
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## FAQ
|
||||||
|
|
||||||
|
### How can I trust the binary installation?
|
||||||
|
|
||||||
|
As mentioned at the bottom of [this manual page][guix/bin-install]:
|
||||||
|
|
||||||
|
> The binary installation tarballs can be (re)produced and verified simply by
|
||||||
|
> running the following command in the Guix source tree:
|
||||||
|
>
|
||||||
|
> make guix-binary.x86_64-linux.tar.xz
|
||||||
|
|
||||||
|
### When will Guix be packaged in debian?
|
||||||
|
|
||||||
|
Vagrant Cascadian has been making good progress on this
|
||||||
|
[here][debian/guix-package]. We have all the pieces needed to put up an APT
|
||||||
|
repository and will likely put one up soon.
|
||||||
|
|
||||||
|
[b17e]: http://bootstrappable.org/
|
||||||
|
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
|
||||||
|
|
||||||
|
[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
|
||||||
|
[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
|
||||||
|
[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
|
||||||
|
[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
|
||||||
|
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
|
||||||
|
[guix/inferiors]: https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html
|
||||||
|
[guix/channels]: https://www.gnu.org/software/guix/manual/en/html_node/Channels.html
|
||||||
|
|
||||||
|
[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
|
||||||
|
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix
|
||||||
39
contrib/guix/guix-build.sh
Executable file
39
contrib/guix/guix-build.sh
Executable file
|
|
@ -0,0 +1,39 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
export LC_ALL=C
|
||||||
|
set -e -o pipefail
|
||||||
|
|
||||||
|
# Determine the maximum number of jobs to run simultaneously (overridable by
|
||||||
|
# environment)
|
||||||
|
MAX_JOBS="${MAX_JOBS:-$(nproc)}"
|
||||||
|
|
||||||
|
# Download the depends sources now as we won't have internet access in the build
|
||||||
|
# container
|
||||||
|
make -C "${PWD}/depends" -j"$MAX_JOBS" download ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
|
||||||
|
|
||||||
|
# Determine the reference time used for determinism (overridable by environment)
|
||||||
|
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
|
||||||
|
|
||||||
|
# Deterministically build Bitcoin Core for HOSTs (overriable by environment)
|
||||||
|
for host in ${HOSTS=i686-linux-gnu x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu}; do
|
||||||
|
|
||||||
|
# Display proper warning when the user interrupts the build
|
||||||
|
trap 'echo "** INT received while building ${host}, you may want to clean up the relevant output and distsrc-* directories before rebuilding"' INT
|
||||||
|
|
||||||
|
# Run the build script 'contrib/guix/libexec/build.sh' in the build
|
||||||
|
# container specified by 'contrib/guix/manifest.scm'
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
guix environment --manifest="${PWD}/contrib/guix/manifest.scm" \
|
||||||
|
--container \
|
||||||
|
--pure \
|
||||||
|
--no-cwd \
|
||||||
|
--share="$PWD"=/bitcoin \
|
||||||
|
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
|
||||||
|
${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
|
||||||
|
-- env HOST="$host" \
|
||||||
|
MAX_JOBS="$MAX_JOBS" \
|
||||||
|
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
|
||||||
|
${V:+V=1} \
|
||||||
|
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
|
||||||
|
bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
|
||||||
|
|
||||||
|
done
|
||||||
206
contrib/guix/libexec/build.sh
Normal file
206
contrib/guix/libexec/build.sh
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
export LC_ALL=C
|
||||||
|
set -e -o pipefail
|
||||||
|
|
||||||
|
# Check that environment variables assumed to be set by the environment are set
|
||||||
|
echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..."
|
||||||
|
echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
|
||||||
|
|
||||||
|
#####################
|
||||||
|
# Environment Setup #
|
||||||
|
#####################
|
||||||
|
|
||||||
|
# The depends folder also serves as a base-prefix for depends packages for
|
||||||
|
# $HOSTs after successfully building.
|
||||||
|
BASEPREFIX="${PWD}/depends"
|
||||||
|
|
||||||
|
# Setup an output directory for our build
|
||||||
|
OUTDIR="${OUTDIR:-${PWD}/output}"
|
||||||
|
[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
|
||||||
|
|
||||||
|
# Setup the directory where our Bitcoin Core build for HOST will occur
|
||||||
|
DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}"
|
||||||
|
if [ -e "$DISTSRC" ]; then
|
||||||
|
echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..."
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
mkdir -p "$DISTSRC"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Given a package name and an output name, return the path of that output in our
|
||||||
|
# current guix environment
|
||||||
|
store_path() {
|
||||||
|
grep --extended-regexp "/[^-]{32}-${1}-cross-${HOST}-[^-]+${2:+-${2}}" "${GUIX_ENVIRONMENT}/manifest" \
|
||||||
|
| head --lines=1 \
|
||||||
|
| sed --expression='s|^[[:space:]]*"||' \
|
||||||
|
--expression='s|"[[:space:]]*$||'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Determine output paths to use in CROSS_* environment variables
|
||||||
|
CROSS_GLIBC="$(store_path glibc)"
|
||||||
|
CROSS_GLIBC_STATIC="$(store_path glibc static)"
|
||||||
|
CROSS_KERNEL="$(store_path linux-libre-headers)"
|
||||||
|
CROSS_GCC="$(store_path gcc)"
|
||||||
|
|
||||||
|
# Set environment variables to point Guix's cross-toolchain to the right
|
||||||
|
# includes/libs for $HOST
|
||||||
|
export CROSS_C_INCLUDE_PATH="${CROSS_GCC}/include:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include"
|
||||||
|
export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include"
|
||||||
|
export CROSS_LIBRARY_PATH="${CROSS_GLIBC}/lib:${CROSS_GLIBC_STATIC}/lib:${CROSS_GCC}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_KERNEL}/lib"
|
||||||
|
|
||||||
|
# Disable Guix ld auto-rpath behavior
|
||||||
|
export GUIX_LD_WRAPPER_DISABLE_RPATH=yes
|
||||||
|
|
||||||
|
# Make /usr/bin if it doesn't exist
|
||||||
|
[ -e /usr/bin ] || mkdir -p /usr/bin
|
||||||
|
|
||||||
|
# Symlink file and env to a conventional path
|
||||||
|
[ -e /usr/bin/file ] || ln -s --no-dereference "$(command -v file)" /usr/bin/file
|
||||||
|
[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env
|
||||||
|
|
||||||
|
# Determine the correct value for -Wl,--dynamic-linker for the current $HOST
|
||||||
|
glibc_dynamic_linker=$(
|
||||||
|
case "$HOST" in
|
||||||
|
i686-linux-gnu) echo /lib/ld-linux.so.2 ;;
|
||||||
|
x86_64-linux-gnu) echo /lib64/ld-linux-x86-64.so.2 ;;
|
||||||
|
arm-linux-gnueabihf) echo /lib/ld-linux-armhf.so.3 ;;
|
||||||
|
aarch64-linux-gnu) echo /lib/ld-linux-aarch64.so.1 ;;
|
||||||
|
riscv64-linux-gnu) echo /lib/ld-linux-riscv64-lp64d.so.1 ;;
|
||||||
|
*) exit 1 ;;
|
||||||
|
esac
|
||||||
|
)
|
||||||
|
|
||||||
|
# Environment variables for determinism
|
||||||
|
export QT_RCC_TEST=1
|
||||||
|
export QT_RCC_SOURCE_DATE_OVERRIDE=1
|
||||||
|
export TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --mtime='@${SOURCE_DATE_EPOCH}' --sort=name"
|
||||||
|
export TZ="UTC"
|
||||||
|
|
||||||
|
####################
|
||||||
|
# Depends Building #
|
||||||
|
####################
|
||||||
|
|
||||||
|
# Build the depends tree, overriding variables that assume multilib gcc
|
||||||
|
make -C depends --jobs="$MAX_JOBS" HOST="$HOST" \
|
||||||
|
${V:+V=1} \
|
||||||
|
${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \
|
||||||
|
i686_linux_CC=i686-linux-gnu-gcc \
|
||||||
|
i686_linux_CXX=i686-linux-gnu-g++ \
|
||||||
|
i686_linux_AR=i686-linux-gnu-ar \
|
||||||
|
i686_linux_RANLIB=i686-linux-gnu-ranlib \
|
||||||
|
i686_linux_NM=i686-linux-gnu-nm \
|
||||||
|
i686_linux_STRIP=i686-linux-gnu-strip \
|
||||||
|
x86_64_linux_CC=x86_64-linux-gnu-gcc \
|
||||||
|
x86_64_linux_CXX=x86_64-linux-gnu-g++ \
|
||||||
|
x86_64_linux_AR=x86_64-linux-gnu-ar \
|
||||||
|
x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \
|
||||||
|
x86_64_linux_NM=x86_64-linux-gnu-nm \
|
||||||
|
x86_64_linux_STRIP=x86_64-linux-gnu-strip \
|
||||||
|
qt_config_opts_i686_linux='-platform linux-g++ -xplatform bitcoin-linux-g++'
|
||||||
|
|
||||||
|
|
||||||
|
###########################
|
||||||
|
# Source Tarball Building #
|
||||||
|
###########################
|
||||||
|
|
||||||
|
# Create the source tarball and move it to "${OUTDIR}/src" if not already there
|
||||||
|
if [ -z "$(find "${OUTDIR}/src" -name 'bitcoin-*.tar.gz')" ]; then
|
||||||
|
./autogen.sh
|
||||||
|
env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" ./configure --prefix=/
|
||||||
|
make dist GZIP_ENV='-9n' ${V:+V=1}
|
||||||
|
mkdir -p "${OUTDIR}/src"
|
||||||
|
mv "$(find "${PWD}" -name 'bitcoin-*.tar.gz')" "${OUTDIR}/src/"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Determine the full path to our source tarball
|
||||||
|
SOURCEDIST="$(find "${OUTDIR}/src" -name 'bitcoin-*.tar.gz')"
|
||||||
|
# Determine our distribution name (e.g. bitcoin-0.18.0)
|
||||||
|
DISTNAME="$(basename "$SOURCEDIST" '.tar.gz')"
|
||||||
|
|
||||||
|
###########################
|
||||||
|
# Binary Tarball Building #
|
||||||
|
###########################
|
||||||
|
|
||||||
|
# Create a spec file to normalize ssp linking behaviour
|
||||||
|
spec_file="$(mktemp)"
|
||||||
|
cat << EOF > "$spec_file"
|
||||||
|
*link_ssp:
|
||||||
|
%{fstack-protector|fstack-protector-all|fstack-protector-strong|fstack-protector-explicit:}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Similar flags to Gitian
|
||||||
|
CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests"
|
||||||
|
HOST_CFLAGS="-O2 -g -specs=${spec_file} -ffile-prefix-map=${PWD}=."
|
||||||
|
HOST_CXXFLAGS="-O2 -g -specs=${spec_file} -ffile-prefix-map=${PWD}=."
|
||||||
|
HOST_LDFLAGS="-Wl,--as-needed -Wl,--dynamic-linker=$glibc_dynamic_linker -static-libstdc++"
|
||||||
|
|
||||||
|
# Make $HOST-specific native binaries from depends available in $PATH
|
||||||
|
export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}"
|
||||||
|
(
|
||||||
|
cd "$DISTSRC"
|
||||||
|
|
||||||
|
# Extract the source tarball
|
||||||
|
tar --strip-components=1 -xf "${SOURCEDIST}"
|
||||||
|
|
||||||
|
# Configure this DISTSRC for $HOST
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" \
|
||||||
|
./configure --prefix=/ \
|
||||||
|
--disable-ccache \
|
||||||
|
--disable-maintainer-mode \
|
||||||
|
--disable-dependency-tracking \
|
||||||
|
${CONFIGFLAGS} \
|
||||||
|
CFLAGS="${HOST_CFLAGS}" \
|
||||||
|
CXXFLAGS="${HOST_CXXFLAGS}" \
|
||||||
|
LDFLAGS="${HOST_LDFLAGS}"
|
||||||
|
|
||||||
|
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool
|
||||||
|
|
||||||
|
# Build Bitcoin Core
|
||||||
|
make --jobs="$MAX_JOBS" ${V:+V=1}
|
||||||
|
|
||||||
|
# Perform basic ELF security checks on a series of executables.
|
||||||
|
make -C src --jobs=1 check-security ${V:+V=1}
|
||||||
|
# Check that executables only contain allowed gcc, glibc and libstdc++
|
||||||
|
# version symbols for Linux distro back-compatibility.
|
||||||
|
make -C src --jobs=1 check-symbols ${V:+V=1}
|
||||||
|
|
||||||
|
# Setup the directory where our Bitcoin Core build for HOST will be
|
||||||
|
# installed. This directory will also later serve as the input for our
|
||||||
|
# binary tarballs.
|
||||||
|
INSTALLPATH="${PWD}/installed/${DISTNAME}"
|
||||||
|
mkdir -p "${INSTALLPATH}"
|
||||||
|
# Install built Bitcoin Core to $INSTALLPATH
|
||||||
|
make install DESTDIR="${INSTALLPATH}" ${V:+V=1}
|
||||||
|
(
|
||||||
|
cd installed
|
||||||
|
|
||||||
|
# Prune libtool and object archives
|
||||||
|
find . -name "lib*.la" -delete
|
||||||
|
find . -name "lib*.a" -delete
|
||||||
|
|
||||||
|
# Prune pkg-config files
|
||||||
|
rm -r "${DISTNAME}/lib/pkgconfig"
|
||||||
|
|
||||||
|
# Split binaries and libraries from their debug symbols
|
||||||
|
{
|
||||||
|
find "${DISTNAME}/bin" -type f -executable -print0
|
||||||
|
find "${DISTNAME}/lib" -type f -print0
|
||||||
|
} | xargs -0 -n1 -P"$MAX_JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg
|
||||||
|
|
||||||
|
cp "${DISTSRC}/doc/README.md" "${DISTNAME}/"
|
||||||
|
|
||||||
|
# Finally, deterministically produce {non-,}debug binary tarballs ready
|
||||||
|
# for release
|
||||||
|
find "${DISTNAME}" -not -name "*.dbg" -print0 \
|
||||||
|
| sort --zero-terminated \
|
||||||
|
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
|
||||||
|
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" \
|
||||||
|
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" && exit 1 )
|
||||||
|
find "${DISTNAME}" -name "*.dbg" -print0 \
|
||||||
|
| sort --zero-terminated \
|
||||||
|
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
|
||||||
|
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" \
|
||||||
|
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" && exit 1 )
|
||||||
|
)
|
||||||
|
)
|
||||||
158
contrib/guix/manifest.scm
Normal file
158
contrib/guix/manifest.scm
Normal file
|
|
@ -0,0 +1,158 @@
|
||||||
|
(use-modules (gnu)
|
||||||
|
(gnu packages)
|
||||||
|
(gnu packages autotools)
|
||||||
|
(gnu packages base)
|
||||||
|
(gnu packages bash)
|
||||||
|
(gnu packages check)
|
||||||
|
(gnu packages commencement)
|
||||||
|
(gnu packages compression)
|
||||||
|
(gnu packages cross-base)
|
||||||
|
(gnu packages file)
|
||||||
|
(gnu packages gawk)
|
||||||
|
(gnu packages gcc)
|
||||||
|
(gnu packages linux)
|
||||||
|
(gnu packages perl)
|
||||||
|
(gnu packages pkg-config)
|
||||||
|
(gnu packages python)
|
||||||
|
(gnu packages shells)
|
||||||
|
(guix build-system trivial)
|
||||||
|
(guix gexp)
|
||||||
|
(guix packages)
|
||||||
|
(guix profiles)
|
||||||
|
(guix utils))
|
||||||
|
|
||||||
|
(define (make-ssp-fixed-gcc xgcc)
|
||||||
|
"Given a XGCC package, return a modified package that uses the SSP function
|
||||||
|
from glibc instead of from libssp.so. Taken from:
|
||||||
|
http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html"
|
||||||
|
(package
|
||||||
|
(inherit xgcc)
|
||||||
|
(arguments
|
||||||
|
(substitute-keyword-arguments (package-arguments xgcc)
|
||||||
|
((#:make-flags flags)
|
||||||
|
`(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
|
||||||
|
|
||||||
|
(define (make-gcc-rpath-link xgcc)
|
||||||
|
"Given a XGCC package, return a modified package that replace each instance of
|
||||||
|
-rpath in the default system spec that's inserted by Guix with -rpath-link"
|
||||||
|
(package
|
||||||
|
(inherit xgcc)
|
||||||
|
(arguments
|
||||||
|
(substitute-keyword-arguments (package-arguments xgcc)
|
||||||
|
((#:phases phases)
|
||||||
|
`(modify-phases ,phases
|
||||||
|
(add-after 'pre-configure 'replace-rpath-with-rpath-link
|
||||||
|
(lambda _
|
||||||
|
(substitute* (cons "gcc/config/rs6000/sysv4.h"
|
||||||
|
(find-files "gcc/config"
|
||||||
|
"^gnu-user.*\\.h$"))
|
||||||
|
(("-rpath=") "-rpath-link="))
|
||||||
|
#t))))))))
|
||||||
|
|
||||||
|
(define (make-cross-toolchain target
|
||||||
|
base-gcc-for-libc
|
||||||
|
base-kernel-headers
|
||||||
|
base-libc
|
||||||
|
base-gcc)
|
||||||
|
"Create a cross-compilation toolchain package for TARGET"
|
||||||
|
(let* ((xbinutils (cross-binutils target))
|
||||||
|
;; 1. Build a cross-compiling gcc without targeting any libc, derived
|
||||||
|
;; from BASE-GCC-FOR-LIBC
|
||||||
|
(xgcc-sans-libc (cross-gcc target
|
||||||
|
#:xgcc base-gcc-for-libc
|
||||||
|
#:xbinutils xbinutils))
|
||||||
|
;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
|
||||||
|
;; from BASE-KERNEL-HEADERS
|
||||||
|
(xkernel (cross-kernel-headers target
|
||||||
|
base-kernel-headers
|
||||||
|
xgcc-sans-libc
|
||||||
|
xbinutils))
|
||||||
|
;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
|
||||||
|
;; derived from BASE-LIBC
|
||||||
|
(xlibc (cross-libc target
|
||||||
|
base-libc
|
||||||
|
xgcc-sans-libc
|
||||||
|
xbinutils
|
||||||
|
xkernel))
|
||||||
|
;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
|
||||||
|
;; BASE-GCC
|
||||||
|
(xgcc (cross-gcc target
|
||||||
|
#:xgcc base-gcc
|
||||||
|
#:xbinutils xbinutils
|
||||||
|
#:libc xlibc)))
|
||||||
|
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
|
||||||
|
;; XGCC
|
||||||
|
(package
|
||||||
|
(name (string-append target "-toolchain"))
|
||||||
|
(version (package-version xgcc))
|
||||||
|
(source #f)
|
||||||
|
(build-system trivial-build-system)
|
||||||
|
(arguments '(#:builder (begin (mkdir %output) #t)))
|
||||||
|
(propagated-inputs
|
||||||
|
`(("binutils" ,xbinutils)
|
||||||
|
("libc" ,xlibc)
|
||||||
|
("libc:static" ,xlibc "static")
|
||||||
|
("gcc" ,xgcc)))
|
||||||
|
(synopsis (string-append "Complete GCC tool chain for " target))
|
||||||
|
(description (string-append "This package provides a complete GCC tool
|
||||||
|
chain for " target " development."))
|
||||||
|
(home-page (package-home-page xgcc))
|
||||||
|
(license (package-license xgcc)))))
|
||||||
|
|
||||||
|
(define* (make-bitcoin-cross-toolchain target
|
||||||
|
#:key
|
||||||
|
(base-gcc-for-libc gcc-5)
|
||||||
|
(base-kernel-headers linux-libre-headers-4.19)
|
||||||
|
(base-libc glibc-2.27)
|
||||||
|
(base-gcc (make-gcc-rpath-link
|
||||||
|
(make-ssp-fixed-gcc gcc-9))))
|
||||||
|
"Convienience wrapper around MAKE-CROSS-TOOLCHAIN with default values
|
||||||
|
desirable for building Bitcoin Core release binaries."
|
||||||
|
(make-cross-toolchain target
|
||||||
|
base-gcc-for-libc
|
||||||
|
base-kernel-headers
|
||||||
|
base-libc
|
||||||
|
base-gcc))
|
||||||
|
|
||||||
|
(packages->manifest
|
||||||
|
(list ;; The Basics
|
||||||
|
bash-minimal
|
||||||
|
which
|
||||||
|
coreutils
|
||||||
|
util-linux
|
||||||
|
;; File(system) inspection
|
||||||
|
file
|
||||||
|
grep
|
||||||
|
diffutils
|
||||||
|
findutils
|
||||||
|
;; File transformation
|
||||||
|
patch
|
||||||
|
gawk
|
||||||
|
sed
|
||||||
|
;; Compression and archiving
|
||||||
|
tar
|
||||||
|
bzip2
|
||||||
|
gzip
|
||||||
|
xz
|
||||||
|
zlib
|
||||||
|
;; Build tools
|
||||||
|
gnu-make
|
||||||
|
libtool
|
||||||
|
autoconf
|
||||||
|
automake
|
||||||
|
pkg-config
|
||||||
|
;; Scripting
|
||||||
|
perl
|
||||||
|
python-3.7
|
||||||
|
;; Native gcc 9 toolchain targeting glibc 2.27
|
||||||
|
(make-gcc-toolchain gcc-9 glibc-2.27)
|
||||||
|
;; Cross gcc 9 toolchains targeting glibc 2.27
|
||||||
|
(make-bitcoin-cross-toolchain "i686-linux-gnu")
|
||||||
|
(make-bitcoin-cross-toolchain "x86_64-linux-gnu")
|
||||||
|
(make-bitcoin-cross-toolchain "aarch64-linux-gnu")
|
||||||
|
(make-bitcoin-cross-toolchain "arm-linux-gnueabihf")
|
||||||
|
;; The glibc 2.27 for riscv64 needs gcc 7 to successfully build (see:
|
||||||
|
;; https://www.gnu.org/software/gcc/gcc-7/changes.html#riscv). The final
|
||||||
|
;; toolchain is still a gcc 9 toolchain targeting glibc 2.27.
|
||||||
|
(make-bitcoin-cross-toolchain "riscv64-linux-gnu"
|
||||||
|
#:base-gcc-for-libc gcc-7)))
|
||||||
|
|
@ -160,7 +160,8 @@ define $(package)_preprocess_cmds
|
||||||
echo "QMAKE_LINK_OBJECT_SCRIPT = object_script" >> qtbase/mkspecs/win32-g++/qmake.conf &&\
|
echo "QMAKE_LINK_OBJECT_SCRIPT = object_script" >> qtbase/mkspecs/win32-g++/qmake.conf &&\
|
||||||
sed -i.old "s|QMAKE_CFLAGS = |!host_build: QMAKE_CFLAGS = $($(package)_cflags) $($(package)_cppflags) |" qtbase/mkspecs/win32-g++/qmake.conf && \
|
sed -i.old "s|QMAKE_CFLAGS = |!host_build: QMAKE_CFLAGS = $($(package)_cflags) $($(package)_cppflags) |" qtbase/mkspecs/win32-g++/qmake.conf && \
|
||||||
sed -i.old "s|QMAKE_LFLAGS = |!host_build: QMAKE_LFLAGS = $($(package)_ldflags) |" qtbase/mkspecs/win32-g++/qmake.conf && \
|
sed -i.old "s|QMAKE_LFLAGS = |!host_build: QMAKE_LFLAGS = $($(package)_ldflags) |" qtbase/mkspecs/win32-g++/qmake.conf && \
|
||||||
sed -i.old "s|QMAKE_CXXFLAGS = |!host_build: QMAKE_CXXFLAGS = $($(package)_cxxflags) $($(package)_cppflags) |" qtbase/mkspecs/win32-g++/qmake.conf
|
sed -i.old "s|QMAKE_CXXFLAGS = |!host_build: QMAKE_CXXFLAGS = $($(package)_cxxflags) $($(package)_cppflags) |" qtbase/mkspecs/win32-g++/qmake.conf && \
|
||||||
|
sed -i.old "s/LIBRARY_PATH/(CROSS_)?\0/g" qtbase/mkspecs/features/toolchain.prf
|
||||||
endef
|
endef
|
||||||
|
|
||||||
define $(package)_config_cmds
|
define $(package)_config_cmds
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue