Merge #15176: docs: Get rid of badly named readme
f24ed6d39f Delete README_osx.md and move its contents into build-osx.md (Martin Erlandsson)
Pull request description:
With its current name, the file `doc/README_osx.md` looks like an entry point README for OSX users, but it only contains specific instructions on how to build a DMG.
This PR deletes the file and moves the contents of the file into `doc/build-osx.md`.
Tree-SHA512: 2636b9da967f2a4c0d68cb9a157fb3db137bdb8fbff5d9d004f28b5d816e9c27fddc5c403e6b0c363d1dc9ddc7cac8b295efa01fc691126c0e36e21bb9b3cbd3This commit is contained in:
Wladimir J. van der Laan
commit
b9d3df8bcd
3 changed files with 101 additions and 98 deletions
|
|
@ -1,97 +0,0 @@
|
||||||
Deterministic macOS DMG Notes.
|
|
||||||
|
|
||||||
Working macOS DMGs are created in Linux by combining a recent clang,
|
|
||||||
the Apple binutils (ld, ar, etc) and DMG authoring tools.
|
|
||||||
|
|
||||||
Apple uses clang extensively for development and has upstreamed the necessary
|
|
||||||
functionality so that a vanilla clang can take advantage. It supports the use
|
|
||||||
of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary
|
|
||||||
when building for macOS.
|
|
||||||
|
|
||||||
Apple's version of binutils (called cctools) contains lots of functionality
|
|
||||||
missing in the FSF's binutils. In addition to extra linker options for
|
|
||||||
frameworks and sysroots, several other tools are needed as well such as
|
|
||||||
install_name_tool, lipo, and nmedit. These do not build under linux, so they
|
|
||||||
have been patched to do so. The work here was used as a starting point:
|
|
||||||
[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
|
|
||||||
|
|
||||||
In order to build a working toolchain, the following source packages are needed
|
|
||||||
from Apple: cctools, dyld, and ld64.
|
|
||||||
|
|
||||||
These tools inject timestamps by default, which produce non-deterministic
|
|
||||||
binaries. The ZERO_AR_DATE environment variable is used to disable that.
|
|
||||||
|
|
||||||
This version of cctools has been patched to use the current version of clang's
|
|
||||||
headers and its libLTO.so rather than those from llvmgcc, as it was
|
|
||||||
originally done in toolchain4.
|
|
||||||
|
|
||||||
To complicate things further, all builds must target an Apple SDK. These SDKs
|
|
||||||
are free to download, but not redistributable.
|
|
||||||
To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
|
|
||||||
|
|
||||||
This file is several gigabytes in size, but only a single directory inside is
|
|
||||||
needed:
|
|
||||||
```
|
|
||||||
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file.
|
|
||||||
To create a tarball suitable for Gitian input, there are two options:
|
|
||||||
|
|
||||||
Using macOS, you can mount the dmg, and then create it with:
|
|
||||||
```
|
|
||||||
$ hdiutil attach Xcode_7.3.1.dmg
|
|
||||||
$ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
Alternatively, you can use 7zip and SleuthKit to extract the files one by one.
|
|
||||||
The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure
|
|
||||||
the dmg file is in the current directory, and then run the script. You may wish
|
|
||||||
to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when
|
|
||||||
you've confirmed the extraction succeeded.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt-get install p7zip-full sleuthkit
|
|
||||||
contrib/macdeploy/extract-osx-sdk.sh
|
|
||||||
rm -rf 5.hfs MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries
|
|
||||||
which are created using these tools. The build process has been designed to
|
|
||||||
avoid including the SDK's files in Gitian's outputs. All interim tarballs are
|
|
||||||
fully deterministic and may be freely redistributed.
|
|
||||||
|
|
||||||
genisoimage is used to create the initial DMG. It is not deterministic as-is,
|
|
||||||
so it has been patched. A system genisoimage will work fine, but it will not
|
|
||||||
be deterministic because the file-order will change between invocations.
|
|
||||||
The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff).
|
|
||||||
No effort was made to fix this cleanly, so it likely leaks memory badly. But
|
|
||||||
it's only used for a single invocation, so that's no real concern.
|
|
||||||
|
|
||||||
genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the
|
|
||||||
libdmg-hfsplus project is used to compress it. There are several bugs in this
|
|
||||||
tool and its maintainer has seemingly abandoned the project. It has been forked
|
|
||||||
and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus).
|
|
||||||
|
|
||||||
The 'dmg' tool has the ability to create DMGs from scratch as well, but this
|
|
||||||
functionality is broken. Only the compression feature is currently used.
|
|
||||||
Ideally, the creation could be fixed and genisoimage would no longer be necessary.
|
|
||||||
|
|
||||||
Background images and other features can be added to DMG files by inserting a
|
|
||||||
.DS_Store before creation. This is generated by the script
|
|
||||||
contrib/macdeploy/custom_dsstore.py.
|
|
||||||
|
|
||||||
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a
|
|
||||||
requirement in order to satisfy the new Gatekeeper requirements. Because this
|
|
||||||
private key cannot be shared, we'll have to be a bit creative in order for the
|
|
||||||
build process to remain somewhat deterministic. Here's how it works:
|
|
||||||
|
|
||||||
- Builders use Gitian to create an unsigned release. This outputs an unsigned
|
|
||||||
dmg which users may choose to bless and run. It also outputs an unsigned app
|
|
||||||
structure in the form of a tarball, which also contains all of the tools
|
|
||||||
that have been previously (deterministically) built in order to create a
|
|
||||||
final dmg.
|
|
||||||
- The Apple keyholder uses this unsigned app to create a detached signature,
|
|
||||||
using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
|
|
||||||
- Builders feed the unsigned app + detached signature back into Gitian. It
|
|
||||||
uses the pre-built tools to recombine the pieces into a deterministic dmg.
|
|
||||||
100
doc/build-osx.md
100
doc/build-osx.md
|
|
@ -105,3 +105,103 @@ Notes
|
||||||
* Tested on OS X 10.10 Yosemite through macOS 10.13 High Sierra on 64-bit Intel processors only.
|
* Tested on OS X 10.10 Yosemite through macOS 10.13 High Sierra on 64-bit Intel processors only.
|
||||||
|
|
||||||
* Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714)
|
* Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714)
|
||||||
|
|
||||||
|
Deterministic macOS DMG Notes
|
||||||
|
-----------------------------
|
||||||
|
|
||||||
|
Working macOS DMGs are created in Linux by combining a recent clang,
|
||||||
|
the Apple binutils (ld, ar, etc) and DMG authoring tools.
|
||||||
|
|
||||||
|
Apple uses clang extensively for development and has upstreamed the necessary
|
||||||
|
functionality so that a vanilla clang can take advantage. It supports the use
|
||||||
|
of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary
|
||||||
|
when building for macOS.
|
||||||
|
|
||||||
|
Apple's version of binutils (called cctools) contains lots of functionality
|
||||||
|
missing in the FSF's binutils. In addition to extra linker options for
|
||||||
|
frameworks and sysroots, several other tools are needed as well such as
|
||||||
|
install_name_tool, lipo, and nmedit. These do not build under linux, so they
|
||||||
|
have been patched to do so. The work here was used as a starting point:
|
||||||
|
[mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
|
||||||
|
|
||||||
|
In order to build a working toolchain, the following source packages are needed
|
||||||
|
from Apple: cctools, dyld, and ld64.
|
||||||
|
|
||||||
|
These tools inject timestamps by default, which produce non-deterministic
|
||||||
|
binaries. The ZERO_AR_DATE environment variable is used to disable that.
|
||||||
|
|
||||||
|
This version of cctools has been patched to use the current version of clang's
|
||||||
|
headers and its libLTO.so rather than those from llvmgcc, as it was
|
||||||
|
originally done in toolchain4.
|
||||||
|
|
||||||
|
To complicate things further, all builds must target an Apple SDK. These SDKs
|
||||||
|
are free to download, but not redistributable.
|
||||||
|
To obtain it, register for a developer account, then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
|
||||||
|
|
||||||
|
This file is several gigabytes in size, but only a single directory inside is
|
||||||
|
needed:
|
||||||
|
```
|
||||||
|
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file.
|
||||||
|
To create a tarball suitable for Gitian input, there are two options:
|
||||||
|
|
||||||
|
Using macOS, you can mount the dmg, and then create it with:
|
||||||
|
```
|
||||||
|
$ hdiutil attach Xcode_7.3.1.dmg
|
||||||
|
$ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
Alternatively, you can use 7zip and SleuthKit to extract the files one by one.
|
||||||
|
The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure
|
||||||
|
the dmg file is in the current directory, and then run the script. You may wish
|
||||||
|
to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when
|
||||||
|
you've confirmed the extraction succeeded.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apt-get install p7zip-full sleuthkit
|
||||||
|
contrib/macdeploy/extract-osx-sdk.sh
|
||||||
|
rm -rf 5.hfs MacOSX10.11.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries
|
||||||
|
which are created using these tools. The build process has been designed to
|
||||||
|
avoid including the SDK's files in Gitian's outputs. All interim tarballs are
|
||||||
|
fully deterministic and may be freely redistributed.
|
||||||
|
|
||||||
|
genisoimage is used to create the initial DMG. It is not deterministic as-is,
|
||||||
|
so it has been patched. A system genisoimage will work fine, but it will not
|
||||||
|
be deterministic because the file-order will change between invocations.
|
||||||
|
The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff).
|
||||||
|
No effort was made to fix this cleanly, so it likely leaks memory badly. But
|
||||||
|
it's only used for a single invocation, so that's no real concern.
|
||||||
|
|
||||||
|
genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the
|
||||||
|
libdmg-hfsplus project is used to compress it. There are several bugs in this
|
||||||
|
tool and its maintainer has seemingly abandoned the project. It has been forked
|
||||||
|
and is available (with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus).
|
||||||
|
|
||||||
|
The 'dmg' tool has the ability to create DMGs from scratch as well, but this
|
||||||
|
functionality is broken. Only the compression feature is currently used.
|
||||||
|
Ideally, the creation could be fixed and genisoimage would no longer be necessary.
|
||||||
|
|
||||||
|
Background images and other features can be added to DMG files by inserting a
|
||||||
|
.DS_Store before creation. This is generated by the script
|
||||||
|
contrib/macdeploy/custom_dsstore.py.
|
||||||
|
|
||||||
|
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a
|
||||||
|
requirement in order to satisfy the new Gatekeeper requirements. Because this
|
||||||
|
private key cannot be shared, we'll have to be a bit creative in order for the
|
||||||
|
build process to remain somewhat deterministic. Here's how it works:
|
||||||
|
|
||||||
|
- Builders use Gitian to create an unsigned release. This outputs an unsigned
|
||||||
|
dmg which users may choose to bless and run. It also outputs an unsigned app
|
||||||
|
structure in the form of a tarball, which also contains all of the tools
|
||||||
|
that have been previously (deterministically) built in order to create a
|
||||||
|
final dmg.
|
||||||
|
- The Apple keyholder uses this unsigned app to create a detached signature,
|
||||||
|
using the script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
|
||||||
|
- Builders feed the unsigned app + detached signature back into Gitian. It
|
||||||
|
uses the pre-built tools to recombine the pieces into a deterministic dmg.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -92,7 +92,7 @@ Ensure gitian-builder is up-to-date:
|
||||||
echo 'f9a8cdb38b9c309326764ebc937cba1523a3a751a7ab05df3ecc99d18ae466c9 inputs/osslsigncode-1.7.1.tar.gz' | sha256sum -c
|
echo 'f9a8cdb38b9c309326764ebc937cba1523a3a751a7ab05df3ecc99d18ae466c9 inputs/osslsigncode-1.7.1.tar.gz' | sha256sum -c
|
||||||
popd
|
popd
|
||||||
|
|
||||||
Create the macOS SDK tarball, see the [macOS readme](README_osx.md) for details, and copy it into the inputs directory.
|
Create the macOS SDK tarball, see the [macOS build instructions](build-osx.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory.
|
||||||
|
|
||||||
### Optional: Seed the Gitian sources cache and offline git repositories
|
### Optional: Seed the Gitian sources cache and offline git repositories
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue