Toucan Linux Project - 15
A Roll Your Own Distribution
Goal 3 – Rebuild Base Realm with Optimization
In this chapter we'll get the package manager ready for primary use. Once complete we can test a few builds, talk about a few compiler options, and then recompile the whole base realm optimized for the platform.Step 1 - Fix a Few Things
If you've been following along week-by-week, there were a few packages that were misnamed. Although package names can be different then the package directories in the archive, I decided to keep them all the same regardless of case. If you've started this project after, I have already fixed these in the text so these commands may fail.pushd /usr/src/paccom/realms/base
mv -v tie-ixhash/ Tie-IxHash
mv -v xml-parser/ XML-Parser
mv -v wpa-supplicant/ wpa_supplicant
mv -v python/ Python
popd
Step 2 - Create URL Files in Realm Base
Though paccom doesn't yet support automated package downloads, it will in the future. The URLs are stored in each package directory under the file name url, the contents being the URL itself. These are stored in are file lists for wget which are in /sources. First consolidate them allcd /sources
cat wget*.txt > wgets.txt
Now we'll write a script that will do most of the work
for file in `find -type d -maxdepth 1`; do \
sfile=`echo $file |cut -f 2 -d '/'` \
url=`grep $file wgets` \
echo "echo '$url' > $sfile/url" \
done > url_list
This will handle most of the packages except those that are found within others like 'tar', 'file', and 'xz.' Those can be hand edited resulting in the following file which can be downloaded (https://drive.google.com/open?id=1OtC52Foy10Eb_I8uCGfnguXCCEZq1sDZ) or using the following command to make it
pushd /usr/src/paccom/realms/base
cat > make_urls.sh << "EOF"
echo "http://ftp.gnu.org/gnu/gmp/gmp-6.1.2.tar.xz" > gmp/url
echo "http://www.mpfr.org/mpfr-4.0.2/mpfr-4.0.2.tar.xz" > mpfr/url
echo "https://ftp.gnu.org/gnu/mpc/mpc-1.1.0.tar.gz" > mpc/url
echo "https://github.com/shadow-maint/shadow/releases/download/4.6/shadow-4.6.tar.xz" > shadow/url
echo "https://www.sourceware.org/pub/bzip2/bzip2-1.0.6.tar.gz" > bzip2/url
echo "https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz" > pkg-config/url
echo "http://download.savannah.gnu.org/releases/sysvinit/sysvinit-2.94.tar.xz" > sysvinit/url
echo "http://download.savannah.gnu.org/releases/attr/attr-2.4.48.tar.gz" > attr/url
echo "http://download.savannah.gnu.org/releases/acl/acl-2.2.53.tar.gz" > acl/url
echo "https://www.kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.27.tar.xz" > libcap/url
echo "https://sourceforge.net/projects/psmisc/files/psmisc/psmisc-23.2.tar.xz" > psmisc/url
echo "http://anduin.linuxfromscratch.org/LFS/iana-etc-2.30.tar.bz2" > iana-etc/url
echo "http://ftp.gnu.org/gnu/bison/bison-3.4.1.tar.xz" > bison/url
echo "https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz" > flex/url
echo "http://ftp.gnu.org/gnu/grep/grep-3.3.tar.xz" > grep/url
echo "http://ftp.gnu.org/gnu/bash/bash-5.0.tar.gz" > bash/url
echo "http://ftp.gnu.org/gnu/gdbm/gdbm-1.18.1.tar.gz" > gdbm/url
echo "http://ftp.gnu.org/gnu/gperf/gperf-3.1.tar.gz" > gperf/url
echo "https://prdownloads.sourceforge.net/expat/expat-2.2.6.tar.bz2" > expat/url
echo "http://ftp.gnu.org/gnu/inetutils/inetutils-1.9.4.tar.xz" > inetutils/url
echo "https://www.cpan.org/src/5.0/perl-5.30.0.tar.xz" > perl/url
echo "https://launchpad.net/intltool/trunk/0.51.0/+download/intltool-0.51.0.tar.gz" > intltool/url
echo "http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz" > autoconf/url
echo "http://ftp.gnu.org/gnu/automake/automake-1.16.1.tar.xz" > automake/url
echo "https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-26.tar.xz" > kmod/url
echo "http://ftp.gnu.org/gnu/gettext/gettext-0.20.1.tar.xz" > gettext/url
echo "https://sourceware.org/ftp/elfutils/0.176/elfutils-0.176.tar.bz2" > elfutils/url
echo "ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz" > libffi/url
echo "https://www.openssl.org/source/openssl-1.1.1c.tar.gz" > openssl/url
echo "https://ftp.gnu.org/gnu/grub/grub-2.02.tar.xz" > grub/url
echo "https://github.com/ninja-build/ninja/archive/v1.9.0/ninja-1.9.0.tar.gz" > ninja/url
echo "https://github.com/mesonbuild/meson/releases/download/0.50.1/meson-0.50.1.tar.gz" > meson/url
echo "http://ftp.gnu.org/gnu/coreutils/coreutils-8.31.tar.xz" > coreutils/url
echo "https://github.com/libcheck/check/releases/download/0.12.0/check-0.12.0.tar.gz" > check/url
echo "http://ftp.gnu.org/gnu/diffutils/diffutils-3.7.tar.xz" > diffutils/url
echo "http://ftp.gnu.org/gnu/findutils/findutils-4.6.0.tar.gz" > findutils/url
echo "http://ftp.gnu.org/gnu/groff/groff-1.22.4.tar.gz" > groff/url
echo "http://www.greenwoodsoftware.com/less/less-530.tar.gz" > less/url
echo "http://ftp.gnu.org/gnu/gzip/gzip-1.10.tar.xz" > gzip/url
echo "https://www.kernel.org/pub/linux/utils/net/iproute2/iproute2-5.1.0.tar.xz" > iproute2/url
echo "https://www.kernel.org/pub/linux/utils/kbd/kbd-2.0.4.tar.xz" > kbd/url
echo "http://download.savannah.gnu.org/releases/libpipeline/libpipeline-1.5.1.tar.gz" > libpipeline/url
echo "http://ftp.gnu.org/gnu/make/make-4.2.1.tar.bz2" > make/url
echo "http://ftp.gnu.org/gnu/patch/patch-2.7.6.tar.xz" > patch/url
echo "http://ftp.gnu.org/gnu/tar/tar-1.32.tar.xz" > tar/url
echo "http://ftp.gnu.org/gnu/texinfo/texinfo-6.6.tar.xz" > texinfo/url
echo "ftp://ftp.vim.org/pub/vim/unix/vim-8.1.tar.bz2" > vim/url
echo "https://sourceforge.net/projects/procps-ng/files/Production/procps-ng-3.3.15.tar.xz" > procps-ng/url
echo "https://www.kernel.org/pub/linux/utils/util-linux/v2.33/util-linux-2.33.2.tar.xz" > util-linux/url
echo "https://downloads.sourceforge.net/project/e2fsprogs/e2fsprogs/v1.45.2/e2fsprogs-1.45.2.tar.gz" > e2fsprogs/url
echo "http://www.infodrom.org/projects/sysklogd/download/sysklogd-1.5.1.tar.gz" > sysklogd/url
echo "https://dev.gentoo.org/~blueness/eudev/eudev-3.2.8.tar.gz" > eudev/url
echo "https://www.kernel.org/pub/linux/utils/fs/xfs/xfsprogs/xfsprogs-5.2.0.tar.xz" > xfsprogs/url
echo "https://www.kernel.org/pub/software/utils/pciutils/pciutils-3.6.2.tar.xz" > pciutils/url
echo "https://github.com/thom311/libnl/releases/download/libnl3_4_0/libnl-3.4.0.tar.gz" > libnl/url
echo "https://www.kernel.org/pub/software/network/iw/iw-5.3.tar.xz" > iw/url
echo "https://ftp.gnu.org/gnu/wget/wget-1.20.3.tar.gz" > wget/url
echo "https://downloads.sourceforge.net/joe-editor/joe-4.6.tar.gz" > joe/url
echo "https://w1.fi/releases/wpa_supplicant-2.9.tar.gz" > wpa_supplicant/url
echo "https://github.com//libusb/libusb/releases/download/v1.0.22/libusb-1.0.22.tar.bz2" > libusb/url
echo "https://www.kernel.org/pub/linux/utils/usb/usbutils/usbutils-012.tar.xz" > usbutils/url
echo "https://roy.marples.name/downloads/dhcpcd/dhcpcd-8.0.2.tar.xz" > dhcpcd/url
echo "https://downloads.sourceforge.net/infozip/zip30.tar.gz" > zip/url
echo "https://downloads.sourceforge.net/infozip/unzip60.tar.gz" > unzip/url
echo "https://tukaani.org/xz/xz-5.2.4.tar.xz" > xz/url
echo "http://ftp.gnu.org/gnu/ncurses/ncurses-6.1.tar.gz" > ncurses/url
echo "http://rpm5.org/files/popt/popt-1.16.tar.gz" > popt/url
echo "https://github.com/dosfstools/dosfstools/releases/download/v4.1/dosfstools-4.1.tar.xz" > dosfstools/url
echo "https://github.com/rhinstaller/efivar/releases/download/30/efivar-30.tar.bz2" > efivar/url
echo "https://github.com/rhinstaller/efibootmgr/releases/download/14/efibootmgr-14.tar.bz2 " > efibootmgr/url
echo "http://ftp.gnu.org/gnu/glibc/glibc-2.29.tar.xz" > glibc/url
echo "http://download.savannah.gnu.org/releases/man-db/man-db-2.8.5.tar.xz" > man-db/url
echo "https://cpan.metacpan.org/authors/id/C/CH/CHORNY/Tie-IxHash-1.23.tar.gz" > Tie-IxHash/url
echo "https://cpan.metacpan.org/authors/id/T/TO/TODDR/XML-Parser-2.44.tar.gz" > XML-Parser/url
echo "http://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.xz" > libtool/url
echo "http://ftp.gnu.org/gnu/sed/sed-4.7.tar.xz" > sed/url
echo "http://www.linuxfromscratch.org/lfs/downloads/development/lfs-bootscripts-20190524.tar.bz2" > lfs-bootscripts-20190524/url
echo "https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz" > Python/url
echo "ftp://ftp.astron.com/pub/file/file-5.37.tar.gz" > file/url
echo "http://ftp.gnu.org/gnu/gawk/gawk-5.0.0.tar.xz" > gawk/url
echo "https://zlib.net/zlib-1.2.11.tar.xz" > zlib/url
echo "http://ftp.gnu.org/gnu/readline/readline-8.0.tar.gz" > readline/url
echo "https://www.kernel.org/pub/linux/docs/man-pages/man-pages-5.01.tar.xz" > man-pages/url
echo "http://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.xz" > m4/url
echo "http://ftp.gnu.org/gnu/bc/bc-1.07.1.tar.gz" > bc/url
echo "http://ftp.gnu.org/gnu/binutils/binutils-2.32.tar.xz" > binutils/url
EOF
Execute it
bash make_urls
Delete it once you are sure it has completed properly
rm make_urls
popd
Step 4 - Create the Manifest File
These are listed in the order we installed them when building the base system's
Download the file (https://drive.google.com/open?id=1EtMDMR1PYuATzMrf4MtN1n_4OQGeXep8) or make it using the following command
echo > /usr/src/paccom/realms/base/manifest << "EOF"
man-pages~man-pages-5.01.tar.xz
glibc~glibc-2.29.tar.xz
zlib~zlib-1.2.11.tar.xz
file~file-5.37.tar.gz
readline~readline-8.0.tar.gz
m4~m4-1.4.18.tar.xz
bc~bc-2.1.1.tar.gz
binutils~binutils-2.32.tar.xz
gmp~gmp-6.1.2.tar.xz
mpfr~mpfr-4.0.2.tar.xz
mpc~mpc-1.1.0.tar.gz
shadow~shadow-4.6.tar.xz
gcc~gcc-9.1.0.tar.xz
bzip2~bzip2-1.0.6.tar.gz
pkg-config~pkg-config-0.29.2.tar.gz
ncurses~ncurses-6.1.tar.gz
attr~attr-2.4.48.tar.gz
acl~acl-2.2.53.tar.gz
libcap~libcap-2.27.tar.xz
sed~sed-4.7.tar.xz
psmisc~psmisc-23.2.tar.xz
iana-etc~iana-etc-2.30.tar.bz2
bison~bison-3.4.1.tar.xz
flex~flex-2.6.4.tar.gz
grep~grep-3.3.tar.xz
bash~bash-5.0.tar.gz
libtool~libtool-2.4.6.tar.xz
gdbm~gdbm-1.18.1.tar.gz
gperf~gperf-3.1.tar.gz
expat~expat-2.2.6.tar.bz2
inetutils~inetutils-1.9.4.tar.xz
perl~perl-5.30.0.tar.xz
XML-Parser~XML-Parser-2.44.tar.gz
intltool~intltool-0.51.0.tar.gz
autoconf~autoconf-2.69.tar.xz
automake~automake-1.16.1.tar.xz
Python~Python-3.7.3.tar.xz
xz~xz-5.2.4.tar.xz
kmod~kmod-26.tar.xz
gettext~gettext-0.20.1.tar.xz
elfutils~elfutils-0.176.tar.bz2
libffi~libffi-3.2.1.tar.gz
openssl~openssl-1.1.1c.tar.gz
ninja~ninja-1.9.0.tar.gz
meson~meson-0.50.1.tar.gz
coreutils~coreutils-8.31.tar.xz
check~check-0.12.0.tar.gz
diffutils~diffutils-3.7.tar.xz
gawk~gawk-5.0.0.tar.xz
findutils~findutils-4.6.0.tar.gz
groff~groff-1.22.4.tar.gz
grub~grub-2.02.tar.xz
less~less-530.tar.gz
gzip~gzip-1.10.tar.xz
iproute2~iproute2-5.1.0.tar.xz
kbd~kbd-2.0.4.tar.xz
libpipeline~libpipeline-1.5.1.tar.gz
make~make-4.2.1.tar.bz2
patch~patch-2.7.6.tar.xz
man-db~man-db-2.8.5.tar.xz
tar~tar-1.32.tar.xz
texinfo~texinfo-6.6.tar.xz
vim~vim81.tar.bz2
procps-ng~procps-ng-3.3.15.tar.xz
util-linux~util-linux-2.33.2.tar.xz
e2fsprogs~e2fsprogs-1.45.2.tar.gz
sysklogd~sysklogd-1.5.1.tar.gz
sysvinit~sysvinit-2.94.tar.xz
eudev~eudev-3.2.8.tar.gz
xfsprogs~xfsprogs-5.2.0.tar.xz
pciutils~pciutils-3.6.2.tar.xz
libnl~libnl-3.4.0.tar.gz
iw~iw-5.3.tar.xz
wget~wget-1.20.3.tar.gz
joe~joe-4.6.tar.gz
dhcpcd~dhcpcd-8.0.2.tar.xz
wpa_supplicant~wpa_supplicant-2.9.tar.gz
Tie-IxHash~Tie-IxHash-1.23.tar.gz
libusb~libusb-1.0.22.tar.bz2
usbutils~usbutils-012.tar.xz
zip~zip30.tar.gz
unzip~unzip60.tar.gz
popt~popt-1.16.tar.gz
dosfstools~dosfstools-4.1.tar.xz
efivar~efivar-37.1pre.tar.gz
efibootmgr~efibootmgr-16.tar.bz2
EOF
Step 5 - Test a Few Builds
The base realm is now setup properly for paccom to handle compiling. Let's check a few packages to make sure everything works. First try the bc package which is the command line calculator we will use for performance testing later.Download the new version of paccom at
https://drive.google.com/open?id=15w4IZKr13AGd1gcgOtCtM0KeNudAA1fp. The file will be called paccom_v0.6.pl. Move it to the proper place
mv paccom_v0.6.pl /usr/bin/paccom
First we'll do a dry run using realm (-R) "base", package (-p) "bc" and a dry run (-d).
paccom -R base -p bc -d
It should return
==> Welcome to Paccom! <==
------------ BEGIN base:bc ------------
Preparing to build base:bc
Script = /var/paccom/build/bc/bc-2.1.1/paccom_go.sh
Executing /var/paccom/build/bc/bc-2.1.1/paccom_go.sh in /var/paccom/build/bc/bc-2.1.1
Would run system(bash /var/paccom/build/bc/bc-2.1.1/paccom_go.sh) except this is a dry run
Contents -----------------------
source /usr/src/paccom/global_settings
#Config
PREFIX=/usr CC=gcc CFLAGS="-std=c99" ./configure.sh -G -O3
#Compile
make $MAKEOPTS
#Test
#Install
make install
Success
-------------- END bc ----------------
This shows paccom will build the package bc in the realm base and displays the build script which is exactly what we want. Rebuild the package with
paccom -R base -p bc
Test angband to be sure a different realm still works
paccom -R games -p angband
Step 6 - Modify Paccom to Create Logs
Of course, all the information paccom prints should be saved in a log. Since it can do multiple packages in a realm as a realm build we'll have two logs. The first will be the global log that will indicate which packages are being built, and each package will have its own log. We'll create a new configuration variable called PACCOM_LOGDIR and set it to the build directory under /logs.echo 'PACCOM_LOGDIR=$PACCOM_BUILD/logs' >> /etc/paccom.conf
Now instead of print we will use a subroutine called line_out() that will print to the terminal and to a file handle that is passed in as the first argument. The remaining arguments are output to standard out and the file as specified by the file handle. We will add two global variables, one for logging the build steps called log_g for log global and another for each package called log_l for log local. The global log is opened as soon as the configuration file is read. The local log will be opened at the beginning of each package build and closed at the end.
Now all calls to print() are replaced with line_out(). See the following code blocks
# Variables
my %opts; # Options hash
my $realm; # The realm we are working in
my @realms; # The realms we will address
my $log_g; # Global log
my $log_l; # Local log per package
Also code to open the global log after reading the configuration with an example of a call to line_out().
my %config=load_config();
# Open the log file if LOGDIR is specified
if($config{PACCOM_LOGDIR}) {
open($log_g, ">", "$config{PACCOM_LOGDIR}/run-$$");
}
line_out($log_g,"==> Welcome to Paccom! <==\n");
The line_out() routine will print to both standard out and the file handle if it is not zero
# Output to standard out and logfile
sub line_out {
my $fh=shift;
print "@_";
print $fh "@_" if($fh);
}
Here the local (package log is open in bic_package()
# Open the local log
open($log_g, ">", "$config{PACCOM_LOGDIR}/run-$date_s-$$.paccom.log")
or $log_l=0;
# Check for the large file to set build area
Finally we need to modify how the external calls are ran in order to capture their output. Instead of using the system() command, we'll use IPC by opening the command with a pipe to send the ouyput to paccom so it can log it. For this we create a run() subroutine in the utilities section of the code.
############ UTILITY SUBROUTINES ###############
#run
# Runs a command and writes its output to $log
sub run
{
my $fh=shift;
my $cmd=shift;
open(IN,"$cmd 2>&1 |");
while(<IN>) {
line_out($fh,$_);
}
close(IN);
}
This is passed in a file handle and the command line. It opens the command and send both standard in and standard error to a pipe with the output of the pipe being the IN file handle. Now we just read all lines from the pipe and send them to the log closing the pipe when there is no more data.
All print keywords now need to be changed to line_out with the first argument either $log_g for the global log or $log_l for the local log. We also modify functions to pass in the log file handle for those subroutines we want to use for both.
Now rebuild bc with
mkdir -p /var/paccom/logs
paccom -R base -p bc
and look in /var/paccom/logs for two new files
base:bc.Mon_Oct_21_10:49:37_2019-7519.paccom.log
run-Mon_Oct_21_10:49:37_2019-7519.paccom.log
The run file will contain the log of the whole build and the base:bc will contain the build log of bc itself. The date, time, and process identification will be different each run.
Last check is to run a full realm as a dry run. Let's try the base realm
paccom -R base -dv
This should build the whole realm. It will stop on binutils because the build area is too small. This is one of the packages that needs the large build area. To set this for the large build create a file called large in the binutils package directory
touch /usr/src/paccom/realms/base/binutils/large
Then rerun the command. It is important to note that even if the archive can be created in the build area there may not be enough space for all the object files that are created. As we progress forward we'll find which packages fail. Right now, as paccom is written, anything that fails will leave a directory in the build area behind. This is on purpose, for the time being, to help determine which packages have problems. Having these directories will also cause paccom to fail since it can't create them in the build area. This, too, is the early versions as we bootstrap the package manager. We can fix this all later.
One to note right away, several packages have patches which we are ignoring currently. We need paccom to apply patches.
Step 7 - Modify Paccom to Apply patches
This will be very simple. Any file in the package directory that ends in .patch will be assumed to be a patch that needs to be applied. All patches are applied using the patch command though the new run() subroutine.# Get a list of patche found in the current (package) directory
sub get_patches
{
my $prefix=shift;
my @patches = glob("*.patch");
my $num=0;
while($patches[$num]) {
print " ******** Found patch $patches[$num]\n";
$patches[$num]="$prefix/$patches[$num]";
$num++;
}
return(@patches);
}
This function gets the list of patches from the package directory. It must be called when the current working directory is correct. It will prepend the complete path to create the absolute path.
# Do patches this assumes the current directory is the package build dir
sub do_patches
{
my @patches=shift;
foreach my $file (@patches) {
line_out($log_l," Applying patch $file\n");
run($log_l,"/usr/bin/patch -Np1 -i $file");
}
}
After switching to the build area which contains the source we need to apply the patches.
Recall we built patches into several config files as a temporary solution for using cci.pl. We need to remove those as they will cause errors since we have moved the patch files. We can remove these to avoid printed errors.
sed -i 's!patch -Np1 -i /sources/kbd-2.1.0-backspace-1.patch!#!' /usr/src/paccom/realms/base/kbd/config
sed -i 's!patch -Np1 -i /usr/src/paccom/realms/base/sources/bzip2-1.0.6-install_docs-1.patch!#!' /usr/src/paccom/realms/base/bzip2/config
sed -i 's!patch -Np1 -i /usr/src/paccom/realms/base/sources/coreutils-8.31-i18n-1.patch!#!' /usr/src/paccom/realms/base/coreutils/config
Step 8 - Test A Patched Build
Now we should test a few compiles with the patches. Again we can start with a dry run to check the patches are being applied. The kbd has a patch and is a reasonable size to use for tests.paccom -R base -p kbd -d
The output shows the patch is being applied.
==> Welcome to Paccom! <==
==> Welcome to Paccom! <==
------------ BEGIN base:kbd ------------
Preparing to build base:kbd
******** Found patch kbd-2.0.4-backspace-1.patch
Script = /var/paccom/build/kbd/kbd-2.0.4/paccom_go.sh
Full path=/var/paccom/build/kbd/kbd-2.0.4
Applying patch /usr/src/paccom/realms/base/kbd/kbd-2.0.4-backspace-1.patch
patching file data/keymaps/i386/dvorak/dvorak-l.map
patching file data/keymaps/i386/dvorak/dvorak-r.map
patching file data/keymaps/i386/fgGIod/tr_f-latin5.map
patching file data/keymaps/i386/qwerty/lt.l4.map
patching file data/keymaps/i386/qwerty/lt.map
patching file data/keymaps/i386/qwerty/no-latin1.map
patching file data/keymaps/i386/qwerty/ru1.map
patching file data/keymaps/i386/qwerty/ru2.map
patching file data/keymaps/i386/qwerty/ru-cp1251.map
patching file data/keymaps/i386/qwerty/ru-ms.map
patching file data/keymaps/i386/qwerty/ru_win.map
patching file data/keymaps/i386/qwerty/se-ir209.map
patching file data/keymaps/i386/qwerty/se-lat6.map
patching file data/keymaps/i386/qwerty/tr_q-latin5.map
patching file data/keymaps/i386/qwerty/ua.map
patching file data/keymaps/i386/qwerty/ua-utf.map
patching file data/keymaps/i386/qwerty/ua-utf-ws.map
patching file data/keymaps/i386/qwerty/ua-ws.map
Executing /var/paccom/build/kbd/kbd-2.0.4/paccom_go.sh in /var/paccom/build/kbd/kbd-2.0.4
Would run system(bash /var/paccom/build/kbd/kbd-2.0.4/paccom_go.sh) except this is a dry run
Contents -----------------------
source /usr/src/paccom/global_settings
#Config
patch -Np1 -i /sources/kbd-2.1.0-backspace-1.patch
sed -i 's/\(RESIZECONS_PROGS=\)yes/\1no/g' configure
sed -i 's/resizecons.8 //' docs/man/man8/Makefile.in
PKG_CONFIG_PATH=/tools/lib/pkgconfig ./configure --prefix=/usr --disable-vlock
#Compile
make $MAKEOPTS
#Test
#Install
make install
Success
-------------- END kbd ----------------
-------------- END kbd ----------------
Then drop the dry tun to make sure it builds properly.
paccom -R base -p kbd
Step 9 - Verify The Realm Files
There are many packages that used the defaults and have no config, compile, test, or install scripts. We should verify that these scripts are sourced form the realm directory. We can test this with a dry run of patch.
paccom -R base -p patch -dv
In the output the following appears
Using realm /usr/src/paccom/realms/base/config
Using global /usr/src/paccom/compile
Using global /usr/src/paccom/install
Script = /var/paccom/build/patch/patch-2.7.6/paccom_go.sh
Executing /var/paccom/build/patch/patch-2.7.6/paccom_go.sh in /var/paccom/build/patch/patch-2.7.6
Would run system(bash /var/paccom/build/patch/patch-2.7.6/paccom_go.sh) except this is a dry run
Contents -----------------------
This shows that the realm config is used along with the global compile and install. This is the expected behavior.
Step 10 - Modify Paccom to Block
There are some packages we don't want to install again. This could be due to a destructive install or even a slow build that we just don't want to wait on. We'll add the ability to block a package from being installed by checking for the presence of a file named *block in the package directory.To test, let's block the GNU C Compiler. Since we didn't use the cci.pl tool to install the compiler we have to install it from scratch
pushd /usr/src/paccom/realms/base
mkdir -v gcc
cd gcc
Make the config
cat > config << "EOF"
sed -e '/m64=/s/lib64/lib/' -i.orig gcc/config/i386/t-linux64
mkdir -v build
cd build
SED=sed \
../configure --prefix=/usr \
--enable-languages=c,c++ \
--disable-multilib \
--disable-bootstrap \
--with-system-zlib
EOF
Make the install which will perform some steps that con't need to be perform after the first install, but won't do any damage except print out errors
cat > install << "EOF"
make install
rm -rf /usr/lib/gcc/$(gcc -dumpmachine)/9.2.0/include-fixed/bits/
chown -v -R root:root /usr/lib/gcc/*linux-gnu/9.2.0/include{,-fixed}
ln -sv ../usr/bin/cpp /lib
ln -sv gcc /usr/bin/cc
install -v -dm755 /usr/lib/bfd-plugins
ln -sfv ../../libexec/gcc/$(gcc -dumpmachine)/9.2.0/liblto_plugin.so /usr/lib/bfd-plugins/
mkdir -pv /usr/share/gdb/auto-load/usr/lib
mv -v /usr/lib/*gdb.py /usr/share/gdb/auto-load/usr/lib
EOF
Finally create our test block file and the large file since GCC will require more space
touch block large
Next modify paccom to honor the block file, the following lines are added in bic_package() after the change to the package directory
#Check for the block file
if ( -f "block" ) {
line_out($log_g," Package is blocked. Remove $pck_dir/block to unblock for build\n");
return(0);
}
This will cause the bic_program() subroutine to return without continuing, but won't stop the main process.
Now try to build gcc to make sure the block works
popd
paccom -R base -p gcc -d
The output shows
==> Welcome to Paccom! <==
------------ BEGIN base:gcc ------------
Preparing to build base:gcc
Package is blocked. Remove /usr/src/paccom/realms/base/gcc/block to unblock for build
Now remove it to check again.
rm /usr/src/paccom/realms/base/gcc/block
paccom -R base -p gcc -d
This time it performs the dry run build. This build is using the realm configuration files because we manually installed GCC the first time. See step 13 below.
Step 11 - Test a Realm Build
We'll dry tun the whole realm as a test. First remove the current logs (we'll add a log purge later).
rm /var/paccom/logs/*
Now build the whole realm with the dry run by not specifying a package at all.
paccom -R base -d
Check the log file in /var/paccom/logs that starts with "run-" for any errors regarding permissions or missing directories.
Step 12 - Setting Up glibc
We need to create the build files for glibc as we did a manually build during the base system build.Create the config
cat > /usr/src/paccom/realms/base/glibc/config << "EOF"
sed -i '/asm.socket.h/a# include <linux/sockios.h>' \
sysdeps/unix/sysv/linux/bits/socket.h
ln -sfv ../lib/ld-linux-x86-64.so.2 /lib64
ln -sfv ../lib/ld-linux-x86-64.so.2 /lib64/ld-lsb-x86-64.so.3
mkdir -v build
cd build
../configure --prefix=/usr \
--disable-werror \
--enable-kernel=3.2 \
--enable-stack-protector=strong \
--with-headers=/usr/include \
libc_cv_slibdir=/lib
EOF
Create the install
cat > /usr/src/paccom/realms/base/glibc/install << EOF
make install
cp -v ../nscd/nscd.conf /etc/nscd.conf
mkdir -pv /var/cache/nscd
EOF
This package requires the large build area
touch /usr/src/paccom/realms/base/glibc/large
Step 13 - Setup GCC
Like glibc we manually installed the compiler. To create its configuration do the following steps.
Create the config
cat > /usr/src/paccom/realms/base/gcc/config << "EOF"
sed -e '/m64=/s/lib64/lib/' -i.orig gcc/config/i386/t-linux64
mkdir -v build
cd build
SED=sed \
../configure --prefix=/usr \
--enable-languages=c,c++ \
--disable-multilib \
--disable-bootstrap \
--with-system-zlib
EOF
To compile the compiler requires the large build area (we did this earlier, but we'll do it again to keep all GCC instructions in the same place.)
touch /usr/src/paccom/realms/base/gcc/large
Create the install
cat > /usr/src/paccom/realms/base/gcc/install << "EOF"
make install
rm -rf /usr/lib/gcc/$(gcc -dumpmachine)/9.2.0/include-fixed/bits/
chown -v -R root:root /usr/lib/gcc/*linux-gnu/9.2.0/include{,-fixed}
ln -sv ../usr/bin/cpp /lib
ln -sv gcc /usr/bin/cc
install -v -dm755 /usr/lib/bfd-plugins
ln -sfv ../../libexec/gcc/$(gcc -dumpmachine)/9.2.0/liblto_plugin.so /usr/lib/bfd-plugins/
mkdir -pv /usr/share/gdb/auto-load/usr/lib
mv -v /usr/lib/*gdb.py /usr/share/gdb/auto-load/usr/lib
EOF
This will perform steps we don't need to do again at each install, but there is no problem with performing them again except there may be errors in the log.
Step 14 - GCC march Options
Our first step to optimizing the code will be to use compiling techniques specifically for the native architecture . To set this the -march option for GCC. With this set, GCC can use both code generation (compiling) and optimization techniques that are specific to the CPU architecture, such as operands that available for the native CPU that may not be existent in others, optimized hardware floating point instructions, and specialty register use such as SIMD . There is also an -mtune option that can further improve code compilation and optimization. Using these options automatically enables or disables many other options based on the chosen architecture as selected by the GCC developers who are, undoubtedly, the experts.Fortunately, it is easy to enable this, even if you don't know the specific CPU architecture in the system. Simply set the -march to native and GCC will probe the CPU to determine the best options. To see what options are affected on your system use the following command
gcc -march=native -Q --help=target
Outputs the following (for my architecture which is an Intel Skylake CPU). At the bottom is also shows the choices available for many of the options.
The following options are target specific:
-m128bit-long-double [enabled]
-m16 [disabled]
-m32 [disabled]
-m3dnow [disabled]
-m3dnowa [disabled]
-m64 [enabled]
-m80387 [enabled]
-m8bit-idiv [disabled]
-m96bit-long-double [disabled]
-mabi= sysv
-mabm [enabled]
-maccumulate-outgoing-args [disabled]
-maddress-mode= long
-madx [enabled]
-maes [enabled]
-malign-data= compat
-malign-double [disabled]
-malign-functions= 0
-malign-jumps= 0
-malign-loops= 0
-malign-stringops [enabled]
-mandroid [disabled]
-march= skylake
-masm= att
-mavx [enabled]
-mavx2 [enabled]
-mavx256-split-unaligned-load [disabled]
-mavx256-split-unaligned-store [disabled]
-mavx5124fmaps [disabled]
-mavx5124vnniw [disabled]
-mavx512bitalg [disabled]
-mavx512bw [disabled]
-mavx512cd [disabled]
-mavx512dq [disabled]
-mavx512er [disabled]
-mavx512f [disabled]
-mavx512ifma [disabled]
-mavx512pf [disabled]
-mavx512vbmi [disabled]
-mavx512vbmi2 [disabled]
-mavx512vl [disabled]
-mavx512vnni [disabled]
-mavx512vpopcntdq [disabled]
-mbionic [disabled]
-mbmi [enabled]
-mbmi2 [enabled]
-mbranch-cost=<0,5> 3
-mcall-ms2sysv-xlogues [disabled]
-mcet-switch [disabled]
-mcld [disabled]
-mcldemote [disabled]
-mclflushopt [enabled]
-mclwb [disabled]
-mclzero [disabled]
-mcmodel= [default]
-mcpu=
-mcrc32 [disabled]
-mcx16 [enabled]
-mdispatch-scheduler [disabled]
-mdump-tune-features [disabled]
-mf16c [enabled]
-mfancy-math-387 [enabled]
-mfentry [disabled]
-mfentry-name=
-mfentry-section=
-mfma [enabled]
-mfma4 [disabled]
-mforce-drap [disabled]
-mforce-indirect-call [disabled]
-mfp-ret-in-387 [enabled]
-mfpmath= sse
-mfsgsbase [enabled]
-mfunction-return= keep
-mfused-madd
-mfxsr [enabled]
-mgeneral-regs-only [disabled]
-mgfni [disabled]
-mglibc [enabled]
-mhard-float [enabled]
-mhle [enabled]
-miamcu [disabled]
-mieee-fp [enabled]
-mincoming-stack-boundary= 0
-mindirect-branch-register [disabled]
-mindirect-branch= keep
-minline-all-stringops [disabled]
-minline-stringops-dynamically [disabled]
-minstrument-return= none
-mintel-syntax
-mlarge-data-threshold=<number> 65536
-mlong-double-128 [disabled]
-mlong-double-64 [disabled]
-mlong-double-80 [enabled]
-mlwp [disabled]
-mlzcnt [enabled]
-mmanual-endbr [disabled]
-mmemcpy-strategy=
-mmemset-strategy=
-mmitigate-rop [disabled]
-mmmx [enabled]
-mmovbe [enabled]
-mmovdir64b [disabled]
-mmovdiri [disabled]
-mmpx [disabled]
-mms-bitfields [disabled]
-mmusl [disabled]
-mmwaitx [disabled]
-mno-align-stringops [disabled]
-mno-default [disabled]
-mno-fancy-math-387 [disabled]
-mno-push-args [disabled]
-mno-red-zone [disabled]
-mno-sse4 [disabled]
-mnop-mcount [disabled]
-momit-leaf-frame-pointer [disabled]
-mpc32 [disabled]
-mpc64 [disabled]
-mpc80 [disabled]
-mpclmul [enabled]
-mpcommit [disabled]
-mpconfig [disabled]
-mpku [disabled]
-mpopcnt [enabled]
-mprefer-avx128
-mprefer-vector-width= none
-mpreferred-stack-boundary= 0
-mprefetchwt1 [disabled]
-mprfchw [enabled]
-mptwrite [disabled]
-mpush-args [enabled]
-mrdpid [disabled]
-mrdrnd [enabled]
-mrdseed [enabled]
-mrecip [disabled]
-mrecip=
-mrecord-mcount [disabled]
-mrecord-return [disabled]
-mred-zone [enabled]
-mregparm= 6
-mrtd [disabled]
-mrtm [disabled]
-msahf [enabled]
-msgx [enabled]
-msha [disabled]
-mshstk [disabled]
-mskip-rax-setup [disabled]
-msoft-float [disabled]
-msse [enabled]
-msse2 [enabled]
-msse2avx [disabled]
-msse3 [enabled]
-msse4 [enabled]
-msse4.1 [enabled]
-msse4.2 [enabled]
-msse4a [disabled]
-msse5
-msseregparm [disabled]
-mssse3 [enabled]
-mstack-arg-probe [disabled]
-mstack-protector-guard-offset=
-mstack-protector-guard-reg=
-mstack-protector-guard-symbol=
-mstack-protector-guard= tls
-mstackrealign [disabled]
-mstringop-strategy= [default]
-mstv [enabled]
-mtbm [disabled]
-mtls-dialect= gnu
-mtls-direct-seg-refs [enabled]
-mtune-ctrl=
-mtune= skylake
-muclibc [disabled]
-mvaes [disabled]
-mveclibabi= [default]
-mvect8-ret-in-mem [disabled]
-mvpclmulqdq [disabled]
-mvzeroupper [enabled]
-mwaitpkg [disabled]
-mwbnoinvd [disabled]
-mx32 [disabled]
-mxop [disabled]
-mxsave [enabled]
-mxsavec [enabled]
-mxsaveopt [enabled]
-mxsaves [enabled]
Known assembler dialects (for use with the -masm= option):
att intel
Known ABIs (for use with the -mabi= option):
ms sysv
Known code models (for use with the -mcmodel= option):
32 kernel large medium small
Valid arguments to -mfpmath=:
387 387+sse 387,sse both sse sse+387 sse,387
Known indirect branch choices (for use with the -mindirect-branch=/-mfunction-return= options):
keep thunk thunk-extern thunk-inline
Known choices for return instrumentation with -minstrument-return=:
call none nop5
Known data alignment choices (for use with the -malign-data= option):
abi cacheline compat
Known vectorization library ABIs (for use with the -mveclibabi= option):
acml svml
Known address mode (for use with the -maddress-mode= option):
long short
Known preferred register vector length (to use with the -mprefer-vector-width= option):
128 256 512 none
Known stack protector guard (for use with the -mstack-protector-guard= option):
global tls
Valid arguments to -mstringop-strategy=:
byte_loop libcall loop rep_4byte rep_8byte rep_byte unrolled_loop vector_loop
Known TLS dialects (for use with the -mtls-dialect= option):
gnu gnu2
Known valid arguments for -march= option:
i386 i486 i586 pentium lakemont pentium-mmx winchip-c6 winchip2 c3 samuel-2 c3-2 nehemiah c7 esther i686 pentiumpro pentium2 pentium3 pentium3m pentium-m pentium4 pentium4m prescott nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client icelake-server cascadelake bonnell atom silvermont slm goldmont goldmont-plus tremont knl knm intel geode k6 k6-2 k6-3 athlon athlon-tbird athlon-4 athlon-xp athlon-mp x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 btver1 btver2 generic native
Known valid arguments for -mtune= option:
generic i386 i486 pentium lakemont pentiumpro pentium4 nocona core2 nehalem sandybridge haswell bonnell silvermont goldmont goldmont-plus tremont knl knm skylake skylake-avx512 cannonlake icelake-client icelake-server cascadelake intel geode k6 athlon k8 amdfam10 bdver1 bdver2 bdver3 bdver4 btver1 btver2 znver1 znver2
As you can see, this is quite extensive but it is all handled by the native setting. While this can be tweaked for various reasons, there is rarely a use case that requires it.
The other option we will use (temporarily only) is -O2 for optimization level 2. This is considered safe and the level that developers try to support. We will investigate this next chapter. The other option is the -pipe option to let the compiler uses pipes instead of files to move data between the stages of compilation. Recall these settings are contain in the /usr/src/paccom/global_settings file.
Step 15 - Compile the Base Realm
Now we can compile the whole realm using the new optimizations for native CPU architecture and level 2 code optimization. Because the compiler, the standard library, and the linker are so integrated, compiling them is a different affair. They will need to be handled differently with a fairly complicated process which we'll handle later. For now just block them alltouch /usr/src/paccom/realms/base/{gcc,glibc,binutils}/block
Then as the root user (paccom will require root privileges) run
rm /var/paccom/logs/*
paccom -R base -vik
Now paccom will build all the packages in the base realm using the new compiler settings except for those we blocked (gcc and glibc.) This will take some time and you'll need to check the log files afterwards. Play some angband if you want and when it's complete do the following
grep -iEl "fail|error" /var/paccom/*
and it will return the name of all files that contain an error or failure. If there are some, then check the config, compile, and install files to be sure they are correct by comparing them to the older chapters when we installed them in the target system. If they are wrong, recreate them and use paccom to compile them individually using the -R and -p options.
Next time we'll talk more about optimization with GCC.
Copyright (C) 2019 Michael R Stute
Next time we'll talk more about optimization with GCC.
Copyright (C) 2019 Michael R Stute
