8 - Build the System (Second half)

Toucan Linux Project - 8

A Roll Your Own Distribution

Goal 1 – Base System
Stage 2 – Building the Base System

We will continue to build the software of the base system. This will contain the tools and utilities we need to maintain the system and recover from crashes. Once complete, will move on to configuring the system and installing a kernel. This post is exactly like the previous We will use are custom build program, cci.pl, to continue the process. Start in the $LFS/cci directory (or where you placed it if you are modifying the process more to your methods.)

Step 1 – Verify the System

Follow all the verification steps of the last post and then check that you still have the confs variable set with

echo $confs

If it doesn’t print anything run

export confs=/sources/confs

The standard warning applies: Remember for this whole section you are working as root and even though your are restricted to the target’s filesystem (due to chroot) it is still very easy to undo a lot of your hard work. Work slowly and carefully as you should anytime you are using the root account and always double-check the command line before you press ENTER. Again you can copy and paste as much as possible.

Step 2 – xz 5.2.4

Make the config

cat > $confs/config.xz << EOF
./configure --prefix=/usr    \
            --disable-static \
            --docdir=/usr/share/doc/xz-5.2.4
EOF

Make the install which will move a few binaries

cat > $confs/install.xz << "EOF"
make install
mv -v   /usr/bin/{lzma,unlzma,lzcat,xz,unxz,xzcat} /bin
mv -v /usr/lib/liblzma.so.* /lib
ln -svf ../../lib/$(readlink /usr/lib/liblzma.so) /usr/lib/liblzma.so
EOF

And build with

./cci.pl -n xz -p xz-5.2.4.tar.xz -c config.xz -i install.xz

Step 3- kmod 26

Create the config

cat > $confs/config.kmod << EOF
./configure --prefix=/usr          \
            --bindir=/bin          \
            --sysconfdir=/etc      \
            --with-rootlibdir=/lib \
            --with-xz              \
            --with-zlib
EOF

The kmod package handles module administration replacing the old module-init-tools which supplied the familiar lsmod, insmod, modprobe, and other module commands. The kmod program replaces all of those and performs the same functionality by using symbolic links which need to be made by the install component. The install creation is as follows

echo > $confs/install.kmod << "EOF"
make install
for target in depmod insmod lsmod modinfo modprobe rmmod; do
  ln -sfv ../bin/kmod /sbin/$target
done
ln -sfv kmod /bin/lsmod
EOF

We can't run the test suite as it has a lot of requirements. Use the null package instead

./cci.pl -n kmod -p kmod-26.tar.xz -c config.kmod -t null -i install.kmod

Step 4 - gettext 0.20.1

Create the config

cat > $confs/config.gettext << EOF
./configure --prefix=/usr    \
            --disable-static \
            --docdir=/usr/share/doc/gettext-0.20.1
EOF

And the install which needs to change permissions

cat > $confs/install.gettext << EOF
make install
chmod -v 0755 /usr/lib/preloadable_libintl.so
EOF

The build with

./cci.pl  -n gettext -p gettext-0.20.1.tar.xz -c config.gettext -i install.gettext

Step 5 - libelf 0.176

The only component LFS uses from the elfutils (Executable and Linking Format) package is libelf. It compiles all components by only installs libelf. The package will use all the defaults except the install. Even though we are installing only libelf, we will keep the name elfutils.

Make the config

cat > $confs/install.elfutils << EOF
make -C libelf install
install -vm644 config/libelf.pc /usr/lib/pkgconfig
EOF

Build with

./cci.pl  -n elfutils -p elfutils-0.176.tar.bz2 -i install.elfutils

Step 6 - libffi 3.2.1

We only need the config component as the others are default

cat > $confs/config.libffi << "EOF"
sed -e '/^includesdir/ s/$(libdir).*$/$(includedir)/' \
    -i include/Makefile.in

sed -e '/^includedir/ s/=.*$/=@includedir@/' \
    -e 's/^Cflags: -I${includedir}/Cflags:/' \
    -i libffi.pc.in
./configure --prefix=/usr \
            --disable-static \
            --with-gcc-arch=native
EOF

Build with

./cci.pl -n libffi -p libffi-3.2.1.tar.gz -c config.libffi

Step 7 - OpenSSL 1.1.1c

Build the config

cat > $confs/config.openssl << "EOF"
sed -i '/\} data/s/ =.*$/;\n    memset(\&data, 0, sizeof(data));/' \
  crypto/rand/rand_lib.c
./config --prefix=/usr         \
         --openssldir=/etc/ssl \
         --libdir=lib          \
         shared                \
         zlib-dynamic
EOF

Create the install which will install the additional documentation. If you would rather not, leave the last two lines before the EOF out of the file

cat > $confs/install.opensll << "EOF"
sed -i '/INSTALL_LIBS/s/libcrypto.a libssl.a//' Makefile
make MANSUFFIX=ssl install
mv -v /usr/share/doc/openssl /usr/share/doc/openssl-1.1.1c
cp -vfr doc/* /usr/share/doc/openssl-1.1.1c
EOF

Install with

./cci.pl -n openssl -p openssl-1.1.1c.tar.gz -c config.openssl -i install.openssl

Step 8 - Python 3.7.4

With this package there is a chance it might run out of space in the build ramdisk. If it does unmount the ramdisk with

umount /sources/build

though I have found it to always work.
Create the config

cat > $confs/config.python << EOF
./configure --prefix=/usr       \
            --enable-shared     \
            --with-system-expat \
            --with-system-ffi   \
            --with-ensurepip=yes
EOF

Create the install

cat > $confs/install.python << EOF
make install
chmod -v 755 /usr/lib/libpython3.7m.so
chmod -v 755 /usr/lib/libpython3.so
ln -sfv pip3.7 /usr/bin/pip3
EOF

There is additional HTML documentation you can install on the LFS page. Add those lines above the EOF in the install.

Build the package with

./cci.pl -n python -p Python-3.7.3.tar.xz -c config.python -i install.python

If it was necessary to umount the ramdisk remount it with:

mount -o size=250M -t tmpfs tmpfs /sources/build

Step 9 - ninja 1.9.0

Ninja uses python as the build language. Since it is non-standard we will create three scripts and use the null script for the config.

Create the compile script

echo "python3 configure.py --bootstrap" >> $confs/compile.ninja

Create the test script

echo "./ninja ninja_test && ./ninja_test --gtest_filter=-SubprocessTest.SetWithLots" >> $confs/test.ninja

Make the install script

cat > $confs/install.ninja << EOF
install -vm755 ninja /usr/bin/
install -vDm644 misc/bash-completion /usr/share/bash-completion/completions/ninja
install -vDm644 misc/zsh-completion  /usr/share/zsh/site-functions/_ninja
EOF

Finally, build with

./cci.pl -n ninja -p ninja-1.9.0.tar.gz -c compile.ninja -t test.ninja -i install.ninja

To avoid a user from exhausting system memory, LFS suggests a modification but it depends on the user creating an environment variable (or allowing the system to supply one) that is easily overridden. This isn't a security patch and I opted not to apply it.

Step 10 - Meson 0.51.1

Meson, like Ninja, is a build system that uses Python. It has only a compile and install.

Make the compile

echo "python3 setup.py build" > $confs/compile.meson

And the install:

echo 'python3 setup.py install --root=dest && cp -rv dest/* /' > $confs/install.meson

The build using null for the config and test scripts

./cci.pl -n meson -p meson-0.50.1.tar.gz -c null -m compile.meson -t null -i install.meson

Step 11 - coreutils 8.31

Again we have a patch we will handle using an absolute path. Our package manager will handle this much better. Create the config with the patch

cat > $confs/config.coreutils << "EOF"
patch -Np1 -i /sources/coreutils-8.31-i18n-1.patch
sed -i '/test.lock/s/^/#/' gnulib-tests/gnulib.mk
autoreconf -fiv
FORCE_UNSAFE_CONFIGURE=1 ./configure \
            --prefix=/usr            \
            --enable-no-install-program=kill,uptime
EOF

The test script runs some tests first as root. A second set requires the ownership to be nobody and the tests ran as that user. The test is somewhat complicated

cat > $confs/test.coreutils << "EOF"
make NON_ROOT_USERNAME=nobody check-root
# Add a test user
echo "dummy:x:1000:nobody" >> /etc/group
chown -Rv nobody . 
su nobody -s /bin/bash \
          -c "PATH=$PATH make RUN_EXPENSIVE_TESTS=yes check"
# Delete the user
sed -i '/dummy/d' /etc/group
EOF

Make the install script which will make the binaries FHS compliant

cat > $confs/install.coreutils << "EOF"
make install
mv -v /usr/bin/{cat,chgrp,chmod,chown,cp,date,dd,df,echo} /bin
mv -v /usr/bin/{false,ln,ls,mkdir,mknod,mv,pwd,rm} /bin
mv -v /usr/bin/{rmdir,stty,sync,true,uname} /bin
mv -v /usr/bin/chroot /usr/sbin
mv -v /usr/share/man/man1/chroot.1 /usr/share/man/man8/chroot.8
sed -i s/\"1\"/\"8\"/1 /usr/share/man/man8/chroot.8
mv -v /usr/bin/{head,nice,sleep,touch} /bin
EOF

Then build with

./cci.pl -n coreutils -p coreutils-8.31.tar.xz -c config.coreutils -t test.coreutils -i install.coreutils

Step 12 - Check 0.12.0

Check uses the defaults except for the install which fixes a script. Make the install

cat > $confs/install.check << EOF
make docdir=/usr/share/doc/check-0.12.0 install
sed -i '1 s/tools/usr/' /usr/bin/checkmk
EOF

Then create with

./cci.pl -n check -p check-0.12.0.tar.gz -i install.check

This package has a long delay in the testing where nothing seems to be occurring but it is running. Patience is required.

Step 13 - diffutils 3.7

This is a completely well behaved package. Just build with

./cci.pl -n diffutils -p diffutils-3.7.tar.xz

Step 14 - gawk 5.0.1

There is optional documentation that can be installed. I have chosen not to install it.
Create the config

cat > $confs/config.gawk << EOF
sed -i 's/extras//' Makefile.in
./configure --prefix=/usr
EOF

Then build with

./cci.pl -n gawk -p gawk-5.0.0.tar.xz -c config.gawk

Step 15 - findutils 4.6.0

The config needs to modify the test programs and apply a fix. Create it with

cat > $confs/config.findutils << "EOF"
sed -i 's/test-lock..EXEEXT.//' tests/Makefile.in
sed -i 's/IO_ftrylockfile/IO_EOF_SEEN/' gl/lib/*.c
sed -i '/unistd/a #include <sys/sysmacros.h>' gl/lib/mountlist.c
echo "#define _IO_IN_BACKUP 0x100" >> gl/lib/stdio-impl.h
./configure --prefix=/usr --localstatedir=/var/lib/locate
EOF

Since our startup scripts (the default LFS scripts) need find it must be moved to the root partition. For this we need a custom install

cat > $confs/install.findutils << "EOF"
make install
mv -v /usr/bin/find /bin
sed -i 's|find:=${BINDIR}|find:=/bin|' /usr/bin/updatedb
EOF

Then build

./cci.pl -n findutils -p findutils-4.6.0.tar.gz -c config.findutils -i install.findutils

Step 16 - groff 1.22.4

The groff package is part of a page formatting system. It needs a paper size. For the US this is most likely "letter" but "A4" in the EU. There is a default that is compiled set by the environment variable PAGE. The default can be changed using a file called /etc/papersize. This example uses "letter" but if you prefer something else, change it in the config script

echo 'PAGE=letter ./configure --prefix=/usr' > $confs/config.groff

This can not be built in parallel, so create a compile to disable it

echo "make -j1" > $confs/compile.groff

There is no test suite. Build it

./cci.pl -n groff -p groff-1.22.4.tar.gz -c config.groff -m compile.groff -t null

Step 17 - GRUB 2.04

GRUB version 2 is the boot loader we will use, like LFS. It is complex but very versatile and very standard.

Create the config

cat > $confs/config.grub << EOF
./configure --prefix=/usr          \
            --sbindir=/sbin        \
            --sysconfdir=/etc      \
            --disable-efiemu       \
            --disable-werror
EOF

Make the install

cat > $confs/install.grub << EOF
make install
mv -v /etc/bash_completion.d/grub /usr/share/bash-completion/completions
EOF

The build with

./cci.pl -n grub -p grub-2.02.tar.xz -c config.grub -t null -i install.grub

Step 18 - less 551

This package only needs a config

echo "./configure --prefix=/usr --sysconfdir=/etc" > $confs/config.less

Since there is no test suite build with-system

./cci.pl -n less -p less-530.tar.gz -c config.less -t null

Step 19 - gzip 1.10

Only the install is changed

echo "make install && mv -v /usr/bin/gzip /bin" > $confs/install.gzip

Then build with

./cci.pl -n gzip -p  gzip.tar.gz -i install.gzip

Step 20 -IPRoute2 5.2.0

This package requires the Berkeley DB for the arpd program and iptables. Per LFS we will disable them for now, and revisit them later is necessary.

Create the config

cat > $confs/config.iproute2 << EOF
sed -i /ARPD/d Makefile
rm -fv man/man8/arpd.8
sed -i 's/.m_ipt.o//' tc/Makefile
EOF

and the install

echo "make DOCDIR=/usr/share/doc/iproute2-5.2.0 install" > $confs/install.iproute

Build with

./cci.pl -n iproute2 -p iproute2-5.1.0.tar.xz -c config.iproute2 -t null -i install.iproute2

Step 21 - kbd 2.1.0

The kbd package contains utilities to control the console which includes keyboard mapping and fonts. One program has the old svgalib as a dependency so LFS disables this since svgalib is now defunct. But a lot of older code, such as svgalib, can still be used on modern kernels but doing so is an experiment so for now, we will go with the LFS recommendation of disabling it. There is also an available patch to make the delete key consistent throughout the kbd utilities. Since cci.pl doesn't handle patches we will use an absolute path and fix it later.

Make the config

cat > $confs/config.kbd << "EOF"
patch -Np1 -i /sources/kbd-2.1.0-backspace-1.patch
sed -i 's/\(RESIZECONS_PROGS=\)yes/\1no/g' configure
sed -i 's/resizecons.8 //' docs/man/man8/Makefile.in
PKG_CONFIG_PATH=/tools/lib/pkgconfig ./configure --prefix=/usr --disable-vlock
EOF

If you want additional documentation do the following

cat > $confs/install.kbd << "EOF"
make install
mkdir -v /usr/share/doc/kbd-2.1.0
cp -R -v docs/doc/* /usr/share/doc/kbd-2.1.0'
EOF

and add "-i install.kbd" to the cci.pl command below.

Build it with:
./cci.pl -n kbd -p kbd-2.0.4.tar.xz -c config.kbd

Step 22 - libpipline 1.5.1

This one uses all the defaults. Build it with

./cci.pl -n libpipeline -p libpipeline-1.5.1.tar.gz

Step 23 - make 4.2.1

Make requires a change in the config and a different method of launching the tests.

Make the config

cat > $confs/config.make << "EOF"
sed -i '211,217 d; 219,229 d; 232 d' glob/glob.c
./configure --prefix=/usr
EOF

And the test

echo 'make PERL5LIB=$PWD/tests/ check' > $confs/test.make

Then install with

./cci.pl -n make -p make-4.2.1.tar.bz2 -c config.make -t test.make

Step 24 - patch 2.7.6

All we need is the defaults

./cci.pl -n patch -p patch-2.7.6.tar.xz

Step 25 - man-db 2.8.5

The application part of the online manual. The only change we are making for TTLP from LFS is to set the default browser to links instead of lynx. We have also modified the location of the browser, vgrind, and grap to /usr/local/bin which will be the location for TTLP when we add these tools.

echo > $confs/config.man-db << EOF
./configure --prefix=/usr                   \
       --docdir=/usr/share/doc/man-db-2.8.5 \
       --sysconfdir=/etc                    \
       --disable-setuid                     \
       --enable-cache-owner=bin             \
       --with-browser=/usr/bin/links        \
       --with-vgrind=/usr/local/bin/vgrind  \
       --with-grap=/usr/local/bin/grap      \
       --with-systemdtmpfilesdir=           \
       --with-systemdsystemunitdir=
EOF

The build then uses the default

./cci.pl -n man-db -p man-db-2.8.5.tar.xz -c config.man-db

Please check www.linuxfromscratch.org/lfs/view/development/chapter06/man-db.html for information about manual pages in language other than English.

Step 26 - tar 1.32

Create the config

cat > $confs/config.tar << EOF
FORCE_UNSAFE_CONFIGURE=1  \
./configure --prefix=/usr \
            --bindir=/bin
EOF

This install doesn't install the additional HTML documentation. Build with

../cci.pl -n tar -p tar-1.32.tar.xz -c config.tar

The test suite is very slow.

Step 27 - texinfo 6.6

Make the config

echo './configure --prefix=/usr --disable-static' > $confs/config.texinfo

We will install the TeK components because they are small and we might want to install TeK later.

cat > $confs/install.texinfo << EOF
make install
make TEXMF=/usr/share/texmf install-tex
EOF

Build with

./cci.pl -n texinfo -p texinfo-6.6.tar.xz -c config.texinfo -i install.texinfo

Step 28 - VIM 8.1.1535 

VIM is one of the few packages that will explode into a directory with a name that doesn't match the file. In this case when you explode vim-8.1.tar.gz it creates a directory called vim81 when the standard is to use vim-8.1. This will be a problem we could handle with the package manager, but it is even easier to simply rename the tarball first

pushd /sources
mv vim-8.1.tar.bz2 vim81.tar.bz2
popd

We will only need to perform that one time and both *cci.pl* and the forth coming package manager will work.

Make the config

cat > $confs/config.vim << EOF
echo '#define SYS_VIMRC_FILE "/etc/vimrc"' >> src/feature.h
./configure --prefix=/usr
EOF

And the test

cat > $confs/test.vim << EOF
chown -Rv nobody .
su nobody -s /bin/bash -c "LANG=en_US.UTF-8 make -j1 test"
EOF

And the install which makes a link for age old vi users. Creating the links will fail on later installs, but that is okay.

cat > $confs/install.vim << "EOF"
make install
ln -sv vim /usr/bin/vi
for L in  /usr/share/man/{,*/}man1/vim.1; do
    ln -sv vim.1 $(dirname $L)/vi.1
done
EOF

The build with

./cci.pl -n vim -p vim81.tar.bz2 -c config.vim -t test.vim -i install.vim

Please be sure to enter ":help iccf" at the VIM prompt.

Step 29 Procps-ng 3.3.15

Create the configure (which disables the kill command which comes from util-linux)

cat > $confs/config.procps-ng << EOF
./configure --prefix=/usr                    \
    --exec-prefix=                           \
    --libdir=/usr/lib                        \
    --docdir=/usr/share/doc/procps-ng-3.3.15 \
    --disable-static                         \
    --disable-kill
EOF

In the partial build we need to modify the tests run per LFS

cat > $confs/test.procps-ng << "EOF"
sed -i -r 's|(pmap_initname)\\\$|\1|' testsuite/pmap.test/pmap.exp
sed -i '/set tty/d' testsuite/pkill.test/pkill.exp
rm testsuite/pgrep.test/pgrep.exp
make check
EOF

And finally the install

cat > $confs/install.procps-ng << "EOF"
make install
mv -v /usr/lib/libprocps.so.* /lib
ln -sfv ../../lib/$(readlink /usr/lib/libprocps.so) /usr/lib/libprocps.so
EOF

Build with

./cci.pl -n procps-ng -p procps-ng-3.3.15.tar.xz -c config.procps-ng -t test.procps-ng -i install.procps-ng

Step 30 - util-linux 2.34

Create the config which makes a directory necessary before the configure script is ran

cat > $confs/config.util-linux << EOF
mkdir -pv /var/lib/hwclock
./configure ADJTIME_PATH=/var/lib/hwclock/adjtime   \
            --docdir=/usr/share/doc/util-linux-2.34 \
            --disable-chfn-chsh  \
            --disable-login      \
            --disable-nologin    \
            --disable-su         \
            --disable-setpriv    \
            --disable-runuser    \
            --disable-pylibmount \
            --disable-static     \
            --without-python     \
            --without-systemd    \
            --without-systemdsystemunitdir
EOF

The install uses the nobody user

cat > $confs/test.util-linux << EOF
chown -Rv nobody .
su nobody -s /bin/bash -c "PATH=$PATH make -k check"
EOF

Then install with

./cci.pl -n util-linux -p util-linux-2.33.2.tar.xz -c config.util-linux -t test.util-linux

Step 31 - e2fsprogs 1.45.3

Though we aren't using ext type file systems, others will (you might have chosen ext4). We will install the utilities for administrating those file system as they are very standard.

Create the config

cat > $confs/config.e2fsprogs << EOF 
mkdir -v build
cd       build
../configure --prefix=/usr           \
             --bindir=/bin           \
             --with-root-prefix=""   \
             --enable-elf-shlibs     \
             --disable-libblkid      \
             --disable-libuuid       \
             --disable-uuidd         \
             --disable-fsck
EOF

Create the install

cat > $confs/install.e2fsprogs << EOF
make install
make install-libs
chmod -v u+w /usr/lib/{libcom_err,libe2p,libext2fs,libss}.a
gunzip -v /usr/share/info/libext2fs.info.gz
install-info --dir-file=/usr/share/info/dir /usr/share/info/libext2fs.info
makeinfo -o      doc/com_err.info ../lib/et/com_err.texinfo
install -v -m644 doc/com_err.info /usr/share/info
install-info --dir-file=/usr/share/info/dir /usr/share/info/com_err.info
EOF

Then build with

./cci.pl -n e2fsprogs -p e2fsprogs-1.45.2.tar.gz -c config.e2fsprogs -i install.e2fsprogs

Step 32 - Sysklogd 1.5.1

Though sysklogd is not our final log daemon it will be what we start with to stay LFS compliant.
Create the config

cat > $confs/config.sysklogd << EOF
sed -i '/Error loading kernel symbols/{n;n;d}' ksym_mod.c
sed -i 's/union wait/int/' syslogd.c
EOF

And the install

echo 'make BINDIR=/sbin install' > $confs/install.sysklogd

And build with

./cci.pl -n sysklogd -p sysklogd-1.5.1.tar.gz -c config.sysklogd -i install.sysklogd 

Step 33 - sysvinit 2.95

Again we need a patch with the absolute path for now

echo 'patch -Np1 -i /sources/sysvinit-2.95-consolidated-1.patch' > $confs/config.sysvinit

Then install with

./cci.pl -n sysvinit -p sysvinit-2.94.tar.xz -c config.sysvinit

Step 34 - eudev 3.2.8

Create the config

cat > $confs/config.eudev << EOF
./configure --prefix=/usr           \
            --bindir=/sbin          \
            --sbindir=/sbin         \
            --libdir=/usr/lib       \
            --sysconfdir=/etc       \
            --libexecdir=/lib       \
            --with-rootprefix=      \
            --with-rootlibdir=/lib  \
            --enable-manpages       \
            --disable-static
EOF

Create the test script

cat > $confs/test.eudev << EOF
mkdir -pv /lib/udev/rules.d
mkdir -pv /etc/udev/rules.d
make check
EOF

Create the install

cat > $confs/install.eudev << EOF
tar -xvf ../udev-lfs-20171102.tar.xz
make -f udev-lfs-20171102/Makefile.lfs install
udevadm hwdb --update
EOF

Build with

./cci.pl -n eudev -p eudev-3.2.8.tar.gz -c config.eudev -t test.eudev -i install.eudev

That completes the system software. We have two very important tasks left. The first is some system  configuration, including the boot process since the init program only launches processes from /etc/init. The second is to build a kernel and get it booted with our boot loader, GRUB. We also have a few packages to add that our not part of the LFS base, but we will need for The Toucan Linux Project.

Copyright (C) 2019 by Michael R Stute