Toucan Linux Project - 13
A Roll Your Own Distribution
Goal 2 – Angband
After all that hard work we finally get to advance to our second goal: Angband. Angband is a classic terminal game based on Tolkien's fantasy novels The Lord of the Rings. It is a spin off of the game Rogue (actually Moria) which started way back in 1979 when Michael Toy and Glenn Wichman (later Ken Arnold who wrote the original curses library to handle efficient updating terminals). This is when terminals were true serial devices that operated over slow wires, often at speeds of 4,800 to 9,600 baud and updating the whole screen constantly made games too slow. Rogue, an expedition into the danger of a randomly generated dungeon hunting for treasure, became the killer application for curses and spawned many, many others which have been lumped into a group now called "Rouge-like."For clarification the modern term is a bit off from the historical term. Rogue had the, some would say evil, concept of having no saved games. It let you play as many games as you liked provided you started as a new character (this is an role playing game where the player takes actions for a "hero") for each game, but once started the player could play until the character died and then forever, that game was gone. It went as far as to save the games in a directory other than the players home directory, and didn't make it easy to copy (or even read) the save file. The file was written and encrypted to make sure a player couldn't just modify it to resurrect the character. Nope, death was final (now dubbed "permadeath") and if you wanted to keep playing you started all over with a new unskilled character. Some versions might allow you to find your last dead character's gear, after it had been picked through by the dungeon denizens, but overall the only evidence your character even existed was in the high scores file--if you got that far.
Today, Rouge-like means any game that doesn't allow you to have saved games but it can be graphical, low-rez graphical, or even text, though originally Rouge-like meant a console game where the text characters (letters, numbers, and symbols) represented all the aspects of the game. With a good terminal, Rogue and its spawn could look pretty decent, having full color, lines for walls, etc. One particular version called Angband became immensely popular because it was tough but balanced and seemed to have the right combination of risk versus reward. The dungeon denizens and unique foes were pulled right from the pages of Lord of the Rings. Even today, it is a very playable game, having been extended to use SDL graphics but still plays well on the console. At least for The Toucan Linux Project, where there is a lot of time spent compiling, it might provide a little entertainment while the system busies itself doing our bidding.
Not that the base system is built, Angband isn't a very difficult target since it requires mostly the standard library and ncurses (at least until you add support for X Windows when it explodes to 67 libraries--oh the cost of graphics and sound). First we'll build it manually as an exercise but then use the package manager as a test for the next build cycle.
But first let's improve the boot screen and learn about grub.
Step 1 - Customizing the Boot Screen
Though the current boot screen has that MSDOS feel which is adequate, it certainly won't win any beauty contests. Converting it into a fully modern boot takes some work, but we can take a few simple steps to improve it using a splash screen. This is also a great opportunity to introduce our mascot Killbill, the keel-billed toucan.This will require work in the /boot directory. Make sure it is mounted
mount /boot
Backup the current grub configuration
cd /boot/grub
mv grub.cfg grub.cfg.basic
Then we need to create a new one. Most systems now have a frame buffer which is a graphics plain for the console. At the dawn of Linux, there were only teletypes that printed the output of the computer on paper. Later terminals came along that had the same print a line capability of the teletype. Later they actually scrolled to make room for a new line and at last came "smart terminals" which could be sent commands to perform certain tasks such as "place the cursor at location X,Y," "scroll up a line," "clear the screen," and so forth. These were still serial terminals that could only display text along with some extended characters. This is what Angband originally supported. Some terminals had a graphic capability that still operated in serial mode. Eventually Unix started appearing on "workstations" which were powerful personal computers (for the time) that had graphics capabilities and Unix switch the video hardware to graphical screen. In the early days of Linux there such libraries as libsvga allowed a programmer to switch the system to graphic mode and display everything in graphics instead of just text. Later support for frame buffers were added.
Frame buffers are simply an abstract memory object that is a source of pixels that can shown on a video device. It is abstract because each different type of hardware might need a different layout of pixels but a generic frame buffer layer can act as an intermediate device. A program puts pixels on the framebuffer using a standard method and a driver then formats it correctly for the real hardware device. The console can be ran on a frame buffer and can also show graphics which can be used to spiffy up the boot screen.
We'll use a picture of our mascot as the boot and keep it pretty simple for now. Later we can make it better. Download the splash image at
https://drive.google.com/open?id=1GJv1avCUsvCYKN1cECYwU1tVSDbGo8OA
or right click on the image and save as /boot/grub/splash_ttlp.png after you have mounted the boot partition.
Then we need to make a new Grub configuration file that will enable graphics. This isn't too difficult but in order to support systems without a frame buffer (or systems without a Linux driver for their frame buffer) we still need to support text based booting. First, we'll examine the file to understand what it's doing then we'll create it. If you want to use a text editor you can follow along.
Initial Settings
set timeout=3
set default=0
These two lines set the timeout to 3 seconds and the default entry to 0. The timeout determines how long Grub waits for you to press a key to choose a different entry before booting the one selected by default.
Grub Environment
set have_grubenv=true
load_env
Since Grub doesn't allow commands to write files to avoid corruption from the boot loader, it provides a mechanism to keep data from the last boot in what is called the "environment block." This is an opaque mechanism to the bootloader but it is maintained in a file called grubenv in the grub directory (/boot/grub/grubenv normally though distros might change this). It is exactly 1K (1,024) bytes in length. You can view this file with
mount /boot
cat /boot/grub/grubenv
# GRUB Environment Block
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
which shows a blank environment block. Data can be written to the environment block using save_env and the whole environment block can be loaded with load_env. This is only available for filesystems that don't use checksums (such as ZFS) and only on a plain disk with BIOS of EFI functions. To write data to the environment block the save_env entry can be used with the list of environment variables. We will use the environment block to save the default menu entry (the one picked last time) and the script will use features we can use later in our work.
Video Setup Function
Next we define a function to load the video. Most systems are supported by one of seven video modules (efi_gop, efi_uga, ieee1275_fb, vbe, vga, video_bochs, or video_cirrus). In fact, the first five will cover just about any system, but grub supports even more video modules. Most systems will use one of the efi modules or the vbe module. Older systems will probably use vga, video_bochs, or video_cirrus. If the variable feature_all_video_module is equal to y then we will load all video modules using the all_video module, otherwise just the seven are loaded, one at a time. This can be used to support the graphical video modes.function load_video {
if [ $feature_all_video_module == y ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
Save Default
We will use a function to save the last entry chosen as the default. This way if the user chooses a different boot entry it will stay as the default until a different one is selected. The function works by using the special variable chosen that will return the name of the menu entry last chosen. We simply set the default to this entry and save it to the environment block. After first setting the dfalt to 0 to make sure it is valid we then load it as seen above.function save_default {
if [ ${have_grubenv} == true ]; then
set default="${chosen}"
save_env default
fi
}
Partition and Filesystems
Next we add the modules to read GPT partitions, MSDOS partitions, and XFS filesystems. Also the root is set to the boot partition (hda1 is assumed).insmod part_gpt
insmod part_msdos
insmod xfs
set root=(hd0,gpt1)
Video Initiation
The font name is stored in a variable to the default unicode.pf2 and the root is set to the boot partition which is (hd0,gpt1) for this system (change if necessary). It is possible to set the root using the UUID but you must first search for it, and since each GPT partition has a unique UUID there isn't a huge advantage unless for some reason the drive devices change.font=/grub/fonts/unicode.pf2
Now we try to load the font and if it works we can know we can move to graphic mode. First we set the variable gfxmode to 1024x768 which sizes the screen. This can be can mode returned by the videoinfo command at the Grub command line. We'll use 1024x768 which all machines but the very old (which probably don't have a framebuffer) can support. The load_video function is called to load the video modules and then the graphics terminal module is loaded (gfxterm). The locale is set by setting the locale_dir directory and setting the lang variable to the locale which is en_US for me. We also use the gettext module to perform any translations that have been included for the locale.
if loadfont $font ; then
set gfxmode=1024x768
load_video
insmod gfxterm
set locale_dir=/grub/locale
set lang=en_US
insmod gettext
fi
The console will still be in text mode at this point. To switch it to the graphic console we issue the terminal_output to the gfxterm module we previously loaded. Keep in mind that this is graphic mode for Grub which comes from the BIOS (Grub gets the video information from the BIOS, for instance, for the videoinfo command), the kernel can choose a different mode entirely or even switch back to the text console. This is determined by setting the gfxpayload variable to either "text" or "keep." The "keep" option tell Grub to keep the graphic console displayed instead of resetting it to the text console. If the option "text" is given Grub will switch to the text console before booting the kernel. This is set as the default at this level, but each menuentry can override this in case it is booting a kernel that doesn't support a framebuffer.
terminal_output gfxterm
set gfxpayload=keep
Next we setup the background. Start by loading the png module which understands how to decode a PNG image file and format it for the framebuffer. This could also be a JPEG image if insmod jpeg is used or a Targa (TGA) if insmod tga. The menu colors are set with normal to yellow on black with a blue on yellow highlight using menu_color_normal and menu_color_highlight. The background image is specified with the background_image command and the return is checked. If it fails then we are in text mode and the colors are set differently with a white on blue normal text and a blue on yellow highlight.
### Begin Background setup ###
insmod png
set menu_color_normal=yellow/black
set menu_color_highlight=blue/yellow
if background_image /grub/splash_ttlp.png; then
true
else
set menu_color_normal=white/blue
set menu_color_highlight=blue/yellow
fi
### END Background setup ###
Now the menu entries are listed below. Currently we only have one, but you might be using Grub to boot more than one OS. You'll need to modify all entries and call save_default in the list. The menu entry for our kernel is
menuentry "Toucan Linux, Linux 5.2.29" {
save_default
linux /vmlinuz-5.2.9-ttlp root=/dev/sda2 ro rootfstype=xfs
}
We no longer need the set root entry as we have set a default above. If the kernel image of an OS is on a different partition then the set root should be changed in that menu entry. We also can drop the insmod xfs as we have loaded it above.
To create the file use a text editor or run the following
mount /boot
mv /boot/grub/grub.cfg grub.cfg.bkup
echo > /boot/grub/grub.cfg << "EOF"
# Begin /boot/grub/grub.cfg for The Toucan Linux Project
set timeout=3
set default=0
set have_grubenv=true
load_env
function load_video {
if [ $feature_all_video_module == true ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
function save_default {
if [ ${have_grubenv} == true ]; then
set default="${chosen}"
save_env default
fi
}
insmod part_msdos
insmod part_gpt
insmod xfs
font=/grub/fonts/unicode.pf2
set root=(hd0,gpt1)
if loadfont $font ; then
set gfxmode=1024x768
load_video
insmod gfxterm
set locale_dir=/grub/locale
set lang=en_US
insmod gettext
fi
terminal_output gfxterm
set gfxpayload=keep
### Begin Background setup ###
insmod png
set menu_color_normal=yellow/black
set menu_color_highlight=blue/yellow
if background_image /grub/splash.png; then
true
else
set menu_color_normal=white/blue
set menu_color_highlight=blue/yellow
fi
### END Background setup ###
set gfxpayload=keep
menuentry "Toucan Linux 5.2.29" {
save_default
linux /vmlinuz-5.2.9-ttlp root=/dev/sda2 ro rootfstype=xfs
}
EOF
Step 2 - Theory of the Package Manager
The package manager needs to be a source level package manager that builds software from source such as Gentoo's portage. Gentoo is certainly the mother of distros since you can set flags and build pretty much any distro you want. Portage is nice and we could adopt it, but over the years it has become more and more complex with its data sources, configuration files, repositories and working directories spread far and between. Unless you stay plain-vanilla Gentoo (no layman, no '~' packages) your soon find your builds to be a complete mess. It is a good system but TTLP needs something simpler. Gentoo makes it difficult to be experimental because doing so often requires theuse of layman and using the '~' mask instead which then means dependencies get horribly complicated.The name of the package manager will be Package Commander or paccom for short. I like this name because it sounds like some secret branch of the government (there probably is PACCOM buried within the deep state). I just like the thought of a phrase like, "Someone call PACCOM on the QT. Now!" Its is also a kind of inside joke because while PACCOM sounds all official and secretive with all the opacity of modern government, I want it to be very transparent.
The most important aspect of building the system for TTLP, being experimental, is to be able to control the compilation of every single package with full granularity and still maintain ease of understanding. To achieve this it will use two principles: hierarchical configuration and single point of information.
Single Point of Information
This means that each package should have all build and configuration files related to the package in one place. The source can be anywhere in a repository but the various files used to configure, compile, build, and install the package should all be in one place separated from all other packages. Since ease of use is a big factor for TTLP, ASCII text files should be the basis for all storage, not SQL databases, Berkley DB databases, any binary formatted file of any type but good old fashion ASCII files. The concept isn't new but is, in fact, adopted from Unix itself. While Windows has the registry, it once had text configuration files. Browsing through the registry one a newly installed system won't give you much insight into what everything is unless you already know the operating system extremely well. The goal of paccom is to put files related to a package all within a compartment and if all those files are ACSII text, then the compartment can be a simple filesystem directory. Those directories can be grouped within another directory to relate them together.We have already started this work with the cci.pl program which took our four configuration files and turned them into text files all stored within the same directory that was the base name of the package. If the patches are also stored in the directory then all we need to build the package is the source tarball. The presence of files can be used as flags or store information that is special under a filename that makes it obvious what the data does.
Hierarchical configuration
This is a simple idea that is very old. Create a hierarchy of information where the deeper you traverse the hierarchy, the more granular the information becomes. At the top level you have general information and at the bottom you have specific information. A good way to think of this might be something likeOS
/ \
/ \
Windows Unix
/\ / \
/ \ / \
NT Pre95 Linux Mach
/ \
/ \
Ubuntu MX
Information about operating systems in general would be in OS. Within Windows would be information that pertained to all version of Windows while Unix would contain information about all Unixes. Under Linux would be information pertaining to Linux that would override what was in Unix, providing information more pertinent to Linux and, of course, beneath it our two distros of Linux which as Ubuntu and MX. Each of these would contain even more specific information about each distro. Each topic would be a file, such as "Initial Program," "Filesystems," "Boot Program," or any topic. If a directory didn't have a file of a given name, then the information from the previous level will be used.
This is exactly how paccom will work. There will be a main directory that will contain the information hierarchy with packages grouped into large groups, called realms, such as "base," "games," "sound,", etc. Underneath these will be one directory per package that will contain all the files related to that package configuration, compile, testing, and installing. Any information, such as the list of files installed, will be in a file with an appropriate names like "Files Installed." Other files will control if the package is to be installed or skipped if the whole realm is built, if the package is installed, if it needs to be built in the large build directory (greater than 250MB), if it is currently installed and what version.
The main directory will be controlled by a configuration file stored in /etc/paccom. The paccom executable will load this configuration file to find all the required paths including the main directory which will be set to /usr/src/paccom. Underneath this main directory will be the global config, compile, test, and install files along with the global settings file called global_settings. There will also be a directory called realms which will contain all the package directories for that realm. There can be an optional config, compile, test, and install files to override the global defaults above, along with a realm settings file, called realms_settings to apply to all package in that realm. Each realm directory will have a directory within it called sources that will contain all the source archives (tarballs and zip files) for that realm to make it easy to distribute the source for a whole realm as a single tarball.
Each package directory will also have optional config, compile, test, and install files (as created currently by cci.ppl) unique for that package. If there is file called large the large build directory will be used for the build. Later we will add and block file to make sure that package is never installed, a files file that will contain all files installed, a installed file that will contain the versions installed. A file called url will contain the download URL for the source archive. To add dependencies we would created a directory called deps which would contain symbolic links to the package directories for its dependencies.
This scheme will allow the settings and build process for each package to be adjusted. If different compiler options are desired for a single package that can be in the config or compile file depending on the build requirements of the package. If the build process is complicated and needs more logic than the simple configuration files, paccom will first look for a file canned bic.sh (short for build-install-compile) it will use it rather than look for the four configuration files.
The last component is a manifest file in each realm directory that contains all the available packages for that realm named manifest. They may not be all installed, but for a distribution it will contain one line for all possible packages in the realm. This will be a CSV-type file with a field delimiter of a tilde (~). The first field is the package name in paccom. The second field is the filename that contains the source archive. Since we want package information to be stored in the packages directory (per single point of information) the manifest is simply a quick way to read all available packages in a realm without reading the whole directory for all entries. The most important part of the manifest file is it contains the order in which the packages should be installed to handle any in-realm dependencies.
That's pretty much the basics for the package manager. It will be in Perl and written to be easy to understand. We can expand it later to include dependencies and a build list in order to determine a reasonable build order.
Paccom Directory Structure
\->paccom - main directory|-> config - global config file
|-> compile - global compile file
|-> install - global install file
|-> test - global test file
|-> global_settings - global environment variables
\->realms - directory containing realm directories
|-> config - realm config file
|-> compile - realm compile file
|-> install - realm install file
|-> test - realm test file
|-> realm_settings - realm environment variables
\-> base - base realm directory
|-> manifest - list of packages in base realm
\-> sources - directory of source archives
|-> config - realm config file
|-> compile - realm compile file
|-> install - realm install file
|-> test - realm test file
\-> acl - package directory for acl package
| |-> large - if file is present use large build area
| |-> block - if file is present don't installation
| |-> bic.sh - if file is present use for build
| |-> url - contains the URL for source download
| |-> installed - if present package is installed (contains version)
| |-> files - list of files in package
| |-> config - package config file
| |-> compile - package conmpile file
| |-> install - package install file
| |-> test - package test file
|-> attr - package directory for attr
|-> (other package directories)
\-> games - realm directory for games
\-> (other realm directories)
Step 3 - Moving the Kernel Source
Start cleaning up the /sources directory by moving the kernel source to its final destination. All source for the kernels will be installed in /usr/src. The whole directory can be moved using one command and if it is on the same partition, it only need to modify inodes making it instantaneous. The commands are (change to your kernel)cd /sources
mv linux-5.2.29 /usr/src
You might as well move our kernel patch as well
mkdir -pv /usr/src/patch
mv -v /sources/ kernel-fix-root-device.patch /usr/src/patch
Step 3 - Preparing for paccom
Perform the following commands to setup the paccom area under /usr/src. This will create the main area for paccom configuration and realm storage.mkdir -vp /usr/src/paccom/realms/{base,games}/sources
Now we'll move all the directories created by using cci.pl to install and build the source packages from building the system previously. This are all considered part of the base and should be located in the realm base.
cd /sources
find -type d -exec mv -v '{}' /usr/src/paccom/realms/base \;
This will move all directories in /sources to /usr/src/realms/base. This is the basis for all the packages we have installed so far. Each include the config, compile, test, and install scripts we created when building the base. This is all the first version of paccom will need to rebuild those files. We'll add some configuration files in the main paccom directory later when we are ready to rebuild the system completely with optimizations using paccom.
Step 4 - Creating Basic Paccom Configuration
Recall in the theory of operation paccom will use a set of configuration files at three different levels: global, realm, and package. There will be a global config, global compile, global test, and global install. There will be a realm config, realm compile, realm test, and realm install. At the very bottom level will be a package config, package compile, package test, and package install. It will use the package first if available, if it doesn't exist it will use the realm configurations if they exist, and if they don't exist it will use the global configurations.Base is the only realm we will install in the root directory as the prefix for all other packages will be in /usr/local (the reason will be revealed later). That means the configuration files we created earlier for cci.pl actually are realm specific to base. We need to put those configuration files in /usr/src/paccom/realms/base and create a set of global ones that will exist in /usr/src/paccom.
Both the realms and global configuration files will follow the same idea. They will overwrite the prefix and other installation directories to install the code where we want it. Many packages use /usr/local/ as the prefix and for TTLP we intend to install everything not in realm base in /usr/local while the realm base will be installed in the root. Recall in the theory of operation we want to be able to swap two different areas for applications so we can test new code or new optimization options for old code. We don’t want to do this with the base realm as that contains the basic tools we need to administrate and maintain the system.
Move the Configuration Files
We created these files earlier for the installation. They were called config, compile, test, and install and were placed in /sources/configs (the value we set in the confs environment variable.) These provide the defaults we need for most packages. But for the realm base we want the prefix to be the root instead of /usr/local so we need to move the file config there.Move it to the realm directory for realm base
mv /sources/configs/config /usr/src/paccom/realms/base
The null script can stay in the global area since we will only need one for all purposes as it is the NOP but only used for cci.pl. The remaining three scripts will serve as a default for all packages as well. But we want to change the compile script in the realm to use $MAKEOPTS instead of -j $CCI_CPUS.
sed -i 's/-j $CCICPUS/$MAKEOPTS/' compile
mv -v /sources/configs/* /usr/src/paccom
Create the paccom Configuration File
The main configuration file will contain the settings used to configure paccom. It should contain the following entriesPACCOM_DIR - the main directory containing the global settings, global config, global compile, global test, global install, and the null script (used by cci.pl).
PACCOM_REALMS - the location of the realms directory
BUILDDIR - the build area where build directories will be created for packages that are not large
LARGE_BUILDDIR - the build where build directories will be created for large packages
There will be other options added later, but for the first version this is all we need. Like the shell, it is possible to define variables and then deference them with $ allowing the administrator to create other variables in the file that paccom doesn't really need, but won't interfere.
By defaults we will put the main area at /usr/src/paccom since this is a source distribution. The realms directory will be located under the main (wherever it is defined.) The build directory and the large build directory will be located under the same directory at /var/paccom. The initial configuration file can be created with
echo > /etc/paccom.conf << "EOF"
PACCOM_DIR="/usr/src/paccom"
PACCOM_REALMS=$PACCOM_DIR/realms
PACCOM_BUILD=/var/paccom
BUILDDIR=$PACCOM_BUILD/build
LARGE_BUILDDIR=$PACCOM_BUILD/large
EOF
Create the paccom Global Defaults
The global defaults file contains the shell environment variables that all the build scripts will use. This can contain anything required by the individual package. There is a global settings in the file $PACCOM_DIR/global_settings and one for each realm in $PACCOM_REALMS/realm_settings. If these files are present they will be added at the top of each build script. There is not a designated settings file at the package level because any additional environment variables can be set or unset in the local config file.For now we'll only set a few options
echo > /usr/src/paccom/global_settings << "EOF"
CPUS=`grep 'family' /proc/cpuinfo |wc -l`
export CFLAGS="-O2 -march=native -pipe"
export CXXFLAGS=$CFLAGS
export MAKEOPTS=-j$CPUS
EOF
The above options set the optimization level to 2, the machine architecture to the native machine, and to use pipes instead of temporary files to pass data between the compilation programs for both C and C++. It counts the number of processors, set in CPUS and sets the make options to use all available processors when compiling.
Step 5 - Create and Mount the Build Directory
The configuration files specifies /var/paccom/build for packages unless they are large in which case it is /var/paccom/large. We need to create these two directoriesmkdir -vp /var/paccom/{build,large}
To increase the speed of the build process we'll use a RAM disk using the tmpfs filesystem. This will work for any package that only requires 250M or less to compile which is almost all of them. To create this mount so it is mounted on every boot do
echo "tmpfs /var/paccom/build tmpfs size=250M 0 0" >> /etc/fstab
mount /var/paccom/build
Using a RAM disk is optional, but it makes compilation much faster.
Step 6 - Adding angband
The current version of Angband is 4.0.5. It is distributed more as rolling release were you simply download the master source that contains whatever is considered the latest release. First thing to note is that Angband is distributed as ZIP file instead of a tarball.For now, we'll set it up in the paccom directory under the games realm so we can build it with paccom later.
Create the games realm, realm sources, and angband directories
mkdir -vp /usr/src/paccom/realms/games/{angband,sources}
Create the manifest file for the realm games
echo "angband~angband-master.zip" > /usr/src/paccom/realms/games/manifest
We are now ready work on the code for paccom. We can use Angband as the test program then move on to testing a complete realm with the base realm. Next time we'll move to our next goal and examine the code of paccom version 1.
Copyright (C) 2019 Michael R Stute
No comments:
Post a Comment