Toucan Linux Project: 14 - Coding Paccom First Version

Toucan Linux Project - 14

A Roll Your Own Distribution

Goal 2 – Angband

Though we retrieved the source for Angband, we chose not to install it manually, but instead will install it as a test for the package manager paccom. In this chapter we’ll look at the Perl code of paccom. While Perl has a reputation for being cryptic and hard to understand, it doesn’t have to be that way. It can be clear an precise, though it certainly can be confusing. As a language that was originally used to program tasks that were complicated to write in shell programs, it is a perfect choice for a system utility. In many cases, we’ll use available programs in the base realm to do the work instead of installing native Perl module (using cpan or from the source) which means paccom will have a few dependencies as possible. The only additional module required is Tie::IxHash which we installed in chapter 9.

Let's look at the code step-by-step.

Step 1 – Import Necessary Modules

#!/usr/bin/perl

use strict;
use Getopt::Std;
use File::Copy;
use File::Path qw(make_path remove_tree);
use Tie::IxHash;

The first line simply tells the shell to run /usr/bin/perl as the interrupter for the script. Without this it would assume /bin/sh. Next modules are imported.

strict - Pragma to implement stricter requirements for references, variables, and subroutines. This requires variables to be declared with a scope before using. Also checks references and subroutine names.

Getopt::Std - standard command option parser

File::Copy - file copy subroutine

File::Path - imports make_path (mkdir -p) and remove_tree (rm -rf) for creating and deleteing directory paths

Tie::IxHash - Used to create order associated arrays which are best thought of as hashes that preserve the order of insertion. This ensures data added to a hash can be retrieved in the same (or reverse) order it was added.

Step 2 Declare Variables

# Variables
my %opts; # Options hash
my $realm; # The realm we are working in
my @realms; # The realms we will address

Global variables are declared next as required by the stricture.
%opts - a hash to store the command line options from GetOpt.
$realm - the realm in which we are currently working as specified on the command line
@realms - an array (list) of realms to operate on. This is used if multiple realm are specified on the command line. In the case that $realm is equal to "@all" (an in the future possibly other meta-groups) this will include all the realms to process

Step 3 – Process Command Line

This will process all command options and create a set of global variables to make the code easier to read. Each possible command option is parsed and assigned to a variable. The command line options are

i - print informational messages (flag)
v - print even more messages (flag)
p <string> - package name to install if a single package is to be installed
d - Go through all steps but don't actually configure, compile, or install. Do a dry run (flag)
r - retrieve the source archives if they are not present in the realm sources directory (flag)
k - continue with the remaining packages if an error is encountered in one
R - the realm to install, required if p is specified
c - config file to use, if not specified it will look for /etc/paccom.conf

getopts("R:c:ivp:dr",\%opts) or die("Options are invalid\n");

The call to getopts specifies which options to allow and a colon following the option indicates there is an argument expected. The second parameter is a reference to a hash in which to store the options in the form $hash{option} = argument.

print "Welcome to Paccom!\n";

# Get switches, set options
my $info=$opts{i}; # Print informative messages
my $verbose=$opts{v}; # Be verbose and print lots more info
my $package=$opts{p}; # package to build if a single package
my $dryrun=$opts{d}; # Dry run, do everything b call the build script
my $autoDL=$opts{r}; # If on and source tarball is missing, fetch it with wget
my $keepgoing=$opts{k}; # Keep going if a package fails, die if a realm does
$realm=$opts{R}; # Which realm we are in

All options (except the c option) are stored in variable names to make more sense in the code.

-i = $info
-v = $verbose
-p = $package
-d = $dryrun
-r = $autoDL (retrieve)
-k = $keepgoing
-R = $realm

Step 4 – The Main Program

############ MAIN ###############
#Set the realm to @all is none is specified
if (!$realm) {
if(!$package) {
print "Realm must be specified to perform work on a package\n";
exit(0);
}
else {
print "No realm specified, assuming \@all\n";
$realm="\@all";
}
}
else {
@realms[0]=$realm;
}

The main program follows parsing the command line options. First we check to be sure $realm has a value if a single package is specified. If not, exit as this is required. If no realm is specified then set the realm to "@all" which means to build all realms which is a complete system rebuild. It is possible to specific a realm without a package in which case only that realm is built.

Step 5 – Load and Print Conifg

my %config=load_config();

if($info || $verbose) {
print "Configuation:\n";
while (my($key, $value) = each (%config)) {
print "$key = $value\n";
}
}

# Verify we have default scripts or exit
if (! (-f "$config{PACCOM_DIR}/config" &&
-f "$config{PACCOM_DIR}/compile" &&
-f "$config{PACCOM_DIR}/test" &&
-f "$config{PACCOM_DIR}/install") ) {
print "Failed to find all four global defaults\n";
print "These are required even if realm defaults are present\n";
exit;
}

Next the configuration file is loaded with load_config(). If $info and $verbose is set print out the configuration values as loaded. They are stored in the %config hash. After which the presence of the global scripts are check to be sure they are present in the directory specified in PACCOM_DIR. If they are not present then exit. This is a sanity check.

Step 6 – Process Realms

# Switch to the realms dir
chdir($config{PACCOM_REALMS});
if($realm && $package) {
bic_package($realm, $package, 0);
}
else {
@realms=get_realms($realm) if($realm eq "\@all");

#Now process each realm
foreach my $realm (@realms) {
print "Doing realm $realm\n";
do_realm($realm);
}
}

Still in the main program we now process the realms one by one. First the current directory is set to the directory containing paccom realms (PACCOM_REALMS). There are two options. First, if a single package is specified in $package with its realm in $realm then the package builder fucntion bic_package() is called directory. Otherwise, @realms is considered to be a list of realms to compile. In the future we'll support a comma-separated list of realms, but for now it can be a single realm or the super-group "@all" which means all realms. In this case, the name of all realms is captured with get_realms() first. Then the realms list is traversed and each one is passed to the realm builder do_realm().

Step 7 – The Build Driver

This is the longest piece of code in the whole package manager. It will change a lot as we add functionality including many calls to abstract options, but for now since we are interested in getting a package manager up and working quickly, it will do most of the work of building packages.
We'll examine the build-install-compile function (bic_package()) in pieces

############ PACKAGE SUBROUTINES ###############
# Build, install, compile - main driver routine
sub bic_package {
my $realm=shift;
my $package=shift;
my $manifest=shift;

print "------------ BEGIN $realm:$package ------------\nPreparing to build $realm:$package\n";

if(!$manifest) {
chdir($realm);
#Check to make sure package is part of the realm
print " Loading manifest\n" if($verbose);
$manifest=load_manifest($realm);
if(!$manifest) {
print ("Can't load manifest from realm $realm\n");
return(0);
}

if(!$manifest->{$package}) {
print "Package $realm:$package not found...skipping\n";
return(0);
}
}

First we parse the arguments which are the name of the realm, the package to install, and the manifest which is a reference to a hash containing the entries in the manifest. If the manifest is not passed (when it is building a single package) it first changes to the realm directory, loads the manifest using load_manifest() and returns FALSE it if can't. Once the manifest it loaded (or provided as a argument) it checks to make sure the package is a valid name and returns FALSE if it is not. This is verifies the package exists in the realm.

Step 8 – Determine Package and Build Areas

# Determine the build area path
my $bld_area=$config{BUILDDIR};
my $pck_dir="$config{PACCOM_REALMS}/$realm/$package";
print " Changing to $pck_dir\n" if($verbose);
if(!chdir($pck_dir)) {
print "Can't change to $pck_dir\n";
return(0);
}

# Check for the large file to set build area
if ( -f "large" ) {
print " Using large build area\n" if($verbose);
$bld_area="$config{LARGE_BUILDDIR}";
}

# Determine the full path to package build area
my $src="$config{PACCOM_REALMS}/$realm/sources/" . $manifest->{$package}->{src};
my $full_path="$bld_area/$package";

The current directory needs to be set to the package directory underneath the realm directory. This directory contains all the configuration files necessary to build the package. First the build area ($bld_area) is set to the default from the configuration file. The package directory is found in the Paccom realms directory under the realm name. This is set as $pck_dir. Once determined it is made the current directory. If it fails return FALSE. Then the code checks for the presence of a file called !large in the current directory, and if it exists, changes the build area to the large build directory from the configuration. The source file should be in the !sources directory underneath the realm directory with the name fond in the manifest. The manifest is stored in a hash of hashes with the first key being the package name and the second key as "src." This is set in the variable $src. Finally the full path to the build area, including the package name, is set by appending the package name to the $bld_area which is set in $full_path.

Step 9 – Prepare the Build Area

# Make the build directory
print " Copying $src to $full_path\n" if($verbose);
if(!make_path($full_path)) {
print "Can't make build path $full_path\n";
return(0);
}

Now the build directory is created using $full_path as the directory name passed to the make_path() function. If this returns false it indicates the directory can't be created so in turn we exit with a FALSE value.

Step 10 – Extract the Package

# unzip or untar it
# It is a zip?
my $bdir;
if($manifest->{$package}->{src} =~ m/.zip$/) {
print " It is a zip file\n" if($verbose);
if(!chdir($full_path)) {
print "Can't change to $full_path to unzip\n";
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}
system("/usr/bin/unzip " . $src);
# Name is the src without the zip
($bdir=$manifest->{$package}->{src}) =~ s/.zip$//;
$bdir="$full_path/$bdir";
print " Build dir is $bdir\n" if($verbose);
}

# It is a tar?
if($manifest->{$package}->{src} =~ m/.tar./) {
print " It is a tarball\n" if($verbose);
if(!chdir($full_path)) {
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}
system("/bin/tar xf " . $src);
# Name is the src without the zip
($bdir=$manifest->{$package}->{src}) =~ s/.tar.*$//;
$bdir="$full_path/$bdir";
print " Build dir is $bdir\n" if($verbose);
}

This section checks it the package is a ZIP file or a tarball. It looks for a .zip ending or a .tar. within the name. For both it changes to the full path and if unsuccessful returns to the realms directory and returns FALSE. On any failure to change to a directory we will change back to the realms directory to be sure we are at a known location. If the directory change works without error, we then call wither unzip or tar to explode the archive in the build directory. The new directory name is derived from the package file name in the manifest using Perl's regular expressions which is, in this case, like sed. This can present a problem if the download embedded directory in the archive is different then name. Those we'll have to handle manually in a a bic.sh script. After this we have the source in an isolated directory in the proper build area, ready for building.

Step 11 – Change to Package Directory

# Switch to package directory in build area
if(!chdir($pck_dir)) {
print "Can't change to $pck_dir\n";
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}

Before we build we need to be in the package directory to find the necessary pieces, change and check for failure switching to our safe home if it fails.

Step 12 – Find or Assemble Build Script

#Start determining which script to use or construct for building
# If there is a script name bic.sh it is the full vuild script
# 1) If there is a bic.sh, run it
# 2) Otherwise call build script to assemble it

#1) A pre-created bic.sh that does the whole build process
my $script;
if( -f "bic.sh" ) {
print " Found complete script: bic.sh\n" if($verbose);
$script="$pck_dir/bic.sh"
}
else {
$script=assemble_script($realm);
open(OUT,">$bdir/paccom_go.sh") or return(0);
print OUT "$script";
close(OUT);
$script="$bdir/paccom_go.sh";
}

print "Script = $script\n";

The next step is to determine how to assemble the build script from the various available pieces. First check for the presence of a file called bic.sh which we should assume is a script that handles the whole build for that package because it is complicated, very different, or perhaps there is a lot of pre-configure work. This script should handle the whole process of configuring, building, and installing the package (testing if required). If there no such file, then the the script is return from a call to assemble_script() which is written out to paccom_go.sh in the build directory.

Step 13 – Change to the Build Area

# Change to the build area and run the script
if(!chdir($bdir)) {
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
retrun(0);
}

We need to be in the build area for the build, change and go home on failure.

Step 14 – Execute the Build Script

# Execute the script or for a dry run, print it
print " Executing $script in $bdir\n";
my $result;
if($dryrun) {
print "Would run system(bash $script) except this is a dry run\n";
print "Contents -----------------------\n";
print read_in($script);
$result=0;
}
else {
system("/bin/bash $script | tee ../../logs/$package.log"");
$result=$?;
}

Since the build area is complete the build script is called to perform the work. If the dryrun flag is on we will just print out the contents of the build script, otherwise we launch the script using a new copy of bash. It is important to start a new bash to make sure the environment is clean and our own copy is unaffected by any changes.

Notice that the system commands pipes the output of the build script to the logs directory in the build area. These logs will build up over time, but will only contain the last build of the packages. They should be cleaned periodically, a task we'll need to handle in the future.

Step 15 – Cleanup the Build Area

# Remove the build directory
chdir($pck_dir);
print " Deleting build area $full_path\n" if($verbose);
remove_tree($full_path);

printf $result!=0 ? "Failed\n" : "Success\n -------------- END $package ----------------\n";
return($?);
}

Whether or not the build worked, we clean the build area by deleting the entire directory tree. We then return the result of the last command ran--the bash for the build.

Step 16 – Assemble Script Subroutine

# Assemble a build script from the pieces found
# Add the global and realm settings first
# Then call find_script to return the given script
sub assemble_script
{
my $realm=shift;
if(!$realm) {
print "Failed! Realm directory not found for $realm\n";
exit;
}


my ($defaults,$configp,$compilep,$installp,$testp);

# Load Global defaults then realm defaults
# Local settings will be in the config file
$defaults="source $config{PACCOM_DIR}/global_settings"
if(-f "$config{PACCOM_DIR}/global_settings");
$defaults.="\nsource $config{PACCOM_REALMS}/$realm/realm_settings"
if(-f "$config{PACCOM_REALMS}/$realm/realm_settings");

$configp = find_script($realm,"config");
$compilep = find_script($realm,"compile");
$testp = find_script($realm,"test") if($config{test});
$installp = find_script($realm,"install");

# Add a newline if no test run
$testp="\n" if(!$testp);

return ("$defaults\n#Config\n$configp\n#Compile\n$compilep\n#Test\n$testp\#Install\n$installp\n");
}

The assemble_script() routine will find the various pieces of the configuration script (config, compile, test, and install) by first sanity checking a realm was passed in. Afterwards it will it will get the defaults script as the global defaults (global_settings) if it is present and then switch to the realms settings if it is present (realm_settings in the realm directory). Then the four component scripts are found using find_script(). We only include the test script if the test flag is set otherwise we put in the newline. We then return the contents of the scripts separated by newlines as the return value.

Step 17 – Find Script Subroutine

# Find a script by first looking in the local dir
# If not there look in the realm dir
# If not there use the default
sub find_script {
my $realm=shift;
my $file=shift;

my $contents;

if(-f "$file") {
print "Found a local $file script\n" if($verbose);
$contents = read_in("$file");
}
else {
if(-f "$config{PACCOM_DIR}/realms/$realm/$file") {
# Use realm
print "Using realm $config{PACCOM_DIR}/realms/$realm/$file\n" if($verbose);
$contents = read_in("$config{PACCOM_DIR}/realms/$realm/$file");
}
else {
# Use global
print "Using global $config{PACCOM_DIR}/$file\n" if($verbose);
$contents = read_in("$config{PACCOM_DIR}/$file");
}
}
return($contents);

}

This subroutine will find the deepest level file with the name of the second argument using the PACCOM_DIR., PACCOM_REALMS/$realm, and the package directory. It starts in the package directory (considered local since it is the current working directory), then the realm directory of the package, and finally the global area. Once found it reads in the contents using read_in() and returns them to the caller. As soon as it finds a valid file it stops searching, returning the deepest file it can (package, realm, global in that order.)

Step 18 – Do a Full Realm Subroutine

################ REALM ROUTINES ######################
# do_realm
# This is the main driver for the program
# 1) chdir into the realm's directory
# 2) Load the manifest which is a list of tarballs to process in the order they are processed
# 3) process each file by calling do_package
sub do_realm {
my $realm=shift;

print "Processing realm $realm\n";
# Change into realm directory
chdir("$config{PACCOM_REALMS}/$realm") or die("Can't change to realm $realm\n");

my $manifest;
$manifest=load_manifest($realm);
if(!$manifest) {
print "Can't load manifest for realm $realm\n";
return(0);
}

# BIC each package in manifest in order
foreach my $entry (keys (%$manifest)) {
bic_package($realm,$entry,$manifest);
return(0) if(!$keepgoing)
}

# Always return to realm dir
chdir($config{PACCOM_REALMS}) or die("Failed to find realm directory\n");
}

This subroutine will build all packages in a realm. It changes to the realm directory and loads the manifest using load_manfest(). Once loaded it calls bic_package() for each entry in the manifest. If the $keepgoing flag is set it will continue even if there is an error. Right now this is dummy code, but we will fully support it in the future. Once complete we return to the realms directory.

Step 19 – Load the Manifest Subroutine

# load_manifest
# Loads the manifest file with is a file with two fields
# name~tarball

sub load_manifest {
my $realm=shift;

print "Loading manifest for realm $realm\n" if($verbose);

# Created a tied hash to preserve build order from manifest
my %packages;
tie %packages, 'Tie::IxHash';

#Open the manifest file and read in
open(IN,"<","manifest") or return(0);
while(<IN>) {
chomp($_);
trim($_);
next if(($_ eq "" )|| ($_ =~ m/^#/)); # Skip lines starting with # and blanks
my ($name,$tarball)=split(/~/,$_);
print " Loaded $name with $tarball\n" if($verbose);
my $hash;
$hash->{src}=$tarball;
$packages{$name}=$hash;
}
close(IN);

return(\%packages);
}

The manifest file is stored as a hash of hashes with first key being the package name. This is a reference to a hash with keys of the field names which are currently only the name of the source file. To access the information use the following key

$manifest->{package name}->{src} = source code tarball name

Note that this is a tied hash allowing use to traverse through the packages in the same order they are in the manifest file. Since the manifest file has the packages ordered in the order they are necessary to install, this acts as the prerequisites within the realm. For multi-realm builds (future work) we will have to order the packages in some sort of sense until we fully implement dependencies. We use a hash for future expansion of the fields.

The load_manifest subroutine assumes the first argument is the name of the manifest file. It creates an IxHash tied hash, opens the file, reads each line and splits it on the tilde (~). The first field is the source name stored under src the second is the URL stored under url.

Step 20 – Get Realms Subroutine

# Reads the realms
sub get_realms
{
# Read the current directory and assume directories are realms
opendir(my $dh,".") or die("Can't open realms directory at $config{PACCOM_REALMS}\n");
my @files=grep { !/^\./ && -d $_} readdir($dh);
close($dh);

print("Realms found: @files\n") if($verbose);
return(@files);
}

This opens the current directory and reads the contents that don'r start with a . and is a directory. This is the list of available realms. It is returned as a list.

Step 21 – Trim Subroutine

############ UTILITY SUBROUTINES ###############
# trim
# Removing leading and trailing space from a string
sub trim
{
my $s;

$s =~ s/^\s+|\s+$//g;
return $s;
}

The first of the utility subroutines, this is the standard subroutine to trim the leading and trailing space from a string.

Step 22 – Read In Subroutine

# read_in
# Read the contents of a file into a single scalar variable
# using standard Perl Slurp and return it
sub read_in
{
local $/ = undef;
open my $fh, $_[0] or die "Can't open $_[0]: $!";
my $slurp = <$fh>;
return $slurp;
}

The read_in() subroutine using Perl's file slurping to read the full contents of a file and return it.

Step 23 – Load Config Subroutine

# Load config
# This loads the global configuration parameters used for building software
# CCFLAGS the flags to use for compiling
# This can be anything the compiler accepts
# but recommended: -O3 -march, etc.
sub load_config
{
my %config;

# Set default config file
my $file="/etc/paccom.conf";
# This can be changed using the -c flag
$file=$opts{c} if($opts{c});

# Open file and read in keys, values
open(IN,"<",$file) or
die("Can't open config file $file\n");
while(my $line=<IN>) {
next if($line =~ m/^#/);
chomp($line);
my($var,$value)=split(/=/,$line,2);
trim($var);
trim($value);
$value =~ s/"//g;
$config{$var}=$value;
}
close(IN);

# Resolve all vars
expand_vars(\%config);
return(%config);
}

The configuration file is read using the default of /etc/paccom.conf. If the -c is supplied it reads that file instead. Since a configuration is required if it is not available the program exits. Each line is read in, the newline removed, split using the equals sign as the field separator (=), both values are trimmed, and quotes removed. Once all operations are complete is it added to the global config hash. Since values can contain other keys the expand_vars() is used to replace all keys in values with the proper value.

Step 24 – Expand Variables Subroutine

sub expand_vars
{
my $vars=shift;

my $done=0;
# Now iterate through the hash and replace any $var with it's value
while(!$done) {
$done=1;

while (my($key, $value) = each (%$vars)) {
if($value =~ /(\$\w+)/) {
$done=0;
my $varname = substr($1,1);
$value =~ s/\$$varname/$vars->{$varname}/g;
$vars->{$key}=$value;
}
}
}
}

This routine is used to handle configuration values to contain other keys, such as PACCOM_REALMS=$PACCOM_DIR/realms. We first need to resolve PACCOM_DIR which might be defined later in the file. For expand all variables, a completions ($done) is set to false. A loop is started that continues as long as the completion flag is false. It is then set to true to end the loop if no remaining variables need to be expanded. It then traverses the config hash looking for the resolve operator ($) and if it finds it clears the completion flag and sets the value of key ($) with its value. This ensures that the outer loop will complete only on a pass where there is no longer any unresolved keys.

Download at https://drive.google.com/open?id=1LImQymYujyQENwWxRBCvUcItZy6CsV2z

To create the full program do the following as the root user

echo > /usr/bin/paccom << "EOF"
#!/usr/bin/perl

# PACCOM - source package manager (builder) for The Toucan Linux Project
# Copyright (C) 2019 Michael R Stute
# Version 0.9

use strict;
use Getopt::Std;
use File::Copy;
use File::Path qw(make_path remove_tree);
use Tie::IxHash;

# Variables
my %opts; # Options hash
my $realm; # The realm we are working in
my @realms; # The realms we will address

getopts("R:c:ivp:dr",\%opts) or die("Options are invalid\n");

print "==> Welcome to Paccom! <==\n";

# Get switches, set options
my $info=$opts{i}; # Print informative messages
my $verbose=$opts{v}; # Be verbose and print lots more info
my $package=$opts{p}; # package to build if a single package
my $dryrun=$opts{d}; # Dry run, do everything b call the build script
my $autoDL=$opts{r}; # If on and source tarball is missing, fetch it with wget
my $keepgoing=$opts{k}; # Keep going if a package fails, die if a realm does
$realm=$opts{R}; # Which realm we are in

############ MAIN ###############
#Set the realm to @all is none is specified
if (!$realm) {
if(!$package) {
print "Realm must be specified to perform work on a package\n";
exit(0);
}
else {
print "No realm specified, assuming \@all\n";
$realm="\@all";
}
}
else {
@realms[0]=$realm;
}

my %config=load_config();

if($info || $verbose) {
print "Configuation:\n";
while (my($key, $value) = each (%config)) {
print "$key = $value\n";
}
}

# Verify we have default scripts or exit
if (! (-f "$config{PACCOM_DIR}/config" &&
-f "$config{PACCOM_DIR}/compile" &&
-f "$config{PACCOM_DIR}/test" &&
-f "$config{PACCOM_DIR}/install") ) {
print "Failed to find all four global defaults\n";
print "These are required even if realm defaults are present\n"
}

# Switch to the realms dir
chdir($config{PACCOM_REALMS});
if($realm && $package) {
bic_package($realm, $package, 0);
}
else {
@realms=get_realms($realm) if($realm eq "\@all");

#Now process each realm
foreach my $realm (@realms) {
print "Doing realm $realm\n";
do_realm($realm);
}
}

############ PACKAGE SUBROUTINES ###############
# Build, install, compile - main driver routine
sub bic_package {
my $realm=shift;
my $package=shift;
my $manifest=shift;

print "------------ BEGIN $realm:$package ------------\nPreparing to build $realm:$package\n";

if(!$manifest) {
chdir($realm);
#Check to make sure package is part of the realm
print " Loading manifest\n" if($verbose);
$manifest=load_manifest($realm);
if(!$manifest) {
print ("Can't load manifest from realm $realm\n");
return(0);
}

if(!$manifest->{$package}) {
print "Package $realm:$package not found...skipping\n";
return(0);
}
}

# Determine the build area path
my $bld_area=$config{BUILDDIR};
my $pck_dir="$config{PACCOM_REALMS}/$realm/$package";
print " Changing to $pck_dir\n" if($verbose);
if(!chdir($pck_dir)) {
print "Can't change to $pck_dir\n";
return(0);
}

# Check for the large file to set build area
if ( -f "large" ) {
print " Using large build area\n" if($verbose);
$bld_area="$config{LARGE_BUILDDIR}";
}

# Determine the full path to package build area
my $src="$config{PACCOM_REALMS}/$realm/sources/" . $manifest->{$package}->{src};
my $full_path="$bld_area/$package";

# Make the build diretory
print " Copying $src to $full_path\n" if($verbose);
if(!make_path($full_path)) {
print "Can't make build path $full_path\n";
return(0);
}

# unzip or untar it
# It is a zip?
my $bdir;
if($manifest->{$package}->{src} =~ m/.zip$/) {
print " It is a zip file\n" if($verbose);
if(!chdir($full_path)) {
print "Can't change to $full_path to unzip\n";
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}

system("unzip " . $src);
# Name is the src without the zip
($bdir=$manifest->{$package}->{src}) =~ s/.zip$//;
$bdir="$full_path/$bdir";
print " Build dir is $bdir\n" if($verbose);
}

# It is a tar?
if($manifest->{$package}->{src} =~ m/.tar./) {
print " It is a tarball\n" if($verbose);
if(!chdir($full_path)) {
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}
system("tar xf " . $src);

# Name is the src without the zip
($bdir=$manifest->{$package}->{src}) =~ s/.tar.*$//;
$bdir="$full_path/$bdir";
print " Build dir is $bdir\n" if($verbose);
}

# Switch to package directory in build area
if(!chdir($pck_dir)) {
print "Can't change to $pck_dir\n";
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
return(0);
}

#Start determining which script to use or construct for building
# If there is a script name bic.sh it is the full vuild script
# 1) If there is a bic.sh, run it
# 2) Otherwise call build script to assemble it

#1) A pre-created bic.sh that does the whole build process
my $script;
if( -f "bic.sh" ) {
print " Found complete script: bic.sh\n" if($verbose);
$script="$pck_dir/bic.sh"
}
else {
$script=assemble_script($realm);
open(OUT,">$bdir/paccom_go.sh") or return(0);
print OUT "$script";
close(OUT);
$script="$bdir/paccom_go.sh";
}

print "Script = $script\n";

# Change to the build area and run the script
if(!chdir($bdir)) {
chdir($config{PACCOM_REALMS});
remove_tree($full_path);
retrun(0);
}

# Execute the script or for a dry run, print it
print " Executing $script in $bdir\n";
my $result;
if($dryrun) {
print "Would run system(bash $script) except this is a dry run\n";
print "Contents -----------------------\n";
print read_in($script);
$result=0;
}
else {
system("bash $script | tee ../../logs/$package.log"");
$result=$?;
}

# Remove the build directory
chdir($pck_dir);
print " Deleting build area $full_path\n" if($verbose);
remove_tree($full_path);

printf $result!=0 ? "Failed\n" : "Success\n -------------- END $package ----------------\n";
return($?);
}

# Assemble a build script from the pieces found
# Add the global and realm settings first
# Then call find_script to return the given script
sub assemble_script
{
my $realm=shift;
if(!$realm) {
print "Failed! Realm directory not found for $realm\n";
exit;
}


my ($defaults,$configp,$compilep,$installp,$testp);

# Load Global defaults then realm defaults
# Local settings will be in the config file
$defaults="source $config{PACCOM_DIR}/global_settings"
if(-f "$config{PACCOM_DIR}/global_settings");
$defaults.="\nsource $config{PACCOM_REALMS}/$realm/realm_settings"
if(-f "$config{PACCOM_REALMS}/$realm/realm_settings");

$configp = find_script($realm,"config");
$compilep = find_script($realm,"compile");
$testp = find_script($realm,"test") if($config{test});
$installp = find_script($realm,"install");

# Add a newline if no test run
$testp="\n" if(!$testp);

return ("$defaults\n#Config\n$configp\n#Compile\n$compilep\n#Test\n$testp\#Install\n$installp\n");
}

# Find a script by first looking in the local dir
# If not there look in the realm dir
# If not there use the default
sub find_script {
my $realm=shift;
my $file=shift;

my $contents;

if(-f "$file") {
print "Found a local $file script\n" if($verbose);
$contents = read_in("$file");
}
else {
if(-f "$config{PACCOM_REALMS}/$realm/$file") {
# Use realm
print "Using realm $config{PACCOM_REALMS}/$realm/$file\n" if($verbose);
$contents = read_in("$config{PACCOM_REALMS}/$realm/$file");
}
else {
# Use global
print "Using global $config{PACCOM_DIR}/$file\n" if($verbose);
$contents = read_in("$config{PACCOM_DIR}/$file");
}
}
return($contents);

}

################ REALM ROUTINES ######################
# do_realm
# This is the main driver for the progra
# 1) chdir into the realm's directory
# 2) Load the manifest which is a list of tarballs to process in the order they are processed
# 3) process each file by calling do_package
sub do_realm {
my $realm=shift;

print "Processing realm $realm\n";
# Change into realm directory
chdir("$config{PACCOM_REALMS}/$realm") or die("Can't change to realm $realm\n");

my $manifest;
$manifest=load_manifest($realm);
if(!$manifest) {
print "Can't load manifest for realm $realm\n";
return(0);
}

# BIC each package in manifest in order
foreach my $entry (keys (%$manifest)) {
bic_package($realm,$entry,$manifest);
return(0) if(!$keepgoing)
}

# Always return to realm dir
chdir($config{PACCOM_REALMS}) or die("Failed to find realm directory\n");
}

# load_manifest
# Loads the manifest file with is a file with two fields
# name~tarball

sub load_manifest {
my $realm=shift;

print "Loading manifest for realm $realm\n" if($verbose);

# Created a tied hash to preserve build order from manifest
my %packages;
tie %packages, 'Tie::IxHash';

#Open the manifest file and read in
open(IN,"<","manifest") or return(0);
while(<IN>) {
chomp($_);
trim($_);
next if(($_ eq "" )|| ($_ =~ m/^#/)); # Skip lines starting with # and blanks
my ($name,$tarball)=split(/~/,$_);
print " Loaded $name with $tarball\n" if($verbose);
my $hash;
$hash->{src}=$tarball;
$packages{$name}=$hash;
}
close(IN);

return(\%packages);
}

# Reads the realms
sub get_realms
{
# Read thhe current directory and assume directories are realms
opendir(my $dh,".") or die("Can't open realms directory at $config{PACCOM_REALMS}\n");
my @files=grep { !/^\./ && -d $_} readdir($dh);
close($dh);

print("Realms found: @files\n") if($verbose);
return(@files);
}

############ UTILITY SUBROUTINES ###############
# trim
# Removing leading and trailing space from a string
sub trim
{
my $s;

$s =~ s/^\s+|\s+$//g;
return $s;
}

# read_in
# Read the contents of a file into a single scalar variable
# using standard Perl Slurp and return it
sub read_in
{
local $/ = undef;
open my $fh, $_[0] or die "Can't open $_[0]: $!";
my $slurp = <$fh>;
return $slurp;
}

# Load config
# This loads the global configuration parameters used for building software
# CCFLAGS the flags to use for compiling
# This can be anything the compiler accepts
# but recommended: -O3 -march, etc.
sub load_config
{
my %config;

# Set default config file
my $file="/etc/paccom.conf";
# This can be changed using the -c flag
$file=$opts{c} if($opts{c});

# Open file and read in keys, values
open(IN,"<",$file) or
die("Can't open config file $file\n");
while(my $line=<IN>) {
next if($line =~ m/^#/);
chomp($line);
my($var,$value)=split(/=/,$line,2);
trim($var);
trim($value);
$value =~ s/"//g;
$config{$var}=$value;
}
close(IN);

# Resolve all vars
expand_vars(\%config);
return(%config);
}

# Find all $ operators in values of config and replace with proper value
sub expand_vars
{
my $vars=shift;

my $done=0;
# Now iterate through the hash and replace any $var with it's value
while(!$done) {
$done=1;
while (my($key, $value) = each (%$vars)) {
if($value =~ /(\$\w+)/) {
$done=0;
my $varname = substr($1,1);
$value =~ s/\$$varname/$vars->{$varname}/g;
$vars->{$key}=$value;
}
}
}
}
EOF

Step 25 - Test Paccom

We can perform a test by doing a dry run with Angbad. But first we need to prepare the system and configure Angband.

Add a games group with your userid added to the group

echo "games:x:60:userid" >> /etc/groups

but be sure to replace userid with your own username

Create the config file

cat > /usr/src/paccom/realms/games/angband/config << EOF
./autogen.sh
./configure --disable-x11 --bindir=/usr/local/games --enable-curses --with-setgid=games
EOF

Make the install which needs to change some permissions for various directories and the executable

echo > /usr/src/paccom/realms/games/angband/config << "EOF"
make install

# Setup for setgid games for shared files
chgrp games /usr/local/games/angband
chmod g+s /usr/local/games/angband
chgrp games /usr/local/var/games/angband
chmod 775 /usr/local/var/games/angband
chgrp -R games /usr/local/var/games/angband/*
chmod 775 /usr/local/var/games/angband/*
EOF

Now try a dry run with

# ./paccom.pl -dv -R games -p angband
==> Welcome to Paccom! <==
Configuation:
PACCOM_BUILD = /var/paccom
CFLAGS = -O2 -march=native -pipe
CXXFLAGS = -O2 -march=native -pipe
BUILDDIR = /var/paccom/build
PACCOM_REALMS = /usr/src/paccom/realms
LARGE_BUILDDIR = /var/paccom/large
PACCOM_DIR = /usr/src/paccom
MAKEOPTS = -j8
------------ BEGIN games:angband ------------
Preparing to build games:angband
Loading manifest
Loading manifest for realm games
Loaded angband with angband-master.zip from URL
Changing to /usr/src/paccom/realms/games/angband
Copying /usr/src/paccom/realms/games/sources/angband-master.zip to /var/paccom/build/angband
It is a zip file
Archive: /usr/src/paccom/realms/games/sources/angband-master.zip
2a840212b67288426a3032f3daa0fd0b25dc83e3
creating: angband-master/
inflating: angband-master/.gitignore
inflating: angband-master/.travis.yml
inflating: angband-master/Makefile
...
extracting: angband-master/tests/trivial/matcher/matcher
creating: angband-master/utils/
inflating: angband-master/utils/codestats
Build dir is /var/paccom/build/angband/angband-master
Found a local config script
Using global /usr/src/paccom/compile
Found a local install script
Script = /var/paccom/build/angband/angband-master/paccom_go.sh
Executing /var/paccom/build/angband/angband-master/paccom_go.sh in /var/paccom/build/angband/angband-master
Would run system(bash /var/paccom/build/angband/angband-master/paccom_go.sh) except this is a dry run
Contents -----------------------
source /usr/src/paccom/global_settings
#Config
./autogen.sh
./configure --disable-x11 --bindir=/usr/local/games --enable-curses --with-setgid=games

#Compile
make $MAKEOPTS

#Test

#Install
make install

# Setup for setgid games for shared files
chgrp games /usr/local/games/angband
chmod g+s /usr/local/games/angband
chgrp games /usr/local/var/games/angband
chmod 775 /usr/local/var/games/angband
chgrp -R games /usr/local/var/games/angband/*
chmod 775 /usr/local/var/games/angband/*

Deleting build area /var/paccom/build/angband
Success
-------------- END angband ----------------

The output should appear as above except I removed a large section of files from the output of the unzip command to shorten it. It shows the most important part at the bottom where it prints the contents of the build script below the line starting with Contents ---. This show it is using the config file within the package directory, the compile file from the global directory, and the install file from the global directory. Note, there is no test as we haven't enabled it. If the output is different check everything carefully from the last chapter and this one until you find the problem.

Step 26 - Install Angband

If everything is good there no reason not to install the package using paccom.

Issue the following

# ./paccom.pl -v -R games -p angband

which is the same as before except the dryrun option (-d) has been removed. This time the build create will be called and there will be a log file in /var/paccom/build which will contain the output of the build script for Angband. You can check it for errors if you want.

To check if everything went according to plan simply try

angband -mgcu

and Angband should run. To find out all about it see http://rephial.org.

That's it for this time. Next week we'll start working on the base realm to make sure we can rebuild the system. Most of it will be configuration work, but when we're done we can perform automated builds of the two realms we have: base and games. Angband might keep you occupied while the wait for the long build process.

Toucan Linux Project

Tuesday, October 15, 2019

14 - Coding Paccom First Version