Table of Contents
When you have multiple machines that should run the same packages, it is wasted time if they all build their packages themselves from source. There is a ways of getting a set of binary packages: The bulk build system, or pbulk ("p" stands for "parallel). This chapter describes how to set it up so that the packages are most likely to be usable later.
Since a bulk build takes several days or even weeks to finish, you should think about the setup before you start everything. Pay attention to at least the following points:
If you want to upload the binary packages to ftp.NetBSD.org, make sure the setup complies to the requirements for binary packages:
To end up on ftp.NetBSD.org, the packages must be built by a NetBSD developer on a trusted machine (that is, where you and only you have root access).
Packages on ftp.NetBSD.org should only be created from the stable branches (like 2009Q1), so that users browsing the available collections can see at a glance how old the packages are.
The packages must be built as root, since some packages require set-uid binaries at runtime, and creating those packages as unprivileged user doesn't work well at the moment.
Make sure that the bulk build cannot break anything in
your system. Most bulk builds run as root, so they should be run at least
in a chroot environment or something even more restrictive, depending on
what the operating system provides. There have been numerous cases where
certain packages tried to install files outside the
LOCALBASE or wanted to edit some files in
/etc. Furthermore, the bulk builds install and
deinstall packages in
/usr/pkg (or whatever
LOCALBASE is) during their operation, so be sure
that you don't need any package during the build.
A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.
40 GB for the distfiles (read-write, remote, temporary)
30 GB for the binary packages (read-write, remote, permanent)
1 GB for the pkgsrc tree (read-only, remote, permanent)
5 GB for
LOCALBASE (read-write, local, temporary)
10 GB for the log files (read-write, remote, permanent)
5 GB for temporary files (read-write, local, temporary)
Running a pbulk-style bulk build works roughly as follows:
First, build the pbulk infrastructure in a fresh pkgsrc location.
Then, build each of the packages from a clean installation directory using the infrastructure.
To simplify configuration we provide helper script
In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create user with name "pbulk".
Fetch and extract pkgsrc. Use a command like one of these:
(cd /usr && ftp -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
(cd /usr && fetch -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
(cd /usr && cvs -Q -z3 -d firstname.lastname@example.org:/cvsroot get -P pkgsrc)
Or any other way that fits (e.g., curl, wget).
Deploy and configure pbulk tools, e.g.:
sh pbulk.sh -n # native (NetBSD)
sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment
sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build
mk.conf.frag is a fragment of
mk.conf that contains settings you want to
apply to packages you build. For instance,
PKG_DEVELOPER= yes # perform more checks X11_TYPE= modular # use pkgsrc X11 SKIP_LICENSE_CHECK= yes # accept all licences (useful when building all packages)
If configured for limited list, replace the list in
with your list of packages one per line without empty lines or comments. E.g.:
www/firefox mail/thunderbird misc/libreoffice4
At this point you can also review configuration in
and make final amendments, if wanted.
After it finishes, you'll have
/mnt filled with distribution files, binary packages, and reports,
plain text summary in
pbulk.sh script does not cover all possible use cases.
While being ready to run, it serves as a good starting point to understand and build more complex setups.
The script is kept small enough for better understanding.
pbulk.sh script supports running
unprivileged bulk build and helps configuring distributed bulk builds.
After your pkgsrc bulk-build has completed, you may wish to
create a CD-ROM set of the resulting binary packages to assist
in installing packages on other machines. The
pkgtools/cdpack package provides
a simple tool for creating the ISO 9660 images.
cdpack arranges the packages on the CD-ROMs in a
way that keeps all the dependencies for a given package on the same
CD as that package.
Complete documentation for cdpack is found in the cdpack(1)
man page. The following short example assumes that the binary
packages are left in
/usr/pkgsrc/packages/All and that
sufficient disk space exists in
hold the ISO 9660 images.
cdpack /usr/pkgsrc/packages/All /u2/images
If you wish to include a common set of files
etc.) on each CD in the collection, then you need to create a
directory which contains these files. e.g.
echo "This is a README" > /tmp/common/README
echo "Another file" > /tmp/common/COPYING
echo "#!/bin/sh" > /tmp/common/bin/myscript
echo "echo Hello world" >> /tmp/common/bin/myscript
chmod 755 /tmp/common/bin/myscript
Now create the images:
cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images
Each image will contain
in their root directories.