Chapter 10. Creating a new pkgsrc package from scratch

Table of Contents

10.1. Common types of packages
10.1.1. Perl modules
10.1.2. KDE3 applications
10.1.3. Python modules and programs
10.2. Examples
10.2.1. How the www/nvu package came into pkgsrc

When you find a package that is not yet in pkgsrc, you most likely have a URL from where you can download the source code. Starting with this URL, creating a package involves only a few steps.

  1. First, install the packages pkgtools/url2pkg and pkgtools/pkglint.

  2. Then, choose one of the top-level directories as the category in which you want to place your package. You can also create a directory of your own (maybe called local). In that category directory, create another directory for your package and change into it.

  3. Run the program url2pkg, which will ask you for a URL. Enter the URL of the distribution file (in most cases a .tar.gz file) and watch how the basic ingredients of your package are created automatically. The distribution file is extracted automatically to fill in some details in the Makefile that would otherwise have to be done manually.

  4. Examine the extracted files to determine the dependencies of your package. Ideally, this is mentioned in some README file, but things may differ. For each of these dependencies, look where it exists in pkgsrc, and if there is a file called buildlink3.mk in that directory, add a line to your package Makefile which includes that file just before the last line. If the buildlink3.mk file does not exist, it must be created first. The buildlink3.mk file makes sure that the package's include files and libraries are provided.

    If you just need binaries from a package, add a DEPENDS line to the Makefile, which specifies the version of the dependency and where it can be found in pkgsrc. This line should be placed in the third paragraph. If the dependency is only needed for building the package, but not when using it, use BUILD_DEPENDS instead of DEPENDS. Your package may then look like this:

    [...]
    
    BUILD_DEPENDS+= libxslt-[0-9]*:../../textproc/libxslt
    DEPENDS+=       screen-[0-9]*:../../misc/screen
    DEPENDS+=       screen>=4.0:../../misc/screen
    
    [...]
    
    .include "../../category/package/buildlink3.mk"
    .include "../../devel/glib2/buildlink3.mk"
    .include "../../mk/bsd.pkg.mk"
    
  5. Run pkglint to see what things still need to be done to make your package a good one. If you don't know what pkglint's warnings want to tell you, try pkglint --explain or pkglint -e, which outputs additional explanations.

  6. In many cases the package is not yet ready to build. You can find instructions for the most common cases in the next section, Section 10.1, “Common types of packages”. After you have followed the instructions over there, you can hopefully continue here.

  7. Run bmake clean to clean the working directory from the extracted files. Besides these files, a lot of cache files and other system information has been saved in the working directory, which may become wrong after you edited the Makefile.

  8. Now, run bmake to build the package. For the various things that can go wrong in this phase, consult Chapter 19, Making your package work.

  9. When the package builds fine, the next step is to install the package. Run bmake install and hope that everything works.

  10. Up to now, the file PLIST, which contains a list of the files that are installed by the package, is nearly empty. Run bmake print-PLIST >PLIST to generate a probably correct list. Check the file using your preferred text editor to see if the list of files looks plausible.

  11. Run pkglint again to see if the generated PLIST contains garbage or not.

  12. When you ran bmake install, the package has been registered in the database of installed files, but with an empty list of files. To fix this, run bmake deinstall and bmake install again. Now the package is registered with the list of files from PLIST.

  13. Run bmake package to create a binary package from the set of installed files.

10.1. Common types of packages

10.1.1. Perl modules

Simple Perl modules are handled automatically by url2pkg, including dependencies.

10.1.2. KDE3 applications

KDE3 applications should always include meta-pkgs/kde3/kde3.mk, which contains numerous settings that are typical of KDE3 packages.

10.1.3. Python modules and programs

Python modules and programs packages are easily created using a set of predefined variables.

If some Python versions are not supported by the software, set the PYTHON_VERSIONS_INCOMPATIBLE variable to the Python versions that are not supported, e.g.

PYTHON_VERSIONS_INCOMPATIBLE=       26

If the packaged software is a Python module, include ../../lang/python/extension.mk. In this case, the package directory should be called py-software and PKGNAME should be set to ${PYPKGPREFIX}-${DISTNAME}, e.g.

DISTNAME=   foopymodule-1.2.10
PKGNAME=    ${PYPKGPREFIX}-${DISTNAME}

If it is an application, include ../../lang/python/application.mk. In order to correctly set the path to the Python interpreter, use the REPLACE_PYTHON variable and set it to the list of files (paths relative to WRKSRC) that must be corrected. For example:

REPLACE_PYTHON=   *.py

Most Python packages use either distutils or easy-setup (eggs). If the software uses distutils, include ../../lang/python/distutils.mk. so pkgsrc will use this framework. distutils uses a script called setup.py, if the distutils driver is not called setup.py, set the PYSETUP variable to the name of the script.

Otherwise, if the packaged software is egg-aware, you only need to include ../../lang/python/egg.mk.

Some Python modules have separate distributions for Python-2.x and Python-3.x support. In pkgsrc this is handled by the versioned_dependencies.mk file. Set PYTHON_VERSIONED_DEPENDENCIES to the list of packages that should be depended upon and include ../../lang/python/versioned_dependencies.mk, then the pkgsrc infrastructure will depend on the appropriate package version. For example:

PYTHON_VERSIONED_DEPENDENCIES=dateutil dns

Look inside versioned_dependencies.mk for a list of supported packages.

10.2. Examples

10.2.1. How the www/nvu package came into pkgsrc

10.2.1.1. The initial package

Looking at the file pkgsrc/doc/TODO, I saw that the nvu package has not yet been imported into pkgsrc. As the description says it has to do with the web, the obvious choice for the category is www.

$ mkdir www/nvu
$ cd www/nvu

The web site says that the sources are available as a tar file, so I fed that URL to the url2pkg program:

$ url2pkg http://cvs.nvu.com/download/nvu-1.0-sources.tar.bz2

My editor popped up, and I added a PKGNAME line below the DISTNAME line, as the package name should not have the word sources in it. I also filled in the MAINTAINER, HOMEPAGE and COMMENT fields. Then the package Makefile looked like that:

# $NetBSD: creating.html,v 1.70 2014/07/16 22:30:47 asau Exp $
#

DISTNAME=       nvu-1.0-sources
PKGNAME=        nvu-1.0
CATEGORIES=     www
MASTER_SITES=   http://cvs.nvu.com/download/
EXTRACT_SUFX=   .tar.bz2

MAINTAINER=     rillig@NetBSD.org
HOMEPAGE=       http://cvs.nvu.com/
COMMENT=        Web Authoring System

# url2pkg-marker (please do not remove this line.)
.include "../../mk/bsd.pkg.mk"

Then, I quit the editor and watched pkgsrc downloading a large source archive:

url2pkg> Running "make makesum" ...
=> Required installed package digest>=20010302: digest-20060826 found
=> Fetching nvu-1.0-sources.tar.bz2
Requesting http://cvs.nvu.com/download/nvu-1.0-sources.tar.bz2
100% |*************************************| 28992 KB  150.77 KB/s00:00 ETA
29687976 bytes retrieved in 03:12 (150.77 KB/s)
url2pkg> Running "make extract" ...
=> Required installed package digest>=20010302: digest-20060826 found
=> Checksum SHA1 OK for nvu-1.0-sources.tar.bz2
=> Checksum RMD160 OK for nvu-1.0-sources.tar.bz2
work.bacc -> /tmp/roland/pkgsrc/www/nvu/work.bacc
===> Installing dependencies for nvu-1.0
===> Overriding tools for nvu-1.0
===> Extracting for nvu-1.0
url2pkg> Adjusting the Makefile.

Remember to correct CATEGORIES, HOMEPAGE, COMMENT, and DESCR when you're done!

Good luck! (See pkgsrc/doc/pkgsrc.txt for some more help :-)

10.2.1.2. Fixing all kinds of problems to make the package work

Now that the package has been extracted, let's see what's inside it. The package has a README.txt, but that only says something about mozilla, so it's probably useless for seeing what dependencies this package has. But since there is a GNU configure script in the package, let's hope that it will complain about everything it needs.

$ bmake
=> Required installed package digest>=20010302: digest-20060826 found
=> Checksum SHA1 OK for nvu-1.0-sources.tar.bz2
=> Checksum RMD160 OK for nvu-1.0-sources.tar.bz2
===> Patching for nvu-1.0
===> Creating toolchain wrappers for nvu-1.0
===> Configuring for nvu-1.0
[...]
configure: error: Perl 5.004 or higher is required.
[...]
WARNING: Please add USE_TOOLS+=perl to the package Makefile.
[...]

That worked quite well. So I opened the package Makefile in my editor, and since it already has a USE_TOOLS line, I just appended perl to it. Since the dependencies of the package have changed now, and since a perl wrapper is automatically installed in the tools phase, I need to build the package from scratch.

$ bmake clean
===> Cleaning for nvu-1.0
$ bmake
[...]
*** /tmp/roland/pkgsrc/www/nvu/work.bacc/.tools/bin/make is not \
GNU Make.  You will not be able to build Mozilla without GNU Make.
[...]

So I added gmake to the USE_TOOLS line and tried again (from scratch).

[...]
checking for GTK - version >= 1.2.0... no
*** Could not run GTK test program, checking why...
[...]

Now to the other dependencies. The first question is: Where is the GTK package hidden in pkgsrc?

$ echo ../../*/gtk*
[many packages ...]
$ echo ../../*/gtk
../../x11/gtk
$ echo ../../*/gtk2
../../x11/gtk2
$ echo ../../*/gtk2/bui*
../../x11/gtk2/buildlink3.mk

The first try was definitely too broad. The second one had exactly one result, which is very good. But there is one pitfall with GNOME packages. Before GNOME 2 had been released, there were already many GNOME 1 packages in pkgsrc. To be able to continue to use these packages, the GNOME 2 packages were imported as separate packages, and their names usually have a 2 appended. So I checked whether this was the case here, and indeed it was.

Since the GTK2 package has a buildlink3.mk file, adding the dependency is very easy. I just inserted an .include line before the last line of the package Makefile, so that it now looks like this:

[...]
.include "../../x11/gtk2/buildlink3.mk"
.include "../../mk/bsd.pkg.mk

After another bmake clean && bmake, the answer was:

[...]
checking for gtk-config... /home/roland/pkg/bin/gtk-config
checking for GTK - version >= 1.2.0... no
*** Could not run GTK test program, checking why...
*** The test program failed to compile or link. See the file config.log for the
*** exact error that occured. This usually means GTK was incorrectly installed
*** or that you have moved GTK since it was installed. In the latter case, you
*** may want to edit the gtk-config script: /home/roland/pkg/bin/gtk-config
configure: error: Test for GTK failed.
[...]

In this particular case, the assumption that every package prefers GNOME 2 had been wrong. The first of the lines above told me that this package really wanted to have the GNOME 1 version of GTK. If the package had looked for GTK2, it would have looked for pkg-config instead of gtk-config. So I changed the x11/gtk2 to x11/gtk in the package Makefile, and tried again.

[...]
cc -o xpidl.o -c -DOSTYPE=\"NetBSD3\" -DOSARCH=\"NetBSD\"  [...]
In file included from xpidl.c:42:
xpidl.h:53:24: libIDL/IDL.h: No such file or directory
In file included from xpidl.c:42:
xpidl.h:132: error: parse error before "IDL_ns"
[...]

The package still does not find all of its dependencies. Now the question is: Which package provides the libIDL/IDL.h header file?

$ echo ../../*/*idl*
../../devel/py-idle ../../wip/idled ../../x11/acidlaunch
$ echo ../../*/*IDL*
../../net/libIDL

Let's take the one from the second try. So I included the ../../net/libIDL/buildlink3.mk file and tried again. But the error didn't change. After digging through some of the code, I concluded that the build process of the package was broken and couldn't have ever worked, but since the Mozilla source tree is quite large, I didn't want to fix it. So I added the following to the package Makefile and tried again:

CPPFLAGS+=              -I${BUILDLINK_PREFIX.libIDL}/include/libIDL-2.0
BUILDLINK_TRANSFORM+=   -l:IDL:IDL-2

The latter line is needed because the package expects the library libIDL.so, but only libIDL-2.so is available. So I told the compiler wrapper to rewrite that on the fly.

The next problem was related to a recent change of the FreeType interface. I looked up in www/seamonkey which patch files were relevant for this issue and copied them to the patches directory. Then I retried, fixed the patches so that they applied cleanly and retried again. This time, everything worked.

10.2.1.3. Installing the package

$ bmake CHECK_FILES=no install
[...]
$ bmake print-PLIST >PLIST
$ bmake deinstall
$ bmake install