Outline
- pkgsrc in 3 minutes
- The old
pkg_install
- Package metadata
- Package APIs
- Integration and conversion
- Outlook
pkgsrc in 3 minutes
- Framework for building third party software
- Cross platform
- System of choice on NetBSD and DragonFly
- Around 6300 packages
- Spun off from FreeBSD ports in 1997
pkgsrc: high-level structure
- Build descriptions (Makefiles)
- Common build rules
- buildlink3/wrapper framework
pkg_install
What is pkg_install?
- pkg_add
- pkg_delete
- pkg_create
- pkg_info
- pkg_admin
- ...
The old pkg_install
- Inherited from FreeBSD
- Extended over the time to add some feature
- Grown code
pkg_install — illustrated
(from Wikipedia)
Problems en detail
- Use of external programs during extraction and removal
- Code full of side effects
- Temporary copy during extraction
- Missing, outdated or incomplete specs
- Redundancy of scripts
- Interaction with high-level tools
Pkgsrc: open issues
- Advanced update facilities
- Coherent behaviour between builds from source and binary packages
The answer
- tech-pkg@NetBSD.org, April 2006: Google SoC
Evaluation of the Status Quo -
Paris, May 2006: pkgsrccon
Detailed requirements for a new pkg_install
Desired structure
(from Wikipedia)
Package patterns (I)
Before
- Plain package names: exact match
- Dewey patterns:
gdm>=2.14<2.14.8
- Fnmatch patterns:
pear-5.0.[0-9]*
- Csh-style alternatives:
sun-{jre,jdk}<1.3.1.0.2
Problems with old patterns
- Which match is better?
- How ambigious is a rule?
- What does
gdm<2.14>2.14.8
mean?
Package patterns (II)
After
- Dewey-like atoms:
- package base name
- operator, version pairs
- relation operator like >=, >, ==, ...
- prefix match operator ~
- Alternatives using |
- Explicit ordering rules
Use of patterns
- Dependencies
- Conflicts
- Vulnerabilities
- Compatibility
Compatibility patterns
- APIs are forward compatible until broken: open dependencies
- ABIs are backward compatible unless broken: closed dependencies
- Problem: point of breakage not known in advance
- Solution: packages specify what they are compatible to
- Back-port planned
Package lists
- No more @exec/@unexec
- Automatic tracking of shared and intermediate directories
- Added first-class support for configuration files
- ...and configuration directories
- ...as well as files and directories under package control
Plists as mini IDS
- Checksums are stored in the Plist
- ... and so are ownership
- ... and permissions
- Tagging of entries for type-specific (de)install hooks
Regeneration and verification of installed packages
Package metadata
- Name and version of the package
- Prefix of the package
- Who maintains the package
- Target OS version and architecture
- Redistribution license
- Descriptions, possible localized
Serialized descriptions
--- %YAML:1.0 !pkgsrc.org,20060813/pkg
pkgname: mad123-0.8.5nb2
prefix: /usr/pkg
platform:
os: NetBSD
os_version: 3.99.22
architecture: i386
abi: 32
depends:
- curl>=7.15.3
- libmad>=0.15.1bnb1
plist:
/usr/pkg/bin/mad123: !pkgsrc.org,20060813/plist/file
checksum:
RMD160: 4b8fd3e645386f1d674603a349613075d2d2377c
modification_time: 2006-11-07 08:16:51
size: 20776
permission: 0555
/usr/pkg/man/man1/mad123.1: !pkgsrc.org,20060813/plist/file
checksum:
RMD160: 6b6a1800c664a7940975ee634b050e2ecc5397c4
modification_time: 2006-11-07 08:16:51
size: 5518
permission: 0444
meta:
descriptions:
en_US: |
mad123 is a simple command line MP3 player using Robert Leslie's
libmad fixed-point MP3 decoder library.
...
comments:
en_US: Command-line MPEG audio player
license: GPL
maintainer: pkgsrc-users@NetBSD.org
build_time: 2006-11-08 14:19:12
Binary package format
- Compressed tarballs
- Strictly ordered
- Top-level index file
- Signature
- package name / subpath for each file in the package
Architecture of pkg_install
- Pattern API: matching, ordering, extraction of package names
- Package description API: accessors for all metadata, verification
- Plist API: new plist entries, iteration, verfication
- Database API: querying, regeneration, modification groups
Integration I: staged installation
- Direct installation to /usr/pkg considered evil:
- Hard to trace implicit changes
- No protection against side-effects
- Correctness of binary packages
- Instead: installation to temporary directory
- Optionally: full package building as normal user
- Utilizes WRKDIR reference check
- Conversion in progress, 50% mark is aim for pkgsrc-2006Q4
Integration II: pattern conversion
- Most nasty wildcard patterns already fixed before pksgrc-2006Q2
- Dewey patterns stay valid
- Csh-style alternatives are expanded and converted independently
- Rest converted with heuristics: 30 exceptions in pkgsrc-2006Q2
Integration III: new pkg flavour
- Callers of pkg_install mostly isolated in mk/flavour/pkg
- Conversion of mk/pkginstall to use tagging and new metadata
- Work-in-progress, as the command interfaces are not finalized
Integration IV: Old packages
- Basic metadata conversion works
- ...and is used for testing the tools
- Problem: conversion of old install scripts
- mk/pkginstall fragments can be reasonably detected and handled
- other fragments are difficult, no clear delimiter
Summary
- The new pkg_install:
- Self-contained tool chain
- Modular architecture
- In-place operation
- Work-in-progress
- Visible changes in pkgsrc:
- Fixed many bogus dependencies
- Staged installation
- On-going efforts to annote special permissions
Q&A
Questions?