# Split debug symbols for pkgsrc builds, final term GSoC 2016 notes In this document I will try to recap everything done during the GSoC 2016 project and all the possible TODOs left out. It is not intended to replace digging inside the source code and commit messages but is intended to help mentors, curiouses and myself to get a brief overview on what was done and what is needed to be done in order to import that in the official pkgsrc tree. ## First mid-term During the community bonding period and first week of first mid-term I have researched how stripping debugging symbols work and gave a look to other related works. Some of the information that I have collected were shared in a blog post appeared in The NetBSD Blog: Functionality for stripping off debug symbols in `.debug' files was implemented in `mk/bsd.debugdata.mk'. Apart that also sanity checks were implemented in `mk/checks/check-debugdata.mk' in order to check that all binaries and libraries had the symbols stripped off and that the corresponding `.debug' files exist. The `.debug' files are dynamically added to the package PLIST and installed as part of the same binary package. In order to test these functionality let's install `local/green-bottles' package prior without and with setting PKG_DEBUGDATA (that turn on the strip of debug data off from binaries and libraries). Let's see what happens without setting PKG_DEBUGDATA: $ cd pkgsrc/local/green-bottles/ $ make install ===> Installing dependencies for green-bottles-0 => Tool dependency checkperms>=1.1: found checkperms-1.11nb1 ===> Overriding tools for green-bottles-0 ===> Extracting for green-bottles-0 [...] ===> Installing binary package of green-bottles-0 => Dropping ``root'' privileges. ...if we try to debug it the binary is stripped by default and so we could not easily use a symbolic debugger for it: $ pkg_info -L green-bottles Information for green-bottles-0: Files: /usr/pkg/bin/green-bottles $ file `which green-bottles` /usr/pkg/bin/green-bottles: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 7.99.30, stripped $ gdb `which green-bottles` GNU gdb (GDB) 7.10.1 Copyright (C) 2015 Free Software Foundation, Inc. [...] For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/pkg/bin/green-bottles...(no debugging symbols found)...done. If we set PKG_DEBUGDATA...: $ cd pkgsrc/local/green-bottles/ $ make clean deinstall [...] $ make install PKG_DEBUGDATA=yes ===> Installing dependencies for green-bottles-0 => Tool dependency checkperms>=1.1: found checkperms-1.11nb1 ===> Overriding tools for green-bottles-0 ===> Extracting for green-bottles-0 [...] => Stripping debug symbols => Installing debug data [...] => Checking for missing debug data in green-bottles-0 [...] ===> Installing binary package of green-bottles-0 [...] $ pkg_info -L green-bottles Information for green-bottles-0: Files: /usr/pkg/bin/green-bottles.debug /usr/pkg/bin/green-bottles $ gdb `which green-bottles` GNU gdb (GDB) 7.10.1 Copyright (C) 2015 Free Software Foundation, Inc. [...] Reading symbols from /usr/pkg/bin/green-bottles...Reading symbols from /usr/pkg/bin/green-bottles.debug...done. done. (gdb) b main Breakpoint 1 at 0xad0: file green-bottles.c, line 29. (gdb) quit We can observe that now to all binaries and libraries a corresponding `.debug' file that only contain debug data is installed. ## Final term During the final term some parts implemented in the first mid-term were polished and fixed. Initial functionality for SUBPACKAGES support was implemented but compared to what it was done in the first mid-term this part is far from being complete. Unlike debugdata functionality the changes for SUBPACKAGES support are scattered over the pkgsrc infrastructure, files that were adjusted: - mk/bsd.pkg.mk - mk/check/check-files.mk - mk/check/check-interpreter.mk - mk/check/check-shlibs.mk - mk/check/check-stripped.mk - mk/check/check-debugdata.mk - mk/install/install.mk - mk/pkgformat/pkg/metadata.mk - mk/pkgformat/pkg/package.mk - mk/pkgformat/pkg/pkgformat.mk - mk/plist/plist.mk To test the SUBPACKAGES support a simple package was written and it is available as `local/frobnitzem'. The package contains three SUBPACKAGES respectively called frobnitzem-{foo,bar,baz} that respectively install just a script of the same name. $ cd pkgsrc/local/frobnitzem/ $ make install ===> Installing dependencies for frobnitzem-0 => Tool dependency checkperms>=1.1: found checkperms-1.11nb1 ===> Overriding tools for frobnitzem-0 ===> Extracting for frobnitzem-0 [...] => Creating binary package /tmp/pkgsrc/local/frobnitzem/work/.packages/frobnitzem-foo-0.tgz => Creating binary package /tmp/pkgsrc/local/frobnitzem/work/.packages/frobnitzem-bar-0.tgz => Creating binary package /tmp/pkgsrc/local/frobnitzem/work/.packages/frobnitzem-baz-0.tgz => Becoming ``root'' to make su-real-package-install (/usr/pkg/bin/sudo) ===> Installing binary package of frobnitzem-foo-0 ===> Installing binary package of frobnitzem-bar-0 ===> Installing binary package of frobnitzem-baz-0 => Dropping ``root'' privileges. $ frobnitzem-bar && frobnitzem-baz && frobnitzem-foo bar baz foo $ pkg_info -L frobnitzem-bar frobnitzem-baz frobnitzem-foo Information for frobnitzem-bar-0: Files: /usr/pkg/bin/frobnitzem-bar Information for frobnitzem-baz-0: Files: /usr/pkg/bin/frobnitzem-baz Information for frobnitzem-foo-0: Files: /usr/pkg/bin/frobnitzem-foo ## Future works ### PKG_DEBUGDATA `mk/bsd.debugdata.mk' implementation was not tested in the wild and probably will need some little adjustments. Apart that all the `.debug' files should be installed in a separate subpackage when the SUBPACKAGES support will be complete. ### SUBPACKAGES SUBPACKAGES support is far from being complete and also far to be in good fit also for asking a code review on tech-pkg@. There are a lot of missing and/or incomplete parts, in particular the following ones: - mk/pkgformat/*/*: initial support was added but a lot of _METADATA_TARGETS still need to be SUBPACKAGES-ified (most of them will need mostly mechanical changes and at least a modus operandi is provided by the ones that were SUBPACKAGES-ified) - mk/pkginstall/*: it is still completely unaware of SUBPACKAGES support - mk/plist/*: plist.mk was adjusted to support SUBPACKAGES but there are other places where the support can be improved (e.g. it would be nice to have a smart way to separate the SUBPACKAGES PLISTs, probably via per-subpackage PRINT_PLIST_AWK) - mk/check/*: was adjusted to support SUBPACKAGES with kludges and support is suboptimal (e.g. `check-files.mk' should probably check installed files per-subpackages not considering *all* the SUBPACKAGES!) - mk/*: other parts (e.g. per-subpackage BUILD_DEPENDS, DEPENDS, etc.) The current plan in order to finish and then polish it is the following: - Complete the SUBPACKAGES support similar to what was done during the GSoC (with SUBPACKAGES and !SUBPACKAGES case code duplication) - When the SUBPACKAGES support is complete we can switch to an implicit (and hidden) subpackage as suggested by in order to get rid of code duplication and having a single control flow relative to SUBPACKAGES. In other words: every package will always have at least one subpackage. - Adapt `mk/bsd.debugdata.mk' to SUBPACKAGES and other possible other candidates like `tex-*-doc', etc. After doing that look at less trivial possible SUBPACKAGES candidate like `databases/postegresql*-*'. ## References - Git repository of pkgsrc `debugpkg' branch (contains all the work done during the GSoC) - `GSoC 2016 Reports: Split debug symbols for pkgsrc builds, part 1' (this can be considered a subset of information contained in the diary) - `Split debug symbols for pkgsrc builds' diary (contains other possible interesting information, brainstorming, etc.)