[COMING SOON] Vendor branches with Mercurial

Vendor branches with Mercurial

Create a new vendor branch
Find our local patches
Prepare a legacy vendor branch from CVS
Import a new version on a vendor branch

Vendor branches with Mercurial

WORK IN PROGRESS -- NOT TESTED OR REVIEWED YET

Create a new vendor branch

You want to import a copy of Acme Widgetmaker. It is Apache2-licensed. It will live in src/external/apache2/widgetmaker, and we will track upstream releases on the vendor branch “vendor/widgetmaker”.

Create a shared repository clone (requires “share” extension):
```
$ hg share -U src src-vendor
```
This is another working tree for the same underlying repository data, so all the same changesets and branches are available. (Not strictly necessary, but it's much faster than waiting for “hg update” to create or delete gigabytes of data across hundreds of thousands of files as you switch between the vendor branch and trunk, and it takes much less disk space than regular “hg clone”.) The “-U” option leaves the working tree empty, checked out at the null revision, to start the vendor branch on its own history; if you forget it, you can do “hg update null” instead.

Unpack the distribution into the right path in the shared repository:

$ cd ../src-vendor
$ mkdir -p external/apache2/widgetmaker/dist
$ tar -C external/apache2/widgetmaker/dist \
    --strip-components=1 \
    -xvzf /path/to/widgetmaker-1.23.tar.gz

Commit it on the vendor branch:

$ hg branch vendor/widgetmaker
$ hg addremove
$ hg commit -m 'Import widgetmaker-1.23'

Merge the vendor branch into trunk—this will populate external/apache2/widgetmaker/dist/ on trunk with the content of Acme Widgetmaker 1.23 on the vendor branch:
```
$ cd ../src
$ hg update trunk
$ hg merge vendor/widgetmaker
$ hg commit -m 'Merge widgetmaker-1.23'
```
Tag it for reference:
```
$ hg tag vendor/widgetmaker-1.23
```

Find our local patches

Developers can commit on trunk to files under external/apache2/widgetmaker/dist/, creating local patches. Local patches incur a maintenance burden and may create bugs, so we should generally try to keep them to a minimum and/or submit them back upstream if applicable. To find the local patches on trunk:

$ cd src/external/apache2/widgetmaker/dist
$ hg diff --from vendor/widgetmaker

A release branch such as “netbsd-11” may not be on the latest version. In that case, you can consult doc/3RDPARTY to find which version it's on, and use the tag to diff:

$ hg diff --from vendor/widgetmaker-1.23

Alternatively, you can diff from the latest revision of the branch that has been merged into the current branch:

$ hg diff --from 'max(ancestors(.) & branch("vendor/widgetmaker"))'

Prepare a legacy vendor branch from CVS

Upstream software that NetBSD ships, such as less(1) and tmux(1), was historically maintained in CVS vendor branches. CVS vendor branches are quirky: “cvs import”, which imports a new of an upstream directory as a release tag on a vendor branch, doesn't delete files on the vendor branch—even if they are deleted from the release tag. For CVS usage, this didn't matter much: the deletion was recorded from one release tag to the next, and that was enough for “cvs checkout -jOLD -jNEW” to delete it on trunk too.

Unfortunately, the conversion to Mercurial is missing two parts:

The sequence of release tags. Although vendor imports with “cvs import external/bsd/... VENDOR release-1-4” happen in some definite sequence of release tags “release-1-3”, “release-1-4”, “release-2-1”, and so on, CVS doesn't record that sequence anywhere. And the release tags themselves don't exactly correspond to states of the vendor branch: “cvs checkout -rrelease-1-4” may not match “cvs checkout -rVENDOR:DATE” for any actual date. So the conversion doesn't have commits representing the release tags as such.
The merge points. When merging changes from one release tag to the next with “cvs checkout -jrelease-1-4 -jrelease-2-1” and then “cvs commit”, CVS never recorded any correspondence between the release tags and that commit. So the conversion doesn't have merge history identifying which release tags were merged into trunk.

To work around this, we can create a fictitious merge of the vendor branch from CVS, so that the next time we do a vendor branch update in Mercurial, it will compute the desired three-way merge between the current latest upstream, our local patches, and the next upstream version.

Merge the vendor branch into trunk, but if any files don't match, take the local version (from trunk), not the branch:
```
$ hg merge -t :local vendor/widgetmaker
```
The vendor branch may still have files that were deleted upstream, because “cvs import” didn't actually delete them on the vendor branch—only on the release tag:
```
$ hg revert -r 'p1()' -C --all
```
Review the diff from trunk to make sure the “merge” isn't actually changing anything:
```
$ hg diff --from 'p1()'
```
Commit:
```
$ hg commit
```

Import a new version on a vendor branch

When Acme Widgetmaker 1.24 is released, it may be time for us to import a new version—and make sure we don't lose any local patches we made, in case they are still applicable.

Create a shared repository for the vendor work tree if you haven't already:
```
$ hg share -U src src-vendor
```

Switch to the vendor branch and replace its content wholesale (except for .hgtags, which is where the previous tag is stored):

$ cd src-vendor
$ hg update vendor/widgetmaker
$ hg revert -r null -C --all -X .hgtags
$ mkdir -p external/apache2/widgetmaker/dist
$ tar -C external/apache2/widgetmaker/dist \
    --strip-components=1 \
    -xvzf /path/to/widgetmaker-1.24.tar.gz

Auto-detect additions, removals, and renames; then commit and tag it for reference:
```
$ hg addremove
$ hg commit -m 'Import widgetmaker-1.24'
$ hg tag vendor/widgetmaker-1.24
```
Note: You can also use, for example, “hg addremove -s 80” to auto-detect renames with 80% similarity; the default is to detect renames only if they have 100% similarity, i.e., if they are byte-for-byte identical. If you lower the similarity threshold, you should verify the renames are correct; somteimes automatic rename detection goes awry, especially with long blocks of copying terms copied and pasted into many files.

Merge the vendor branch into trunk:

$ cd ../src
$ hg merge vendor/widgetmaker

If there are merge conflicts, resolve them and mark them resolved with “hg resolve”:
```
$ edit sys/kern/sys_descrip.c
$ hg resolve -m sys/kern/kern_descrip.c
```
You can also use it to re-merge with different merge tools, e.g. “-t :local” to take our local version and discard upstream changes, or “-t :other” to take the upstream version and discard local changes.
Review the new changes being imported, and review our local changes from upstream:
```
$ hg diff
$ hg diff --from 'p2()'
```
Commit the merge, and push it:
```
$ hg diff
$ hg commit -m 'Merge widgetmaker-1.24'
$ hg push
```
(You can also push it to a draft topic for review first.)