Jörg Sonnenberger

News from the VCS front

NetBSD Developer Summit, EuroBSDCon 2017


  1. lochley.NetBSD.org
  2. Mercurial performance
  3. Check list
  4. Pending work


  • Former home of cvs.NetBSD.org
  • Lots of RAM, lots of disk space
  • New home of the automated CVS -> Fossil -> Git conversion
  • ...and now home of the CVS -> Fossil -> Mercurial conversion, too

lochley and Mercurial: challenges

  • fastimport used one directory with one file per blob
  • ...with uncompressed content
  • Duplicate commit check happens after transaction locks
  • fastimport assumed markers are persistent
  • ...tagging commits with markers, breaking duplicate detection
  • fastimport missed support for its own mark file
  • ...no incremental conversion without fixing it

lochley and Mercurial: status quo

  • Conversion fully integrated now
  • Similar time to the git cycle
  • Results pushed to Bitbucket
  • ...beware that src is pushing some scalability limits for BB

Mercurial performance relative to Git (I)

  • Basic testing on E5 E3-1245v5 under Illumos with 100Mbit/s+ connectivity
  • hg init + unbundle from cdn.NetBSD.org: ~8min, CPU bound (zstd)
  • ...with zstd as revlog compression engine: 6min
  • hg clone from https://bitbucket.org/netbsd/src: 10min (bzip2)
  • ...with zstd as revlog compression engine: 7min
  • git clone: ~5min with 3min local CPU time
  • git clone needs 50% more bandwidth than the bundle

Mercurial performance relative to Git (II)

  • hg co -r trunk after initial clone: 40s (2min CPU time)
  • hg co -r netbsd-8 and back to trunk: 6s / 8s (16s / 22s CPU time)
  • git checkout netbsd-8 and back to trunk: 5s / 8s (6s / 8s CPU time)
  • hg commit vs hg commit BUILDING: 2s / 0.8s
  • git commit -a vs git commit BUILDING: 0.7s / 2s
  • hg blame UPDATING: 0.7s
  • git blame UPDATING: 2s

Check list: compared to FreeBSD's VCS wiki

  • CVS migration: solved
  • History removal: possible, but tricky with convert
  • Base system integration: not planned
  • Database recovery: double check transaction assumptions
  • Partial clones: needs investigation, shallow and remotefilelog extensions

Check list: core@ requirements (I)

  • Performance: testing on low-end hardware in progress
  • ...but comfortable on the 90$ Pinebook under Linux
  • ...peak RSS on Linux ~800MB
  • User interface and terminology mostly compatible with CVS
  • Concurrent use of CVS: not planned
  • ...but preparation and testing of infrastructure against Mercurial
  • ...expected down time for final migration: around 2 days

Check list: core@ requirements (II)

  • hg ↔ git bridges exists, performance and limitations need to be checked
  • Automatic clean-up of CVS revision references: not feasible
  • Repository conversion: character set conversion in CVS before final migration
  • ...no detection of moves/copies planned
  • ...no retro-patching of SCCS history planned

Pending work (I)

  • Provide documented workflows for common operations, using modern best practises
  • ...include merge vs rebase discussion for local changes
  • ...staging repository vs "core" repository
  • Work out the build and testing infrastructure
  • ...move towards more off-the-shelf components
  • ...somewhat easier with changesets
  • ...currently under active investigation

Pending work (II)

  • Figure out new release engineering workflow
  • ...easier with changesets
  • Current approach: push changes to staging repository
  • ...CI verifies building (and maybe passing tests)

Pending work (III)

  • Test repository server
  • ...automatic pushing of clone bundles to ftp.NetBSD.org
  • ...test build host attached
  • ...allow pushing commits for authentication testing
  • ...provide source-changes alternative based on hg notify
  • Investigate Security Team ↔ Releng interaction