Ports+: Yet Another FreeBSD Ports Upgrade Utility: uyota 匠の一手

As I got tired of long waiting of portupgrade trying to resolve dependencies, I came up with yet another tool for upgrading FreeBSD ports system. Unlink other tools, it tries to maximize existing resource to maximize its performance. This program attempts to wrap around with another 'make' and expand use of FreeBSD ports system. The heart of ports+ is parsing INDEX and +CONTENTS files. The rest is handed to GNU make. I think it comes to a point where I seek wider audiences to test with it.

Some short summaries of ports+ * Magnitude quicker dependency resolving compare to portupgrade. * Dependencies are resolved only once for all upgrades where portupgrade does before each 'make deinstall and make install' of ports. * Commands to execute for each ports are very flexible. * Allow concurrent builds of ports with -j option. (yet experimental)

Some dry runs comparisons with about 500 ports installed on Mobile Intel(R) Celeron(R) CPU 2.40GHz. Note that programs were run twice in a row after rebooting for each program. That is to kill disk-cache effect from the other program. The first run takes longer to read disk; however, the second run only accessed via file-cache.

# time portupgrade -n -a -O
60.901u 36.208s 2:30.08 64.6%   147+13408k 4840+59io 77pf+0w
# time portupgrade -n -a -O
83.581u 47.805s 2:51.34 76.6%   155+13197k 547+33io 0pf+0w
# time gmake -n all
4.784u 1.982s 0:22.06 30.6%     258+2485k 1528+126io 9pf+0w
# time gmake -n all
1.551u 0.205s 0:01.85 94.5%     179+8540k 0+0io 0pf+0w

Three advises before trying:

1. Read the description in GNUmakefile and set PORTS+_IGNORE in ports+.conf for ones you don't want to upgrade.
2. At least, backup /var/db/pkg so that you can try again even with other tools.
3. Do not 'make clean' so that you can try installing again.

Ports+ requires:

1. GNU make, gmake.
2. GNU awk, gawk.
3. Up to date INDEX file.

Known Issues:

1. Concurrent build with -j is still experimental although I always use it myself. There is a race condition when more than two ports updating the same +CONTENTS files at the same time. This could result +CONTENTS file out of sync. However, you could try at your risk, too.
2. If OPTIONS changes dependences of a port, ports+ won't handle it at this moment.
I don't know if I have to maintain any other file other than +CONTENTS.

For these who want to give a shot even without reading further. Try these commands below. These two gmake examples are dry runs and won't harm your system. You can try with 'gmake -n' and once you are confident, you can take -n option out.

# fetch http://www.ne.jp/asahi/export/uyota/src/ports+-1.59.tar.bz2
# tar xvf ports+-1.59.tar.bz2
# cd ports+1.59
# gmake test
# gmake -n all

Quoted from the pacakge:

# Name: ports+, FreeBSD ports extension
#
# FreeBSD ports upgrading utility that takes advantages of the ports
# infrastructure to extend use of the ports system.
#                                 Written by Yoshihiro Ota <ota@j.email.ne.jp>

#
# Short Summaries
#  * Magnitude quicker dependency resolving compare to portupgrade.
#  * Dependencies are resolved only once for all upgrades where portupgrade
#    does before each 'make deinstall and make install' of ports.
#  * Commands to execute for each ports are very flexible.
#  * Allow concurrent builds of ports with -j option. <yet experimental>
#
# Description
#   The ports+ command is used to upgrade installed packages via ports.
#
#  In short, ports+ simulates what portupgrade does, but ports+ resolves
#  port dependencies only once at start time and its operation is magnitude
#  faster than portupgrade.
#   As the number of ports increased as well as the number of ports
#  installed in each machines increased over years, the amount of
#  time portupgrade spends to resolve dependencies among various time for
#  each port increased significantly.  That is most apparent when script
#  language based ports are being upgraded.  Instead,  ports+ only performs
#  the minimum but necessary package database maintenance and dependency
#  resolution for upgrade to work.
#
# Requirements
#  Ports+ depends on GNU make and GNU awk.
#  Ports+ requires the ports INDEX file.
#
# Known Problems
#  * If 'OPTIONS' change causes different dependencies from the INDEX, they
#  are not tracked.
#
# Suggestions before Using
#  Ports+ is still under developments and there are possibilities for bugs.
#  Two things are suggested.
#  1. Backup /var/db/pkg.  I.e. tar cvfj /var/db/pkg.tar.bz2 /var/db/pkg
#  2. No 'make clean' or equivalent so that you can run portupgrade or any
#  other tools to redo the install.
#
# Preparation
#   * ports+.conf is the configuration file for customizing its behavior
# and values.  Store any personal change in this file.
#   * It is a good idea to start with ignore list to prevent unwanted upgrades.
# PORTS+_IGNORE is a list of the awk style regular expressions to denote
# ones need to be ignored.  They are different from shell expansions and
# it is recommended to check 'man gawk.'
#
# Basic Usage
#
#  Available targets: all, package names, and package fullpaths.
#
#  The 'all' target rebuilds all ports installed on the system.
#  In the process of upgrades, ports are installed in run-dependency order.
#  In other word, build-dependencies are not handled by PORTS+.
#  If new dependencies include uninstalled ports, they will be installed as
#  well.
#  # gmake all
#
#  The package names are also available.  For example, www/seamonkey has
#  its target name seamonkey, and mail/sylpheed as sylpheed.  There are some
#  ports in multilingual and they have suffixes like ja-, for example, ja-xpdf
#  for japanese/xpdf and xpdf for graphics/xpdf.  When this type
#  of target is used, all packages the target requires are reinstalled in order.
#  # gmake seamonkey sylpheed ja-xpdf
#
#  The package fullpath targets are available, like /var/db/pkg/xosview-1.8.3.
#  For upgrading, either the new version number or old version number can
#  be used.
#  # gmake /var/db/pkg/xosview-1.8.3
#  # gmake /var/db/pkg/p5-*
#
#
#  Operations : ${NEW}, ${UPGRADE}, ${NOOP}
#
#  While scanning installed and available ports, PORTS+ groups ports into
#  three categories and operates based on these: NEW that are ones not
#  installed, UPGRADE that are ones installed already and need upgrades, and
#  NOOP that are ones already latest port is installed and not necessary to
#  upgrade.  Note that new dependences of ports require not-installed
#  ports, sometime.
#    These commands are default for regular upgrade.  However, these commands
#  can be overridden by configuration file or command line such that it can
#  do anything.
#    Due to the differences how arguments are handled between BSD make and
#  GNU make $(BSDMAKE) needs to be used.
#
#  Extract and patch source of all ports that are installed and need upgrade.
#  # gmake all UPGRADE='$(BSDMAKE) patch' NEW='$(BSDMAKE) patch' \
#      BSDMAKE_OPTIONS='NO_CHECKSUM=YES'
#
#  Extract source of all ports that seamonkey requires.
#  # gmake seamonkey UPGRADE='$(BSDMAKE) extract' \
#      NOOP='$(BSDMAKE) extract' NEW='$(BSDMAKE) extract'
#
#  Advanced Usage: Parallel Build (Experimental)
#
#    Modem make utilities provide -j option to run multiple jobs
#  simultaneously.  Thanks to that, ports+ support concurrent building of
#  non-depending ports.  Some of ports require 'make config' to run before
#  building.  This is interactive process and does not work well with
#  concurrent builds as each process is detached from the terminal.
#  Therefore, run the 'config' for all.  Then, pass 'CONFIG_DONE=1' to
#  BSDMAKE so that the port system does not invoke 'dialog' for your input.
#
#  # gmake UPGRADE='$(BSDMAKE) config' NEW='$(BSDMAKE) config' all
#  # gmake -j 3 all BSDMAKE_OPTIONS='CONFIG_DONE=1'
#
#  Tips:
#
#  Ports+ create a package backup before deinstalling a port.  Ports+ does
#  not delete any of these backup files unless 'gmake clean' is performed.
#  In addition, ports+ does not create a backup if one already exists.
#  Sometimes, pkg_create -b fails due some errors such as md5 checksum mismatch
#  or missing file. In this case, ports+ will stop because pkg_create failed
#  but they will leave its package backup.  When ports+ runs again, the backup
#  file already exists so that ports+ will upgrade the port.  This behavior is
#  chosen because if backup fails, it needs to stop at least once.  However,
#  if someone retries to upgrade the same port, it will not a big matter that
#  old package was not created right because he/she is installing a newer
#  version, anyway.
#
#  Ports+ does not care about obsolete ports.  It will not deinstall.  In some
#  cases, obsoleted port causes a conflict with another port that is required
#  by other port.  If that happens, the obsolete port has to be deleted
#  manually and the new port has to be installed (probably by hand).
#

Next