On Sat, Dec 22, 2007, Jeff Johnson wrote:

> In order to automate a vcheck invocation through a run-time dependency
> probe, I'm likely going to need to either propagate a %track section
> to all binary subpkgs of a build, or generate the %track section from
> N and EVR arguments in the BuildRequires: dependency.
>
> Currently the %track section is resident only in the *.src.rpm header,
> making
> it rather awkward to retrieve the RPMTAG_TRACK section where it is needed
> in lib/depends.c. Adding the tag to binary headers would permit retrieval
> from
> rpmdb headers (and also generalizes rpmbuild -bt nicely, as many %track
> sections, not just the section in the current *.src.rpm Header could be
> probed.)
>
> OTOH, its unclear whether there is sufficient benefit to justify adding the
> 100-200
> bytes of a %track section to binary headers. Yes, 100-200b is not very
> much, but
> I'm sure there will be complaints.
>
> The other approach is to generate the vcheck stanza from the {N,EVR,Flags}
> triple. A vcheck stanza looks like
>     %track
>     prog popt = {
>         version   = 1.12
>         url       = http://rpm5.org/files/%{name}
>         regex     = %{name}-(\d+\.\d+)\.tar\.gz
>    }
>
> The "version" can be parsed from the dependency EVR tag.
>
> Which leaves the N field to carry both the "url" and "regex" pattern. While
> that may sound gory and ugly, the typical usage case for automatically
> checking that Source0: was latest available would look something like
>   Source0: http://rpm5.org/files/%{name}/popt-1.12.tar.gz
>   ...
>   BuildRequires: vcheck(%SOURCEURL0) = %version
> and the regex pattern would be generated by a heuristic, replacing
> digit strings with \d patterns for matching.
>
> Any opinions before I hack something up?

Hmmmm... I'm still thinking about all of this, but I'm still not
entirely convinced whether it is worth the effort and the right
direction. Sure, it sounds sexy that the version is automatically
checked via a dependency, but...

- until now all dependencies were AFAIK all "local" ones, i.e.,
  RPM was able to _quickly_ and _locally_ check them. With the
  vcheck(..) this would result in a _slow_ and _remote_ check.
  This has to be not bad, just a little bit surprisingly perhaps.

- it might be not reasonable to _force_ a version check through
  a dependency. My experience with over 6 years of %track'ing is that
  although the tracking _logically_ is the first build step before doing
  the preparation/build/install/clean, one usually always wants it do
  execute _manually_. Both because it can be a slow step (between a few
  seconds and about half a minute in case of timeouts, etc) and one
  not always wants to track the version over and over again while one
  OTOH usually rather often might build the package. Additionally, very
  often (e.g. for all(!) sourceforge.net projects) one has to track a
  different URL than the URL from which to download.

- vcheck(1) provides really lots of features. Usually one just need
  the basic features to perfectly track about 90% of all vendor
  distribution files. So, on the first spot it certainly looks
  sexy to just on-the-fly generate the tracking configuration. But
  I'm not convinced that this heuristic generation really works in
  practice, because one usually always needs slight variations.
  Especially in version matching. In OpenPKG we always start with
  the regex (meta-variable) __VER__, then we often have to turn it
  into \d+\.\d+\.\d+ and then we something recognize that instead of
  1.2.0 just 1.2 is used by the vendor and hence we rewrite it to
  \d+\.\d+(\.\d+)*. Or we change back a \d+\.\d+\.\d+ to __VER__ in
  order to also be able to track beta or release candidate versions. Or
  in short: in practice one usually never can just say that this or that
  is the right pattern. Over the life-time of a package one regularily
  has to adjust it. With a generated approach this would be not possible
  at all and hence I think the resulting version-checking dependency
  would be just fail. Additionally, more complex trackings are
  not possible at all, too.

So, I think the currently implemented %track support is good and a major
step forward in practice as it (as OpenPKG clearly shows since years)
allows one to automatically track the upstream version. But I personally
would let the current implementation just settle for RPM 5.0 and see
what feedback comes. Based on this feedback I would decide what further
improvements should be done for RPM 5.0.

Additionally, what would be IMHO a lot more useful in practice than
additional _tracking_ functionality would be to have the companion
feature which is still missing in RPM: the possibility to automatically
download those distribution files which are still not present on the
local filesystem! In OpenPKG we are using a silly external Perl utility
for this ("openpkg dev fetch *.spec"), but to complement the vendor
tracking to have this functionality built directly into RPM would be
phantastic.

Here is the specification:

1. Usage:
   $ rpm --fetch *.spec

2. Functionality:
   RPM parses all SourceN and PatchN tag values.
   There are four possibilities:

     (a) SourceX: <file>
         This is the regular local file. RPM checks for
         %{_sourcedir}/<file> on the local filesystem. If it is not
         existing, it complains with an error.

     (b) SourceX: <url>
         This is a regular remote URL. RPM checks for
         %{_sourcedir}/basename(<url>) on the local filesystem. If
         it is not existing, it downloads remote <url> to local
         %{_sourcedir}/basename(<url>). If it cannot download it, it
         complains with an error.

     (c) SourceX: <url>:<file>
         This is a remote URL where the local <file> is different
         from basename(<url>) (very important as this is usually
         the case when downloading unversioned files, etc). RPM
         checks for %{_sourcedir}/<file> on the local filesystem.
         If it is not existing, it downloads remote <url> to local
         %{_sourcedir}/<file>. If it cannot download it, it complains
         with an error.

     (d) SourceX: <url>#<file>
         This is a remote URL where the local <file> cannot be
         downloaded via <url> automatically at all. This is usually
         used in conjunction with a "NoSource: X" header and
         indicates that <url> has to be visited _manually_, searched
         for <file> manually, downloaded manually and then stored
         under %{_sourcedir}/<file> manually. RPM checks for
         %{_sourcedir}/<file> existance and if it doesn't exist is
         complains with an error containing a hint that the user has to
         visit <url>.

I guess that implementing this via rpmio should be rather straight
forward for you, Jeff. At least *THIS* would be IMHO the second *KILLER
FEATURE* for RPM 5 beside the version tracking. So, if you want to
really further improve the version tracking, please implement this
companion functionality now and suspend the vcheck(...) dependency
feature for RPM 5.1 and until we received the feedback of the community.

                                       Ralf S. Engelschall
                                       rse@engelschall.com
                                       www.engelschall.com