On Dec 23, 2007, at 6:56 AM, Ralf S. Engelschall wrote:

> On Sat, Dec 22, 2007, Jeff Johnson wrote:
>
>> In order to automate a vcheck invocation through a run-time  
>> dependency
>> probe, I'm likely going to need to either propagate a %track section
>> to all binary subpkgs of a build, or generate the %track section from
>> N and EVR arguments in the BuildRequires: dependency.
>>
>> Currently the %track section is resident only in the *.src.rpm  
>> header,
>> making
>> it rather awkward to retrieve the RPMTAG_TRACK section where it is  
>> needed
>> in lib/depends.c. Adding the tag to binary headers would permit  
>> retrieval
>> from
>> rpmdb headers (and also generalizes rpmbuild -bt nicely, as many % 
>> track
>> sections, not just the section in the current *.src.rpm Header  
>> could be
>> probed.)
>>
>> OTOH, its unclear whether there is sufficient benefit to justify  
>> adding the
>> 100-200
>> bytes of a %track section to binary headers. Yes, 100-200b is not  
>> very
>> much, but
>> I'm sure there will be complaints.
>>
>> The other approach is to generate the vcheck stanza from the  
>> {N,EVR,Flags}
>> triple. A vcheck stanza looks like
>>     %track
>>     prog popt = {
>>         version   = 1.12
>>         url       = http://rpm5.org/files/%{name}
>>         regex     = %{name}-(\d+\.\d+)\.tar\.gz
>>    }
>>
>> The "version" can be parsed from the dependency EVR tag.
>>
>> Which leaves the N field to carry both the "url" and "regex"  
>> pattern. While
>> that may sound gory and ugly, the typical usage case for  
>> automatically
>> checking that Source0: was latest available would look something like
>>   Source0: http://rpm5.org/files/%{name}/popt-1.12.tar.gz
>>   ...
>>   BuildRequires: vcheck(%SOURCEURL0) = %version
>> and the regex pattern would be generated by a heuristic, replacing
>> digit strings with \d patterns for matching.
>>
>> Any opinions before I hack something up?
>
> Hmmmm... I'm still thinking about all of this, but I'm still not
> entirely convinced whether it is worth the effort and the right
> direction. Sure, it sounds sexy that the version is automatically
> checked via a dependency, but...
>

Thinking here too ;-)

> - until now all dependencies were AFAIK all "local" ones, i.e.,
>   RPM was able to _quickly_ and _locally_ check them. With the
>   vcheck(..) this would result in a _slow_ and _remote_ check.
>   This has to be not bad, just a little bit surprisingly perhaps.
>

There's a whole class of run-time probes already implemented
in rpm-4.4.9 that few are using yet.

Because of rpmio, anyplace that a file path is used (several places
with run-time probes already), a URL can be used. Whether one
choses to use a URL (or a run-time probe) is left to the packager.

> - it might be not reasonable to _force_ a version check through
>   a dependency. My experience with over 6 years of %track'ing is that
>   although the tracking _logically_ is the first build step before  
> doing
>   the preparation/build/install/clean, one usually always wants it do
>   execute _manually_. Both because it can be a slow step (between a  
> few
>   seconds and about half a minute in case of timeouts, etc) and one
>   not always wants to track the version over and over again while one
>   OTOH usually rather often might build the package. Additionally,  
> very
>   often (e.g. for all(!) sourceforge.net projects) one has to track a
>   different URL than the URL from which to download.
>

RPM is not "forcing" anyone to add anything whatsoever to spec files.  
And sure there
are timeouts and radically different failure modes and additional
sysadmin complexity with network access in rpm.


> - vcheck(1) provides really lots of features. Usually one just need
>   the basic features to perfectly track about 90% of all vendor
>   distribution files. So, on the first spot it certainly looks
>   sexy to just on-the-fly generate the tracking configuration. But
>   I'm not convinced that this heuristic generation really works in
>   practice, because one usually always needs slight variations.
>   Especially in version matching. In OpenPKG we always start with
>   the regex (meta-variable) __VER__, then we often have to turn it
>   into \d+\.\d+\.\d+ and then we something recognize that instead of
>   1.2.0 just 1.2 is used by the vendor and hence we rewrite it to
>   \d+\.\d+(\.\d+)*. Or we change back a \d+\.\d+\.\d+ to __VER__ in
>   order to also be able to track beta or release candidate  
> versions. Or
>   in short: in practice one usually never can just say that this or  
> that
>   is the right pattern. Over the life-time of a package one regularily
>   has to adjust it. With a generated approach this would be not  
> possible
>   at all and hence I think the resulting version-checking dependency
>   would be just fail. Additionally, more complex trackings are
>   not possible at all, too.
>

Yep, the heuristic I proposed will never be able to guess all  
patterns correctly.
See the alternative, pushing RPMTAG_TRACK into binary packages,
though.

> So, I think the currently implemented %track support is good and a  
> major
> step forward in practice as it (as OpenPKG clearly shows since years)
> allows one to automatically track the upstream version. But I  
> personally
> would let the current implementation just settle for RPM 5.0 and see
> what feedback comes. Based on this feedback I would decide what  
> further
> improvements should be done for RPM 5.0.
>
> Additionally, what would be IMHO a lot more useful in practice than
> additional _tracking_ functionality would be to have the companion
> feature which is still missing in RPM: the possibility to  
> automatically
> download those distribution files which are still not present on the
> local filesystem! In OpenPKG we are using a silly external Perl  
> utility
> for this ("openpkg dev fetch *.spec"), but to complement the vendor
> tracking to have this functionality built directly into RPM would be
> phantastic.
>

Um, rpm has had the ability to automatically download, with
digest/signature verification of the downloaded content
through a similar run-time probe as vcheck, since 4.4.6.
(digesy/signature verification in 4.4.9).

Fragile and largely untested yes. But already implemented.

> Here is the specification:
>
> 1. Usage:
>    $ rpm --fetch *.spec
>

Instead of an explicit CLI option, rpm splits configuration
to local and remote, and checks that a local copy is present.
If not present, the fetch from remote URL is attempted.

> 2. Functionality:
>    RPM parses all SourceN and PatchN tag values.
>    There are four possibilities:
>
>      (a) SourceX: <file>
>          This is the regular local file. RPM checks for
>          %{_sourcedir}/<file> on the local filesystem. If it is not
>          existing, it complains with an error.
>
>      (b) SourceX: <url>
>          This is a regular remote URL. RPM checks for
>          %{_sourcedir}/basename(<url>) on the local filesystem. If
>          it is not existing, it downloads remote <url> to local
>          %{_sourcedir}/basename(<url>). If it cannot download it, it
>          complains with an error.
>
>      (c) SourceX: <url>:<file>
>          This is a remote URL where the local <file> is different
>          from basename(<url>) (very important as this is usually
>          the case when downloading unversioned files, etc). RPM
>          checks for %{_sourcedir}/<file> on the local filesystem.
>          If it is not existing, it downloads remote <url> to local
>          %{_sourcedir}/<file>. If it cannot download it, it complains
>          with an error.
>
>      (d) SourceX: <url>#<file>
>          This is a remote URL where the local <file> cannot be
>          downloaded via <url> automatically at all. This is usually
>          used in conjunction with a "NoSource: X" header and
>          indicates that <url> has to be visited _manually_, searched
>          for <file> manually, downloaded manually and then stored
>          under %{_sourcedir}/<file> manually. RPM checks for
>          %{_sourcedir}/<file> existance and if it doesn't exist is
>          complains with an error containing a hint that the user  
> has to
>          visit <url>.
>

The functionality currently implemented is nowhere near as fancy.  
basename
of the URL is assumed identical between local and remote, the remote
URL is in the spec file, the local URL can be configured in macros.

> I guess that implementing this via rpmio should be rather straight
> forward for you, Jeff. At least *THIS* would be IMHO the second  
> *KILLER
> FEATURE* for RPM 5 beside the version tracking. So, if you want to
> really further improve the version tracking, please implement this
> companion functionality now and suspend the vcheck(...) dependency
> feature for RPM 5.1 and until we received the feedback of the  
> community.
>

I'll likely dust off the bit rot and set up a 3rd demonstration of the
ability for rpm to fetch sources/patches/icons.

I do need to permit remote spec files. ATM, the spec file parsing
is still line-by-line, which doesn't work well with the URL caching
and persistent http connections. Adding an equivalent vcheck-and- 
fetch step
to get a local copy of a remote spec file, and reopening the spec file
won't be hard.

The URL parser likely needs to be dusted off and brought into the 21st
century. No ?options, and no %20 character handling, are the major  
flaws.

Thanks for comments. Suspending vcheck to 5.1 is perfectly OK w me.

73 de Jeff