[MOVE THIS THREAD TO rpm-devel@ FOR FURTHER DISCUSSION. PLEASE REPLY HERE NOW]
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sat, 26 May 2007 19:55:14 -0400
(from a #rpm conversation with rsc)
One very easy way to prepare drop-in parallel rpm packaging on
all platforms is to add "5" after all path collisions.
So
/usr/include/rpm5
/usr/lib/rpm5
/bin/rpm5
/usr/bin/rpmbuild5
/etc/rpm5
to name the most significant path collisions (libraries already have
*-4.5* instead).
The other features (or incomaptibilities) that would need to be
addressed
are
1) reverting btree -> hash for rpmdb indices (/var/lib/rpm can
then be shared)
2) making the parent directory and file linkto dependencies opt-
in rather than opt-out.
and likely one or two more small changes that are equally easy to
revert.
The largest remaining issue that I can think of is the name of the
rpm-python module.
I can't think of better atm. It is absolutely essential that there is
a way
to install rpm5.org rpm next to vendor distributed rpm, as vendors
currently
control rpm's distribution channels.
I wince at the number of typo's I'm going to make if /bin/rpm -> /bin/
rpm5.
Oh well, its likely necessary. Lemme sleep on it and then I'm likely
to do
the Grand Renaming on the rpm-4_5 branch tomorrow before attempting
per-platform packaging.
73 de Jeff
-----------------------------------------------------------------------------
From: "Ralf S. Engelschall" <rse+rpm-team@rpm5.org>
Date: Sun, 27 May 2007 11:22:03 +0200
On Sat, May 26, 2007, Jeff Johnson wrote:
> (from a #rpm conversation with rsc)
>
> One very easy way to prepare drop-in parallel rpm packaging on
> all platforms is to add "5" after all path collisions.
>
> So
> /usr/include/rpm5
> /usr/lib/rpm5
> /bin/rpm5
> /usr/bin/rpmbuild5
> /etc/rpm5
> to name the most significant path collisions (libraries already have *-4.5*
> instead).
Sorry if I'm too heretic here, but that's IMHO just the hard-coded POV
of a usual Linux distribution where everything is merged together into
the same namespace starting at /.
The traditional Unix approach here is to just use a different
"prefix" and that's it. The default might be still prefix=/, but
one could install RPM with prefix=/tmp/rpm5 (for quick testing) or
prefix=/usr/local/rpm5 or whatever else and this way files end up in
/usr/local/rpm5/{include/rpm,lib/rpm,bin,usr/bin,etc} and everythig is
still fine. For instance OpenPKG installs RPM under arbitrary filesystem
prefixes since many years and although it looks strange having to run
the "rpm" command with a fully qualified path or add a directory to
$PATH at the first spot, it usually is totally harmless.
Plus, what to do if someone needs RPM 5.0.0, 5.0.1 plus other versions
installed _AT THE SAME TIME_ for comparison and development reasons? So,
I think we need both: prefix-based and suffix-based namespace collision
resolving. The usual way would be:
$ make install [DESTDIR=...] [prefix=...] [suffix=...]
The default can be DESTDIR="", prefix="/" and suffix="" to be backward
compatible, of course. With DESTDIR we support the packers, prefix
allows to use arbitrary prefixes and suffix allows one to install RPM
5.0.0 with suffix=500 and RPM 5.0.1 with suffix=501 side-by-side.
This way we are flexible enough and don't have to hard-code any
particular namespace resolving approach. And best of all is that DESTDIR
and prefix are already supported anyway. I just wanted to raise the
issue that suffix should be not hard-coded and that one usually not even
needs it as prefix already exists. But too have all three is certainly
the best.
Ralf S. Engelschall
rse@engelschall.com
www.engelschall.com
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 08:27:02 -0400
On May 27, 2007, at 5:22 AM, Ralf S. Engelschall wrote:
> On Sat, May 26, 2007, Jeff Johnson wrote:
>
>> (from a #rpm conversation with rsc)
>>
>> One very easy way to prepare drop-in parallel rpm packaging on
>> all platforms is to add "5" after all path collisions.
>>
>> So
>> /usr/include/rpm5
>> /usr/lib/rpm5
>> /bin/rpm5
>> /usr/bin/rpmbuild5
>> /etc/rpm5
>> to name the most significant path collisions (libraries already
>> have *-4.5*
>> instead).
>
> Sorry if I'm too heretic here, but that's IMHO just the hard-coded POV
> of a usual Linux distribution where everything is merged together into
> the same namespace starting at /.
>
;-) You're no heretic.
I see the problem as giving the consumer what they want rather
than explaining why something else is better.
In this case, linux people are used to everything merged together.
OTOH, the Mac OS X case is different for rpm because users wish
autonomous prefix-based installations like /opt/local and /sw.
My answer for Mac OS X is likely a bit heretical as well. For a system
wide installer like rpm to be effective, there must be a common
shared database, and the path to the database doesn't really
fit into a prefix based, executable/development path model.
So on Mac OS X I plan to stake a claim on the currently unused
real estate of /bin/rpm and /var/lib/rpm; i.e. no prefix.
> The traditional Unix approach here is to just use a different
> "prefix" and that's it. The default might be still prefix=/, but
> one could install RPM with prefix=/tmp/rpm5 (for quick testing) or
> prefix=/usr/local/rpm5 or whatever else and this way files end up in
> /usr/local/rpm5/{include/rpm,lib/rpm,bin,usr/bin,etc} and everythig is
> still fine. For instance OpenPKG installs RPM under arbitrary
> filesystem
> prefixes since many years and although it looks strange having to run
> the "rpm" command with a fully qualified path or add a directory to
> $PATH at the first spot, it usually is totally harmless.
>
> Plus, what to do if someone needs RPM 5.0.0, 5.0.1 plus other versions
> installed _AT THE SAME TIME_ for comparison and development
> reasons? So,
> I think we need both: prefix-based and suffix-based namespace
> collision
> resolving. The usual way would be:
>
> $ make install [DESTDIR=...] [prefix=...] [suffix=...]
>
> The default can be DESTDIR="", prefix="/" and suffix="" to be backward
> compatible, of course. With DESTDIR we support the packers, prefix
> allows to use arbitrary prefixes and suffix allows one to install RPM
> 5.0.0 with suffix=500 and RPM 5.0.1 with suffix=501 side-by-side.
>
> This way we are flexible enough and don't have to hard-code any
> particular namespace resolving approach. And best of all is that
> DESTDIR
> and prefix are already supported anyway. I just wanted to raise the
> issue that suffix should be not hard-coded and that one usually not
> even
> needs it as prefix already exists. But too have all three is certainly
> the best.
>
Yep, all sensible and consistent with a traditional unix model.
There are two other approaches to software installation that rpm
needs to start to handle/manage soon imho
1) stow based symlinks
2) plan9 mounts
Automating stow/plan9 installations within rpm will be much more
heretical
than DESTDIR, prefix, suffix ;-)
73 de Jeff
-----------------------------------------------------------------------------
From: =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb@algonet.se>
Date: Sun, 27 May 2007 22:02:09 +0200
Jeff Johnson wrote:
> OTOH, the Mac OS X case is different for rpm because users wish
> autonomous prefix-based installations like /opt/local and /sw.
>
> My answer for Mac OS X is likely a bit heretical as well. For a system
> wide installer like rpm to be effective, there must be a common
> shared database, and the path to the database doesn't really
> fit into a prefix based, executable/development path model.
>
> So on Mac OS X I plan to stake a claim on the currently unused
> real estate of /bin/rpm and /var/lib/rpm; i.e. no prefix.
The previous Mac packages use /usr/local and /var/local,
mostly because those were easier to wipe when something
went wrong but also cause they're the default locations...
(unless counting OpenDarwin, that still used main /usr)
It does have a bunch of problems when system is not using
RPM itself, such as interfering with local installations.
So, I've grown to like the /opt/local and /sw prefixes...
Even better with something like /openpkg or slashpackage.
But I think your ambition on Mac OS X is higher than mine,
I only envisioned add-on packages and not the full system.
I'm most likely going to use /opt/local at first (MacPorts),
and then maybe switch to something portable like /openpkg.
So I'm not going to like "rpm5" more than I like "gmd5sum",
but then again I don't have Fink paths enabled by default.
(either I use the regular /usr/local or I use /opt/local,
and try to not mix the two up since it always ends up bad)
--anders
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 16:09:00 -0400
On May 27, 2007, at 4:02 PM, Anders F Bj=F6rklund wrote:
> Jeff Johnson wrote:
>
>> OTOH, the Mac OS X case is different for rpm because users wish
>> autonomous prefix-based installations like /opt/local and /sw.
>>
>> My answer for Mac OS X is likely a bit heretical as well. For a =20
>> system
>> wide installer like rpm to be effective, there must be a common
>> shared database, and the path to the database doesn't really
>> fit into a prefix based, executable/development path model.
>>
>> So on Mac OS X I plan to stake a claim on the currently unused
>> real estate of /bin/rpm and /var/lib/rpm; i.e. no prefix.
>
> The previous Mac packages use /usr/local and /var/local,
> mostly because those were easier to wipe when something
> went wrong but also cause they're the default locations...
> (unless counting OpenDarwin, that still used main /usr)
>
> It does have a bunch of problems when system is not using
> RPM itself, such as interfering with local installations.
> So, I've grown to like the /opt/local and /sw prefixes...
> Even better with something like /openpkg or slashpackage.
>
>
> But I think your ambition on Mac OS X is higher than mine,
> I only envisioned add-on packages and not the full system.
> I'm most likely going to use /opt/local at first (MacPorts),
> and then maybe switch to something portable like /openpkg.
>
;-) yes ambitious, perhaps overly. we'll see in abt 4-6 weeks.
> So I'm not going to like "rpm5" more than I like "gmd5sum",
> but then again I don't have Fink paths enabled by default.
> (either I use the regular /usr/local or I use /opt/local,
> and try to not mix the two up since it always ends up bad)
>
as long as the path to the database can be shared, then
all of /opt/local, /sw, and / installed rpm should interoperate
nicely and smoothly.
separate databases is doable but rather clunky and unnecessary imho.
73 de Jeff
-----------------------------------------------------------------------------
From: Robert Scheck <robert@rpm5.org>
Date: Sun, 27 May 2007 11:32:21 +0200
Hi folks,
On Sat, 26 May 2007, Jeff Johnson wrote:
> One very easy way to prepare drop-in parallel rpm packaging on
> all platforms is to add "5" after all path collisions.
the main reason for Jeff's mail was my purposal to import rpm5.org into
ex-"Fedora Extras" as rpm5-4.5-0.x package with the possibility to co-exist
with current "rpm" package. But currently I'm in doubt whether this a good
idea, because - as Jeff mentioned - python seems tricky. And when importing
rpm5 into Fedora I really would like to provide rpm5 libraries also to yum
users without bigger modifications to yum until "wnh" is ready to replace
the stuff. Note that yum/createrepo upstream is very arrogant and only sees
rpm.org as rpm upstream. The other way simply would be to conflict or even
obsolete rpm when the rpm5 package gets installed. Thoughts?
Personally I'm not happy with a possible global renaming of rpm to rpm5, I
would do this for the Fedora case to show especially the Red Hat folks that
good things exist and maybe lucky users are switching over.
Nevertheless "we" as rpm5.org shouldn't provide binaries, there are AFAIR
too much variations at downstream and most vendors only accept packages
built and/or signed-off by their packagers. Instead of providing binaries
we should try to get the vendors on our boat and not to rpm.org - that IMHO
would help much (especially when bigger distributions like openSUSE etc.
are switching from the "Red Hat fork"). Last of it means of course, that
we've to hear more what the distributions care about...but just my 2 ct.
Greetings,
Robert
-----------------------------------------------------------------------------
From: "Ralf S. Engelschall" <rse+rpm-team@rpm5.org>
Date: Sun, 27 May 2007 13:20:55 +0200
On Sun, May 27, 2007, Robert Scheck wrote:
> [...]
> Nevertheless "we" as rpm5.org shouldn't provide binaries, there are AFAIR
> too much variations at downstream and most vendors only accept packages
> built and/or signed-off by their packagers.
> [...]
Yes, exactly! Jeff, what is your personal opinion on this?
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 08:41:42 -0400
On May 27, 2007, at 7:20 AM, Ralf S. Engelschall wrote:
> On Sun, May 27, 2007, Robert Scheck wrote:
>
>> [...]
>> Nevertheless "we" as rpm5.org shouldn't provide binaries, there
>> are AFAIR
>> too much variations at downstream and most vendors only accept
>> packages
>> built and/or signed-off by their packagers.
>> [...]
>
> Yes, exactly! Jeff, what is your personal opinion on this?
>
Short term, the common linux distribution paradigm is binary
packages, and I don't think rpm5.org has any choice other than
to distribute binary packages for certain vendor distros.
I think its time to start spec'ing out what would be needed
to deliver *.rpm packages to build systems like openpkg,
OpenSUSE, Cooker, MacPorts, mock/mach/koji/whatever and
other logical candidates.
And there's an underlying design flaw in rpmbuild that needs to
be answered as well
Which drives a build, a srpm or a spec file?
that I need to find an answer for.
FYI: Michael Jennings has done a pretty good job with mezzanine,
which essentially pulls package build elements from cvs/svn, and
pushes the build elements to a build system for building.
73 de Jeff
-----------------------------------------------------------------------------
From: "Thomas Lotterer" <thomas+rpm5@lotterer.net>
Date: Sun, 27 May 2007 17:54:51 +0200
n3npq@mac.com wrote on 2007-05-27 14:41:
> Which drives a build, a srpm or a spec file?
>
My POV is clearly the spec file.
The SRPM is an archive only, carrying the spec file and everything else needed
for a build. It could have been any other archive format, plain filesystem or
version control checkout. However, SRPM has good capability to access metadata.
I want to point out that Ralf and I discussed the notion of an "archive" with
Richard Stallman. It is absolutely in compliance with GPL to ship proprietary
and copylefted GNU material together in any kind of archive, including ZIP,
tar, cpio and SRPM. This adheres to the "mere aggregation" terminology of GPL
and is very important for commercial use cases. Needless to say, binary RPMs
are a different story.
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 13:05:43 -0400
On May 27, 2007, at 11:54 AM, Thomas Lotterer wrote:
> n3npq@mac.com wrote on 2007-05-27 14:41:
>> Which drives a build, a srpm or a spec file?
>>
> My POV is clearly the spec file.
Perfectly understood. Your view is the majority POV too.
There are build systems that do
rpmbuild --rebuild foo*.src.rpm
and there are many GNOME users that like to do
rpmbuild -ta foo*.tgz
which is where the confusion starts.
Sure a spec file is being used underneath the hood.
> The SRPM is an archive only, carrying the spec file and everything
> else needed
> for a build. It could have been any other archive format, plain
> filesystem or
> version control checkout. However, SRPM has good capability to
> access metadata.
>
Again, perfectly understood. While a SRPM has good capability to
access metadata,
unfortunately the actual scripts that are executed to perform a build
are not in
the metadata, but rather in the spec file in the cpio archive payload.
What I think needs to happen is to clearly define a flow of control
spec+tarball+patches -> srpm -> build -> binary packages
to permit srpm's to be more easily created from non-spec file sources
(like MacPorts Portfiles, or ant or ...), and drive the build solely
from
srpm metadata, not from a parse of a spec file.
The reason to do so is largely to support alternative, non-rpm, means
to drive a build. Portfiles are a quite pleasant and useful alternative
recipe. There's also xmlspec, another replacment for the current
rpmbuild spec file parser.
FWIW, my wraptastic.org logs show tarballs are considerably more popular
than srpm's as a distribution format.
Which largely means to me that srpm's as is are failing in the market.
> I want to point out that Ralf and I discussed the notion of an
> "archive" with
> Richard Stallman. It is absolutely in compliance with GPL to ship
> proprietary
> and copylefted GNU material together in any kind of archive,
> including ZIP,
> tar, cpio and SRPM. This adheres to the "mere aggregation"
> terminology of GPL
> and is very important for commercial use cases. Needless to say,
> binary RPMs
> are a different story.
>
Licensing is a whole different can of worm's, particularly when the
spec file has
different licensing than the tarballs and the patches.
Sure you can aggregate, but what purpose? The commercial issues
instantly
need to be figgered for all the elements of the aggregation.
73 de Jeff
-----------------------------------------------------------------------------
From: "Thomas Lotterer" <thomas+rpm5@lotterer.net>
Date: Sun, 27 May 2007 19:35:29 +0200
n3npq@mac.com wrote on 2007-05-27 19:05:
> While a SRPM has good capability to access metadata, unfortunately the
> actual scripts that are executed to perform a build are not in the metadata,
> but rather in the spec file in the cpio archive payload.
>
Ah - now we get to the point!
In fact, we are having problems at OpenPKG with information from the spec file
not making it into the SRPM and sometimes we'd even need to carry it forward
into the binary RPM. Causes us to have some ugly workarounds in place, the
worst I remember is a code fragment that somewhat emulates rpm2cpio but aborts
reading after it caught the spec file for performance reasons. Brrr.
I'd love to see *all* spec file content being available as SRPM metadata. This
would support creation from other sources than spec, too. We seriously thought
about putting the whole spec file into a single header string. But please carry
all the information forward to the binary RPM metadata. Much of it is waste but
some may be important, e.g. losing BuildPreReq is regularly a pain, making
build reproducibility inaccurate.
--
http://thomas.lotterer.net
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 13:46:32 -0400
On May 27, 2007, at 1:35 PM, Thomas Lotterer wrote:
> n3npq@mac.com wrote on 2007-05-27 19:05:
>> While a SRPM has good capability to access metadata, unfortunately
>> the
>> actual scripts that are executed to perform a build are not in the
>> metadata,
>> but rather in the spec file in the cpio archive payload.
>>
> Ah - now we get to the point!
>
> In fact, we are having problems at OpenPKG with information from
> the spec file
> not making it into the SRPM and sometimes we'd even need to carry
> it forward
> into the binary RPM. Causes us to have some ugly workarounds in
> place, the
> worst I remember is a code fragment that somewhat emulates rpm2cpio
> but aborts
> reading after it caught the spec file for performance reasons. Brrr.
>
Yep. The rpmbuild code has been neglected for years, and I'd kinda
like to see rpm outlive rpmbuild if you know what I mean.
> I'd love to see *all* spec file content being available as SRPM
> metadata. This
> would support creation from other sources than spec, too. We
> seriously thought
> about putting the whole spec file into a single header string. But
> please carry
> all the information forward to the binary RPM metadata. Much of it
> is waste but
> some may be important, e.g. losing BuildPreReq is regularly a pain,
> making
> build reproducibility inaccurate.
>
The core issue is that the spec file parser has side effects, some of
them quite
surprising. E.g. the current demonstration for a remote build system at
ftp://wraptastic.org/F
is fetching sources+patches as a side effect of the spec file parse,
kinda
very not good.
There's also the very well known issue that spec files, indeed, have
no grammar.
Which means that all development is driven by meeting expectations,
some of
which expectations are kinda whacko. ;-)
Start making a list of the info that you want to leak from srpm ->
binary rpm please.
73 de Jeff
-----------------------------------------------------------------------------
From: "Thomas Lotterer" <thomas+rpm5@lotterer.net>
Date: Sun, 27 May 2007 23:56:26 +0200
n3npq@mac.com wrote on 2007-05-27 19:46:
> Start making a list of the info that you want to leak from srpm ->
> binary rpm please.
>
Thanks for the word "Start". Disburdens me from elaborate engineering ;-)
To be honest, I'm thinking about what to omit. Maybe nothing.
spec/BuildPreReq -> SRPM -> RPM
spec/%(prep|build|install|...) -> SRPM -> RPM
This is meant to catch the "want" case.
In addition, I want to see the expanded values in the binary RPM, too.
This is meant to catch the "have" case.
I'm thinking about the ability to extend dumb installer software to
build+install+test (*) software by allowing it not only to find the latest and
greatest binaries (I've read the wnh doc) but also to enable it to build
missing binaries from sources. Currently this needs support from an external
database to provide information being lost during the spec -> SRPM transition.
Worse, after binaries have been built it might be desirable to reproduce them
(e.g. rebuilding them because a static library from another package has
changed). This is currently impossible. Even with an external database in place
the "have" values from the point in time of the build are missing entirely,
making decisions impossible.
Hypothetical example: assume gcc 4.2 was recently superceded by gcc 4.3 because
it was discovered that gcc 4.2 produces vulnerable code. The persons
responsible for security need to know which packages were build with the
vulnerable version and trigger a rebuild, even if absolutely nothing changed in
the code or spec of the affected package.
(*) I can imagine we'll see a TestPreReq and a %test some day ;-)
--
http://thomas.lotterer.net
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Mon, 28 May 2007 03:45:32 -0400
(let's see if I can move this discussion to rpm-devel)
On May 27, 2007, at 5:56 PM, Thomas Lotterer wrote:
> n3npq@mac.com wrote on 2007-05-27 19:46:
>> Start making a list of the info that you want to leak from srpm ->
>> binary rpm please.
>>
> Thanks for the word "Start". Disburdens me from elaborate
> engineering ;-)
> To be honest, I'm thinking about what to omit. Maybe nothing.
>
Running a look-aside cache of SRPM headers, each of which is named
by its SRPM pkgid, solves the original request in principle. Each binary
package carries the SRPM pkgid from which it was built, and that pkgid
can be used as a retrieval key on the look aside cache.
> spec/BuildPreReq -> SRPM -> RPM
> spec/%(prep|build|install|...) -> SRPM -> RPM
> This is meant to catch the "want" case.
>
> In addition, I want to see the expanded values in the binary RPM, too.
> This is meant to catch the "have" case.
>
> I'm thinking about the ability to extend dumb installer software to
> build+install+test (*) software by allowing it not only to find the
> latest and
> greatest binaries (I've read the wnh doc) but also to enable it to
> build
> missing binaries from sources. Currently this needs support from an
> external
> database to provide information being lost during the spec -> SRPM
> transition.
> Worse, after binaries have been built it might be desirable to
> reproduce them
> (e.g. rebuilding them because a static library from another package
> has
> changed). This is currently impossible. Even with an external
> database in place
> the "have" values from the point in time of the build are missing
> entirely,
> making decisions impossible.
>
> Hypothetical example: assume gcc 4.2 was recently superceded by gcc
> 4.3 because
> it was discovered that gcc 4.2 produces vulnerable code. The persons
> responsible for security need to know which packages were build
> with the
> vulnerable version and trigger a rebuild, even if absolutely
> nothing changed in
> the code or spec of the affected package.
>
OK, I better understand what you want now.
A build process goes through stages, each of which has inputs and
outputs.
E.g. one stage in the process has a static library SRPM as input, and
produces
a static library binary rpm as output.
Another stage in the process uses the output (the installed static
library) of a previous
stage as input, and produces another binary rpm as output.
You wish to be able to carry sufficient information through the
process stages input/output
so that process precursor stages can be restarted when necessary.
You have already realized that not only the inputs/outputs of the
process stages, but
also the tools used during each stage, are necessary information to
reproduce a
given process stage reliably. (That's my somewhat overly formal
translation of your
gcc 4.2/4.3 example).
I also point out that not only the tools used, but also how the tools
were invoked, is
relevant. Consider your gcc example and assume the vulnerability was
produce iff
-O5 were used.
Finally, I will carry the analysis ad absurdum to make a point: The
environment values of lm_sensors
for fan_control may be relevant as the root cause of the -O5
vulnerability in your gcc 4.2 example.
The point is that there is -- of necessity -- information loss in any
process flow.
For software tools, binaries can be marked not only with symbols, but
also with the tools
and even the arguments to the tools, that were used to produce the
binaries.
rpm (and packaging) is relevant to the process forensics only because
packages
are the common input/output format for the build/install stages, and
because certain
packaging transforms, like stripping binaries, are responsible for
the loss of the information
markers added by compiler tools. The additional issue is that rpm
currently does
not attempt to extract and preserve the markers in, say, header
metadata.
Is the above a fair statement of the problem you are trying to solve?
I can see several
possible concrete applications within rpm:
1) identifying executables and libraries that were not built with
$RPM_OPT_FLAGS
(or more generally, preserving what compiler flags were used)
2) tracking which executables have linked which libraries
statically (note that soname dependencies
are not sufficient for tracking certain flaws with dynamic
libraries, there are aliasing issues).
3) tracking API/ABI symbol usage more precisely, much like recent
2.6 linux kernel ABI's are tracked.
I do question whether the problem is worth solving. Sometimes brute
force techniques
like "make world" get the job done efficiently even if the build
takes a week. Do you
really need to efficiently restart just the necessary process stages
that are affected by, say,
a single static library flaw?
Disassembling every file looking for a a common flaw signature, and
then rebuilding
the package that contains the path, is another efficient technique
that does not require
detailed process flow information.
So I definitely question whether detailed process tracking
information belongs in binary packaging
headers. Its just not that hard to use a package identifier and build
process stage identifier
to save necessary information details outside of the common input/
output format used in build
process stages (i.e. *.rpm headers).
But I can think of possible solutions to your problem now that I
think I better understand
what you want.
73 de Jeff
-----------------------------------------------------------------------------
From: "Ralf S. Engelschall" <rse+rpm-team@rpm5.org>
Date: Sun, 27 May 2007 20:15:24 +0200
On Sun, May 27, 2007, Jeff Johnson wrote:
> [...]
> FWIW, my wraptastic.org logs show tarballs are considerably more popular
> than srpm's as a distribution format.
> [...]
Oh, this reminds me that in the past sometimes there were no tarball
distributed corresponding to the distributed SRPM. Just a reminder for
the future: for every version of RPM there always should be a tarball
with a _unique_ version in the filename. Currently there are SRPMs
which contain different snapshots tarballs of RPM, all under the same
filename.
| # ls -l
| total 367896
| -rw-r--r-- 1 rpm rpm 84750 May 21 15:03 popt-1.10.9-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 84875 May 23 01:48 popt-1.10.9-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 85207 May 26 03:50 popt-1.11-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 860335 May 21 15:03 rpm-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 14133128 May 23 20:57 rpm-4.5-0.1.src.rpm
| -rw-r--r-- 1 rpm rpm 860097 May 23 01:48 rpm-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 14119424 May 23 01:48 rpm-4.5-0.2.src.rpm
| -rw-r--r-- 1 rpm rpm 862260 May 26 03:50 rpm-4.5-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 14115330 May 26 03:50 rpm-4.5-0.3.src.rpm
| -rw-r--r-- 1 rpm rpm 389925 May 21 15:03 rpm-build-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 390023 May 23 01:48 rpm-build-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 390526 May 26 03:50 rpm-build-4.5-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 42528099 May 21 15:03 rpm-devel-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 42528235 May 23 01:48 rpm-devel-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 42582170 May 26 03:50 rpm-devel-4.5-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 4437012 May 21 15:03 rpm-libs-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 4435500 May 23 01:48 rpm-libs-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 4442437 May 26 03:50 rpm-libs-4.5-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 32823 May 21 15:03 rpm-perl-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 32937 May 23 01:48 rpm-perl-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 33128 May 26 03:50 rpm-perl-4.5-0.3.i386.rpm
| -rw-r--r-- 1 rpm rpm 160831 May 21 15:03 rpm-python-4.5-0.1.i386.rpm
| -rw-r--r-- 1 rpm rpm 160992 May 23 01:48 rpm-python-4.5-0.2.i386.rpm
| -rw-r--r-- 1 rpm rpm 161135 May 26 03:50 rpm-python-4.5-0.3.i386.rpm
| # openpkg rpm2cpio rpm-4.5-0.1.src.rpm | cpio -it 2>/dev/null | grep tar.gz
| rpm-4.5.tar.gz
| # openpkg rpm2cpio rpm-4.5-0.2.src.rpm | cpio -it 2>/dev/null | grep tar.gz
| rpm-4.5.tar.gz
| # openpkg rpm2cpio rpm-4.5-0.3.src.rpm | cpio -it 2>/dev/null | grep tar.gz
| rpm-4.5.tar.gz
You see what I mean? First, no tarball for download and even if
one extracts the tarball from the SRPM is had no unique filename.
This scheme for instance still doesn't allow downstream vendors
like OpenPKG to _easily_ and _cleanly_ use RPM 4.5. The approach OpenPKG here
has to do is instead of using an obvious and wished
SourceX: http://rpm5.org/files/rpm/rpm-4.5/rpm-4.5-0.3.tar.gz
we had to manually extract e.g. the tarball of rpm-4.5-0.3.src.rpm,
rename it from rpm-4.5.tar.gz to rpm-4.5-0.3.tar.gz, store it on in the
upstream vendor copy area (CPY) of our own FTP server and then use
SourceX: ftp://ftp.openpkg.org/sources/CPY/rpm/rpm-4.5-0.3.tar.gz
in our "openpkg.spec". Sure, this technically is no problem and already
had to be done for a few upstream vendors. But for about 1100 packages
and about 2500 upstream files (which are not in our CVS but fetched
directly from upstream when building the SRPM) this happens only for the
"less nice" vendors -- which fortunately is under 1%. So, if we want
to be nicer for our downstream users my recommendation is to
1. consistently provide tarballs of _ALL_ RPM versions (even for snapshots)
2. consistently name tarballs according to the RPM version and snapshot.
Ralf S. Engelschall
rse@engelschall.com
www.engelschall.com
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 14:25:28 -0400
On May 27, 2007, at 2:15 PM, Ralf S. Engelschall wrote:
> On Sun, May 27, 2007, Jeff Johnson wrote:
>
>> [...]
>> FWIW, my wraptastic.org logs show tarballs are considerably more
>> popular
>> than srpm's as a distribution format.
>> [...]
>
> Oh, this reminds me that in the past sometimes there were no tarball
> distributed corresponding to the distributed SRPM. Just a reminder for
> the future: for every version of RPM there always should be a tarball
> with a _unique_ version in the filename. Currently there are SRPMs
> which contain different snapshots tarballs of RPM, all under the same
> filename.
>
I don't distribute development tar balls for exactly the reason you
mention.
Use rpm2cpio.sh on the *.src.rpm if you want development tar balls, or
wait for the final tarball if you want released.
Adding goop to file names that aren't *.rpm is kinda pointless busy
work (for me).
> | # ls -l
> | total 367896
> | -rw-r--r-- 1 rpm rpm 84750 May 21 15:03
> popt-1.10.9-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 84875 May 23 01:48
> popt-1.10.9-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 85207 May 26 03:50 popt-1.11-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 860335 May 21 15:03 rpm-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 14133128 May 23 20:57 rpm-4.5-0.1.src.rpm
> | -rw-r--r-- 1 rpm rpm 860097 May 23 01:48 rpm-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 14119424 May 23 01:48 rpm-4.5-0.2.src.rpm
> | -rw-r--r-- 1 rpm rpm 862260 May 26 03:50 rpm-4.5-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 14115330 May 26 03:50 rpm-4.5-0.3.src.rpm
> | -rw-r--r-- 1 rpm rpm 389925 May 21 15:03 rpm-
> build-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 390023 May 23 01:48 rpm-
> build-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 390526 May 26 03:50 rpm-
> build-4.5-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 42528099 May 21 15:03 rpm-
> devel-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 42528235 May 23 01:48 rpm-
> devel-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 42582170 May 26 03:50 rpm-
> devel-4.5-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 4437012 May 21 15:03 rpm-
> libs-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 4435500 May 23 01:48 rpm-
> libs-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 4442437 May 26 03:50 rpm-
> libs-4.5-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 32823 May 21 15:03 rpm-
> perl-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 32937 May 23 01:48 rpm-
> perl-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 33128 May 26 03:50 rpm-
> perl-4.5-0.3.i386.rpm
> | -rw-r--r-- 1 rpm rpm 160831 May 21 15:03 rpm-
> python-4.5-0.1.i386.rpm
> | -rw-r--r-- 1 rpm rpm 160992 May 23 01:48 rpm-
> python-4.5-0.2.i386.rpm
> | -rw-r--r-- 1 rpm rpm 161135 May 26 03:50 rpm-
> python-4.5-0.3.i386.rpm
> | # openpkg rpm2cpio rpm-4.5-0.1.src.rpm | cpio -it 2>/dev/null |
> grep tar.gz
> | rpm-4.5.tar.gz
> | # openpkg rpm2cpio rpm-4.5-0.2.src.rpm | cpio -it 2>/dev/null |
> grep tar.gz
> | rpm-4.5.tar.gz
> | # openpkg rpm2cpio rpm-4.5-0.3.src.rpm | cpio -it 2>/dev/null |
> grep tar.gz
> | rpm-4.5.tar.gz
>
> You see what I mean? First, no tarball for download and even if
> one extracts the tarball from the SRPM is had no unique filename.
>
Yep. Those are development "Caveat emptor" snapshot releases.
> This scheme for instance still doesn't allow downstream vendors
> like OpenPKG to _easily_ and _cleanly_ use RPM 4.5. The approach
> OpenPKG here
> has to do is instead of using an obvious and wished
>
> SourceX: http://rpm5.org/files/rpm/rpm-4.5/rpm-4.5-0.3.tar.gz
>
> we had to manually extract e.g. the tarball of rpm-4.5-0.3.src.rpm,
> rename it from rpm-4.5.tar.gz to rpm-4.5-0.3.tar.gz, store it on in
> the
> upstream vendor copy area (CPY) of our own FTP server and then use
>
> SourceX: ftp://ftp.openpkg.org/sources/CPY/rpm/rpm-4.5-0.3.tar.gz
>
> in our "openpkg.spec". Sure, this technically is no problem and
> already
> had to be done for a few upstream vendors. But for about 1100 packages
> and about 2500 upstream files (which are not in our CVS but fetched
> directly from upstream when building the SRPM) this happens only
> for the
> "less nice" vendors -- which fortunately is under 1%. So, if we want
> to be nicer for our downstream users my recommendation is to
>
> 1. consistently provide tarballs of _ALL_ RPM versions (even for
> snapshots)
> 2. consistently name tarballs according to the RPM version and
> snapshot.
>
3) teach rpmbuild to unpack *.src.rpm's
Truly, this is rpm, not tar, development ;-)
73 de Jeff
-----------------------------------------------------------------------------
From: "Ralf S. Engelschall" <rse+rpm-team@rpm5.org>
Date: Sun, 27 May 2007 20:40:35 +0200
On Sun, May 27, 2007, Jeff Johnson wrote:
> [...]
> 3) teach rpmbuild to unpack *.src.rpm's
>
> Truly, this is rpm, not tar, development ;-)
Ok, accepted!
In OpenPKG the problem was that we had to _bootstrap_ (still no RPM or
GNU cpio available, but GNU tar), but we could use a rpm2cpio emulation
(http://www.openpkg.com/adventcalendar/2006/16) and build GNU cpio
earlier in our procedure. Ok, as I was too biased by OpenPKG's problems
here I hereby withdraw my request for snapshot tarballs as long as we
then at least _consistently_ provide tarballs for _ALL_ release version.
Ralf S. Engelschall
rse@engelschall.com
www.engelschall.com
-----------------------------------------------------------------------------
From: =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb@algonet.se>
Date: Sun, 27 May 2007 22:05:22 +0200
Ralf S. Engelschall wrote:
> Oh, this reminds me that in the past sometimes there were no tarball
> distributed corresponding to the distributed SRPM.
I recall this clearly*, and the downloading and extracting a couple
of hundred megs worth of source RPMs to see what was going on... :-)
> In OpenPKG the problem was that we had to _bootstrap_ (still no RPM or
> GNU cpio available, but GNU tar), but we could use a rpm2cpio emulation
> (http://www.openpkg.com/adventcalendar/2006/16) and build GNU cpio
> earlier in our procedure.
Same (rpm2cpio.pl) happened with the Mac OS X / Darwin builds of RPM.
Think there was even talk about a stand-alone C version at one point.
--anders
* See e.g. http://sourceforge.net/forum/forum.php?forum_id=411568
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 17:57:02 -0400
On May 27, 2007, at 2:40 PM, Ralf S. Engelschall wrote:
> On Sun, May 27, 2007, Jeff Johnson wrote:
>
>> [...]
>> 3) teach rpmbuild to unpack *.src.rpm's
>>
>> Truly, this is rpm, not tar, development ;-)
>
> Ok, accepted!
>
> In OpenPKG the problem was that we had to _bootstrap_ (still no RPM or
> GNU cpio available, but GNU tar), but we could use a rpm2cpio
> emulation
> (http://www.openpkg.com/adventcalendar/2006/16) and build GNU cpio
> earlier in our procedure. Ok, as I was too biased by OpenPKG's
> problems
> here I hereby withdraw my request for snapshot tarballs as long as we
> then at least _consistently_ provide tarballs for _ALL_ release
> version.
>
FYI: rpm-4.4.5 (iirc) and later supports tar payloads. Be careful
with hard links
within packages, cpio and tar handle differently, and I was too lazy
to do
the necessary PartialHardlinkSet computation.
I was very surprised to find that either of GNU tar/cpio can be
plugged onto
the back side of rpm2cpio with either format.
XAR format will break all of that uglix interoperability of course,
can't be helped.
On the positive side, there will be 2 tools (at least) that will be
able to read *.rpm format
natively.
73 de Jeff
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 09:03:52 -0400
On May 27, 2007, at 5:32 AM, Robert Scheck wrote:
> Hi folks,
>
> On Sat, 26 May 2007, Jeff Johnson wrote:
>> One very easy way to prepare drop-in parallel rpm packaging on
>> all platforms is to add "5" after all path collisions.
>
> the main reason for Jeff's mail was my purposal to import rpm5.org
> into
> ex-"Fedora Extras" as rpm5-4.5-0.x package with the possibility to
> co-exist
> with current "rpm" package. But currently I'm in doubt whether this
> a good
> idea, because - as Jeff mentioned - python seems tricky. And when
> importing
> rpm5 into Fedora I really would like to provide rpm5 libraries also
> to yum
> users without bigger modifications to yum until "wnh" is ready to
> replace
> the stuff. Note that yum/createrepo upstream is very arrogant and
> only sees
> rpm.org as rpm upstream. The other way simply would be to conflict
> or even
> obsolete rpm when the rpm5 package gets installed. Thoughts?
>
There's a couple things that can be done with rpm5-python:
1) First of all, since the underlying methods are the same, its quite
possible
to supply a patch for rpm that does
import rpm5 as rpm
2) The idea of a single rpm module isn't really best for python
objects. For
starters, there is no "rpm" object per-se. The preliminaries to split
rpm-python
into different object specific modules was all done years ago: already
there are sub-modules (if you will)
_rpm.hdr
_rpm.al
_rpm.ds
_rpm.fd
_rpm.fts
_rpm.fi
_rpm.mi
_rpm.mi
_rpm.ps
_rpm.rc
_rpm.te
_rpm.ts
_rpm.spec
that, if made precisely python modules, will not conflict with
existing vendor rpm-python.
What that means is that most of the python code could be easily
installed on different paths,
and an import can be used in a python module wrapper to load all the
sub-modules in order
to assemble the functional equivalent of an rpm module without
patching yum or anaconda
or other tools other than minimally.
Tricky, and my Python Fu is rusty, but appears to me to be feasible.
> Personally I'm not happy with a possible global renaming of rpm to
> rpm5, I
> would do this for the Fedora case to show especially the Red Hat
> folks that
> good things exist and maybe lucky users are switching over.
>
I have mixed feelings about renaming too. The overriding issue for me
is to
re-establish a distribution channel for rpm.
> Nevertheless "we" as rpm5.org shouldn't provide binaries, there are
> AFAIR
> too much variations at downstream and most vendors only accept
> packages
> built and/or signed-off by their packagers. Instead of providing
> binaries
> we should try to get the vendors on our boat and not to rpm.org -
> that IMHO
> would help much (especially when bigger distributions like openSUSE
> etc.
> are switching from the "Red Hat fork"). Last of it means of course,
> that
> we've to hear more what the distributions care about...but just my
> 2 ct.
>
Downstream variations need to be pulled upstream.
Inviting packagers from vendor-peculier build systems and actively
helping with the packaging would likely be welcomed, particularly
for rpm, which is a tough and confusing build.
I'm open to suggestions however. I may hack in /bin/rpm5 today just
to see how ugly. I have no problems reverting the change the day after
tomorrow, or next week. Getting a delivery system in place, not renaming
/bin/rpm -> /bin/rpm5, is my primary goal.
73 de Jeff
-----------------------------------------------------------------------------
From: "Thomas Lotterer" <thomas+rpm5@lotterer.net>
Date: Sun, 27 May 2007 17:38:15 +0200
robert@rpm5.org wrote on 2007-05-27 11:32:
> Nevertheless "we" as rpm5.org shouldn't provide binaries, [...]
>
If we can agree on this than it is worth mentioning in the PR.
Sounds like a clean plan which is a plus.
> Instead of providing binaries we should try to get the vendors on our boat
>
We may distribute their rpm* (experimental) packages through rpm5.org later, to
show vendor support or at least availability to everyone. Little technical
use, more a marketing advantage. Only useful once we have a good portion of
builds available.
--
http://thomas.lotterer.net
-----------------------------------------------------------------------------
From: Jeff Johnson <n3npq@mac.com>
Date: Sun, 27 May 2007 12:51:28 -0400
On May 27, 2007, at 11:38 AM, Thomas Lotterer wrote:
> robert@rpm5.org wrote on 2007-05-27 11:32:
>> Nevertheless "we" as rpm5.org shouldn't provide binaries, [...]
>>
> If we can agree on this than it is worth mentioning in the PR.
> Sounds like a clean plan which is a plus.
>
>> Instead of providing binaries we should try to get the vendors on
>> our boat
>>
> We may distribute their rpm* (experimental) packages through
> rpm5.org later, to
> show vendor support or at least availability to everyone. Little
> technical
> use, more a marketing advantage. Only useful once we have a good
> portion of
> builds available.
>
I don't disagree. Which is a polite way of saying I don't entirely
agree either. ;-)
The rpm-4_5 branch needs to demonstrate that rpm can be reliably and
compatibly
replaced everywhere. I knopw of no issue that cannot be fixed or
reverted.
That's a very different issue then whether rpm5.org distributes
binaries, or
whether vendors adopt rpm5.org, use rpm.org, or continue doing their own
thing.
Distributing binary builds for distro platforms is just to jump start
the process.
I *really* want to see rpm-4.5 in CentOS5 add-ons next month, not
wait for 3+ years
to see what RHEL6 does.
Does that clarify?
BTW, dries of rpmforge is already figgering how to package rpm-4.5 for
rpmforge, and I'm pretty sure that it will be OK to distribute
through OpenSUSE,
and Cooker, and ...
Getting there as fast as I can type.
-----------------------------------------------------------------------------
From: Robert Scheck <robert@rpm5.org>
Date: Sun, 27 May 2007 19:59:34 +0200
On Sun, 27 May 2007, Jeff Johnson wrote:
> I *really* want to see rpm-4.5 in CentOS5 add-ons next month, not
> wait for 3+ years to see what RHEL6 does.
>
> Does that clarify?
Well, when a usable rpm5 package gets accepted for ex-"Fedora Extras", I'm
able to branch for EPEL which means, we'll reach RHEL (including CentOS).
Maybe we can work together with the CentOS guys to make things more easy.
Oh and I would like to see rpm-4.5/rpm5 in any rpm-based distribution and
obsoleting the mostly-used 4.4.2 (or even older) anywhere, no question.
Greetings,
Robert
Received on Mon May 28 10:20:57 2007