RPM Community Forums

Mailing List Message of <rpm-devel>

Re: malloc returns NULL when using yum

From: Jeff Johnson <n3npq@mac.com>
Date: Sat 17 Jan 2009 - 15:03:38 CET
Message-id: <BD0CF3B2-303F-47DA-8CDF-F1A1264B26FA@mac.com>

On Jan 17, 2009, at 8:41 AM, Anders F Björklund wrote:

> Jeff Johnson wrote:
>> A similar issue was reported @rpm.org, yum needing 2.5Gb for an  
>> upgrade.
>> That problem  was diagnosed as memory fragmentation within  
>> rpmdbGrowIterator(),
>> mostly because its inconceivable to most that yum could somehow be  
>> flawed.
>> There's a patch that can do a power of 2 block allocation in  
>> rpmdbGrowIterator() @rpm.org,
>> not hard.
>> The failure is seen only with yum and python, and shows up with  
>> yum's peculier
>> usage of rpmlib, yum chooses to call rpmtsRun() a second time.
> Supposedly (have not verified it's the same issue) this also happens
> with Smart when using rpm.org and python, as reported for in:
> https://bugs.launchpad.net/smart/+bug/244943

Thanks for ptr.

Yes its very hard to verify a reproducer because
	1) I was testing PCRE when I saw the malloc flaw, since vanished.
	2) different rpmlib's are involved
	3) its a fragmentation, not a leak, problem. rpmlib approaches 0 leaks
	and afaik has no recurrent leaks (there are 1-time allocations that  
are not freed).

> The --stepped option mentioned causes Smart to do multiple commits
> (after splitting the ChangeSet), with each commit() doing a ts.run

Is smart calling ts.run() once or twice per-commit?

For hysterical up2date reasons, yum is likely detecting file conflicts
with a call to ts.run(), then calling ts.run() a 2nd time to
install/erase packages. The 2nd call is triggering the
fragmentation in the bug report.

There's likely some other code rearrangements in python, deleting
objects that assist in removing fragmentation. But that likely
cannot be done with yum, but I'm perfctly happy to help with smart.

Yes, I would expect the problem to show up with --stepped, which
has more persistence, and a better chance to cause fragmentation.
Deleting a rpmts to close/open an rpmdb per-stepped transaction
will free whatever memory happens to be in use by rpmlib, and
as long as there are <100 stepped transactions, close/open an rpmdb
won't hurt much (yum was doing open-get-close cycles on every access,
that too is/was very peculier).

Anyways, I'll try to get a --memstats option drilled into rpmlib.
There's also nothing wrong with the rpm.org power-of-2 block allocator  
that I believe there's a deeper issue causing the fragmentation
that will be buried if a block allocator is lashed on top.

73 de Jeff
Received on Sat Jan 17 15:03:42 2009
Driven by Jeff Johnson and the RPM project team.
Hosted by OpenPKG and Ralf S. Engelschall.
Powered by FreeBSD and OpenPKG.