RPM Community Forums

Mailing List Message of <rpm-users>

Re: I killed my RPMDB --- any rescue possible?

From: Jeff Johnson <n3npq@mac.com>
Date: Mon 13 Jun 2011 - 16:49:35 CEST
Message-id: <875BA37E-834A-4FC9-BCA1-800562065BF6@mac.com>

On Jun 13, 2011, at 9:58 AM, Eric MSP Veith wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Sunday 12 June 2011, Jeff Johnson <n3npq@mac.com> wrote:
>>> However, after using "rpm -r /my/chroot -Uvh ...", the RPMDB *outside*
>>> the chroot is corrupted. When running "rpm -qa", this is what I get:
>> 
>> So which rpmdb needs fixing? The rpm inside or outside the chroot?
> 
> Outside. The host system's RPMDB is broken (after rpm -r ... -Uvh ...). The 
> chroot's RPMDB is intact, but the package I wanted to install is missing, 
> not so surprisingly.  :-)
> 

OK.

>> And what versions of rpm are in use inside and outside the chroot?
> 
> 5.3.10 in both cases.
> 

OK.

>> Underlying what?
> 
> Sorry, wrong wording: The bdb version linked with the current rpm-5.3.10 
> install is 5.1.25. The RPMDB was originally created using bdb 4.7.something, 
> and after installing rpm-5.3.10 (with bdb-5.1.25), I immediateley ran 
> dbconvert.sh.
> 

OK. There are no significant format changes in Berkeley DB
that are NOT handled by db-utils, specifically db_dump  | db_load.

What is subtle is that one MUST choose the correct version specific
db_dump/db_load to upgrade (and downgrade). I'm not at all
sure that dbconvert.sh is properly implemented to Just Work with
	1) a outer <-> inner chroot "system"
	2) multiple rpmdb paths
	3) multiple database versions

The "safest" way to avoid problems is to do
	chroot /your/chroot /bin/bash
and just run dbconvert.sh within.


>> Clean-up and save what you have in /var/lib/rpm (...)
>> Create a new directory and copy what you have (...)
>> Check (as root) that Packages is at least mostly intact:
>> 	rpm -qavv
> 
> I did as advised, and rpm -qavv outputs this:
> 
> - ---%<---
> D: pool fd:     created size 208 limit -1 flags 0
> D: pool iob:    created size 20 limit -1 flags 0
> D: pool mire:   created size 84 limit -1 flags 0
> D: pool lua:    created size 32 limit -1 flags 0
> D: pool ts:     created size 720 limit -1 flags 0
> D: pool gi:     created size 92 limit -1 flags 0
> D: pool db:     created size 184 limit -1 flags 0
> D: pool dbi:    created size 284 limit -1 flags 0
> D: opening  db index       /var/lib/rpm/Packages thread:rdonly mode=0x0
> D: pool mi:     created size 88 limit -1 flags 0
> D: pool h:      created size 216 limit -1 flags 0
> libXfixes-4.0.3-1ev.i686
> error: rpmdb: header #33554432 cannot be loaded -- skipping.
> (none)-(none)-(none)
> ==> munmap(0xb65d0000[180811702]) error(22): Invalid argument
> error: rpmdb: header #67108864 cannot be loaded -- skipping.
> error: rpmdb: header #83886080 cannot be loaded -- skipping.
> error: rpmdb: header #100663296 cannot be loaded -- skipping.
> error: rpmdb: header #117440512 cannot be loaded -- skipping.
> error: rpmdb: header #134217728 cannot be loaded -- skipping.
> error: rpmdb: header #150994944 cannot be loaded -- skipping.
> error: rpmdb: header #167772160 cannot be loaded -- skipping.
> error: rpmdb: header #201326592 cannot be loaded -- skipping.
> error: rpmdb: header #218103808 cannot be loaded -- skipping.
> error: rpmdb: header #234881024 cannot be loaded -- skipping.
> error: rpmdb: header #268435456 cannot be loaded -- skipping.
> error: rpmdb: header #285212672 cannot be loaded -- skipping.
> error: rpmdb: header #301989888 cannot be loaded -- skipping.
> (none)-(none)-(none)
> (none)-(none)-(none)
> (none)-(none)-(none)
> error: rpmdb: header #385875968 cannot be loaded -- skipping.
> (none)-(none)-(none)
> Segmentation fault
> - --->%---
> 

OK. You have a damaged database for whatever reason. All of those
"header #12345678" are too big so you likely have NOT succeeded
in byte swabbing the header instances (i.e. what dbconvert.sh is doing).

(alternatively) You have run dbconvert.sh an even number of times
and so two applications of dbconvert.sh has undone the byte swabbing
originally intended.

You can try reading the rpmdb with pre-rpm-5.3 to assess how much data
is savable.

Doing rpm --rebuilddb (possibly with --nosignature --nodigest --nohdrchk disablers
to get past the segfault) WILL discard damaged headers that you can then replace later.

While doing --rebuilddb, make sure you do either
	cd /var/lib/rpm
	rm -f __db*
or
	cd /var/lib/rpm
	db_recover -v
(note no -e passed to db_recover to remove the __db cache)

A damaged cache in __db* files can/will screw a --rebuilddb worser.
	

>> 
>> Re-create the indices (and check that dbconvert.sh swapped the bytes):
>> 	rpm --rebuilddb --vv
> 
> - ---%<---
> D: pool fd:     created size 208 limit -1 flags 0
> D: pool iob:    created size 20 limit -1 flags 0
> D: pool mire:   created size 84 limit -1 flags 0
> D: pool lua:    created size 32 limit -1 flags 0
> D: pool ts:     created size 720 limit -1 flags 0
> D: pool db:     created size 184 limit -1 flags 0
> D: pool dbi:    created size 284 limit -1 flags 0
> D: rpmdb: cpus 2 physmem 3026Mb
> D: opening  db environment /var/lib/rpm/Packages thread:lock:log:mpool:txn
> rpmdb: Thread/process 18824/3062732608 failed: Thread died in Berkeley DB 
> library
> error: db_init:db3.c:1020: dbenv->failchk(-30973): DB_RUNRECOVERY: Fatal 
> error, run database recovery
> Re-opening dbenv with DB_RECOVER ...
> D: opening  db environment /var/lib/rpm/Packages 
> thread:lock:log:mpool:txn:recover
> Finding last valid log LSN: file: 1 offset 9104
> recovery 28% completeRecovery starting from [1][162]
> recovery 95% completeRecovery complete at Mon Jun 13 13:54:23 2011
> Maximum transaction ID 80000003 Recovery checkpoint [1][9296]
> .
> recovery succeeded.
> D: opening  db index       /var/lib/rpm/Packages create:thread:auto_commit 
> mode=0x2
> D: opening  db index       /var/lib/rpm/Name create:thread:auto_commit 
> mode=0x2
> D: pool h:      created size 216 limit -1 flags 0
> error: db3: header #33554432 cannot be loaded -- skipping.
> error: db3: header #67108864 cannot be loaded -- skipping.
> error: db3: header #83886080 cannot be loaded -- skipping.
> error: db3: header #100663296 cannot be loaded -- skipping.
> error: db3: header #117440512 cannot be loaded -- skipping.
> error: db3: header #134217728 cannot be loaded -- skipping.
> error: db3: header #150994944 cannot be loaded -- skipping.
> error: db3: header #167772160 cannot be loaded -- skipping.
> error: db3: header #201326592 cannot be loaded -- skipping.
> error: db3: header #218103808 cannot be loaded -- skipping.
> error: db3: header #234881024 cannot be loaded -- skipping.
> error: db3: header #268435456 cannot be loaded -- skipping.
> error: db3: header #285212672 cannot be loaded -- skipping.
> error: db3: header #301989888 cannot be loaded -- skipping.
> error: db3: header #385875968 cannot be loaded -- skipping.
> error: db3: header #486539264 cannot be loaded -- skipping.
> rpm: header.c:1050: headerLoad: Assertion `(rpmint32_t)rdl >= 0' failed.
> Aborted
> - --->%---
> 

And this is fully automated database recovery that makes the database
accessible but can NOT fix damaged data.

> Do you have any hint/idea why "rpm -r /my/chroot -Uvh /some/package.rpm" 
> kills the RPMDB *outside* of the chroot? Did I miss anything here?
> 

I don't know what operations were run, and with what utilities, so
I cannot hazard a guess.

If you believe that rpm is confused about outer <-> inner chroot(2)
behavior, then you need to run
	strace -e open -o /tmp/xyzzy -f rpm 
and examine the paths that were opened.

In _ALL_ cases, doing
	chroot /your/chroot /bin/bash
and avoiding -r "works".

hth

73 de Jeff



  • application/pkcs7-signature attachment: smime.p7s
Received on Mon Jun 13 16:50:07 2011
Driven by Jeff Johnson and the RPM project team.
Hosted by OpenPKG and Ralf S. Engelschall.
Powered by FreeBSD and OpenPKG.