Cpio - input files (from list) are stored in different order inside cpio archive - why?


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Cpio - input files (from list) are stored in different order inside cpio archive - why?
# 1  
Old 03-01-2018
Sun Cpio - input files (from list) are stored in different order inside cpio archive - why?

Due to budget constraints I have to reinvent an Enterprise backup system in a SPARC (sun4v) Solaris estate (10 & 11). (yep - reinvent wheel, fun but time consuming. Is this wise?! Smilie )

For each filesystem of interest, to try to capture a 'catalog' at the front of each cpio archive (for an easy scripted restore system that I will write later), I touch a file .${DIR}/.${OFILE}.fullscan that will hold a full filesystem scan. Then I run this to populate the first 'record' in the catalog file, so that the catalog file always sits at the head of the archive:
Code:
find .${DIR}/.${OFILE}.* -type f -mtime -1 -ls > .${DIR}/${OFILE}.fullscan

This populates our file list:
Code:
find .${DIR} -xdev -local -ls >> .${DIR}/.${OFILE}.fullscan

This then runs the actual cpio operation:
Code:
awk '{$1=$2=$3=$4=$5=$6=$7=$8=$9=$10=""; print $0}' .${DIR}/.${OFILE}.fullscan|\
  cut -c11- |cpio -oc 2>>/dev/null|gzip -qc1 ->${OUTFILE}

9 times out of 10 my ${OFILE}.fullscan appears in the first few files in the cpio archive. Occasionally it's a few files 'lower down' but always in the first 20. So good so far.

Today, on a Solaris 11.2 system I found the fullscan file over 1000 files into one of the cpio archives and another one over 6400 files into the archive. (In another they appeared at the end but I'm still checking that's not a 'code' issue!) Smilie Why?! Help!

I checked the text file content to make sure the top record was as expected for the examples that put the fullscan file much farther down the archive.

My worst case scenario is a 57M file filesystem (yep - source code repo) which generates a 9.5GB 'fullscan' file (over 11 hours) and due to other bits and pieces I need to do, I really, really don't want my catalogs appearing half way through that one. (This supersized filesystem backup will inevitably be broken up into smaller tasks but for now I'm just asking.)

Is this a multithreading effect? I don't believe awk or cut would reorder the list and so cpio would receive ${OFILE}.fullscan as the first argument. The file in question is being read but that wouldn't generate an exclusive lock to prevent access or anything like that from another read process. Note: for what it's worth these 2 anomalies occurred in /var of a guest Solaris zone that's visible from the global zone.

Can anyone think of a way to:
  1. Assuming there isn't a trivial answer to this - debug this easily for an explanation? (Bear in mind it's intermittent.)
  2. Workaround this? I want the catalog to always be quickly and easly accessed from (the top of?) many 100GB+ gzipped cpio archives! How can we persuade cpio to load files into the archive in exactly the order its file list is fed to it?
Your thoughts much appreciated.
Alex
# 2  
Old 03-02-2018
I think your problem is that find takes the order in the directory.
In a simple file system initialy the order in the directory defaults to the order in which files are created. But if files are deleted, a new file can take the position of a deleted entry.
Not to speak of more complex file systems that use a hash table.

What is your file system type?
Perhaps you do not need to reinvent the wheel. There is ufsdump/ufsrestore for the ufs file system, and certainly another method for a zfs file system.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Cpio archive help

Hi I'm new to the forum and looking for some help with cpio archive creation. I have a bunch of directories that I need to make into a cpio archive. The problem I'm having is that when I input the commands I get the cpio archive but it creates a duplicate of the archive inside the archive... (4 Replies)
Discussion started by: reeves1985
4 Replies

2. Shell Programming and Scripting

Cpio all *.txt-files out of folders to just one directory

I need a hint for reading manpage (I did rtfm really) of cpio to do this task as in the headline described. I want to put all files of a certain type, lets say all *.txt files or any other format. Spread in more than hundreds of subdirectories in one directory I would like to select them and just... (3 Replies)
Discussion started by: 1in10
3 Replies

3. HP-UX

cpio deleting source files. Can't figure out why!

So I ran into a strange problem the other day when using cpio. I'm not really sure this needed to be in an HPUX specific forum, but wasn't sure where else to put it. I'm running HPUX 11.11 PARISC. I was using cpio to copy a directory and its entire contents from one server, we'll call it... (7 Replies)
Discussion started by: paqman
7 Replies

4. Shell Programming and Scripting

Appending a CPIO to an existing archive

I created a CPIO archive I wanted to add addition data to it but am having issues: -rw-r--r-- 1 test test 629295104 2011-10-28 12:41 /home/test/Downloads/test.cpio I tried: sudo find /tmp -depth | cpio -oAO /home/test/Downloads/test.cpio cpio: premature end of file and (1 Reply)
Discussion started by: metallica1973
1 Replies

5. UNIX for Dummies Questions & Answers

Unable to restore cpio archive to a directory

Hello Every one, I want to back up all passwd files to /xyz/passfiles.cpio and Then restore them to /abc directory. Here is what I wrote: find / -name passwd | cpio -oc > /tmp/passwd.cpio and to restore cd abc cpio -ium < /tmp/passwd.cpio I can not find the files restored to /abc... (2 Replies)
Discussion started by: drdigital_m
2 Replies

6. HP-UX

cpio.cat.z files????

what are cpio.cat.z files??? Can I delete them??? (3 Replies)
Discussion started by: ldaliosmane
3 Replies

7. Red Hat

Method to Unpack cpio files

Hi all, I want to unpack some files .Files and their sizes are: 1. Linux9i_Disk1.cpio -- 500m 2. Linux9i_Disk2.cpio--- 600m 3.Linux9i_Disk3.cpio---- 250m I used cpio -idmv Linux9i_Disk1.cpio command to unpack the files. But Its taking more time to unpack the files.What could be the... (2 Replies)
Discussion started by: William1482
2 Replies

8. Shell Programming and Scripting

cpio - files > 2gb

Hi, Currently a backup script copies compressed files to tape using cpio command (on AIX 5.2). Recently we've had a compressed file which has gone over 2 GB in size resulting in an error while copying this file onto the tape using cpio. Any suggestions on relevant workarounds would be much... (2 Replies)
Discussion started by: dnicky
2 Replies

9. AIX

cpio - files > 2 GB

Hi, Currently a backup script copies compressed files to tape using cpio command (on AIX 5.2). Recently we've had a compressed file which has gone over 2 GB in size resulting in an error while copying this file onto the tape using cpio. Any suggestions on relevant workarounds would be much... (0 Replies)
Discussion started by: dnicky
0 Replies

10. UNIX for Dummies Questions & Answers

Selective restore from a cpio tape archive

Hi, I use following command to restore data from my cpio tape archive: $cpio -icvd < /dev/rct0 But this'll restore all tape contents to the current path, what if I want only selected files from the tape, suppose I want /home/compdir/home2/Rev83/data/PL/01/*.* files to be restored... (8 Replies)
Discussion started by: tayyabq8
8 Replies
Login or Register to Ask a Question