AIX

Dear All,

I am trying to backup Databases in my company which installed on AIX oracle 10g. From the start they have not used the archives to backup DB. Instead they use expdp utility to export the data in to dump file and rename the file extension to .tmp and ftp to another machine to compress the file. (Don't ask why they change extension to .tmp)

I am trying to short the process by zip the file before sending file to Storage (NAS). But when compress large database dump files (oracle dump, used ZIP,gzip commands) I get below error. File size more than 20GB.

zip error: Entry too big to split, read, or write (file exceeds Zip's 4GB uncompressed size limit)

I have read increase ulimit will help on this issue but my machine ulimit values are:

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

AIX version - 5.3

is there anyway i can compress the dump file without using pipe method?

gzip 1.2.4
Zip 2.32

Could you direct the export/dump to a pipe file instead? Create one with:-

Code:

mknod /tmp/my_pipe p

After this exists, start the following (in the background):-

Code:

dd if=/tmp/my_pipe | compress > /path/to/large/dump.tmp.Z &

You may well be able to substitute compress with gzip or other tools. You will want the flags that read and write standard input/output. I think it's -c for gzip, but haven't got it, so can't be sure.


After this is running, start the export/dump/whatever.




Robin

Thanks for the reply. Since the setup is very old I don't want to change much as not sure what are AIX configurations and what patches they applied (Work Around scripts). That's why i don't want to use pipe file system. It would be great if i can use GZIP or ZIP instead.

You might want to try a newer gzip-version instead of this (which is saying 1993). Go to Michael Perzls Freeware for AIX - gzip download and select:

Code:

May 30 2008 00:00       114504 gzip-1.3.12-1.aix5.1.ppc.rpm
Apr 22 2010 00:00       120293 gzip-1.4-1.aix5.1.ppc.rpm
Apr 03 2013 00:00       138092 gzip-1.5-1.aix5.1.ppc.rpm

I hope this helps.

bakunin

I'm not sure why patch levels would affect this. We've been running it on AIX 4.3.3 ( oh the shame) for ages and have migrated databases across the network from AIX 5.1 to RHEL 6.5 (export to pipe, dd across the network to a pipe, import from a pipe) It's just standard stuff.

The process also removes the need to store a huge intermediate file and then have space to compress it to along with the time saving of doing it all in one. On the migration to RHEL, we saved 23 hours over a process that was taking about 40.



If you really want to stick with a similar process to what you have and can't even upgrade gzip, perhaps you could split the file into smaller chunks, compress each, then concatenate them on the other side, so on the source server:-

Code:

split -a 5 -b 500m exp.dat exp.dat.split
for splitfile in exp.dat.split*
do
   gzip $splitfile        # Add whatever normal options you use
done

Then transfer all the exp.dat.split*zip files to the target server and:-

Code:

for splitfile in exp.dat.split*
do
   gunzip $splitfile        # Add whatever normal options you use
done

cat exp.dat.split* > exp.dat

..... and load your data in the normal way.

You may need to experiment with the values of -a & -b for the split so you get a managable number of files and of a suitable size to still allow gzip to work. Have a read of the split manual pages for more information.


Robin

Thanks Robin and Bakunin for helping me out. I will upgrade the ZIP and GZIP and see how it goes.

Robin as the system is not stable setup (overall) if some issue fired up they will think its the new setup (trying to save my self while fixing ). Any Known issues with split and concatenate in another machine? This sounds interesting as well.