I have a unix script to create a TAR which is invoked from Mainframe job.
Code:
#!/bin/kshecho "Start Time : " $(date +%m-%d.%H.%M.%S)echo "*************************"tdate=$(date +_%m%d_%H%M%S)cmdfil="/usr/lpp/web-data/mfg/nct/file-data/ftpcmd.dat"in_rec="/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat"tardirctry="/usr/lpp/web-data/mfg/nct/file-data/purgeDocs"cnt=1cat ${in_rec} | \while read namelistdocd /usr/lpp/web-data/mfg/nct/file-dataecho "File name value is : " $namelistif [[ $cnt -eq 1 ]] ; thentar cvf $tardirctry/nctill_arch.tar $namelistcnt=2elsetar rvf $tardirctry/nctill_arch.tar $namelistfidoneecho "done"echo "End Time : " $(date +%m-%d.%H.%M.%S
There are 19000 reords to be TAR and its taking around 14hrs for the same . the records are in diff formats like DP100001.txt,DP100002.JPG,DP100003.PDF....
IS there any way to optimize this script so that it takes less time ?
Running tar for each of the 19K files seems very inefficient. You might try something like this:
Code:
#!/usr/bin/env ksh
echo "Start Time : " $(date +%m-%d.%H.%M.%S)
echo "*************************"
tdate=$(date +_%m%d_%H%M%S)
cmdfil="/usr/lpp/web-data/mfg/nct/file-data/ftpcmd.dat" # is this needed?
in_rec="/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat"
tardirctry="/usr/lpp/web-data/mfg/nct/file-data/purgeDocs"
xargs <$in_rec tar -rvf $tardirctry/nctill_arch.tar
echo "done"
echo "End Time : " $(date +%m-%d.%H.%M.%S
Changes in bold. Note: there is NOT a missing pipe before the tar command.
Xargs will read the lines from the file and invoke tar for a group of files. If xargs is unable to fit all filenames on the command line, it will invoke tar multiple times (unavoidable, but far less than once per file).
The problem in that thread was a bit different (no files to tar) but the solution turns out to be similar to what you might want, so it's worth a read.
Additional thought: It still may take a bit of time to archive 19K files depending on the volume of data that is to be processed. I believe that this is as efficient as it can get unless you want to split up the files and generate multiple archive files concurrently. Concurrent processing still might be limited by I/O rates depending on location of your input.
The performance problem is because the "-r" parameter to "tar" asks for an append to an existing archive. The append is slow. The "xargs" idea in an earlier post certainly reduces the number of appends.
The script implies that all the files for "tar" are in this tree: /usr/lpp/web-data/mfg/nct/file-data
/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat
What is the criteria for a file being in this list? Are there any files in the tree which are not in the list? If you are just archiving the entire contents of the directory, the "tar" can be done in one pass.
The main aim is to delete data which are lesser than 2009 . I have some mainframe jobs that will create this data set with all 19k records and FTP to unix as a dat file . Inside the file nctillist.dat I have 19krecords ,some ex records St7000012.jpg,st75432.xls,dp25842.doc...... I need to read the file nctillist.dat and create a TAR file with all the records inside nctillist.dat.
I tried the xargs command but it's saying the destination file nctill_arch.tar : Edc5141i read only file system
---------- Post updated at 08:38 AM ---------- Previous update was at 08:38 AM ----------
The main aim is to delete data which are lesser than 2009 . I have some mainframe jobs that will create this data set with all 19k records and FTP to unix as a dat file . Inside the file nctillist.dat I have 19krecords ,some ex records St7000012.jpg,st75432.xls,dp25842.doc...... I need to read the file nctillist.dat and create a TAR file with all the records inside nctillist.dat.
I tried the xargs command but it's saying the destination file nctill_arch.tar : Edc5141i read only file system
hi all,
i have created a for loop, it looks like this -
#!/bin/bash
cd /mnt/local/data/tars
for tar in *
do
base=$(basename "$tar")
"$base" -tvf >> /mnt/local/data/logs/"$base".csv
done
but i get this error -
./tar_loop.sh: line 8:... (2 Replies)
Hi all. I'm hitting a problem creating a tar archive in one directory from files located in a different directory. It fails when I replace the absolute paths with variables in the script but works if I just run tar on the cmdln. E.g.
#!/bin/ksh
BASE=$PWD
STAGE=$BASE/stage
LOG=$BASE/log... (4 Replies)
I am trying to archive directories based on their last modified date. When I tar and compress the directory it makes copies of whats inside, I don't know how to fix this.
Here is my code.
#!/bin/bash
#AUTODRUNDISABLE
VERSION="0.2"
cd /desired/directory/to/archive
find . -type d -newermt... (3 Replies)
Hi,
How to create a .tar file in windows OS..
I need to ftp tat <filename>.tar file into AIX and untar it there and will use for futher actions..
Please help.. Thanks in advance.. (2 Replies)
Hello all.
I have a tar file that contains a number of files that are stored in different directories.
If I extract this tar file with -xvf , the directories get created.
Is there a way to extract all of the files into one directory without creating the directories stored in the tar file. (9 Replies)
Hi
I have done a search for this but couldn't find much on it.
I am creating a tar file with the command below
tar cvf /export/home/user/backup/*Will this is being created I have a job spooling to 5 texts files in the following directory /export/home/user/backup/STATS/
The tar files... (1 Reply)
hi,
How do i create a tar file of a directory excluding the links in that particular directory and its sub-directories.
The below command doesnt work for me.
tar -cvf abc.tar /dir1 --exclude"^l" (1 Reply)
Hi All,
I am not one of the super users / root for AIX 5.3 system.
There is a filesystem
Say
/DIR1 and its has several subdirs in it say SUBDIR1, SUBDIR2, SUBDIR3.
Can I create a tar file for all files under DIR1 and SUBDIR1, SUBDIR3. Excluding SIBDIR2?
Also how can I preserve... (2 Replies)
I received a tar file of a directory with 50,000 files in it. Is it possible to extract the files in the tar file without first creating the directory?
ie. Doing tar -xvf filename.tar extracts as follows:
x directory/file1.txt
x directory/file2.txt
.
.
.
I would like to avoid... (4 Replies)
Hi,
I am getting following error while running script:
tar: can't change directories to /home/xyz/script: No such file or directory
tar: compress: No such file or directory
tar: getts.tar same as archive file
Following is my script:
#!/bin/bash
echo "You want:"
echo "1. zip... (1 Reply)
