Creating a TAR


 
Thread Tools Search this Thread
# 1  
Creating a TAR

Hello ,

I have a unix script to create a TAR which is invoked from Mainframe job.
Code:
#!/bin/ksh echo "Start Time : " $(date +%m-%d.%H.%M.%S) echo "*************************" tdate=$(date +_%m%d_%H%M%S) cmdfil="/usr/lpp/web-data/mfg/nct/file-data/ftpcmd.dat" in_rec="/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat" tardirctry="/usr/lpp/web-data/mfg/nct/file-data/purgeDocs" cnt=1 cat ${in_rec} | \ while read namelist do cd /usr/lpp/web-data/mfg/nct/file-data echo "File name value is : " $namelist if [[ $cnt -eq 1 ]] ; then tar cvf $tardirctry/nctill_arch.tar $namelist cnt=2 else tar rvf $tardirctry/nctill_arch.tar $namelist fi done echo "done" echo "End Time : " $(date +%m-%d.%H.%M.%S

There are 19000 reords to be TAR and its taking around 14hrs for the same . the records are in diff formats like DP100001.txt,DP100002.JPG,DP100003.PDF....

IS there any way to optimize this script so that it takes less time ?

Can anyone suggest me as im new to UNIX

Thanks
# 2  
how about the file sizes ? ( all 19000 files )
# 3  
Running tar for each of the 19K files seems very inefficient. You might try something like this:

Code:
#!/usr/bin/env ksh
echo "Start Time : " $(date +%m-%d.%H.%M.%S)
echo "*************************"
tdate=$(date +_%m%d_%H%M%S)
cmdfil="/usr/lpp/web-data/mfg/nct/file-data/ftpcmd.dat"   # is this needed?
in_rec="/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat"
tardirctry="/usr/lpp/web-data/mfg/nct/file-data/purgeDocs"


xargs <$in_rec tar -rvf $tardirctry/nctill_arch.tar    


echo "done"
echo "End Time : " $(date +%m-%d.%H.%M.%S

Changes in bold. Note: there is NOT a missing pipe before the tar command.

Xargs will read the lines from the file and invoke tar for a group of files. If xargs is unable to fit all filenames on the command line, it will invoke tar multiple times (unavoidable, but far less than once per file).

There is a related discussion here:
https://www.unix.com/shell-programmin...y-archive.html

The problem in that thread was a bit different (no files to tar) but the solution turns out to be similar to what you might want, so it's worth a read.

Additional thought: It still may take a bit of time to archive 19K files depending on the volume of data that is to be processed. I believe that this is as efficient as it can get unless you want to split up the files and generate multiple archive files concurrently. Concurrent processing still might be limited by I/O rates depending on location of your input.

Last edited by agama; 02-02-2012 at 12:56 AM..
# 4  
Thanks for your suggestion Agama , I tried giving xargs but I'm getting an error saying in_rec not found ,

Xargs $in_rec tar -rvf $tardirctry/nctill_arc.tar
# 5  
Code:
xargs <$in_rec tar -rvf $tardirctry/nctill_arch.tar

# 6  
The performance problem is because the "-r" parameter to "tar" asks for an append to an existing archive. The append is slow. The "xargs" idea in an earlier post certainly reduces the number of appends.

The script implies that all the files for "tar" are in this tree:
/usr/lpp/web-data/mfg/nct/file-data

/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat
What is the criteria for a file being in this list? Are there any files in the tree which are not in the list? If you are just archiving the entire contents of the directory, the "tar" can be done in one pass.
# 7  
The main aim is to delete data which are lesser than 2009 . I have some mainframe jobs that will create this data set with all 19k records and FTP to unix as a dat file . Inside the file nctillist.dat I have 19krecords ,some ex records St7000012.jpg,st75432.xls,dp25842.doc...... I need to read the file nctillist.dat and create a TAR file with all the records inside nctillist.dat.

I tried the xargs command but it's saying the destination file nctill_arch.tar : Edc5141i read only file system

---------- Post updated at 08:38 AM ---------- Previous update was at 08:38 AM ----------

The main aim is to delete data which are lesser than 2009 . I have some mainframe jobs that will create this data set with all 19k records and FTP to unix as a dat file . Inside the file nctillist.dat I have 19krecords ,some ex records St7000012.jpg,st75432.xls,dp25842.doc...... I need to read the file nctillist.dat and create a TAR file with all the records inside nctillist.dat.

I tried the xargs command but it's saying the destination file nctill_arch.tar : Edc5141i read only file system
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Programming

Creating for loop for tar -tvf doesnt work

hi all, i have created a for loop, it looks like this - #!/bin/bash cd /mnt/local/data/tars for tar in * do base=$(basename "$tar") "$base" -tvf >> /mnt/local/data/logs/"$base".csv done but i get this error - ./tar_loop.sh: line 8:... (2 Replies)
Discussion started by: robertkwild
2 Replies

2. Shell Programming and Scripting

Problem creating a tar ball in different directories

Hi all. I'm hitting a problem creating a tar archive in one directory from files located in a different directory. It fails when I replace the absolute paths with variables in the script but works if I just run tar on the cmdln. E.g. #!/bin/ksh BASE=$PWD STAGE=$BASE/stage LOG=$BASE/log... (4 Replies)
Discussion started by: user052009
4 Replies

3. Shell Programming and Scripting

Tar creating copies of files

I am trying to archive directories based on their last modified date. When I tar and compress the directory it makes copies of whats inside, I don't know how to fix this. Here is my code. #!/bin/bash #AUTODRUNDISABLE VERSION="0.2" cd /desired/directory/to/archive find . -type d -newermt... (3 Replies)
Discussion started by: jrymer
3 Replies

4. Windows & DOS: Issues & Discussions

[SOLVED] Creating .tar file

Hi, How to create a .tar file in windows OS.. I need to ftp tat <filename>.tar file into AIX and untar it there and will use for futher actions.. Please help.. Thanks in advance.. (2 Replies)
Discussion started by: Priya Amaresh
2 Replies

5. Solaris

How to extract files from a tar file without creating the directories?

Hello all. I have a tar file that contains a number of files that are stored in different directories. If I extract this tar file with -xvf , the directories get created. Is there a way to extract all of the files into one directory without creating the directories stored in the tar file. (9 Replies)
Discussion started by: gkb
9 Replies

6. UNIX for Dummies Questions & Answers

Creating a Tar file while files are spooling

Hi I have done a search for this but couldn't find much on it. I am creating a tar file with the command below tar cvf /export/home/user/backup/*Will this is being created I have a job spooling to 5 texts files in the following directory /export/home/user/backup/STATS/ The tar files... (1 Reply)
Discussion started by: sgarvan
1 Replies

7. Shell Programming and Scripting

Creating tar excluding links

hi, How do i create a tar file of a directory excluding the links in that particular directory and its sub-directories. The below command doesnt work for me. tar -cvf abc.tar /dir1 --exclude"^l" (1 Reply)
Discussion started by: yesmani
1 Replies

8. UNIX for Dummies Questions & Answers

Creating tar file for subdirs, excluding one and preserving user info

Hi All, I am not one of the super users / root for AIX 5.3 system. There is a filesystem Say /DIR1 and its has several subdirs in it say SUBDIR1, SUBDIR2, SUBDIR3. Can I create a tar file for all files under DIR1 and SUBDIR1, SUBDIR3. Excluding SIBDIR2? Also how can I preserve... (2 Replies)
Discussion started by: Hangman2
2 Replies

9. UNIX for Dummies Questions & Answers

extract tar files without creating directory

I received a tar file of a directory with 50,000 files in it. Is it possible to extract the files in the tar file without first creating the directory? ie. Doing tar -xvf filename.tar extracts as follows: x directory/file1.txt x directory/file2.txt . . . I would like to avoid... (4 Replies)
Discussion started by: here2learn
4 Replies

10. Shell Programming and Scripting

Error while creating .tar file using script

Hi, I am getting following error while running script: tar: can't change directories to /home/xyz/script: No such file or directory tar: compress: No such file or directory tar: getts.tar same as archive file Following is my script: #!/bin/bash echo "You want:" echo "1. zip... (1 Reply)
Discussion started by: Rakesh Bhat
1 Replies

Featured Tech Videos