Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Creating a TAR
Top Forums UNIX for Dummies Questions & Answers Creating a TAR Post 302595019 by agama on Wednesday 1st of February 2012 11:51:07 PM
Old 02-02-2012
Running tar for each of the 19K files seems very inefficient. You might try something like this:

Code:
#!/usr/bin/env ksh
echo "Start Time : " $(date +%m-%d.%H.%M.%S)
echo "*************************"
tdate=$(date +_%m%d_%H%M%S)
cmdfil="/usr/lpp/web-data/mfg/nct/file-data/ftpcmd.dat"   # is this needed?
in_rec="/usr/lpp/web-data/mfg/nct/file-data/nctilllist.dat"
tardirctry="/usr/lpp/web-data/mfg/nct/file-data/purgeDocs"


xargs <$in_rec tar -rvf $tardirctry/nctill_arch.tar    


echo "done"
echo "End Time : " $(date +%m-%d.%H.%M.%S

Changes in bold. Note: there is NOT a missing pipe before the tar command.

Xargs will read the lines from the file and invoke tar for a group of files. If xargs is unable to fit all filenames on the command line, it will invoke tar multiple times (unavoidable, but far less than once per file).

There is a related discussion here:
https://www.unix.com/shell-programmin...y-archive.html

The problem in that thread was a bit different (no files to tar) but the solution turns out to be similar to what you might want, so it's worth a read.

Additional thought: It still may take a bit of time to archive 19K files depending on the volume of data that is to be processed. I believe that this is as efficient as it can get unless you want to split up the files and generate multiple archive files concurrently. Concurrent processing still might be limited by I/O rates depending on location of your input.
Last edited by agama; 02-02-2012 at 12:56 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Error while creating .tar file using script

Hi, I am getting following error while running script: tar: can't change directories to /home/xyz/script: No such file or directory tar: compress: No such file or directory tar: getts.tar same as archive file Following is my script: #!/bin/bash echo "You want:" echo "1. zip... (1 Reply)
Discussion started by: Rakesh Bhat
1 Replies

2. UNIX for Dummies Questions & Answers

extract tar files without creating directory

I received a tar file of a directory with 50,000 files in it. Is it possible to extract the files in the tar file without first creating the directory? ie. Doing tar -xvf filename.tar extracts as follows: x directory/file1.txt x directory/file2.txt . . . I would like to avoid... (4 Replies)
Discussion started by: here2learn
4 Replies

3. UNIX for Dummies Questions & Answers

Creating tar file for subdirs, excluding one and preserving user info

Hi All, I am not one of the super users / root for AIX 5.3 system. There is a filesystem Say /DIR1 and its has several subdirs in it say SUBDIR1, SUBDIR2, SUBDIR3. Can I create a tar file for all files under DIR1 and SUBDIR1, SUBDIR3. Excluding SIBDIR2? Also how can I preserve... (2 Replies)
Discussion started by: Hangman2
2 Replies

4. Shell Programming and Scripting

Creating tar excluding links

hi, How do i create a tar file of a directory excluding the links in that particular directory and its sub-directories. The below command doesnt work for me. tar -cvf abc.tar /dir1 --exclude"^l" (1 Reply)
Discussion started by: yesmani
1 Replies

5. UNIX for Dummies Questions & Answers

Creating a Tar file while files are spooling

Hi I have done a search for this but couldn't find much on it. I am creating a tar file with the command below tar cvf /export/home/user/backup/*Will this is being created I have a job spooling to 5 texts files in the following directory /export/home/user/backup/STATS/ The tar files... (1 Reply)
Discussion started by: sgarvan
1 Replies

6. Solaris

How to extract files from a tar file without creating the directories?

Hello all. I have a tar file that contains a number of files that are stored in different directories. If I extract this tar file with -xvf , the directories get created. Is there a way to extract all of the files into one directory without creating the directories stored in the tar file. (9 Replies)
Discussion started by: gkb
9 Replies

7. Windows & DOS: Issues & Discussions

[SOLVED] Creating .tar file

Hi, How to create a .tar file in windows OS.. I need to ftp tat <filename>.tar file into AIX and untar it there and will use for futher actions.. Please help.. Thanks in advance.. (2 Replies)
Discussion started by: Priya Amaresh
2 Replies

8. Shell Programming and Scripting

Tar creating copies of files

I am trying to archive directories based on their last modified date. When I tar and compress the directory it makes copies of whats inside, I don't know how to fix this. Here is my code. #!/bin/bash #AUTODRUNDISABLE VERSION="0.2" cd /desired/directory/to/archive find . -type d -newermt... (3 Replies)
Discussion started by: jrymer
3 Replies

9. Shell Programming and Scripting

Problem creating a tar ball in different directories

Hi all. I'm hitting a problem creating a tar archive in one directory from files located in a different directory. It fails when I replace the absolute paths with variables in the script but works if I just run tar on the cmdln. E.g. #!/bin/ksh BASE=$PWD STAGE=$BASE/stage LOG=$BASE/log... (4 Replies)
Discussion started by: user052009
4 Replies

10. Programming

Creating for loop for tar -tvf doesnt work

hi all, i have created a for loop, it looks like this - #!/bin/bash cd /mnt/local/data/tars for tar in * do base=$(basename "$tar") "$base" -tvf >> /mnt/local/data/logs/"$base".csv done but i get this error - ./tar_loop.sh: line 8:... (2 Replies)
Discussion started by: robertkwild
2 Replies

LEARN ABOUT CENTOS

th_ischr

th_get_pathname(3)						  C Library Calls						th_get_pathname(3)

NAME

       th_get_pathname,   th_get_uid,	th_get_gid,   th_get_mode,   th_get_crc,   th_get_size,  th_get_mtime,	th_get_devmajor,  th_get_devminor,
       th_get_linkname - extract individual fields of a tar header

       TH_ISREG, TH_ISLNK, TH_ISSYM, TH_ISCHR, TH_ISBLK, TH_ISDIR, TH_ISFIFO - determine what kind of file a tar header refers to

       TH_ISLONGNAME, TH_ISLONGLINK - determine whether the GNU extensions are in use

SYNOPSIS

       #include <libtar.h>

       char *th_get_linkname(TAR *t);

       char *th_get_pathname(TAR *t);

       mode_t th_get_mode(TAR *t);

       uid_t th_get_uid(TAR *t);

       gid_t th_get_gid(TAR *t);

       int th_get_crc(TAR *t);

       off_t th_get_size(TAR *t);

       time_t th_get_mtime(TAR *t);

       major_t th_get_devmajor(TAR *t);

       minor_t th_get_devminor(TAR *t);

       int TH_ISREG(TAR *t);

       int TH_ISLNK(TAR *t);

       int TH_ISSYM(TAR *t);

       int TH_ISCHR(TAR *t);

       int TH_ISBLK(TAR *t);

       int TH_ISDIR(TAR *t);

       int TH_ISFIFO(TAR *t);

       int TH_ISLONGNAME(TAR *t);

       int TH_ISLONGLINK(TAR *t);

VERSION

       This man page documents version 1.2 of libtar.

DESCRIPTION

       The th_get_*() functions extract individual fields from the current tar header associated with the TAR handle t.

       The TH_IS*() macros are used to evaluate what kind of file is pointed to by the current tar header associated with the TAR handle t.

       The TH_ISLONGNAME() and TH_ISLONGLINK() macros evaluate whether or not the GNU extensions are used by the  current  tar	header	associated
       with the TAR handle t.  This is only relevant if the TAR_GNU option was used when tar_open() was called.

SEE ALSO

       tar_open(3)

University of Illinois						     Jan 2001							th_get_pathname(3)
All times are GMT -4. The time now is 02:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy