Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Massive Copy With Base Directory

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Massive Copy With Base Directory
# 1  
Old 07-24-2012
Massive Copy With Base Directory
I have a script that I am using to copy around 40-70k files to a NFS NAS.

I have posted my code below in hopes that someone can help me figure out a faster way of achieving this.

At the end of the script i need to have all the files in the list, copied over to the nas with source directory structure intact.

Code:
for FILENAME in `$CAT $TEMPFILE`
do
BASEFILENAME=`$BASENAME $FILENAME`
BASEDIR=`$DIRNAME $FILENAME`
if [[ $BASEFILENAME != $SCRIPTNAME ]]
then
[[ -d $DESTDIR/$BASEDIR ]] || $MKDIR -p "$DESTDIR"/"$BASEDIR"
$CP $FILENAME $DESTDIR/$BASEDIR/$BASEFILENAME
fi
done

TEMPFILE Excerpt
Quote:
/var/sadm/pkg/SUNWcsr/install/copyright
/var/sadm/pkg/SUNWcsr/install/preremove
/var/sadm/pkg/SUNWcsr/install/r.manifest
I am thinking of trying to use the cpio or pax command, but was not sure if one would be faster than the other.

I know on some systems that CP has a --parents option, but it doesnt seem to work on my Solaris Boxes.

Ideally this would work on Solaris, HP-UX and Redhat.
# 2  
Old 07-24-2012
I would expect a simple, single invocation of pax to be preferable to that shell script.

Quote:
Originally Posted by nitrobass24
Ideally this would work on Solaris, HP-UX and Redhat.
Of pax, cpio, and tar, pax is the only utility that is standardized by POSIX. So if your hope is that a single command runs identically on each of those operating systems, pax seems a good choice.

Regards,
Alister
# 3  
Old 07-24-2012
Once you get beyond the barrier of running cp 70,000 times to copy 70,000 files, the main bottleneck is going to be your network, not pax.
# 4  
Old 07-24-2012
Quote:
Originally Posted by Corona688
Once you get beyond the barrier of running cp 70,000 times to copy 70,000 files, the main bottleneck is going to be your network, not pax.
Is there a way around that barrier of CP/PAX 70k time?

Im open to ideas.

I need it to be simple/work on all systems/be automated.
Once i have it finished we will be running on approx 1000 servers.
# 5  
Old 07-24-2012
Quote:
Originally Posted by nitrobass24
Is there a way around that barrier of CP/PAX 70k time?

Im open to ideas.
You could try using rsync instead, to avoid copying files which don't actually need updating, though rsync usually takes folders, not lists.

Otherwise, there really isn't a "go faster" setting, it comes down to speeding up your network and disks.
# 6  
Old 07-24-2012
Quote:
Originally Posted by nitrobass24
Is there a way around that barrier of CP/PAX 70k time?

Im open to ideas.

I need it to be simple/work on all systems/be automated.
Once i have it finished we will be running on approx 1000 servers.
You don't need to call pax 70k times, i.e. once per file. You invoke pax once and it reads the list of files to copy from stdin. Using your variable names:
Code:
pax -rw "$DESTDIR" < "$TEMPFILE"

That said, Corona's rsync suggestion may be the best alternative if you will be updating the target after (possibly minor) changes in the source.

Regards,
Alister
This User Gave Thanks to alister For This Post:
nitrobass24
# 7  
Old 07-25-2012
I went with the pax command Allister posted above.

this is a one time copy, so no need to update using rsync in my case.

So i am using similar logic somewhere else, but am filtering it through EGREP.

Could i use pax,cpio, in a similar fashion to speed up this piece of the process as well?

Code:
for FILENAME in `$CAT $TEMPFILE`
do
BASEFILENAME=`$BASENAME $FILENAME`
BASEDIR=`$DIRNAME $FILENAME`
SIZE=`$LS -l $FILENAME | $AWK -F" " '{print $5}'`
if [[ $BASEFILENAME != $SCRIPTNAME ]]
then
for COMMAND in $INSECCOM
do
COUNT=`$EGREP -c $COMMAND $FILENAME`
if [[ $COUNT -ne "0" ]]
then
$ECHO "$HOSTNAME" "\t" "$FILENAME" "\t" "$SIZE" "\t" "$COMMAND" "\t" "$COUNT" >> $LOGFILE
[[ -d $DESTDIR/$BASEDIR ]] || $MKDIR -p "$DESTDIR"/"$BASEDIR"
$CP $FILENAME $DESTDIR/$BASEDIR/$BASEFILENAME
fi
done
fi
done
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies

2. Shell Programming and Scripting

Delete directory on date base

Hi, I have below directories Direct 2013-08-12 23123 Direct 2013-08-13 24121 Direct 2013-08-14 34513 Direct 2013-08-31 15435 ........... Direct 2013-09-12 53145 Direct 2013-09-30 65234 Direct 2013-09-30 89642 (11 Replies)
Discussion started by: learnbash
11 Replies

3. Shell Programming and Scripting

Copy same name data from other file base on column

HI I have input file A.txt X Y Z File B.txt 1 X 10 AAA 11123 2 Y 22 PlD 4563 3 Z 55 PlD 54645 4 Z 66 PlD 15698 5 F 44 PlD 154798 6 C 55 PlD 12554 7 Z 88 PlD 23265 8 C 99 PlD 151654 9 C 11 PlD 21546546 I need New File C.txt (1 Reply)
Discussion started by: pareshkp
1 Replies

4. Shell Programming and Scripting

Copy the files in directory and sub folders as it is to another directory.

How to copy files from one directory to another directory with the subfolders copied. If i have folder1/sub1/sub2/* it needs to copy files to folder2/sub1/sub2/*. I do not want to create sub folders in folder2. Can copy command create them automatically? I tried cp -a and cp -R but did... (4 Replies)
Discussion started by: santosh2626
4 Replies

5. Shell Programming and Scripting

how to copy the directory but not copy certain file

Hi experts cp bin root src /mnt but not copy bin/bigfile any help? ( I post this thread in the "redhat" forum wrongly, I don't know how to withdraw that question in that wrong forum) Thanks (6 Replies)
Discussion started by: yanglei_fage
6 Replies

6. Shell Programming and Scripting

Base directory of the package

Hello, I have an RPM package for which I need to know the base directory /opt/appdir. But if the package is relocated, then it will be different. I need to get the value dynamically. I tried to query the RPM with INSTPREFIXES but couldn't get the value. The only way I see is to do 'rpm -ql RPM... (2 Replies)
Discussion started by: chiru_h
2 Replies

7. Shell Programming and Scripting

Copy Last modified directory to another directory

Hi, I have to copy last modified Directory to another directory.How can i Do using pipe or redirect cp -R 'ls -lrt|tail -1' test is not working. please help. (3 Replies)
Discussion started by: shaveta
3 Replies

8. Solaris

Missing Base Directory

Hey all, I'm attempting to migrate a zone from one system to another, but I noticed that on the new system ssl BASEDIR was missing. I was wondering if creating the directory path for this would fix the issue when I migrate files over. SSL is installed on the new system but does not show a... (1 Reply)
Discussion started by: em23
1 Replies

9. UNIX for Dummies Questions & Answers

Find command to run only in the base directory

Hi. I'm trying to get my find command to only search in the directory i tell it to, but i don't want it to search in the sub directories as well... For example, i have a /data/files/ and /data/files/old I want to search for all .sav files within /data/files but i don't want it to drill... (4 Replies)
Discussion started by: Stephan
4 Replies
Login or Register to Ask a Question