Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Optimize my mv script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimize my mv script
# 1  
Old 11-04-2014
Optimize my mv script
Hello,

I'm wondering if there is a quicker way of doing this.

Here is my mv script.

Code:
d=/conversion/program/out
cd $d
ls $d > /home/tempuser/$$tmp
while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        if [ ! -d "/processed/$b" ];then mkdir /processed/$b;fi
        if [ ! -d "/processed/$b/$a" ];then mkdir /processed/$b/$a;fi
        mv $line /processed/$b/$a/
done < /home/tempuser/$$tmp
rm -f /home/tempuser/$$tmp

Script takes files that have been dropped into /conversion/program/out gets some info from the filename and uses that to move the file into it's proper folder.

a= a 4 or 5 digit number
b = YYYYMM eg. 201411

Thanks for any assistance.
Last edited by joeyg; 11-04-2014 at 11:19 AM.. Reason: corrected spelling of title
# 2  
Old 11-04-2014
Before thinking about optimizing...
Must offer a word of caution to a cd command within a script.
You do not have anything checking to verify that the cd command worked - and I have seen many nasty things happen because the user assumed the cd command was successful.
# 3  
Old 11-04-2014
Try this and please test it thoroughly to ensure that it is producing the same result as your initial script.
Code:
d=/conversion/program/out
ls $d | while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        [ ! -d "$d/processed/$b/$a" ] && mkdir -p $d/processed/$b/$a
        mv $d/$line $d/processed/$b/$a/
done

mkdir's -p option means "no error if existing, make parent directories as needed"

---------- Post updated at 05:53 PM ---------- Previous update was at 05:48 PM ----------

Creating and using an additional variable, e.g. dp="$d/processed", might also be useful to avoid repetitions.
This User Gave Thanks to junior-helper For This Post:
whegra
# 4  
Old 11-04-2014
Quote:
Originally Posted by junior-helper
Code:
d=/conversion/program/out
ls $d | while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        [ ! -d "$d/processed/$b/$a" ] && mkdir -p $d/processed/$b/$a
        mv $d/$line $d/processed/$b/$a/
done

Don't you mean?
Code:
[ ! -d "$d/processed/$b/$a" ] && mkdir -p $d/processed/$b/$a && mv $d/$line $d/processed/$b/$a/

Or it is OK to try to move the file even if the directory does not exist?
# 5  
Old 11-04-2014
Well, that's not a bad idea for hardening this script.

Of course it is not OK to move the file to a non-existent directory, but if the directory does not exist, it gets created anyway prior to the mv command. Feel free to correct me if I'm wrong, at any time.
# 6  
Old 11-04-2014
Quote:
Originally Posted by junior-helper
Well, that's not a bad idea for hardening this script.

Of course it is not OK to move the file to a non-existent directory, but if the directory does not exist, it gets created anyway prior to the mv command. Feel free to correct me if I'm wrong, at any time.
Is it guarantee that mkdir -p would create the directory tree?
# 7  
Old 11-04-2014
Here are the results of a test run.

To move 500 files:

New code from junior-helper
Code:
$ time ./new.sh

real    5m16.228s
user    0m12.693s
sys     0m35.638s

Old code
Code:
$ time ./old.sh

real    5m25.931s
user    0m11.627s
sys     0m37.098s

Looks like I'm almost getting a 10sec savings per 500 files.

Also using this line

Code:
[ ! -d "$d/processed/$b/$a" ] && mkdir -p $d/processed/$b/$a && mv $d/$line $d/processed/$b/$a/

Looked to do something odd, it only moved files when the folder did not already exist, so it only moved about 120 files then completed running.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help Optimize the Script Further

Hi All, I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB. I'm relatively new to shell scripting. Please help me optimize this script further. (0 Replies)
Discussion started by: narayanv
0 Replies

2. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1... (8 Replies)
Discussion started by: SkySmart
8 Replies

3. Shell Programming and Scripting

Delete unique rows - optimize script

Hi all, I have the following input - the unique row key is 1st column cat file.txt A response C request C response D request C request C response E request The desired output should be C request (7 Replies)
Discussion started by: varu0612
7 Replies

4. UNIX for Dummies Questions & Answers

optimize if block : shell script

Hi, I need a shell script to determine if a no. is either even, greater than 4, less than 8 SHELL : ksh OS : RHEL 6 this is the if block of the script mod=`expr $num % 2` if || || then echo "No. is either even or greater than 4 or less than 8" fi this code works... (2 Replies)
Discussion started by: sam05121988
2 Replies

5. Shell Programming and Scripting

Can someone please help me optimize my code (script searches subdirectories)?

Here is my code. What it does is it reads an input file (input.txt which contains roughly 2,000 search phrases) and searches a directory for files that contains the search phrase. The directory contains roughly 1900 files and 84 subdirectories. The output is a file (output.txt) that shows only the... (23 Replies)
Discussion started by: jl487
23 Replies

6. Emergency UNIX and Linux Support

Help to optimize script running time

Dear Forum experts I have the below script which I made to run under bash shell, it runs perfectly for low records number, let us say like 100000. when I put all records (3,000,000), it's takes hours can you please suggest anything to optimize or to run in different way :-| {OFS="|";... (6 Replies)
Discussion started by: yahyaaa
6 Replies

7. Shell Programming and Scripting

Optimize and Speedup the script

Hi All, There is a script (test.sh) which is taking more CPU usage. I am attaching the script in this thread. Could anybody please help me out to optimize the script in a better way. Thanks, Gobinath (6 Replies)
Discussion started by: ntgobinath
6 Replies

8. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

I've a script to do some snapshots but the time it does so is very different... once i got a snapshot under 1 sec, on the other hand it took 3 sec, but nothing else changed, i didnt even move the cursor or something. I put the script on a ramdisk and its faster, but still swing from under 1... (1 Reply)
Discussion started by: mcW
1 Replies

9. UNIX for Dummies Questions & Answers

Can we optimize this simple script ?

Hi All , I am just a new bie in Unix/Linux . With help of tips from 'here and there' , I just created a simple script to 1. declare one array and some global variables 2. read the schema names from user (user input) and want2proceed flag 3. if user want to proceed , keep reading user... (8 Replies)
Discussion started by: rajavu
8 Replies

10. Shell Programming and Scripting

optimize the script

Hi, I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt. I want to optimize this... (3 Replies)
Discussion started by: amitrajvarma
3 Replies
Login or Register to Ask a Question