Optimize my mv script Post: 302923842

Sponsored Content

Top Forums Shell Programming and Scripting Optimize my mv script Post 302923842 by jim mcnamara on Tuesday 4th of November 2014 07:04:45 PM

11-04-2014

Registered User

Code:

d=/conversion/program/out
cnt=0
background_limit=10ls $d | while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        if [ ! -d "$d/processed/$b/$a" ] ; then
              mkdir -p $d/processed/$b/$a  || continue
        fi
        mv $d/$line $d/processed/$b/$a/ &
        cnt=$(( $cnt + 1 ))
        [ $(( $cnt %  $backgound_limit )) -eq 0 ] && wait
done
wait

My contention is that mv is taking tons of time, for reasons I don't understand.
Is this an NFS mount?

If $d/processed is on the same filesystem as this $d, then the mv operation should be REALLY fast. Unless the directories involved are really munged with way too many files in which case ls would be super slow and so would any directory operation, like mv or [ -d dirname ] --

I ran the modified code on an unloaded system with 1500+ files in a directory, took less than 2 minutes - 1TB 7200rpm sata disk. They were dummy files of 1 block size - 8192.

Without doing a & on the mv call.

IF mv is mv-ing a file around on a single filesystem then the call amounts to a small, quick change to directory entries. And Achenle has a great point - child processes like that use resources and time.

So I suspect filesystem/directory problems.

What OS and filesystem do you have?

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

optimize the script

Hi, I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt. I want to optimize this...

2. UNIX for Dummies Questions & Answers

Can we optimize this simple script ?

Hi All , I am just a new bie in Unix/Linux . With help of tips from 'here and there' , I just created a simple script to 1. declare one array and some global variables 2. read the schema names from user (user input) and want2proceed flag 3. if user want to proceed , keep reading user...

3. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

I've a script to do some snapshots but the time it does so is very different... once i got a snapshot under 1 sec, on the other hand it took 3 sec, but nothing else changed, i didnt even move the cursor or something. I put the script on a ramdisk and its faster, but still swing from under 1...

4. Shell Programming and Scripting

Optimize and Speedup the script

Hi All, There is a script (test.sh) which is taking more CPU usage. I am attaching the script in this thread. Could anybody please help me out to optimize the script in a better way. Thanks, Gobinath

5. Emergency UNIX and Linux Support

Help to optimize script running time

Dear Forum experts I have the below script which I made to run under bash shell, it runs perfectly for low records number, let us say like 100000. when I put all records (3,000,000), it's takes hours can you please suggest anything to optimize or to run in different way :-| {OFS="|";...

6. Shell Programming and Scripting

Can someone please help me optimize my code (script searches subdirectories)?

Here is my code. What it does is it reads an input file (input.txt which contains roughly 2,000 search phrases) and searches a directory for files that contains the search phrase. The directory contains roughly 1900 files and 84 subdirectories. The output is a file (output.txt) that shows only the...

7. UNIX for Dummies Questions & Answers

optimize if block : shell script

Hi, I need a shell script to determine if a no. is either even, greater than 4, less than 8 SHELL : ksh OS : RHEL 6 this is the if block of the script mod=`expr $num % 2` if || || then echo "No. is either even or greater than 4 or less than 8" fi this code works...

8. Shell Programming and Scripting

Delete unique rows - optimize script

Hi all, I have the following input - the unique row key is 1st column cat file.txt A response C request C response D request C request C response E request The desired output should be C request

9. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1...

10. Shell Programming and Scripting

Help Optimize the Script Further

Hi All, I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB. I'm relatively new to shell scripting. Please help me optimize this script further.

LEARN ABOUT MOJAVE

set_color

set_color(1)							       fish							      set_color(1)

NAME

       set_color - set_color - set the terminal color

set_color - set the terminal color
   Synopsis
       set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR]

   Description
       Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple,
       cyan, white and normal.

       o -b, --background Set the background color

       o -c, --print-colors Prints a list of all valid color names

       o -h, --help Display help message and exit

       o -o, --bold Set bold or extra bright mode

       o -u, --underline Set underlined mode

       o -v, --version Display version and exit

       Calling set_color normal will set the terminal color to whatever is the default color of the terminal.

       Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey
       font color, while set_color --bold white will result in a white font color.

       Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator.

       set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and
       incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of
       ncurses and recompile fish against it in order to fix this issue.

Version 1.23.1							  Sun Jan 8 2012						      set_color(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

optimize the script

Discussion started by: amitrajvarma

2. UNIX for Dummies Questions & Answers

Can we optimize this simple script ?

Discussion started by: rajavu

3. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

Discussion started by: mcW

4. Shell Programming and Scripting

Optimize and Speedup the script

Discussion started by: ntgobinath