Optimize my mv script Post: 302923842

Sponsored Content

Top Forums Shell Programming and Scripting Optimize my mv script Post 302923842 by jim mcnamara on Tuesday 4th of November 2014 07:04:45 PM

11-04-2014

Registered User

Code:

d=/conversion/program/out
cnt=0
background_limit=10ls $d | while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        if [ ! -d "$d/processed/$b/$a" ] ; then
              mkdir -p $d/processed/$b/$a  || continue
        fi
        mv $d/$line $d/processed/$b/$a/ &
        cnt=$(( $cnt + 1 ))
        [ $(( $cnt %  $backgound_limit )) -eq 0 ] && wait
done
wait

My contention is that mv is taking tons of time, for reasons I don't understand.
Is this an NFS mount?

If $d/processed is on the same filesystem as this $d, then the mv operation should be REALLY fast. Unless the directories involved are really munged with way too many files in which case ls would be super slow and so would any directory operation, like mv or [ -d dirname ] --

I ran the modified code on an unloaded system with 1500+ files in a directory, took less than 2 minutes - 1TB 7200rpm sata disk. They were dummy files of 1 block size - 8192.

Without doing a & on the mv call.

IF mv is mv-ing a file around on a single filesystem then the call amounts to a small, quick change to directory entries. And Achenle has a great point - child processes like that use resources and time.

So I suspect filesystem/directory problems.

What OS and filesystem do you have?

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

optimize the script

Hi, I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt. I want to optimize this...

2. UNIX for Dummies Questions & Answers

Can we optimize this simple script ?

Hi All , I am just a new bie in Unix/Linux . With help of tips from 'here and there' , I just created a simple script to 1. declare one array and some global variables 2. read the schema names from user (user input) and want2proceed flag 3. if user want to proceed , keep reading user...

3. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

I've a script to do some snapshots but the time it does so is very different... once i got a snapshot under 1 sec, on the other hand it took 3 sec, but nothing else changed, i didnt even move the cursor or something. I put the script on a ramdisk and its faster, but still swing from under 1...

4. Shell Programming and Scripting

Optimize and Speedup the script

Hi All, There is a script (test.sh) which is taking more CPU usage. I am attaching the script in this thread. Could anybody please help me out to optimize the script in a better way. Thanks, Gobinath

5. Emergency UNIX and Linux Support

Help to optimize script running time

Dear Forum experts I have the below script which I made to run under bash shell, it runs perfectly for low records number, let us say like 100000. when I put all records (3,000,000), it's takes hours can you please suggest anything to optimize or to run in different way :-| {OFS="|";...

6. Shell Programming and Scripting

Can someone please help me optimize my code (script searches subdirectories)?

Here is my code. What it does is it reads an input file (input.txt which contains roughly 2,000 search phrases) and searches a directory for files that contains the search phrase. The directory contains roughly 1900 files and 84 subdirectories. The output is a file (output.txt) that shows only the...

7. UNIX for Dummies Questions & Answers

optimize if block : shell script

Hi, I need a shell script to determine if a no. is either even, greater than 4, less than 8 SHELL : ksh OS : RHEL 6 this is the if block of the script mod=`expr $num % 2` if || || then echo "No. is either even or greater than 4 or less than 8" fi this code works...

8. Shell Programming and Scripting

Delete unique rows - optimize script

Hi all, I have the following input - the unique row key is 1st column cat file.txt A response C request C response D request C request C response E request The desired output should be C request

9. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1...

10. Shell Programming and Scripting

Help Optimize the Script Further

Hi All, I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB. I'm relatively new to shell scripting. Please help me optimize this script further.

LEARN ABOUT DEBIAN

gchem3d

GCHEM3D(1)						       gnome-chemistry-utils							GCHEM3D(1)

NAME

       gchem3d - a small chemical viewer application

SYNOPSIS

       gchem3d [OPTION(S)...] [FILE...]

DESCRIPTION

       gchem3d is a small chemical viewer application, which can show several chemical file formats.

OPTIONS

       The following options are accepted:

       -b COLOR, --bgcolor=COLOR
	   Use the given color as background color.  COLOR can be one of "black" (default), "white", "#rrggbb" (don't forget to escape the "#"
	   character in the shell).

       -d MODEL, --display3d=MODEL
	   Choose how molecules are displayed.	MODEL can be one of "BallnStick" (default), "SpaceFill".

       -?, --help
	   Show application help options.

       --help-all, --help-*
	   Print all or just a group of help options. These options are not documented here. Instead see gtk-options(7) and gnome-options(7).

       -v, --version
	   Print gchem3d version information.

SEE ALSO

       gnome-options(7), gtk-options(7)

AUTHORS

       Jean Brefort <jean.brefort@normalesup.org>
	   Program author.

       Daniel Leidert <daniel.leidert@wgdd.de>
	   Manpage author.

COPYRIGHT

       Copyright (C) 2002-2007 Jean Brefort
       Copyright (C) 2004-2007 Daniel Leidert

       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 or any
       later version published by the Free Software Foundation.

gcu 0.12				      $Date: 2009-03-19 10:53:47 +0100 (jeu. 19 mars 2009) $					GCHEM3D(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

optimize the script

Discussion started by: amitrajvarma

2. UNIX for Dummies Questions & Answers

Can we optimize this simple script ?

Discussion started by: rajavu

3. UNIX for Dummies Questions & Answers

optimize shell script (snapshots)

Discussion started by: mcW

4. Shell Programming and Scripting

Optimize and Speedup the script

Discussion started by: ntgobinath