Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Need help optimizing this piece of code (Shell script Busybox)
Top Forums Shell Programming and Scripting Need help optimizing this piece of code (Shell script Busybox) Post 302561229 by snappy46 on Monday 3rd of October 2011 03:13:40 PM
Old 10-03-2011
Quote:
Originally Posted by Corona688
The performance problem is not due to your redirection -- that's not ideal, but doesn't really HURT you that much.

It's mostly due to running grep | sed | awk | cut | kitchen | sink repeatedly...

For instance I might do something like
Code:
grep "$STRING" | awk ... | while read ITEM1 ITEM2 ITEM3 ; do ... ; done

Running one awk for the whole batch is much faster than running 10,000 awks for 10,000 lines. And you may be able to use awk for more than one thing here, fed into several variables a loop, which would help you do more things yet with one program. You could probably even move the grep inside awk.

You can also move your redirection out of the loop completely:

Code:
grep "$STRING" | awk ... | while read ITEM1 ITEM2 ITEM3 ; do echo something ; done >> appended_file

For any further processing needed, it's much more efficient to use shell builtins than awk | cut | sed | kitchen | sink. This being busybox and not bash the builtins you have available will be rather few, though, but it's still not impossible.

Unfortunately, as you have not posted your input data, I have no idea what data needs to be processed into what. What those XML files look like, and exactly what the output from that grep looks like, would be good.
Unfortunately this script need to run on a media player and awk is not available on it. Of course I could always copy it there but I rather use the default tools available on the media player because it makes it easier to distribute.

The input file for this loop as I mentioned on the first post is the list of all my movies on my hard drive created using the find command:

Code:
 
find "$MOVIESPATH" | egrep -i '\.(asf|avi|dat|divx|flv|img|iso|m1v|m2p|m2t|m2ts|m2v|m4v|mkv|mov|mp4|mpg|mts|qt|rm|rmp4|rmvb|tp|trp|ts|vob|wmv)$' | egrep -iv "$FILTER" > $TMP

Then sorted and saves as $SORTEDTMP

hence input is:

/USB/movies/science-fiction/alien/alien.avi
/USB/movies/comedy/funnymovie/funnymovie.avi
/USB/movies/science-fiction/Ironman/IronMan.avi
etc ....

grep output is based on $MOVIESPATH so if it is equal to "/USB/movies/science-fiction/" then two output will be produce by the grep command:

/USB/movies/science-fiction/alien/alien.avi
/USB/movies/science-fiction/Ironman/IronMan.avi

Thank you for your inputs.

Snappy46
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

a piece of code, plz help to review

use "getopts" to get params from command. Need replace black with a specified string like "%20 DEFAULT_DELIM＝％20 ... while getopts dek:f:t:vh OPTION do case $OPTION in t) DELIM=`tvar=/'"$OPTARG"'/ svar="$DEFAULT_DELIM" awk 'BEGIN{T=ENVIRON;S=ENVIRON; while(index(T,S)!=0){S=S"0"};print... (0 Replies)
Discussion started by: anypager
0 Replies

2. Shell Programming and Scripting

what does this piece of code do?

Hi All, I am trying to understand and change some code written by some programmer a while ago. There are following three lines of code that I am unable to grasp. Could anybody please help me understand it? 1) cd - > /dev/null 2) fname=`basename "$1"` where $1 = /dirA/dirB/a.txt ... (3 Replies)
Discussion started by: Vikas Sood
3 Replies

3. Shell Programming and Scripting

script or piece of code where the data returned by a stored procedure you are writing

hi fndz. Can you please help me with the code if I call a stored procedure from my shell script and stored procedure returns a cursor, cursor output should be saved to a file (3 Replies)
Discussion started by: enigma_83
3 Replies

4. Shell Programming and Scripting

Enabling sh shell in BusyBox

Hi, Does anybody know how to enable the shell sh while creating Ramdisk fs using BusyBox? while creating a configuration using the GUI, I see options only for the ash shell. Is there some option in the config file that gets created with which I can enable the sh shell also apart from the ash... (0 Replies)
Discussion started by: jake24
0 Replies

5. Shell Programming and Scripting

Looking for guidance (comments) on a piece of code

Hello -- I am trying to learn to do a little sed and awk scripting to search for text and numbers in text files (text processing/manipulation). My professor gave me a piece of uncommented code and I am very unfamiliar w/ the language. Can someone help me with comments so I can understand what is... (2 Replies)
Discussion started by: smithan05
2 Replies

6. Programming

what is the name of this piece of code

while ((numRead = read(inputFd, buf, BUF_SIZE)) > 0) if (write(outputFd, buf, numRead) != numRead) fatal("couldn't write whole buffer"); if (numRead == -1) errExit("read"); if (close(inputFd) == -1) errExit("close input"); if (close(outputFd) == -1) errExit("close output"); ... (1 Reply)
Discussion started by: fwrlfo
1 Replies

7. Shell Programming and Scripting

Optimizing the code

Hi, I have two files in the format listed below. I need to find out all values from field 12 to field 20 present in file 2 and list them in file3(format as file2) File1 : FEIN,CHRISTA... (2 Replies)
Discussion started by: nua7
2 Replies

8. Shell Programming and Scripting

Bash Script to Ash (busybox) - Beginner

Hi All, I have a script that I wrote on a bash shell, I use it to sort files from a directory into various other directories. I have an variable set, which is an array of strings, I then check each file against the array and if it is in there the script sorts it into the correct folder. But... (5 Replies)
Discussion started by: sgtbobie
5 Replies

9. Shell Programming and Scripting

Need a piece of shell scripting to remove column from a csv file

Hi, I need to remove first column from a csv file and i can do this by using below command. cut -f1 -d, --complement Mytest.csv I need to implement this in shell scripting, Whenever i am using the above command alone in command line it is working fine. I have 5 files in my directory and... (3 Replies)
Discussion started by: Samah
3 Replies

10. Shell Programming and Scripting

Optimizing the Shell Script [Expert Advise Needed]

I have prepared a shell script to find the duplicates based on the part of filename and retain latest. #!/bin/bash if ; then mkdir -p dup fi NOW=$(date +"%F-%H:%M:%S") LOGFILE="purge_duplicate_log-$NOW.log" LOGTIME=`date "+%Y-%m-%d %H:%M:%S"` echo... (6 Replies)
Discussion started by: gold2k8
6 Replies

LEARN ABOUT SUSE

tcl_allowexceptions

Tcl_AllowExceptions(3)					      Tcl Library Procedures					    Tcl_AllowExceptions(3)

__________________________________________________________________________________________________________________________________________________

NAME

       Tcl_AllowExceptions - allow all exceptions in next script evaluation

SYNOPSIS

       #include <tcl.h>

       Tcl_AllowExceptions(interp)

ARGUMENTS

       Tcl_Interp *interp (in)		Interpreter in which script will be evaluated.
_________________________________________________________________

DESCRIPTION

       If  a  script is evaluated at top-level (i.e. no other scripts are pending evaluation when the script is invoked), and if the script termi-
       nates with a completion code other than TCL_OK, TCL_ERROR or TCL_RETURN, then Tcl normally converts this into a TCL_ERROR  return  with	an
       appropriate  message.  The particular script evaluation procedures of Tcl that act in the manner are Tcl_EvalObjEx, Tcl_EvalObjv, Tcl_Eval,
       Tcl_EvalEx, Tcl_GlobalEval, Tcl_GlobalEvalObj, Tcl_VarEval and Tcl_VarEvalVA.

       However, if Tcl_AllowExceptions is invoked immediately before calling one of those a procedures, then arbitrary completion codes  are  per-
       mitted from the script, and they are returned without modification.  This is useful in cases where the caller can deal with exceptions such
       as TCL_BREAK or TCL_CONTINUE in a meaningful way.

KEYWORDS

       continue, break, exception, interpreter

Tcl									7.4						    Tcl_AllowExceptions(3)
All times are GMT -4. The time now is 07:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy