Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Need help optimizing this piece of code (Shell script Busybox)
Top Forums Shell Programming and Scripting Need help optimizing this piece of code (Shell script Busybox) Post 302561193 by Corona688 on Monday 3rd of October 2011 12:34:34 PM
Old 10-03-2011
The performance problem is not due to your redirection -- that's not ideal, but doesn't really HURT you that much.

It's mostly due to running grep | sed | awk | cut | kitchen | sink repeatedly...

For instance I might do something like
Code:
grep "$STRING" | awk ... | while read ITEM1 ITEM2 ITEM3 ; do ... ; done

Running one awk for the whole batch is much faster than running 10,000 awks for 10,000 lines. And you may be able to use awk for more than one thing here, fed into several variables a loop, which would help you do more things yet with one program. You could probably even move the grep inside awk.

You can also move your redirection out of the loop completely:

Code:
grep "$STRING" | awk ... | while read ITEM1 ITEM2 ITEM3 ; do echo something ; done >> appended_file

For any further processing needed, it's much more efficient to use shell builtins than awk | cut | sed | kitchen | sink. This being busybox and not bash the builtins you have available will be rather few, though, but it's still not impossible.

Unfortunately, as you have not posted your input data, I have no idea what data needs to be processed into what. What those XML files look like, and exactly what the output from that grep looks like, would be good.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

a piece of code, plz help to review

use "getopts" to get params from command. Need replace black with a specified string like "%20 DEFAULT_DELIM＝％20 ... while getopts dek:f:t:vh OPTION do case $OPTION in t) DELIM=`tvar=/'"$OPTARG"'/ svar="$DEFAULT_DELIM" awk 'BEGIN{T=ENVIRON;S=ENVIRON; while(index(T,S)!=0){S=S"0"};print... (0 Replies)
Discussion started by: anypager
0 Replies

2. Shell Programming and Scripting

what does this piece of code do?

Hi All, I am trying to understand and change some code written by some programmer a while ago. There are following three lines of code that I am unable to grasp. Could anybody please help me understand it? 1) cd - > /dev/null 2) fname=`basename "$1"` where $1 = /dirA/dirB/a.txt ... (3 Replies)
Discussion started by: Vikas Sood
3 Replies

3. Shell Programming and Scripting

script or piece of code where the data returned by a stored procedure you are writing

hi fndz. Can you please help me with the code if I call a stored procedure from my shell script and stored procedure returns a cursor, cursor output should be saved to a file (3 Replies)
Discussion started by: enigma_83
3 Replies

4. Shell Programming and Scripting

Enabling sh shell in BusyBox

Hi, Does anybody know how to enable the shell sh while creating Ramdisk fs using BusyBox? while creating a configuration using the GUI, I see options only for the ash shell. Is there some option in the config file that gets created with which I can enable the sh shell also apart from the ash... (0 Replies)
Discussion started by: jake24
0 Replies

5. Shell Programming and Scripting

Looking for guidance (comments) on a piece of code

Hello -- I am trying to learn to do a little sed and awk scripting to search for text and numbers in text files (text processing/manipulation). My professor gave me a piece of uncommented code and I am very unfamiliar w/ the language. Can someone help me with comments so I can understand what is... (2 Replies)
Discussion started by: smithan05
2 Replies

6. Programming

what is the name of this piece of code

while ((numRead = read(inputFd, buf, BUF_SIZE)) > 0) if (write(outputFd, buf, numRead) != numRead) fatal("couldn't write whole buffer"); if (numRead == -1) errExit("read"); if (close(inputFd) == -1) errExit("close input"); if (close(outputFd) == -1) errExit("close output"); ... (1 Reply)
Discussion started by: fwrlfo
1 Replies

7. Shell Programming and Scripting

Optimizing the code

Hi, I have two files in the format listed below. I need to find out all values from field 12 to field 20 present in file 2 and list them in file3(format as file2) File1 : FEIN,CHRISTA... (2 Replies)
Discussion started by: nua7
2 Replies

8. Shell Programming and Scripting

Bash Script to Ash (busybox) - Beginner

Hi All, I have a script that I wrote on a bash shell, I use it to sort files from a directory into various other directories. I have an variable set, which is an array of strings, I then check each file against the array and if it is in there the script sorts it into the correct folder. But... (5 Replies)
Discussion started by: sgtbobie
5 Replies

9. Shell Programming and Scripting

Need a piece of shell scripting to remove column from a csv file

Hi, I need to remove first column from a csv file and i can do this by using below command. cut -f1 -d, --complement Mytest.csv I need to implement this in shell scripting, Whenever i am using the above command alone in command line it is working fine. I have 5 files in my directory and... (3 Replies)
Discussion started by: Samah
3 Replies

10. Shell Programming and Scripting

Optimizing the Shell Script [Expert Advise Needed]

I have prepared a shell script to find the duplicates based on the part of filename and retain latest. #!/bin/bash if ; then mkdir -p dup fi NOW=$(date +"%F-%H:%M:%S") LOGFILE="purge_duplicate_log-$NOW.log" LOGTIME=`date "+%Y-%m-%d %H:%M:%S"` echo... (6 Replies)
Discussion started by: gold2k8
6 Replies

LEARN ABOUT DEBIAN

installsieve

INSTALLSIEVE(1) 					      General Commands Manual						   INSTALLSIEVE(1)

 *

NAME

       installsieve - user utility for managing sieve scripts

SYNOPSIS

       installsieve

DESCRIPTION

       installsieve is a utility that allows users to manage their sieve scripts kept on the server.

OPTIONS

       -v <name> View script with the given name. The script if retrieved sucessfully is output to standard output.

       -l  List  all  of  the scripts currently on the server. If one of the scripts is active a arrow is printed indicating that it is the active
       script.

       -p <port> Port to connect to. If left off this defaults to "sieve" as defined in /etc/services.

       -i <file> Install a file onto the server. If a script with the same name already exists on the server it is overwritten.  Upon  sucessfully
       putting	the  script on the server the script is set active. If <file> has the extension .script it is chopped when put on the server since
       sieve names may not contain a '.'.

       -a <name> Set <name> as the active script. The list of available names can be obtained from the '-l' option..

       -d <name> Delete the sieve script on the server with <name>.

       -m <mechanism> Force installsieve to use <mechanism> for authentication. If not specified the strongest authentication mechanism is chosen.

       -g <name> Get the sieve script with <name> and save it to disk with a ".script" extension. If a file with that name already  exists  it	is
       overwritten.

       -u <user> Userid/Authname to use for authentication; by default, the current user.

CMU
								   Project Cyrus						   INSTALLSIEVE(1)
All times are GMT -4. The time now is 12:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy