Sponsored Content
Top Forums Shell Programming and Scripting sed / grep / for statement performance - please help Post 302368350 by TehOne on Wednesday 4th of November 2009 04:41:10 PM
Old 11-04-2009
Error sed / grep / for statement performance - please help

I'm searching the most effective way of doing the following task, so if someone can either provide a working solution with sed or one totally different but more effective then what I've got so far then please go ahead!

The debugme directory has 3 subdirectorys and each of them has one .txt file with about 48 entrys each.

Code:
time (
for FILE1 in `find debugme -name "*.txt"` ;do
    for FILE2 in `cat "$FILE1" | awk '{print $1}' | grep -i '^\([0-9]\+\)$'` ;do
    #for FILE2 in `sed 's/\([0-9]\+\) \([a-zA-Z0-9]\+\)/\1/i' "$FILE"` ;do
        CHECK=`grep "$FILE2" "debug.files"`
        if [ "$CHECK" = "" ]; then
            ## ADD MISSING ENTRY BLABLA
            echo "$FILE2 was missing, added!"
        fi
    done
done
)

Avg result:
real    0m0.174s
user    0m0.052s
sys     0m0.128s

time (
for FILE1 in `find debugme -name "*.txt"` ;do
    FILE2() {
        S=`grep $1 debug.files`
        if [ "$S" = "" ] ; then
            ## ADD MISSING ENTRY BLABLA
            echo "$FILE2 was missing, added!"
        fi
    }
    while read cola colb ; do
        $(FILE2 $cola)
    done < $FILE1
done
)

Avg result:
real    0m0.269s
user    0m0.064s
sys     0m0.228s

The .txt files are in this format:

Code:
[03:53:22] root:~# cat example.txt
52352578 ABF2778ABD^
73534536 LASDM337lA^
83523422 JFAASMM31^

And debug.files in this:

Code:
[03:53:25] root:~# cat debug.files
52352578
73534536

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using grep in if statement

Can somebody please guide me towards right syntax: #!/bin/ksh if i = $(grep $NAME filename) echo "Name Found" else echo " Name not Found" fi I need to grep for $NAME in the file, and if it returns false, execute a series of commands and if true, exit out. The above is not the right... (3 Replies)
Discussion started by: chiru_h
3 Replies

2. Shell Programming and Scripting

How to improve grep performance...

Hi All, I am using grep command to find string "abc" in one file . content of file is *********** abc = xyz def= lmn ************ i have given the below mentioned command to redirect the output to tmp file grep abc file | sort -u | awk '{print #3}' > out_file Then i am searching... (2 Replies)
Discussion started by: pooga17
2 Replies

3. UNIX for Advanced & Expert Users

sed performance

hello experts, i am trying to replace a line in a 100+mb text file. the structure is similar to the passwd file, id:value1:value2 and so on. using the sed command sed -i 's/\(123\):\(\{1,\}\):/\1:bar:/' data.txt works nicely, the line "123:foo:" is replaced by "123:bar:". however, it takes... (7 Replies)
Discussion started by: f3k
7 Replies

4. Shell Programming and Scripting

Grep statement

Hi All, Please can somebody advise that if I want to search a pattern xyz the grep command should only select xyz and not any other pattern containing xyz (ex abxyzcd) Regards (1 Reply)
Discussion started by: Shazin
1 Replies

5. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time... (3 Replies)
Discussion started by: vikash_k
3 Replies

6. Programming

Help with improve the performance of grep

Input file: #content_1 12314345345 242467 #content_14 436677645 576577657 #content_100 3425546 56 #content_12 243254546 1232454 . . Reference file: content_100 (1 Reply)
Discussion started by: cpp_beginner
1 Replies

7. Shell Programming and Scripting

Use awk/sed/grep with goto statement!

Hi, I have an array with characters and I am looking for specific character in that array and if those specific character not found than I use goto statment which is define somehwhere in the script. My code is: set a = (A B C D E F) @ i = 0 while ($i <= ${#a}) if ($a != "F" || $a != "D")... (3 Replies)
Discussion started by: dixits
3 Replies

8. Shell Programming and Scripting

Using GREP in IF Statement

Hello All, I have 2 different pieces of code, I am confused why the Code1 is giving me the correct result where as the Code2 is not giving me correct result. It gives me always result as "Failure" irrespective of the "ERROR" word exists in logfile or not. may I know the reason why? I am using Bash... (17 Replies)
Discussion started by: Ariean
17 Replies

9. Shell Programming and Scripting

Performance improvement in grep

Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script. Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies

10. Shell Programming and Scripting

Convert Update statement into Insert statement in UNIX using awk, sed....

Hi folks, I have a scenario to convert the update statements into insert statements using shell script (awk, sed...) or in database using regex. I have a bunch of update statements with all columns in a file which I need to convert into insert statements. UPDATE TABLE_A SET COL1=1 WHERE... (0 Replies)
Discussion started by: dev123
0 Replies
TEST(1) 							   User Commands							   TEST(1)

NAME
test - check file types and compare values SYNOPSIS
test EXPRESSION test [ EXPRESSION ] [ ] [ OPTION DESCRIPTION
Exit with the status determined by EXPRESSION. --help display this help and exit --version output version information and exit An omitted EXPRESSION defaults to false. Otherwise, EXPRESSION is true or false and sets exit status. It is one of: ( EXPRESSION ) EXPRESSION is true ! EXPRESSION EXPRESSION is false EXPRESSION1 -a EXPRESSION2 both EXPRESSION1 and EXPRESSION2 are true EXPRESSION1 -o EXPRESSION2 either EXPRESSION1 or EXPRESSION2 is true -n STRING the length of STRING is nonzero STRING equivalent to -n STRING -z STRING the length of STRING is zero STRING1 = STRING2 the strings are equal STRING1 != STRING2 the strings are not equal INTEGER1 -eq INTEGER2 INTEGER1 is equal to INTEGER2 INTEGER1 -ge INTEGER2 INTEGER1 is greater than or equal to INTEGER2 INTEGER1 -gt INTEGER2 INTEGER1 is greater than INTEGER2 INTEGER1 -le INTEGER2 INTEGER1 is less than or equal to INTEGER2 INTEGER1 -lt INTEGER2 INTEGER1 is less than INTEGER2 INTEGER1 -ne INTEGER2 INTEGER1 is not equal to INTEGER2 FILE1 -ef FILE2 FILE1 and FILE2 have the same device and inode numbers FILE1 -nt FILE2 FILE1 is newer (modification date) than FILE2 FILE1 -ot FILE2 FILE1 is older than FILE2 -b FILE FILE exists and is block special -c FILE FILE exists and is character special -d FILE FILE exists and is a directory -e FILE FILE exists -f FILE FILE exists and is a regular file -g FILE FILE exists and is set-group-ID -G FILE FILE exists and is owned by the effective group ID -h FILE FILE exists and is a symbolic link (same as -L) -k FILE FILE exists and has its sticky bit set -L FILE FILE exists and is a symbolic link (same as -h) -O FILE FILE exists and is owned by the effective user ID -p FILE FILE exists and is a named pipe -r FILE FILE exists and read permission is granted -s FILE FILE exists and has a size greater than zero -S FILE FILE exists and is a socket -t FD file descriptor FD is opened on a terminal -u FILE FILE exists and its set-user-ID bit is set -w FILE FILE exists and write permission is granted -x FILE FILE exists and execute (or search) permission is granted Except for -h and -L, all FILE-related tests dereference symbolic links. Beware that parentheses need to be escaped (e.g., by backslashes) for shells. INTEGER may also be -l STRING, which evaluates to the length of STRING. NOTE: [ honors the --help and --version options, but test does not. test treats each of those as it treats any other nonempty STRING. NOTE: your shell may have its own version of test and/or [, which usually supersedes the version described here. Please refer to your shell's documentation for details about the options it supports. GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report test translation bugs to <http://translationproject.org/team/> AUTHOR
Written by Kevin Braunsdorf and Matthew Bradburn. COPYRIGHT
Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
The full documentation for test is maintained as a Texinfo manual. If the info and test programs are properly installed at your site, the command info coreutils 'test invocation' should give you access to the complete manual. GNU coreutils 8.22 June 2014 TEST(1)
All times are GMT -4. The time now is 03:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy