Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

need to process for loop faster

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting need to process for loop faster
# 1  
Old 05-17-2012
need to process for loop faster
I have the following code running against a file. The file can have upwards of 10000 lines.

problem is, the for loop takes a while to go through all those lines. is there a faster way to go about it?


Code:
for line in `grep -P "${MONTH} ${DAY}," file | ${AWK} -F" " '{print $4}' | awk -F":" '{print $1":"$2}' | sort | uniq`

do
.....
.....
done

file is:

Code:
<Mar 31, 2012 4:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 5:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 6:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 7:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 8:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 9:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 10:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>
<Mar 31, 2012 11:40:40 PM UTC> <Warning> <Socket> <BEA-00670450> <Socket 211 internal data record unavailable (probable closure due idle timeout), event received 17>

# 2  
Old 05-17-2012
It's not the 'for' that's slow. It's the four pipes, the sort, and the useless use of backticks.

awk is an entire language. You can probably replace that all with one awk and one sort, and use a while-read loop.

Since you didn't say what this is supposed to do, I'm reverse engineering it.
Last edited by Corona688; 05-17-2012 at 03:33 PM..
This User Gave Thanks to Corona688 For This Post:
SkySmart
# 3  
Old 05-17-2012
I can even make it so it sorts properly by adding leading zeroes to the hours...

Code:
awk -v R="$MONTH $DAY" '$0 ~ R { split($4, A, ":"); printf("%02d:%02d\n", A[1], A[2]) }' filename | sort -u |
while read LINE
do
        echo "Line was $LINE"
done

This User Gave Thanks to Corona688 For This Post:
SkySmart
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make faster loop in multiple directories?

Hello, I am under Ubuntu 18.04 Bionic. I have one shell script run.sh (which is out of my topic) to run files under multiple directories and one file to control all processes running under those directories (control.sh). I set a cronjob task to check each of them with two minutes of intervals.... (3 Replies)
Discussion started by: baris35
3 Replies

2. Shell Programming and Scripting

Process files in loop which have spaces in name

I have a folder with files and I have to process them in a loop. However the filenames have space characters, so the list get split. $ touch "File Number_1" $ touch "File Number_2" $ ls "/tmp/File Number"_* /tmp/File Number_1 /tmp/File Number_2 I tried following (sorry for using the... (3 Replies)
Discussion started by: Wernfried
3 Replies

3. Shell Programming and Scripting

Script to loop process

As I would like to test the open files usage , I would like to have a process that use the open files up to a certain amount eg. 1000 . If I want to have a script ( may be run in a loop ) that could repeatly use open files resource , so that the usage of open files increases , may I know how to... (10 Replies)
Discussion started by: ust
10 Replies

4. UNIX for Dummies Questions & Answers

For loop in process each file

Hi I have following codecd /tmp/test/ for vfile in `ls -1` do for vlink in `ls -l /tmp/testfile/*|bin/grep "local/init\.d/$vfile$"|bin/awk -F"->" '{print($1)}'|bin/awk -F"/" '{print($NF)}'` I know `ls -1` list only file, but I don't... (3 Replies)
Discussion started by: stew
3 Replies

5. Shell Programming and Scripting

[SOLVED] for loop to process files

I need to process a dirtree containing ms office files such that each file is stored as a variable and also, just the file file stem. Why? They will be using as input and output parameters for another script. For example /path/to/second_script -i filename.docx -o filename Here's what I... (1 Reply)
Discussion started by: graysky
1 Replies

6. Shell Programming and Scripting

Process checking loop

Hi, I want to create a script who will check if the java process is running & if it finds the process is still there it continues to execute & when the process completes it exit from the script. I have written a code to check & notify the process existence but i am not getting how to write... (4 Replies)
Discussion started by: d8011
4 Replies

7. Shell Programming and Scripting

How to loop this process?

for two txt files, f1 and f2, I like to do the following grep "abcde" f1 > abcde$f1 grep "xyz" f1 > xyz$f1 can I use a loop to get this done? Thanks for i in f1 f2 do grep "abcde" $i > abcde$i grep "xyz" $i > xyz$i ... done (11 Replies)
Discussion started by: ksgreen
11 Replies

8. Shell Programming and Scripting

loop when process running

Hi Gurus, Could you please help me to create a shell script that will be started by a cron job once every night at 24.00 h (that should bee easy:)) The shell script should control every 30 seconds the name of a process, and when the process doesn't run anymore it should execute a few further... (12 Replies)
Discussion started by: blackwire
12 Replies

9. UNIX for Advanced & Expert Users

faster way to loop?

Sample Log file IP.address Date&TimeStamp GET/POST URL ETC 123.45.67.89 MMDDYYYYHHMM GET myURL http://ABC.com 123.45.67.90 MMDDYYYYHHMM GET myURL http://XYZ.com I have a very huge web server log file (about 1.3GB) that contains entries like the one above. I need to get the last entries of... (9 Replies)
Discussion started by: tads98
9 Replies

10. Programming

mmap vs shared memory - which is faster for reading data between multiple process

Between mmap and shared memory which is the best method of sharing data between multiple applications, interms of speed? (1 Reply)
Discussion started by: nmds
1 Replies
Login or Register to Ask a Question