Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Optimizing bash loop

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimizing bash loop
# 1  
Old 06-21-2018
Optimizing bash loop
now, i have to search for a pattern within a particular time frame which the user will provide in the following format:

Code:
19/Jun/2018:07:04,21/Jun/2018:21:30

it is easy to get tempted to attempt this search with a variation of the following awk command:

Code:
awk '/19\/Jun\/2018:07:04/,/21\/Jun\/2018:21:30/' datafile

but i think it can be more efficient than that. can i do this without having to call external utilities like awk?

im writing this code in /bin/sh. i intend to use this script on different unix flavors which includes but certainly isnt limited to ubuntu, redhat, fedora, debian.

here is the code im using to try to get the time from the log that is the closest to the time specified by the user:
Code:
        echo "${AllEpochTimes}" | awk -F"," -v c=1 -v t=${GoBackTimeEndin} '{a[NR]=$c}END{
                asort(a);d=a[NR]-t;d=d<0?-d:d;v = a[NR]
                for(i=NR-1;i>=1;i--){
                        m=a[i]-t;m=m<0?-m:m
                        if(m<d){
                        d=m;v=a[i]
                 }
                }
                print v
                }'

AllEpochTimes - contains all the converted time stamps of the log. as you can imagine, if the log file is huge, the content of this variable will skyrocket in size.
Last edited by SkySmart; 06-30-2018 at 03:00 PM..
# 2  
Old 06-21-2018
You'll need to convertall times to (arithmetically) comparable numbers or strings, either epoch seconds, or e.g. "YYYYmmddHHMM" format.
# 3  
Old 06-22-2018
Perhaps a function like this will help you:

Code:
awk '
function toYMD(tm) {
   split(tm, vals, "[/:]");
   vals[2] = (index("JanFebMarAprMayJunJulAugSepOctNovDec", vals[2]) - 1) / 3 + 1
   return sprintf("%04d%02d%02d%02d%02d%02d", vals[3], vals[2], vals[1], vals[4], vals[5], vals[6])
}

BEGIN { print toYMD("21/Jun/2018:03:46:54") } '

This User Gave Thanks to Chubler_XL For This Post:
SkySmart
# 4  
Old 06-22-2018
You can use /bin/egrep or /bin/grep -E
Code:
$ /bin/egrep "19\/Jun\/2018:07:04|21\/Jun\/2018:21:30" datafile | /bin/egrep -v "^$"
41.27.21.19 - - [19/Jun/2018:07:04:28 +0000]
41.27.21.19 - - [21/Jun/2018:21:30:28 +0000]
$ /bin/egrep Murugesan datafile | /bin/egrep -v "^$"
$ Ret=$?
$ if [ $Ret -eq 0 ]
  then
    echo Pass result
  else
    echo Fail result
  fi
Fail result
$ /bin/grep -E "19\/Jun\/2018:07:04|21\/Jun\/2018:21:30" datafile | /bin/grep -E -v "^$"
41.27.21.19 - - [19/Jun/2018:07:04:28 +0000]
41.27.21.19 - - [21/Jun/2018:21:30:28 +0000]
$

Last edited by murugesandins; 06-22-2018 at 07:02 AM.. Reason: Replaced \: With :
This User Gave Thanks to murugesandins For This Post:
SkySmart
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Optimizing for loop with awk or anything similar and portable

The variable COUNTPRO contains: COUNTPRO='Error__posting__message__to__EMR__Queue=0 Error__parsing__ReceiptSummary=0 xinetd__=4327 HTTP__1_1__500___=0 START__=2164 Marshaller__exception__while__converting__to__Receipt__xml=0 MessagePublisher__is__not__configured__correctly=0... (9 Replies)
Discussion started by: SkySmart
9 Replies

2. Shell Programming and Scripting

Help on for loop in bash

Hi, In the code "for loop" has been used to search for files (command line arguments) in directories and then produce the result to the standard output. However, I want when no files are named on the command line, it should read a list of files from standard input and it should use the command... (7 Replies)
Discussion started by: Ra26k
7 Replies

3. Shell Programming and Scripting

Optimizing bash script

any way the following code can be optimized? FIRSTIN=$( HKIPP=$(echo ${TMFR} | egrep -v "mo|MO|Mo" | egrep "m |M ") HRAMH=$(echo ${TMFR} | egrep "h|H") HRAMD=$(echo ${TMFR} | egrep "d|D") HRAMW=$(echo ${TMFR} | egrep "w|W") HKIPPO=$(echo ${TMFR} |... (5 Replies)
Discussion started by: SkySmart
5 Replies

4. Shell Programming and Scripting

Speed up bash loop?

I am running the below bash loop on all the files of a specific type (highlighted in bold) in a directory. There are 4 awk commands that use the input files to search another and look for a match. The input files range from 27 - 259 and are a list of names. The file that is searched is... (11 Replies)
Discussion started by: cmccabe
11 Replies

5. Shell Programming and Scripting

Bash For Loop Help

This is a short program I wrote to search through a directory and move files containing the keyword to a folder. #!/bin/bash echo 'What is the directory?' read DIR echo -e '\n' echo 'What is the keyword?' read KEY echo -e '\n' cd $DIR rmdir 'relevant_files' mkdir 'relevant_files'... (5 Replies)
Discussion started by: zenyoul
5 Replies

6. Shell Programming and Scripting

If loop in bash

Hello, I have a script that runs a series of commands. Halfway through the script, I want it to check whether everything is going alright: if it is, to proceed with the script, if it isn't to repeat the last step until it gets it right. My code so far looks like this, simplified a bit: ... (3 Replies)
Discussion started by: Leo_Boon
3 Replies

7. Shell Programming and Scripting

BASH loop inside a loop question

Hi all Sorry for the basic question, but i am writing a shell script to get around a slightly flaky binary that ships with one of our servers. This particular utility randomly generates the correct information and could work first time or may work on the 12th or 100th attempt etc !.... (4 Replies)
Discussion started by: rethink
4 Replies

8. Shell Programming and Scripting

Using variables created sequentially in a loop while still inside of the loop [bash]

I'm trying to understand if it's possible to create a set of variables that are numbered based on another variable (using eval) in a loop, and then call on it before the loop ends. As an example I've written a script called question (The fist command is to show what is the contents of the... (2 Replies)
Discussion started by: DeCoTwc
2 Replies

9. Shell Programming and Scripting

bash and ksh: variable lost in loop in bash?

Hi, I use AIX (ksh) and Linux (bash) servers. I'm trying to do scripts to will run in both ksh and bash, and most of the time it works. But this time I don't get it in bash (I'm more familar in ksh). The goal of my script if to read a "config file" (like "ini" file), and make various report.... (2 Replies)
Discussion started by: estienne
2 Replies

10. Shell Programming and Scripting

Simple loop in Bash

Hi, I want to do a simple loop where I have one column of text in a file and I want the loop to read each line of the file and do a simple command. The text file will be something like this: hostname1 hostname2 hostname3 hostname4 I am using Bash and have already come up with this to... (1 Reply)
Discussion started by: BrewDudeBob
1 Replies
Login or Register to Ask a Question