Printing the output of sed using a loop


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Printing the output of sed using a loop
# 1  
Old 01-17-2019
Printing the output of sed using a loop

So I am writing a bash script that will search a file line by line for unix timestamps, store all of the timestamps into an array, then check how many of those timestamps were created within the last hour, and finally increment a counter every time it finds a timestamp created within the last hour.

whenever I try to run the script I get an error like this one

Code:
2dispoCounter.sh: line 19: [@],-1547770608: syntax error: operand expected (error token is "[@],-1547770608")

when I run the command echo $epochTimes I get thousands of epoch times that look like this

Code:
547765257 1547765347 1547765577 1547765582 1547765956 1547765962 1547765949 1547766001 1547766074 1547766089 1547766158 1547766242 1547766242 1547766527

Here is my script:

Code:
#!/bin/bash
 
a=(0 16 272 512 768 2048 2304 1024 1280)
counter=(0 0 0 0 0 0 0 0 0)
 
#grab # of results from log file and store each value in the array
j=0
for c in ${a[@]}
do
    epochTimes=()
    epochTimes=("$epochTimes[@]", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))
    echo "${epochTimes[@]}"
    now=$(date +%s)
    for aTime in "${epochTimes[@]}"
    do
        echo $aTime
        echo $now
        diff=$(($aTime-$now))
        if [ $diff < 3600 ]
        then
            ((counter[$j]++))
        fi
    done
    echo ${counter[$j]}
    ((j++))
done
python testPrint.py "${counter[@]}"

So how would I go about make aTime within the script be one of these timestamps and not [@], ?
Moderator's Comments:
Mod Comment Please use CODE tags around sample input and output as well as around code segments.

Last edited by Don Cragun; 01-17-2019 at 09:33 PM..
# 2  
Old 01-17-2019
You might consider changing:
Code:
    epochTimes=("$epochTimes[@]", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

to:
Code:
    epochTimes=("${epochTimes[@]}", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

and let us know if that makes any difference.

If not, change the second line of your script to:
Code:
set -xv

and see if you can figure out what is going on by watching the tracing output as your script runs. If not, show us the output you get with tracing turned on and maybe we can help you further diagnose your problem.
These 2 Users Gave Thanks to Don Cragun For This Post:
# 3  
Old 01-18-2019
I figured it out.
As it turns out, for some reason the first argument returned was always [@]. So in order to solve the problem all I had to do was to start the loop at the second argument

Something like

Code:
for aTime in "${epochTimes[@]:2}"

instead of

Code:
for aTime in "${epochTimes[@]}"

# 4  
Old 01-18-2019
Quote:
Originally Posted by jsikarin
Code:
   epochTimes=("$epochTimes[@]", $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

no comma separates the array elements
Code:
   epochTimes=("${epochTimes[@]}" $(grep 'cn2='$c' ' /opt/crowdstrike/log/output | sed -n 's/.*rt=\([0-9]\{10\}\).*/\1\n/p'))

--- Post updated at 06:32 ---

so you just lose some of the data along with the error
Quote:
for aTime in "${epochTimes[@]:2}"

Last edited by nezabudka; 01-18-2019 at 02:12 AM..
These 2 Users Gave Thanks to nezabudka For This Post:
# 5  
Old 01-18-2019
Note that if you have made the change I suggested in post #2 in this thread, there wouldn't have been a [@], in the array (but as nezabudka noted there would have been one element in the array that ended in a comma that would not work correctly.

And, in addition to what nezabudka has already said, note that anytime you have a loop containing a repeated pipeline of grep ... | sed ..., there is a good chance that you could replace that entire loop with a single awk script that could do what you're trying to do much more efficiently. (But, of course, we'd have to know what the format of your input file (/opt/crowdstrike/log/output) is to help you do that.

And, if we knew the format of the output you're trying to produce, that could probably also be built into that awk script without needing to invoke python after the loop (or awk) is done.

You might also note that the construct 'cn2='$c' ' in your grep command, would be more readable for many people if you wrote it as "cnt=$c ". But, of course, the result of using either quoting form will be the same as long as the strings assigned to $c are numeric strings as in your example.

Knowing what operating system you're using would probably also help us give you suggestions that will work correctly in your environment.

And, having a reasonably short sample input file and a corresponding sample output file showing the output you're hoping to produce would be a huge help for anyone trying to help you achieve your goal. Please help us help you.
This User Gave Thanks to Don Cragun For This Post:
# 6  
Old 01-19-2019
I would use awk if I knew how, but for now grep and sed is fine. I have this book on sed and awk that i am reading. I am just glad that I figured out how to use sed.

I will consider changing
Code:
'cn2='$c' '

to
Code:
"cn=$c "

As for what I am doing with the code,
1. I am taking the results and types of detections from an antivirus program
2. and separating them based on the type of detection represented by cn2=$c
3.and cutting out the timestamp from each result.
4. I then compare the timestamp on the detection to the current time to see if its under an hour old.
5. For every detection that is under an hour old a counter within an array is incremented based on the type of virus that was detected.
# 7  
Old 01-20-2019
Based on a few assumptions on the structure of your input file, like
- the numerical "detection type" is introduced by the string cn2=, up to four digits, and terminated by a space char.
- there's no more than one time stamp per line, introduced by the partial string rt=, 10 digits long


here's an awk proposal, working on a mock up input file, that might satisfy your request:
Code:
awk '
BEGIN                   {HRago = srand() - 3600                                                                                 # srand is seeded from the clock, and returns that value
                         for (DMX = n = split ("0 16 272 512 768 2048 2304 1024 1280", DETTYPE); n; n--) CNT[DETTYPE[n]] = 0;   # det. type, and counter (indexed by det type) arrays
                         TIMPAT = "rt=[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"                                       # define epoch time pattern with introducing string
                        }
match ($0, TIMPAT)      {if ((substr($0, RSTART+3, 10) > HRago                  ) &&                                            # epoch time greater than one hour ago AND
                             (match ($0, /cn2=[0-9]* /)                         ) &&                                            # det.type introducer found AND
                             ((DT = substr($0, RSTART+4, RLENGTH-5)) in CNT     )       ) CNT[DT]++                             # det type known in counter array --> incr. counter
                        }
END                     {for (i=1; i<=DMX; i++) print "Detection type:", DETTYPE[i], ", count:", CNT[DETTYPE[i]]                # output detection types and counters
                        }
' file
Detection type: 0 , count: 1
Detection type: 16 , count: 0
Detection type: 272 , count: 0
Detection type: 512 , count: 1
Detection type: 768 , count: 1
Detection type: 2048 , count: 0
Detection type: 2304 , count: 0
Detection type: 1024 , count: 0
Detection type: 1280 , count: 0


Last edited by RudiC; 01-20-2019 at 08:33 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Array not printing values if used in a loop

Hello! I'm making an English to Morse Code translator and I was able to mostly get it all working by looking through older posts here; however, I have one small problem. When I run it it's just printing spaces for where the characters should be. It runs the right amount of times, and if I try... (3 Replies)
Discussion started by: arcoleman10
3 Replies

2. Shell Programming and Scripting

Awk: printing column using for loop

Hello: I've input data: Input data --- 3:60069:C:T 60069 C T 1 0 0 1 0 0 1 0 0 1 0 0 1 --- 3:60079:A:G 60079 A G 1 0 0 0.988 0.012 0 1 0 0 1 0 0 1 --- rs186476240:60157:G:A 60157 G A 1 0 0 1 0 0 1 0 0 1 0 0 1 I edit/make first few columns before numbers (6th column) and want to... (4 Replies)
Discussion started by: genome
4 Replies

3. Shell Programming and Scripting

For Loop in shellscript - Printing Output for every iteration

for VGLIST in `lsvg -o` do CLOSED_OUT=`echo $VGLIST | lsvg -l $VGLIST | awk '{print $6 " " $7}' | grep closed` if ]; then echo "Filesystems $CLOSED_OUT in VG that are in Closed status" else echo "\n Some message" fi Above Code is working fine, but echo "Filesystems $CLOSED_OUT... (8 Replies)
Discussion started by: chandu123
8 Replies

4. Shell Programming and Scripting

Help, while loop and sed not producing desired output

Hello everyone, I need some assistance with what I thought would have been a very simple script. Purpose of Script: Script will parse through a source file and modify (search/replace) certain patterns and output to stdout or a file. Script will utilize a "control file" which will contain... (12 Replies)
Discussion started by: packetjockey
12 Replies

5. Shell Programming and Scripting

printing portion of the output usind sed/awk

friends, i am a newbie in scripting. could someone help me in selecting only the last column of below ps command output ? mqm 14 16466 0 Sep 15 ? 0:01 /opt/mqm/bin/runmqlsr -r -m QMGR.INBOUNDSSL -t TCP -p 1415 -i 5.1.26.5 mqm 12 16700 0 Sep 15 ? 0:00... (4 Replies)
Discussion started by: unahb1
4 Replies

6. Shell Programming and Scripting

printing positional charaters in a loop

#!/bin/bash usep=`df -hT | awk '{ print $5 }'` for (1=1,1<8,i++) output=`echo $usep | awk '{ print $i }'| cut -d'%' -f1` echo $output if then echo "critical value" i need to echo critical value if disk usage pecentage xceeds 10 and i am face problem in position marked red here i... (9 Replies)
Discussion started by: josgeorge
9 Replies

7. Shell Programming and Scripting

sed in while loop producing arithmetic output

Hi all Just wondering if someone can help me with this. I'm trying to write a script that processes the output of another file and prints only the lines I want from it. This is only the second script I have written so please bare with me here. I have referred to the literature and some of the... (3 Replies)
Discussion started by: javathecat
3 Replies

8. Shell Programming and Scripting

Sed subsitution on loop output in shell script

I have a forloop which checks a log for a set of 6 static IP addresses and each IP found is logged to a file which is then mailed to me. After the forloop I always have a text file that may contain up to 6 IP addresses or may contain 0. What I want to do is substitute the IP addresses (if any)... (2 Replies)
Discussion started by: Moxy
2 Replies

9. Shell Programming and Scripting

printing with awk through while loop

ive input file contains to clums a and b spreated by pipe a | b 123|456 323|455 and xyz contains other info about a and b now i want to print as follows: a | b | "info from xyz" but "info from xyz" might be more than 1 line and i want to keep the format to 3 cloums. how to do it?... (3 Replies)
Discussion started by: windows
3 Replies

10. UNIX for Dummies Questions & Answers

Silly question on printing for loop in AWK

Hi, One silly question. I would like to add statement like below and append to a file. I used the below code; however, it does not work. Can anyone please tell me what mistakes I have made? awk ' { for (i=1;i<=563;i++) print i }'>>output.txt Thanks. -Jason (1 Reply)
Discussion started by: ahjiefreak
1 Replies
Login or Register to Ask a Question