Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract & Manipulate continous data stream-- tcpdump

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract & Manipulate continous data stream-- tcpdump
# 1  
Old 04-14-2014
Extract & Manipulate continous data stream-- tcpdump
Hello;

I have this rather tricky problem to solve --(to me, anyways) ..

I am processing the following one liner with tcpdump..

Code:
 tcpdump -i T3501 -A ether host 00:1e:49:29:fc:c9 or ether host 00:1b:2b:86:ec:1b or ether host 00:21:1c:98:a4:08 and net 149.83.6.0/24 | grep --line-buffered -B 20 IBM-32 | awk '/IBM-32/ || ( /IP/ && /Flags \[\.\]\, ack/ )'

The output looks as follows:

Code:
11:55:33.824133 IP 167.26.199.44.playsta2-app > 149.83.6.1.8023: Flags [.], ack 2792, win 63227, length 0
11:55:33.825247 IP 167.26.227.168.4693 > 149.83.6.1.8023: Flags [.], ack 5307, win 64512, length 0
11:55:33.826140 IP 168.108.221.122.57406 > 149.83.6.64.8023: Flags [.], ack 1274289, win 513, length 0
11:55:33.826355 IP 168.108.220.104.50909 > 149.83.6.64.8023: Flags [.], ack 1531837, win 256, length 0
11:55:33.829913 IP 199.198.231.57.58935 > 149.83.6.64.8023: Flags [.], ack 111302, win 64512, length 0
E@.D.!@.t........S.@.$.W79p....dP......BM-3278-2-E.CC218085..
11:55:33.845867 IP 199.198.231.57.34945 > 149.83.6.128.8023: Flags [.], ack 1064, win 63449, length 0
E..D....9....S.@.....W.$...d79p,P.........(..IBM-3278-2-E.CC218085..
11:55:53.395263 IP 199.198.231.57.10464 > 149.83.6.64.8023: Flags [.], ack 16186, win 64512, length 0
11:55:53.400435 IP 168.108.220.104.50909 > 149.83.6.64.8023: Flags [.], ack 2096906, win 256, length 0
E@.D..@.t..R.....S.....Wi!.4.$.8P......BM-3278-2-E.CC210147..
11:55:53.417919 IP 167.26.104.157.stat-scanner > 149.83.6.64.8023: Flags [.], ack 15970, win 64512, length 0
11:55:53.418914 IP 168.108.221.122.57407 > 149.83.6.64.8023: Flags [.], ack 40988, win 509, length 0
11:55:53.425586 IP 199.198.231.57.10498 > 149.83.6.64.8023: Flags [.], ack 274360, win 63452, length 0
11:55:53.431282 IP 168.108.221.122.57406 > 149.83.6.64.8023: Flags [.], ack 1739414, win 513, length 0
E..DhC..9..e.S.......W...$.8i!.PP.........(..IBM-3278-2-E.CC210147..

I need to extract, for each unique IBM-3278 expr, the previous ip_addr before the ">" sign..so that hopefilly end up with, e.g :

Code:
IBM-3278-2-E.CC210147,  IP 168.108.221.122.57407, IP 199.198.231.57.10498, IP 168.108.221.122.57406

So I tried reversing the output with "tac" in the end but nothing happened

Then I thought of using csplit with the IBM as the delimiter but its compaling ..
Code:
 tcpdump -i T3501 -A ether host 00:1e:49:29:fc:c9 or ether host 00:1b:2b:86:ec:1b or ether host 00:21:1c:98:a4:08 and net 149.83.6.0/24 | grep --line-buffered -B 20 IBM-32 | awk '/IBM-32/ || ( /IP/ && /Flags \[\.\]\, ack/ )'|xargs csplit /IBM/

Code:
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
csplit: cannot open `/IBM/' for reading: No such file or directory

so that I can process the split files using a cron job or a daemon..

Any ideas are apptreciated.. Thank you
Last edited by vbe; 04-14-2014 at 01:27 PM.. Reason: code tags please not icode
# 2  
Old 04-14-2014
let's just save each "host > host" line to a variable, and print it when we find IBM-32?
i'm assuming we can find those because they start with a time vs whitespace

Code:
tcpdump | awk '
  /^[0-9]*:[0-9]*/ {src=$3}
  match($0, /IBM-32[^.]*\.[^.]*/) {
    str=substr($0,RSTART,RLENGTH)]
    ibm[str]=src,ibm[str]
  }
  END {
    for (i in ibm) print i, ibm[i]
  }
'

This uses END so you don't get an output until it's DONE which wouldn't really work for realtime output from tcpdump.

Code:
$ cat tcpdump | awk '/^[0-9]*:[0-9]*/ {src=$3} match($0, /BM-32[^.]*\.[^.]*/) { str=substr($0,RSTART,RLENGTH); ibm[str]=src FS ibm[str] } END { for (i in ibm)  { print i, ibm[i] } }'
BM-3278-2-E.CC218085 199.198.231.57.34945 199.198.231.57.58935
BM-3278-2-E.CC210147 168.108.221.122.57406 168.108.220.104.50909

some of them were missing the I in your output here
# 3  
Old 04-14-2014
Thank you very much but I am not getting any output..

Code:
 tcpdump | awk '/^[0-9]*:[0-9]*/ {src=$3} match($0, /IBM-32[^.]*\.[^.]*/) { str=substr($0,RSTART,RLENGTH);\
>  ibm[str]=src FS ibm[str] } END { for (i in ibm)  { print i, ibm[i] } }'
tcpdump: WARNING: Shared: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on Shared, link-type EN10MB (Ethernet), capture size 65535 bytes

# 4  
Old 04-14-2014
Just as a matter of interest there are 4 off "BM-3278" of which 2 are "IBM-3278".

Do we take it that the other two "BM-3278" parts are not required?
OR...
Are all 4 required?
# 5  
Old 04-15-2014
sorry .. that was my typo.. all should be "IBM-3278"

Thnx

---------- Post updated 04-15-14 at 10:38 AM ---------- Previous update was 04-14-14 at 07:04 PM ----------

ok so I made some headway ..added bit more awk filtering:

Code:
tcpdump -i T3501 -A ether host 00:1e:49:29:fc:c9 or ether host 00:1b:2b:86:ec:1b or ether host 00:21:1c:98:a4:08 and net 149.83.6.0/24 | grep --line-buffered -B 20 IBM-32 | awk '/IBM-32/ || ( /IP/ && /Flags \[\.\]\, ack/ )' | awk '/IBM/ {print $0} /IP/ {print "Source-IP= "$3}'

and the output is like:

Code:
Source-IP= 199.198.231.57.59033
E@.D..@.p.g......S.@.G.W$.....W{P....^....(..IBM-3278-2-E.CDC13117..
Source-IP= 168.108.167.244.50411
Source-IP= 168.108.167.244.50413
E..DL0..9....S.@.....W.G..W{$...P....9....(..IBM-3278-2-E.CDC13117..
Source-IP= 199.198.231.57.34947
Source-IP= 168.108.167.244.50411

So my nextt task is, how to do further filtering so that I end up with, e.g.:

Code:
Source-IP= 199.198.231.57.59033
IBM-3278-2-E.CDC13117..
Source-IP= 168.108.167.244.50411
Source-IP= 168.108.167.244.50413
IBM-3278-2-E.CDC13117..
Source-IP= 199.198.231.57.34947
Source-IP= 168.108.167.244.50411

Thnx
# 6  
Old 04-16-2014
finally solved it..

Code:
tcpdump -i T3501 -A ether host 00:1e:49:29:fc:c9 or ether host 00:1b:2b:86:ec:1b or ether host 00:21:1c:98:a4:08 and net 149.83.6.0/24 \
| grep --line-buffered -B 20 IBM-32 | awk '/IBM-32/ || ( /IP/ && /Flags \[\.\]\, ack/ )' \
| awk '/IBM/{ split($0,A,"IBM");   system("date");  print "Term-ID= IBM-"A[2] }   /IP/{ print "Source-IP= "$3 }'

Code:
Wed Apr 16 11:19:58 EDT 2014
Term-ID= IBM--3278-2-E.CC214070..
Source-IP= 199.198.231.57.12596
Source-IP= 168.108.167.244.60976
Source-IP= 199.198.231.57.59263
Wed Apr 16 11:19:58 EDT 2014
Term-ID= IBM--3278-2-E.CC214070..
Source-IP= 168.108.220.104.57107
Source-IP= 168.108.167.244.60976
Source-IP= 168.108.221.122.49326
Source-IP= 167.26.185.245.krb5gatekeeper
Wed Apr 16 11:19:58 EDT 2014
Term-ID= IBM--3278-2-E.CDC06151..
Wed Apr 16 11:19:58 EDT 2014
Term-ID= IBM--3278-2-E.CDC06151..
Source-IP= 168.108.167.244.60976
Wed Apr 16 11:19:58 EDT 2014
Term-ID= IBM--3278-2-E.cdc18155..
Source-IP= 168.108.220.104.57107
Source-IP= 168.108.167.244.60976
Source-IP= 168.108.167.244.60980

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need help to manipulate data using script

Hi i want to manipulate my data to convert row to column name 600 Slno vlan 1 600 2 609 3 700 name 700 Slno vlan 1 600 2 609 3 700 (8 Replies)
Discussion started by: nith_anandan
8 Replies

2. Shell Programming and Scripting

Manipulate XML File Continous STRING by each Order Line using SHELL

heres sample File: <?xml version="1.0"?> <!DOCTYPE cXML SYSTEM "www"><cXML.............................................. <OrderRequest>USE UNIX.com</Extrinsic><Extrinsic name="UniqueName">Peter@UNIX.com</Extrinsic><Extrinsic name="ContractingEntity">UNIX... (3 Replies)
Discussion started by: Pete.kriya
3 Replies

3. Shell Programming and Scripting

Extract a specific line from a stream

Hello, I'm trying to code a bash script and I was wondering how to extract a specific line from a stream. E.g. My file "file" contains three lines and i'd like to find a function f which returns after execution a specific line like the second line, which would be : f(file, 2) = Second... (4 Replies)
Discussion started by: Oddant
4 Replies

4. Shell Programming and Scripting

Extract data from an XML file & write into a CSV file

Hi All, I am having an XML tag like: <detail sim_ser_no_1="898407109001000090" imsi_1="452070001000090"> <security>ADM1=????</security> <security>PIN1=????</security> <security>PIN2=????</security> ... (2 Replies)
Discussion started by: ss_ss
2 Replies

5. OS X (Apple)

Text stream K&R exercises

Hello, ladies, gentlemen. First I suppose I should introduce myself. I've been poking at C since a long time ago, somewhere around 1990. (Don't misinterpret that. "Poking at C", in this statement, means that I jumped on it, studied it for anything from a day to a weekend to a finished "Hello,... (21 Replies)
Discussion started by: Jammer Six
21 Replies

6. Shell Programming and Scripting

Extract File line and manipulate

How can I print a section of each line in a text file. Eg CODE1 XYR Test2 10319389 CODE2 XYR Test2 10319389 CODE3 XYR Test2 10319389 CODE4 XYR Test2 10319389 CODE5 XYR Test2 10319389 First thing that would be nice would a new file like, awk sed and substring may help but can't figure it... (6 Replies)
Discussion started by: kelseyh
6 Replies

7. Shell Programming and Scripting

how to manipulate with lines while playing with data

hello everyone, well I have a file which contains data, I want to add the data on hourly basis, like my file contains data for 24 hours, (so a total of 1440 ) lines. Now i want to add the data on hourly basis to get average values. like if I use (head) command it is ok for first go, but... (5 Replies)
Discussion started by: jojo123
5 Replies

8. Shell Programming and Scripting

manipulate & format the output of spool command

Hi All, I am spooling the data some sql queries into a single file but wanted to know how to format the data of the file generated by spool. #!/bin/sh unset -f USAGE USAGE () { clear echo "############################USAGE#######################\n" echo "Incorrect number of... (2 Replies)
Discussion started by: ss_ss
2 Replies

9. UNIX for Dummies Questions & Answers

Excel data manipulate

All, I have the following format of data in a spreadsheet A 1 2 3 4 B 1 2 3 4 where 'A' is value of 'A1', '1 2 3 4' is value of cell B1, 'B' is value of cell A2, and '1 2 3 4' is value of cell B2. There... (12 Replies)
Discussion started by: rahulrathod
12 Replies

10. Shell Programming and Scripting

manipulate data with specific format

Hi everybody: I have a problem with how I have to manipulate the data which have specific format like this: 249. 0.30727021E+05 0.30601627E+05 0.37470780E-01 -0.44745335E+02 0.82674536E+03 248. 0.30428182E+05 0.30302787E+05 0.40564921E-01 -0.45210293E+02 ... (5 Replies)
Discussion started by: tonet
5 Replies
Login or Register to Ask a Question