Bash/shell merge similar lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash/shell merge similar lines
# 1  
Old 02-17-2016
Bash/shell merge similar lines

Hello,

I've been working on a bash script to parse through firewall logs (cisco). I'm nearing the end and have a dilemma.

My data looks as such (actual data is several gigs worth of logs - without the headers):
sourceIP destinationIP destinationProtocol destinationPort
Code:
1.1.1.1   2.2.2.2         TCP                       22
1.1.1.1   2.2.2.2         TCP                       31
1.1.1.1   2.2.2.2         TCP                       45
1.1.1.1   2.2.2.2         TCP                       67
1.1.1.3   2.2.2.2         TCP                       22
1.1.1.3   2.2.2.2         TCP                       89
1.1.1.3   2.2.2.2         TCP                       78
1.1.1.1   2.2.2.3         TCP                       78
1.1.1.1   2.2.2.3         TCP                       79


I would like to script it so that the ports are put on a single line for matching IPs, like so:
sourceIP destinationIP destinationProtocol destinationPort
Code:
1.1.1.1   2.2.2.2         TCP                       22, 31, 45, 67
1.1.1.3   2.2.2.2         TCP                       22, 89, 78
1.1.1.1   2.2.2.3         TCP                       78, 79

Would awk or sed be able to do what I'm looking for? How?

Any help would be much appreciated.

Last edited by vgersh99; 02-17-2016 at 07:48 PM.. Reason: code tags, please!
# 2  
Old 02-17-2016
something along these lines - a bit succinct on explanation, but... :
Code:
awk '
  # idx is an index constructed by concatenating first 3 fields in the line
  {idx=$1 FS $2 FS $3} 

  # a - an array indexed by 'idx'.
  # if idx is in a, add forth field to it; if not, assign forth field as the first entry in array a
  {a[idx]=(idx in a)?a[idx] OFS $4:$4} 

  # after all records/lines have been process...
  # iterate through all the indecies in array a, print out the value of the index and
  # the corresponding value stored in the array with the current index.
  END {for(i in a) print i FS a[i]}' OFS=, myInputFileGoesHere


Last edited by vgersh99; 02-17-2016 at 08:39 PM..
This User Gave Thanks to vgersh99 For This Post:
# 3  
Old 02-17-2016
cool, that works! mind if you give a quick explanation of what the command is doing?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies

2. Shell Programming and Scripting

Shell script to merge and delete lines

POLY_STATS { EqName 103_tri Id 123 act_polyCount 1 act_polyValue 0 } POLY_STATS { EqName 103_tri Id 123 pass_polyCount 2 pass_polyValue 0 } POLY_STATS { EqName 103 Id 123 mes_polyCount 2 mes_polyValue 0 (5 Replies)
Discussion started by: Jag02
5 Replies

3. Shell Programming and Scripting

Merge two files with similar column entries

Hi , I have few files which contains user name and data transfer rate in MBs and this data is collected for year and for each month report is saved in 12 different files I have to merge all the files to prepare the final report Files are as below Similarly I have 10 more files ... (5 Replies)
Discussion started by: pratapsingh
5 Replies

4. Shell Programming and Scripting

Help with merge two file based on similar column content

Input file 1: A1BG A1BG A1BG A1CF A1CF BCAS BCAS A2LD1 A2M A2M HAT . . Input file 2: A1BG All A1CF TEMP (5 Replies)
Discussion started by: perl_beginner
5 Replies

5. UNIX for Dummies Questions & Answers

merge lines within a file that start with a similar pattern

Hello! i have a text file.. which contains the data as follows i want to merge the declarations lines pertaining to one datatype in to a single line as follows i've searched the forum for help.. but couldn't find much help.. how can i do this?? (1 Reply)
Discussion started by: a_ba
1 Replies

6. UNIX for Dummies Questions & Answers

Merge two files with two columns being similar

Hi everyone. How can I merge two files, where each file has 2 columns and the first columns in both files are similar? I want all in a file of 4 columns; join command removes the duplicate columns. 1 Dave 2 Mark 3 Paul 1 Apple 2 Orange 3 Grapes to get it like this in the 3rd file:... (9 Replies)
Discussion started by: Atrisa
9 Replies

7. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

8. Shell Programming and Scripting

merge similar rows

I have a large file (10M lines) that contains two columns: a frequency and a string, ex: 3 aaaaa 4 bbbbb 2 ccccc 5 aaaaa 1 ddddd 4 ccccc I need to merge the lines whose string part is the same, while updating the frequency. The output should look like this: 8 aaaaa 4 bbbbb 5 ccccc... (2 Replies)
Discussion started by: tootles564
2 Replies

9. Shell Programming and Scripting

Help with comparing 2 lines in 2 different file in shell bash

Hi guys i need help with comparing lines in 2 separate files. Both files contain the same amount of lines and i need to output the difference into the 2nd file. The 1st file is always correct. 1st file (Expected.e): Tuesday, 11 August 2009 Wednesday, 12 August 2009 Thursday, 13 August 2009... (2 Replies)
Discussion started by: kcrap
2 Replies

10. Shell Programming and Scripting

Urgent : Merge similar lines

Hi, I have a file like this. please notice that ./usr/orders1/order_new_2627 appears more than once, thus needs to be merged. I would like to merge the lines where the first column match so the output should be like this: Please help (2 Replies)
Discussion started by: rakeshou
2 Replies
Login or Register to Ask a Question