Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge lines with varying characters

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge lines with varying characters
# 1  
Old 02-18-2016
Merge lines with varying characters
Hi, I have a large set of data (firewall logs) that I'm trying to summarize. I've been able to write a script to consolidate the ports, now am looking to conslidate even further, based on IP.

Code:
Source Destination Type Port
192.168.5.108 192.168.11.12 TCP 1, 2, 3, 4, 5, 15
192.168.5.109 192.168.11.12 TCP 6, 7, 8, 9, 10, 11
192.168.5.110 192.168.11.12 TCP 12, 13
192.168.6.23 192.168.11.12 TCP 14, 15
192.168.5.108 192.168.11.13 TCP 10, 12, 13, 14, 15, 5
192.168.5.109 192.168.11.13 TCP 16, 17, 18, 19, 110, 111
192.168.5.110 192.168.11.13 TCP 112, 113
192.168.6.108 192.168.11.14 TCP 20, 22, 23, 24, 25, 6
192.168.6.109 192.168.11.14 TCP 26, 27, 28, 29, 210, 211
192.168.7.110 192.168.11.14 TCP 212, 213
192.168.6.23 192.168.11.14 TCP 214, 215

I'd like to script it so that the output would group all the source IP's, and their destination ports, going to the same destination IP:
SourceIP1,IP2,IP3,IP4 TCP DestinationIP DestinationPort1,P2,P3,P4,P5,P6......

example, the first destination of 192.168.11.12 would be summarized to look like so:
Code:
192.168.5.108,192.168.5.109,192.168.5.110,192.168.5.23 192.168.11.12 TCP 1, 2, 3, 4, 5, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15

Any help would be greatly appreciated!
Last edited by jim mcnamara; 02-18-2016 at 02:06 PM..
# 2  
Old 02-18-2016
An awk approach:-
Code:
awk '
        {
                port = $0
                sub(/.*[a-zA-Z]+ /,x,port)
                if ( ! ( ( $2 FS $1 ) in A_S_IP ) )
                        A_R_S_IP[$2] = A_R_S_IP[$2] ? A_R_S_IP[$2] FS $1 : $1
                if ( ! ( ( $2 FS $3 ) in A_TYPE ) )
                        A_R_TYPE[$2] = A_R_TYPE[$2] ? A_R_TYPE[$2] FS $3 : $3
                if ( ! ( ( $2 FS port ) in A_PORT ) )
                        A_R_PORT[$2] = A_R_PORT[$2] ? A_R_PORT[$2] ", " port : port

                A_D_IP[$2]
                A_S_IP[$2 FS $1]
                A_TYPE[$2 FS $3]
                A_PORT[$2 FS port]
        }
        END {
                for ( k in A_D_IP )
                        print A_R_S_IP[k], k, A_R_TYPE[k], A_R_PORT[k]
        }
' file

# 3  
Old 02-18-2016
Why don't you work immediately on the input file with the structure of your recent post
Code:
awk '
NR == 1 {print
         next
        }

        {IX = $2 FS $3
         if (!CT[$1 FS $2 FS $3]++) a[IX] = a[IX]?a[IX] "," $1:$1
         b[IX] = b[IX]?b[IX] "," $4:$4
        }

END     {for (i in a) print a[i] FS i FS b[i]
        }
'  file
Source Destination Type Port
192.168.5.108,192.168.5.109,192.168.5.110,192.168.6.23 192.168.11.12 TCP 1,2,3,4,5,15,6,7,8,9,10,11,12,13,14,15
192.168.5.108,192.168.5.109,192.168.5.110 192.168.11.13 TCP 10,12,13,14,15,5,16,17,18,19,110,111,112,113
192.168.6.108,192.168.6.109,192.168.7.110,192.168.6.23 192.168.11.14 TCP 20,22,23,24,25,6,26,27,28,29,210,211,212,213,214,215

There may be duplicate ports in the output which are not eliminated.
# 4  
Old 02-18-2016
Thanks Rudi.

This somewhat works. The output I get combines the source IPs just fine, however the destination ports are incomplete.

Code:
192.168.5.108 192.168.11.12 TCP 1, 2, 3, 4, 5, 15
192.168.5.109,192.168.5.110,192.168.6.23 192.168.11.12 TCP 6,,12,,14,
192.168.5.108,192.168.5.109,192.168.5.110 192.168.11.13 TCP 10,,16,,112,
192.168.6.108,192.168.6.109,192.168.7.110,192.168.6.23 192.168.11.14 TCP 20,,26,,212,,214,

Last edited by Don Cragun; 02-21-2016 at 02:53 AM.. Reason: Add missing CODE tags.
# 5  
Old 02-19-2016
If you change the line:
Code:
         b[IX] = b[IX]?b[IX] "," $4:$4

in RudiC's script to:
Code:
	 gsub(/, /, ",")
         b[IX] = b[IX]?b[IX] "," $4:$4

I think you'll get something closer to what you wanted.
# 6  
Old 02-19-2016
thanks! that made the output much better. Is there a way to tag a thread as "solved"?
# 7  
Old 02-19-2016
Quote:
Originally Posted by umang2382
thanks! that made the output much better. Is there a way to tag a thread as "solved"?
I'm glad that RudiC's suggestion and my minor tweak helped you get the output you wanted.

If you look at the tags attached to this thread at the top of this thread, you'll note that there are two tags already attached AND there is the following note:
Quote:
Mark threads "solved" by adding a "solved" tag to the thread tags (above); and always take a few seconds to properly tag all threads with appropriate technical terms and keywords. - Thank you.
So, just click on the words Edit Tags at the top right corner of the tags associated with this thread, and add the tag solved to mark the thread as solved.
This User Gave Thanks to Don Cragun For This Post:
Neo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge Lines

Hello I have an input as below this is test we are( ) one end of description I am looking for output this is test we are () one end of description (2 Replies)
Discussion started by: Tomlight
2 Replies

2. Shell Programming and Scripting

Merge lines

Hello I have a file with CAR 23 COLOR 12 CAR 44 COLOR 12 CAR 55 COLOR 20 SIZE BIG CAR 56 CAR 57 COLOR 11 How can merge the CAR and the COLOR + SIZE (if there are COLOR,SIZE) CAR 23 COLOR 12 CAR 44 COLOR 12 CAR 55 COLOR 20 SIZE BIG CAR 56 CAR 57 COLOR 11 Every line begin in... (4 Replies)
Discussion started by: sharong
4 Replies

3. Shell Programming and Scripting

Sed - merge lines bw 2 specific characters

Hi, I have a bash script and I am looking for a command that will merge specific lines together. Sample Data: registration time = 1300890272 Id = 1 setd = 0 tagunt = 26 tagId=6, length=8, value= tagId=9, length=5, value= tagId=7, length=2, value= tagId=16, length=2, value= tagId=32,... (8 Replies)
Discussion started by: Winsarc
8 Replies

4. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

5. UNIX for Dummies Questions & Answers

To merge a few lines to 1 line

Hi Experts, This is my input file. input.txt 0 /dev/fd 25 /var 1 /tmp 1 /var/run 1. If this file has single line, then leave it, print the single line else merge the 4 lines above into 1 line as below e.g (6 Replies)
Discussion started by: streddy
6 Replies

6. Shell Programming and Scripting

merge lines

Hi guys in input every 1st line 1st ID value located in 2nd line 1st occurrence . I need to print them down accordingly.. Thanx in advance input rs1040480_XXXXX.value rs1040481_YYYYY.value rs1040482_TXXXX.value 0.7408157 0.3410044 0.7408157 ... (7 Replies)
Discussion started by: stateperl
7 Replies

7. Shell Programming and Scripting

Merge two lines

Hi I have two lines of data formatted as displayed below shop, price, remark, date a,#N/A,order to -fd, 20091011 and would like it to be shop:a price:#N/A remark:order to -fd date:20091011 How can I do it? Many thanks (2 Replies)
Discussion started by: lalelle
2 Replies

8. Shell Programming and Scripting

Merge lines in Flat file based on first 5 characters

Hi I have the fixed width flat file having the following data 12345aaaaaaaaaabbbbbbbbbb 12365sssssssssscccccccccc 12365sssss 12367ddddddddddvvvvvvvvvv 12367 vvvvv Here the first column is length 5 second is length 10 third is length 10 if the second or third column exceeds... (3 Replies)
Discussion started by: Brado
3 Replies

9. Shell Programming and Scripting

Merge lines into one

Source data file from oracle, terminated by ",". 'Cause some of fields have \r\n, then those lines were splitted into multiple lines in the expoted data file. Just vi this file, and found ^M. How to concatenate these line into one if it has a ^M at then end. thanks, (7 Replies)
Discussion started by: anypager
7 Replies

10. Shell Programming and Scripting

deleting a varying amount of lines from a list of files

I did search the posts for info on this and while there were some in the ballpark, none addressed this specifically. (also I tried to post this once it said I was logged out, so hopefully I'm not sending a duplicate here). I have a set of files (250 +/-) where I need to delete the first "$x"... (4 Replies)
Discussion started by: benair
4 Replies
Login or Register to Ask a Question