Sponsored Content
Top Forums Shell Programming and Scripting Bash/shell merge similar lines Post 302966904 by umang2382 on Wednesday 17th of February 2016 06:43:00 PM
Old 02-17-2016
Bash/shell merge similar lines

Hello,

I've been working on a bash script to parse through firewall logs (cisco). I'm nearing the end and have a dilemma.

My data looks as such (actual data is several gigs worth of logs - without the headers):
sourceIP destinationIP destinationProtocol destinationPort
Code:
1.1.1.1   2.2.2.2         TCP                       22
1.1.1.1   2.2.2.2         TCP                       31
1.1.1.1   2.2.2.2         TCP                       45
1.1.1.1   2.2.2.2         TCP                       67
1.1.1.3   2.2.2.2         TCP                       22
1.1.1.3   2.2.2.2         TCP                       89
1.1.1.3   2.2.2.2         TCP                       78
1.1.1.1   2.2.2.3         TCP                       78
1.1.1.1   2.2.2.3         TCP                       79


I would like to script it so that the ports are put on a single line for matching IPs, like so:
sourceIP destinationIP destinationProtocol destinationPort
Code:
1.1.1.1   2.2.2.2         TCP                       22, 31, 45, 67
1.1.1.3   2.2.2.2         TCP                       22, 89, 78
1.1.1.1   2.2.2.3         TCP                       78, 79

Would awk or sed be able to do what I'm looking for? How?

Any help would be much appreciated.

Last edited by vgersh99; 02-17-2016 at 07:48 PM.. Reason: code tags, please!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Urgent : Merge similar lines

Hi, I have a file like this. please notice that ./usr/orders1/order_new_2627 appears more than once, thus needs to be merged. I would like to merge the lines where the first column match so the output should be like this: Please help (2 Replies)
Discussion started by: rakeshou
2 Replies

2. Shell Programming and Scripting

Help with comparing 2 lines in 2 different file in shell bash

Hi guys i need help with comparing lines in 2 separate files. Both files contain the same amount of lines and i need to output the difference into the 2nd file. The 1st file is always correct. 1st file (Expected.e): Tuesday, 11 August 2009 Wednesday, 12 August 2009 Thursday, 13 August 2009... (2 Replies)
Discussion started by: kcrap
2 Replies

3. Shell Programming and Scripting

merge similar rows

I have a large file (10M lines) that contains two columns: a frequency and a string, ex: 3 aaaaa 4 bbbbb 2 ccccc 5 aaaaa 1 ddddd 4 ccccc I need to merge the lines whose string part is the same, while updating the frequency. The output should look like this: 8 aaaaa 4 bbbbb 5 ccccc... (2 Replies)
Discussion started by: tootles564
2 Replies

4. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

5. UNIX for Dummies Questions & Answers

Merge two files with two columns being similar

Hi everyone. How can I merge two files, where each file has 2 columns and the first columns in both files are similar? I want all in a file of 4 columns; join command removes the duplicate columns. 1 Dave 2 Mark 3 Paul 1 Apple 2 Orange 3 Grapes to get it like this in the 3rd file:... (9 Replies)
Discussion started by: Atrisa
9 Replies

6. UNIX for Dummies Questions & Answers

merge lines within a file that start with a similar pattern

Hello! i have a text file.. which contains the data as follows i want to merge the declarations lines pertaining to one datatype in to a single line as follows i've searched the forum for help.. but couldn't find much help.. how can i do this?? (1 Reply)
Discussion started by: a_ba
1 Replies

7. Shell Programming and Scripting

Help with merge two file based on similar column content

Input file 1: A1BG A1BG A1BG A1CF A1CF BCAS BCAS A2LD1 A2M A2M HAT . . Input file 2: A1BG All A1CF TEMP (5 Replies)
Discussion started by: perl_beginner
5 Replies

8. Shell Programming and Scripting

Merge two files with similar column entries

Hi , I have few files which contains user name and data transfer rate in MBs and this data is collected for year and for each month report is saved in 12 different files I have to merge all the files to prepare the final report Files are as below Similarly I have 10 more files ... (5 Replies)
Discussion started by: pratapsingh
5 Replies

9. Shell Programming and Scripting

Shell script to merge and delete lines

POLY_STATS { EqName 103_tri Id 123 act_polyCount 1 act_polyValue 0 } POLY_STATS { EqName 103_tri Id 123 pass_polyCount 2 pass_polyValue 0 } POLY_STATS { EqName 103 Id 123 mes_polyCount 2 mes_polyValue 0 (5 Replies)
Discussion started by: Jag02
5 Replies

10. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies
FAITH(4)						   BSD Kernel Interfaces Manual 						  FAITH(4)

NAME
faith -- IPv6-to-IPv4 TCP relay capturing interface SYNOPSIS
device faith DESCRIPTION
The faith interface captures IPv6 TCP traffic, for implementing userland IPv6-to-IPv4 TCP relay like faithd(8). Each faith interface is created at runtime using interface cloning. This is most easily done with the ifconfig(8) create command or using the cloned_interfaces variable in rc.conf(5). Special action will be taken when IPv6 TCP traffic is seen on a router, and the routing table suggests to route it to the faith interface. In this case, the packet will be accepted by the router, regardless of the list of IPv6 interface addresses assigned to the router. The packet will be captured by an IPv6 TCP socket, if it has the IN6P_FAITH flag turned on and matching address/port pairs. As a result, faith will let you capture IPv6 TCP traffic to some specific destination addresses. Userland programs, such as faithd(8) can use this behavior to relay IPv6 TCP traffic to IPv4 TCP traffic. The program can accept some specific IPv6 TCP traffic, perform getsockname(2) to get the IPv6 destination address specified by the client, and perform application-specific address mapping to relay IPv6 TCP to IPv4 TCP. The IN6P_FAITH flag on a IPv6 TCP socket can be set by using setsockopt(2), with level IPPROTO_IPV6 and optname IPv6_FAITH. To handle error reports by ICMPv6, some ICMPv6 packets routed to an faith interface will be delivered to IPv6 TCP, as well. To understand how faith can be used, take a look at the source code of faithd(8). As the faith interface implements potentially dangerous operations, great care must be taken when configuring it. To avoid possible misuse, the sysctl(8) variable net.inet6.ip6.keepfaith must be set to 1 prior to using the interface. When net.inet6.ip6.keepfaith is 0, no packets will be captured by the faith interface. The faith interface is intended to be used on routers, not on hosts. SEE ALSO
inet(4), inet6(4), faithd(8) Jun-ichiro itojun Hagino and Kazu Yamamoto, An IPv6-to-IPv4 transport relay translator, RFC3142. HISTORY
The FAITH IPv6-to-IPv4 TCP relay translator first appeared in the WIDE hydrangea IPv6 stack. BSD
April 10, 1999 BSD
All times are GMT -4. The time now is 11:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy