Sponsored Content
Top Forums Shell Programming and Scripting Scripting to Duplicate Lines Based on Variable Post 302735147 by sjrupp on Friday 23rd of November 2012 02:40:52 PM
Old 11-23-2012
Scripting to Duplicate Lines Based on Variable

Greeting all! I could use some assistance please. Smilie

I've been searching for the best way to duplicate a line based on a variable in the next line.

Sample Data:
Code:
Nov 22 00:00:19 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:19 10.10.10.1 last message repeated 2 times
Nov 22 00:00:24 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:24 10.10.10.1 last message repeated 5 times
Nov 22 00:00:29 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:29 10.10.10.1 last message repeated 3 times

I want to duplicate the lines based on how many times the log reports it was was repeated. So in the above example, line 1 would be duplicated 2 times, line 3 would be duplicated 5 times and line 5 would be duplicated 3 times.

I'm at a loss how to accomplish my goal. Anyone have a suggestion?

Thanks

Last edited by Scrutinizer; 11-23-2012 at 05:56 PM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding duplicate lines and deleting folders based on them

Hi, I have research data, which is organized to 100 folders numbered 00-99. I have many sets of 100 folders, for different values of initial parameters. For some reason, the computer that ran the program to gather the data, didn't always create a unique seed for each folder. I anticipated that... (1 Reply)
Discussion started by: Jopi
1 Replies

2. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

3. Shell Programming and Scripting

AWK Duplicate lines multiple times based on a calculated value

Hi, I'm trying to create an XML sitemap of our dynamic ecommerce sites SEO Friendly URLs and am trying to create the initial page listing. I have a CSV file that looks like the following and need duplicate the lines based on a value which needs calculating. ... (2 Replies)
Discussion started by: jamesfx
2 Replies

4. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

5. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies

6. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

7. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

8. UNIX for Dummies Questions & Answers

awk solution to duplicate lines based on column

Hi experts, I have a tab-delimited file with one column containing values separated by a comma. I wish to duplicate the entire line for every value in that comma-delimited field. For example: $cat file 4444 4444 4444 4444 9990 2222,7777 6666 2222 ... (3 Replies)
Discussion started by: torchij
3 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Removing duplicate lines on first column based with pipe delimiter

Hi, I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines Command : sort -t'|' -nuk1 file.txt Input : 38376KZ|09/25/15|1.057 38376KZ|09/25/15|1.057 02006YB|09/25/15|0.859 12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies
DNET(8) 						    BSD System Manager's Manual 						   DNET(8)

NAME
dnet -- dumb networking library test program SYNOPSIS
dnet command args [...] DESCRIPTION
dnet is a simple test program for the dnet(3) library. It can be used to compose and transmit network datagrams as a Unix-style filter (e.g. reading from or writing to files and pipes) or modify the local system network configuration (including the ARP cache, firewall ruleset, net- work interfaces, and routing table). Payload generation commands addr address [...] Convert the address (specified as a hostname, IP address, or MAC address) into its binary representation on standard output. hex string [...] Convert the C-style escaped string (shellcode, for instance) into its binary representation on standard output. rand len Write len random bytes to standard output. Packet encapsulation commands eth [type type] [src mac] [dst mac] Prepend the data read from standard input with an Ethernet header on standard output. The Ethernet type may be specified as 'arp', 'ip', or as a hex, octal, or decimal number. arp [op op] [sha mac] [spa host] [tha mac] [tpa host] Prepend the data read from standard input with an ARP header on standard output. The ARP op may be specified as 'req', 'rep', 'revreq', 'revrep', or as a hex, octal, or decimal number. ip [tos num] [id num] [off offset] [ttl num] [proto protocol] [src host] [dst dst] Prepend the data read from standard input with an IP header on standard output. The fragmentation offset may be specified as a decimal number (optionally concatenated with '+' to indicate more fragments) or as a hex number. The protocol may be specified by name, or as a hex, octal, or decimal number. icmp [type num] [code num] Prepend the data read from standard input with an ICMP header on standard output. tcp [sport port] [dport port] [flags flags] [seq num] [ack num] [win num] [urp num] Prepend the data read from standard input with a TCP header on standard output. A port may be specified by name or hex, octal, or deci- mal number. The TCP flags may be specified as some combination of the characters in the set 'SAFRPU' or as a hex number. udp [sport port] [dport port] Prepend the data read from standard input with a UDP header on standard output. A port may be specified by name or hex, octal, or deci- mal number. Packet transmission commands send [device] Read a packet from standard input and send it over the network. If no device is specified, the packet is assumed to be an IP datagram and routed to its destination. Otherwise, the packet is assumed to be an Ethernet frame and is transmitted on the specified interface. Kernel interface commands arp show Display the kernel ARP cache. arp get host Display the kernel ARP entry for host. arp add host mac Add an ARP entry mapping the mac address for host. arp delete host Delete the ARP entry for host. fw show Display the kernel firewall ruleset. fw add|delete action direction device protocol src[:port[-max]] dst[:port[-max]] [type[/code]] Add a rule to or delete a rule from the active firewall ruleset. The action must be either 'allow' or 'block'. The direction must be either 'in' or 'out'. The device may specify an interface name, or 'any'. The protocol may be specified by name, or as a decimal num- ber. For TCP and UDP protocols, a port (or range, if specified with a max value) may be specified in decimal and appended to the source and/or destination address. For ICMP, a type (and optional code) may be specified in decimal. intf show Display the configuration of all network interfaces. intf get device Display the configuration for the interface specified by device. intf set device [alias host] [dst host] [inet host] [link mac] [up|down] [arp|noarp] Configure the interface specified by device. route show Display the kernel routing table. route get dst Display the route for the destination dst, specified as a hostname, IP address, or network prefix in CIDR notation. route add dst gw Add a route for the destination dst through the gateway gw. route delete dst Delete the route for the destination dst. EXAMPLES
Send a UDP datagram containing random shellcode: dnet hex "xebx1fx5ex89x76x08x31xc0x88x46x07x89" "x46x0cxb0x0bx89xf3x8dx4ex08x8dx56x0cxcdx80" "x31xdbx89xd8x40xcdx80xe8xdcxffxffxff/bin/sh" | dnet udp sport 555 dport 666 | dnet ip proto udp src 1.2.3.4 dst 5.6.7.8 | dnet send Save an ARP request in a file and send it twice: dnet arp op req sha 0:d:e:a:d:0 spa 10.0.0.3 tpa 10.0.0.4 | dnet eth type arp src 0:d:e:a:d:0 dst ff:ff:ff:ff:ff:ff > arp.pkt dnet send fxp0 < arp.pkt dnet send fxp0 < arp.pkt Send a fragmented ping packet: # Create ping packet with IP header, to set ICMP checksum echo "monkey monkey monkey monkey" | dnet icmp type 8 code 0 | dnet ip proto icmp src 1.2.3.4 dst 5.6.7.8 > ping.pkt # Chop off IP header dd if=ping.pkt of=ping.data bs=20 skip=1 # Fragment IP payload split -b 24 ping.data p. # Send fragments dnet ip id 1 off 0+ proto icmp src 1.2.3.4 dst 5.6.7.8 < p.aa | dnet send dnet ip id 1 off 24 proto icmp src 1.2.3.4 dst 5.6.7.8 < p.ab | dnet send SEE ALSO
dnet(3) AUTHORS
Dug Song <dugsong@monkey.org> BSD
October 17, 2001 BSD
All times are GMT -4. The time now is 09:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy