Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Scripting to Duplicate Lines Based on Variable

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Scripting to Duplicate Lines Based on Variable
# 1  
Old 11-23-2012
Scripting to Duplicate Lines Based on Variable
Greeting all! I could use some assistance please. Smilie

I've been searching for the best way to duplicate a line based on a variable in the next line.

Sample Data:
Code:
Nov 22 00:00:19 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:19 10.10.10.1 last message repeated 2 times
Nov 22 00:00:24 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:24 10.10.10.1 last message repeated 5 times
Nov 22 00:00:29 10.10.10.1 "%ASA-4-313005: No matching connection for ICMP error message: icmp src Outside:1.2.3.4 dst Inside:69.161.160.162 (type 3, code 2) on Outside interface.  Original IP payload: <unknown>."
Nov 22 00:00:29 10.10.10.1 last message repeated 3 times

I want to duplicate the lines based on how many times the log reports it was was repeated. So in the above example, line 1 would be duplicated 2 times, line 3 would be duplicated 5 times and line 5 would be duplicated 3 times.

I'm at a loss how to accomplish my goal. Anyone have a suggestion?

Thanks
Last edited by Scrutinizer; 11-23-2012 at 05:56 PM.. Reason: code tags
# 2  
Old 11-23-2012
Code:
awk ' { line=$0; getline; times=$(NF-1); i=1; while(i<=times) { print line; i++; } } '  logfile

This User Gave Thanks to Yoda For This Post:
sjrupp
# 3  
Old 11-23-2012
This is fantastic! Thank you ever so much!
# 4  
Old 11-23-2012
Code:
awk '/last message/{for(i=1; i<$(NF-1); i++) print p; next} {p=$0}1' infile

This User Gave Thanks to Scrutinizer For This Post:
sjrupp
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicate lines on first column based with pipe delimiter

Hi, I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines Command : sort -t'|' -nuk1 file.txt Input : 38376KZ|09/25/15|1.057 38376KZ|09/25/15|1.057 02006YB|09/25/15|0.859 12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies

2. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

3. UNIX for Dummies Questions & Answers

awk solution to duplicate lines based on column

Hi experts, I have a tab-delimited file with one column containing values separated by a comma. I wish to duplicate the entire line for every value in that comma-delimited field. For example: $cat file 4444 4444 4444 4444 9990 2222,7777 6666 2222 ... (3 Replies)
Discussion started by: torchij
3 Replies

4. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

5. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

6. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies

7. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

8. Shell Programming and Scripting

AWK Duplicate lines multiple times based on a calculated value

Hi, I'm trying to create an XML sitemap of our dynamic ecommerce sites SEO Friendly URLs and am trying to create the initial page listing. I have a CSV file that looks like the following and need duplicate the lines based on a value which needs calculating. ... (2 Replies)
Discussion started by: jamesfx
2 Replies

9. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

10. Shell Programming and Scripting

Finding duplicate lines and deleting folders based on them

Hi, I have research data, which is organized to 100 folders numbered 00-99. I have many sets of 100 folders, for different values of initial parameters. For some reason, the computer that ran the program to gather the data, didn't always create a unique seed for each folder. I anticipated that... (1 Reply)
Discussion started by: Jopi
1 Replies
Login or Register to Ask a Question