Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Placing Duplicate Lines per section into another file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Placing Duplicate Lines per section into another file
# 1  
Old 01-27-2010
Placing Duplicate Lines per section into another file
Hello,

I need help in putting duplicate lines within a section into another file. Here is what I'm struggling with:

Using this file “data.txt”:
Code:
 
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 587.6580
ABC5 594.0000 588.0971
ABC5 594.5000 588.5361
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Header Table
ABC4
ABC5 594.5000 588.5361 <= to keep in this section
ABC5 601.0000 594.1603
ABC5 601.5000 594.6121
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642 <= dup need to remove

Duplicates need to be placed in “data.dup”
Code:
 
ABC1 012345 header
ABC5 593.5000 587.6580 
ABC1 67890 header
ABC5 602.0000 595.0642

I've been using the current code to find the duplicates, but have yet to find a method to get the duplicates into the “data.dup” file according to the section the duplicate was found.

Code:
# This will remove the duplicate lines and place the unique lines in “data.out”
awk '{ if ($1=="ABC1") {delete arr}
      if ( !arr[$0]++ ) { print $0 } }' data.txt > data.out
 
# This will extract the duplicates into the “data.dup” file.  This works, but I need it 
# to be by section, with a section header
awk '/ABC1/{ ABC1 = $2 } ++arr[ABC1,$0] > 1' data.txt > data.dup

I am using Bourne Shell (/bin/sh)

Any help would be GREATL:Y appreciated.
# 2  
Old 01-27-2010
Try this one:
Code:
awk 'NF==3 && /header/; $0!=s{s=$0;next}1' file

# 3  
Old 01-27-2010
Franklin52 - You Are Awsome!!! I've spent a good 8 hours trying to figure this out (spanding over a few days). May I ask what book I can learn more on advanced scripting?
# 4  
Old 01-27-2010
Quote:
Originally Posted by petersf
Franklin52 - You Are Awsome!!! I've spent a good 8 hours trying to figure this out (spanding over a few days). May I ask what book I can learn more on advanced scripting?
You can check one of the links here or Google for more:

https://www.unix.com/answers-frequent...tutorials.html

And this important, stay tuned here, here is where you can master the stuff! Smilie

Regards
# 5  
Old 01-27-2010
Thanks Man! I appreciate your time. Don't worry about my leaving. This is the real work-related stuff here!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Adding a lines to specific section of the file.

Hello, I have to a add 2 lines to /etc/sudoers file under this section below, can someone please suggest script to add these two lines when execute this remotely on to a multiple servers. before ## Allow root to run any commands anywhere root ALL=(ALL) ALL After ## Allow root... (2 Replies)
Discussion started by: bobby320
2 Replies

2. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

3. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

4. Shell Programming and Scripting

Extract section of file based on word in section

I have a list of Servers in no particular order as follows: virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection). 9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS... (2 Replies)
Discussion started by: jelloir
2 Replies

5. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

6. Shell Programming and Scripting

Duplicate Line Report per Section

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance. Here are the file.dat contents: ABC1... (4 Replies)
Discussion started by: petersf
4 Replies

7. Shell Programming and Scripting

Removing Duplicate Lines per Section

Hello, I am in need of removing duplicate lines from within a file per section. File: ABC1 012345 header ABC2 7890-000 ABC3 012345 Header Table ABC4 ABC5 593.0000 587.4800 ABC5 593.5000 587.6580 <= dup need to remove ABC5 593.5000 ... (5 Replies)
Discussion started by: petersf
5 Replies

8. Shell Programming and Scripting

Using Sed to duplicate a section of a file....

hello all, I have a file like this: section 1 blah1 blah2 section 2 blah1 blah2 section 3 blah1 blah2 and I want to use sed to duplicate section 2, like this: section 1 blah1 blah2 section 2 blah1 blah2 section 2 blah1 (2 Replies)
Discussion started by: nick26
2 Replies

9. Shell Programming and Scripting

Placing lines file in array

Hiya I am fairly new to UNIX and have been asked to write some scripts.... I am working in the korn shell. What I am trying to do is to go through a delimited file: Testingline1;test;test Testingline2;test;test and place the lines into an Array so: Array = Testingline1;test;test Array... (4 Replies)
Discussion started by: ThomasvdH
4 Replies

10. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies
Login or Register to Ask a Question