Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Placing Duplicate Lines per section into another file
Top Forums Shell Programming and Scripting Placing Duplicate Lines per section into another file Post 302390289 by petersf on Wednesday 27th of January 2010 12:24:56 PM
Old 01-27-2010
Placing Duplicate Lines per section into another file
Hello,

I need help in putting duplicate lines within a section into another file. Here is what I'm struggling with:

Using this file “data.txt”:
Code:
 
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 587.6580
ABC5 594.0000 588.0971
ABC5 594.5000 588.5361
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Header Table
ABC4
ABC5 594.5000 588.5361 <= to keep in this section
ABC5 601.0000 594.1603
ABC5 601.5000 594.6121
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642 <= dup need to remove

Duplicates need to be placed in “data.dup”
Code:
 
ABC1 012345 header
ABC5 593.5000 587.6580 
ABC1 67890 header
ABC5 602.0000 595.0642

I've been using the current code to find the duplicates, but have yet to find a method to get the duplicates into the “data.dup” file according to the section the duplicate was found.

Code:
# This will remove the duplicate lines and place the unique lines in “data.out”
awk '{ if ($1=="ABC1") {delete arr}
      if ( !arr[$0]++ ) { print $0 } }' data.txt > data.out
 
# This will extract the duplicates into the “data.dup” file.  This works, but I need it 
# to be by section, with a section header
awk '/ABC1/{ ABC1 = $2 } ++arr[ABC1,$0] > 1' data.txt > data.dup

I am using Bourne Shell (/bin/sh)

Any help would be GREATL:Y appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

2. Shell Programming and Scripting

Placing lines file in array

Hiya I am fairly new to UNIX and have been asked to write some scripts.... I am working in the korn shell. What I am trying to do is to go through a delimited file: Testingline1;test;test Testingline2;test;test and place the lines into an Array so: Array = Testingline1;test;test Array... (4 Replies)
Discussion started by: ThomasvdH
4 Replies

3. Shell Programming and Scripting

Using Sed to duplicate a section of a file....

hello all, I have a file like this: section 1 blah1 blah2 section 2 blah1 blah2 section 3 blah1 blah2 and I want to use sed to duplicate section 2, like this: section 1 blah1 blah2 section 2 blah1 blah2 section 2 blah1 (2 Replies)
Discussion started by: nick26
2 Replies

4. Shell Programming and Scripting

Removing Duplicate Lines per Section

Hello, I am in need of removing duplicate lines from within a file per section. File: ABC1 012345 header ABC2 7890-000 ABC3 012345 Header Table ABC4 ABC5 593.0000 587.4800 ABC5 593.5000 587.6580 <= dup need to remove ABC5 593.5000 ... (5 Replies)
Discussion started by: petersf
5 Replies

5. Shell Programming and Scripting

Duplicate Line Report per Section

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance. Here are the file.dat contents: ABC1... (4 Replies)
Discussion started by: petersf
4 Replies

6. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

7. Shell Programming and Scripting

Extract section of file based on word in section

I have a list of Servers in no particular order as follows: virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection). 9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS... (2 Replies)
Discussion started by: jelloir
2 Replies

8. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

9. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

10. Shell Programming and Scripting

Adding a lines to specific section of the file.

Hello, I have to a add 2 lines to /etc/sudoers file under this section below, can someone please suggest script to add these two lines when execute this remotely on to a multiple servers. before ## Allow root to run any commands anywhere root ALL=(ALL) ALL After ## Allow root... (2 Replies)
Discussion started by: bobby320
2 Replies

LEARN ABOUT DEBIAN

mdbfontsize

mdbFontSize(5)							 The m17n Library						    mdbFontSize(5)

NAME

       mdbFontSize - Font Size

DESCRIPTION

       In some case, a font contains incorrect information about its size (typically in the case of a hacked TrueType font), which results in a
       bad text layout when such a font is used in combination with the other fonts. To overcome this problem, the m17n library loads information
       about font-size adjustment from the m17n database by the tags <font, resize>. The data is loaded as a plist of this format.

       FONT-SIZE-ADJUSTMENT ::= PER-FONT *

       PER-FONT ::= '(' FONT-SPEC ADJUST-RATIO ')'

       FONT-SPEC ::=
	   '(' [ FOUNDRY FAMILY
	      [ WEIGHT [ STYLE [ STRETCH [ ADSTYLE ]]]]]
	    REGISTRY ')'

       ADJUST-RATIO ::= INTEGER

       FONT-SPEC is to specify properties of a font. FOUNDRY to REGISTRY are symbols corresponding to Mfoundry to Mregistry property of a font.
       See m17nFont for the meaning of each property.

       ADJUST-RATIO is an integer number specifying by percentage how much the font-size must be adjusted. For instance, this PER-FONT:

	   ((devanagari-cdac) 150)

       instructs the font handler of the m17n library to open a font of 1.5 times bigger than a requested size on opening a font whose registry is
       'devanagari-cdac'.

COPYRIGHT

       Copyright (C) 2001 Information-technology Promotion Agency (IPA)
       Copyright (C) 2001-2011 National Institute of Advanced Industrial Science and Technology (AIST)
       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License
       <http://www.gnu.org/licenses/fdl.html>.

Version 1.6.2							    12 Jan 2011 						    mdbFontSize(5)
All times are GMT -4. The time now is 11:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy