Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Placing Duplicate Lines per section into another file
Top Forums Shell Programming and Scripting Placing Duplicate Lines per section into another file Post 302390289 by petersf on Wednesday 27th of January 2010 12:24:56 PM
Old 01-27-2010
Placing Duplicate Lines per section into another file
Hello,

I need help in putting duplicate lines within a section into another file. Here is what I'm struggling with:

Using this file “data.txt”:
Code:
 
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 587.6580
ABC5 594.0000 588.0971
ABC5 594.5000 588.5361
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Header Table
ABC4
ABC5 594.5000 588.5361 <= to keep in this section
ABC5 601.0000 594.1603
ABC5 601.5000 594.6121
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642 <= dup need to remove

Duplicates need to be placed in “data.dup”
Code:
 
ABC1 012345 header
ABC5 593.5000 587.6580 
ABC1 67890 header
ABC5 602.0000 595.0642

I've been using the current code to find the duplicates, but have yet to find a method to get the duplicates into the “data.dup” file according to the section the duplicate was found.

Code:
# This will remove the duplicate lines and place the unique lines in “data.out”
awk '{ if ($1=="ABC1") {delete arr}
      if ( !arr[$0]++ ) { print $0 } }' data.txt > data.out
 
# This will extract the duplicates into the “data.dup” file.  This works, but I need it 
# to be by section, with a section header
awk '/ABC1/{ ABC1 = $2 } ++arr[ABC1,$0] > 1' data.txt > data.dup

I am using Bourne Shell (/bin/sh)

Any help would be GREATL:Y appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

2. Shell Programming and Scripting

Placing lines file in array

Hiya I am fairly new to UNIX and have been asked to write some scripts.... I am working in the korn shell. What I am trying to do is to go through a delimited file: Testingline1;test;test Testingline2;test;test and place the lines into an Array so: Array = Testingline1;test;test Array... (4 Replies)
Discussion started by: ThomasvdH
4 Replies

3. Shell Programming and Scripting

Using Sed to duplicate a section of a file....

hello all, I have a file like this: section 1 blah1 blah2 section 2 blah1 blah2 section 3 blah1 blah2 and I want to use sed to duplicate section 2, like this: section 1 blah1 blah2 section 2 blah1 blah2 section 2 blah1 (2 Replies)
Discussion started by: nick26
2 Replies

4. Shell Programming and Scripting

Removing Duplicate Lines per Section

Hello, I am in need of removing duplicate lines from within a file per section. File: ABC1 012345 header ABC2 7890-000 ABC3 012345 Header Table ABC4 ABC5 593.0000 587.4800 ABC5 593.5000 587.6580 <= dup need to remove ABC5 593.5000 ... (5 Replies)
Discussion started by: petersf
5 Replies

5. Shell Programming and Scripting

Duplicate Line Report per Section

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance. Here are the file.dat contents: ABC1... (4 Replies)
Discussion started by: petersf
4 Replies

6. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

7. Shell Programming and Scripting

Extract section of file based on word in section

I have a list of Servers in no particular order as follows: virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection). 9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS... (2 Replies)
Discussion started by: jelloir
2 Replies

8. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

9. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

10. Shell Programming and Scripting

Adding a lines to specific section of the file.

Hello, I have to a add 2 lines to /etc/sudoers file under this section below, can someone please suggest script to add these two lines when execute this remotely on to a multiple servers. before ## Allow root to run any commands anywhere root ALL=(ALL) ALL After ## Allow root... (2 Replies)
Discussion started by: bobby320
2 Replies

LEARN ABOUT DEBIAN

mdbfontset

mdbFontset(5)							 The m17n Library						     mdbFontset(5)

NAME

       mdbFontset - Fontset

DESCRIPTION

       The m17n library loads a fontset definition from the m17n database by the tags <fontset, FONTSET-NAME>. The plist format of the data is as
       follows:

       FONTSET ::= PER-SCRIPT * PER-CHARSET * FALLBACK *

       PER-SCRIPT ::= '(' SCRIPT PER-LANGUAGE + ')'

       PER-LANGUAGE ::= '(' LANGUAGE FONT-SPEC-ELEMENT + ')'

       PER-CHARSET ::= '(' CHARSET FONT-SPEC-ELEMENT + ')'

       FALLBACK ::= FONT-SPEC-ELEMENT

       FONT-SPEC-ELEMENT ::= '(' FONT-SPEC [ FLT-NAME ] ')'

       FONT-SPEC ::=
	    '(' [ FOUNDRY FAMILY
		  [ WEIGHT [ STYLE [ STRETCH [ ADSTYLE ]]]]]
		REGISTRY
	     [ OTF-SPEC ] [ LANG-SPEC ] ')'

       SCRIPT is a symbol of script name (e.g. latin, han) or nil. LANGUAGE is a two-letter symbol of language name code defined by ISO 639 (e.g.
       ja, zh) or nil.

       FONT-SPEC is to specify properties of a font. FOUNDRY to REGISTRY are symbols corresponding to Mfoundry to Mregistry property of a font.
       See m17nFont for the meaning of each property.

       OTF-SPEC is a symbol specifyng the required OTF features. The symbol name has the following syntax.

	 OTF-SPEC-NAME ::= ':otf=' SCRIPT LANGSYS ? GSUB-FEATURES ? GPOS-FEATURES ?

	 SCRIPT ::= SYMBOL
	 LANGSYS ::= '/' SYMBOL

	 GSUB-FEATURES ::= '=' FEATURE-LIST ?

	 GPOS-FEATURES ::= '+' FEATURE-LIST ?

	 FEATURE-LIST ::= '~' ? FEATURE ( ',' '~' ? FEATURE ',' )

       Here, FEATURE is a four-letter Open Type feature.

       LANG-SPEC is a symbol specifying the required language support. The symbol name has the following syntax.

	 LANG-SPEC-NAME ::= ':lang=' LANG

       Here, LANG is a two or three-letter ISO-639 language code.

       FLT-NAME is a name of Font Layout Table (Font Layout Table).

EXAMPLE

       This is an example of PER_SCRIPT.

       (han
	 (ja
	   ((jisx0208.1983-0)))
	 (zh
	   ((gb2312.1980-0)))
	 (nil
	   ((big5-0))))

       It instructs the font selector to use a font of registry 'jisx0208.1983-0' for a 'han' character (i.e. a character whose Mscript property
       is 'han') if the character has Mlanguage text property 'ja' in an M-text and the character is in the repertories of such fonts. Otherwise,
       try a font of registry 'gb2312.1980-0' or 'big5-0'. If that 'han' character does not have Mlanguage text property, try all three fonts.

       See the function mdraw_text() for the detail of how a font is selected.

COPYRIGHT

       Copyright (C) 2001 Information-technology Promotion Agency (IPA)
       Copyright (C) 2001-2011 National Institute of Advanced Industrial Science and Technology (AIST)
       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License
       <http://www.gnu.org/licenses/fdl.html>.

Version 1.6.2							    12 Jan 2011 						     mdbFontset(5)
All times are GMT -4. The time now is 10:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy