Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Placing Duplicate Lines per section into another file
Top Forums Shell Programming and Scripting Placing Duplicate Lines per section into another file Post 302390289 by petersf on Wednesday 27th of January 2010 12:24:56 PM
Old 01-27-2010
Placing Duplicate Lines per section into another file
Hello,

I need help in putting duplicate lines within a section into another file. Here is what I'm struggling with:

Using this file “data.txt”:
Code:
 
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 587.6580
ABC5 594.0000 588.0971
ABC5 594.5000 588.5361
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Header Table
ABC4
ABC5 594.5000 588.5361 <= to keep in this section
ABC5 601.0000 594.1603
ABC5 601.5000 594.6121
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642 <= dup need to remove

Duplicates need to be placed in “data.dup”
Code:
 
ABC1 012345 header
ABC5 593.5000 587.6580 
ABC1 67890 header
ABC5 602.0000 595.0642

I've been using the current code to find the duplicates, but have yet to find a method to get the duplicates into the “data.dup” file according to the section the duplicate was found.

Code:
# This will remove the duplicate lines and place the unique lines in “data.out”
awk '{ if ($1=="ABC1") {delete arr}
      if ( !arr[$0]++ ) { print $0 } }' data.txt > data.out
 
# This will extract the duplicates into the “data.dup” file.  This works, but I need it 
# to be by section, with a section header
awk '/ABC1/{ ABC1 = $2 } ++arr[ABC1,$0] > 1' data.txt > data.dup

I am using Bourne Shell (/bin/sh)

Any help would be GREATL:Y appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

2. Shell Programming and Scripting

Placing lines file in array

Hiya I am fairly new to UNIX and have been asked to write some scripts.... I am working in the korn shell. What I am trying to do is to go through a delimited file: Testingline1;test;test Testingline2;test;test and place the lines into an Array so: Array = Testingline1;test;test Array... (4 Replies)
Discussion started by: ThomasvdH
4 Replies

3. Shell Programming and Scripting

Using Sed to duplicate a section of a file....

hello all, I have a file like this: section 1 blah1 blah2 section 2 blah1 blah2 section 3 blah1 blah2 and I want to use sed to duplicate section 2, like this: section 1 blah1 blah2 section 2 blah1 blah2 section 2 blah1 (2 Replies)
Discussion started by: nick26
2 Replies

4. Shell Programming and Scripting

Removing Duplicate Lines per Section

Hello, I am in need of removing duplicate lines from within a file per section. File: ABC1 012345 header ABC2 7890-000 ABC3 012345 Header Table ABC4 ABC5 593.0000 587.4800 ABC5 593.5000 587.6580 <= dup need to remove ABC5 593.5000 ... (5 Replies)
Discussion started by: petersf
5 Replies

5. Shell Programming and Scripting

Duplicate Line Report per Section

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance. Here are the file.dat contents: ABC1... (4 Replies)
Discussion started by: petersf
4 Replies

6. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

7. Shell Programming and Scripting

Extract section of file based on word in section

I have a list of Servers in no particular order as follows: virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection). 9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS... (2 Replies)
Discussion started by: jelloir
2 Replies

8. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

9. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

10. Shell Programming and Scripting

Adding a lines to specific section of the file.

Hello, I have to a add 2 lines to /etc/sudoers file under this section below, can someone please suggest script to add these two lines when execute this remotely on to a multiple servers. before ## Allow root to run any commands anywhere root ALL=(ALL) ALL After ## Allow root... (2 Replies)
Discussion started by: bobby320
2 Replies

LEARN ABOUT DEBIAN

extract_font_range

extract_font_range(3alleg4)					  Allegro manual				       extract_font_range(3alleg4)

NAME

       extract_font_range - Extracts a range of characters from a font. Allegro game programming library.

SYNOPSIS

       #include <allegro.h>

       FONT *extract_font_range(FONT *f, int begin, int end)

DESCRIPTION

       This  function  extracts  a  character range from a font and returns a new font that contains only the range of characters selected by this
       function. You can pass -1 for either the lower or upper bound if you want to select all characters from the start or  to  the  end  of  the
       font.  Example:

	  FONT *myfont;
	  FONT *capitals;
	  FONT *fontcopy;
	  ...
	  /* Create a font of only capital letters */
	  capitals = extract_font_range(myfont, 'A', 'Z');

	  /* Create a copy of the font */
	  fontcopy = extract_font_range(myfont, -1, -1);
	  ...
	  destroy_font(capitals);
	  destroy_font(fontcopy);

RETURN VALUE

       Returns a pointer to the new font or NULL on error. Remember that you are responsible for destroying the font when you are finished with it
       to avoid memory leaks.

SEE ALSO

       get_font_range_begin(3alleg4), get_font_range_end(3alleg4), merge_fonts(3alleg4), transpose_font(3alleg4), exfont(3alleg4)

Allegro 							   version 4.4.2				       extract_font_range(3alleg4)
All times are GMT -4. The time now is 08:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy