Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Concatenate and sort to remove duplicates
Top Forums Shell Programming and Scripting Concatenate and sort to remove duplicates Post 303027526 by Don Cragun on Monday 17th of December 2018 02:49:25 PM
Old 12-17-2018
I partially agree with nezabudka, awk is the way to go here; but we need to make each block a record, not each line. By setting RS to an empty string, we can tell awk that records are separated by sequences of a <newline> followed by one or more blank lines. Given this fact, the following should work:
Code:
awk '
BEGIN {	RS = ""
}
!($0 in seen) {
	seen[$0]
	printf("%s%s\n", (NR == 1) ? "" : "\n", $0)
}' file

which will print the first occurrence of each duplicated record found in the file named file to its output. Note that the above code does not print an empty line before the 1st output record or after the last output record. The code could be simplified if you always want to print an empty line after each output record.

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates [sort , uniq]

Hey Guys, I have file which looks like this, Contig201#numbPA Contig1452#nmdynD6PA dm022p15.r#CG6461PA dm005e16.f#SpatPA IGU001_0015_A06.f#CG17593PA I need to remove duplicates based on the chracter matching upto '#'. for example if we consider this.. Contig201#numbPA... (4 Replies)
Discussion started by: sharatz83
4 Replies

2. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script... (5 Replies)
Discussion started by: Amruta Pitkar
5 Replies

3. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

4. UNIX for Dummies Questions & Answers

sort by date and concatenate first three

Hi: I am trying to create some script that sorts the files in a subdirectory by date and concatenates the thre most recently created files. SAy, file1 date1 file2 date2 file3 date3 file4 date4 file5 date5 file6 date6 i only want to concatenate the first three which are the most... (4 Replies)
Discussion started by: jlarios
4 Replies

5. Solaris

concatenate/sort/cut

I have the following requirement. 1. I have to concatenate the 10 fixed width files. 2. sort based on first 10 characters 3. after that i have remove first 10 chacters from the file. can you please tell me how to do it. Thanks in Advance Samba (1 Reply)
Discussion started by: samba
1 Replies

6. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

7. Shell Programming and Scripting

Sort data by date first and then remove duplicates

Hi , I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points : (1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column) (2)after sorting the file based on latest date... (3 Replies)
Discussion started by: samrat dutta
3 Replies

8. Shell Programming and Scripting

Bash - remove duplicates without sort

I need to use bash to remove duplicates without using sort first. I can not use: cat file | sort | uniq But when I use only cat file | uniq some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies

9. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

10. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

LEARN ABOUT MOJAVE

datetime::locale::en_gu

DateTime::Locale::en_GU(3)				User Contributed Perl Documentation				DateTime::Locale::en_GU(3)

NAME

       DateTime::Locale::en_GU

SYNOPSIS

	 use DateTime;

	 my $dt = DateTime->now( locale => 'en_GU' );
	 print $dt->month_name();

DESCRIPTION

       This is the DateTime locale package for English Guam.

DATA

       This locale inherits from the DateTime::Locale::en locale.

       It contains the following data.

   Days
       Wide (format)

	 Monday
	 Tuesday
	 Wednesday
	 Thursday
	 Friday
	 Saturday
	 Sunday

       Abbreviated (format)

	 Mon
	 Tue
	 Wed
	 Thu
	 Fri
	 Sat
	 Sun

       Narrow (format)

	 M
	 T
	 W
	 T
	 F
	 S
	 S

       Wide (stand-alone)

	 Monday
	 Tuesday
	 Wednesday
	 Thursday
	 Friday
	 Saturday
	 Sunday

       Abbreviated (stand-alone)

	 Mon
	 Tue
	 Wed
	 Thu
	 Fri
	 Sat
	 Sun

       Narrow (stand-alone)

	 M
	 T
	 W
	 T
	 F
	 S
	 S

   Months
       Wide (format)

	 January
	 February
	 March
	 April
	 May
	 June
	 July
	 August
	 September
	 October
	 November
	 December

       Abbreviated (format)

	 Jan
	 Feb
	 Mar
	 Apr
	 May
	 Jun
	 Jul
	 Aug
	 Sep
	 Oct
	 Nov
	 Dec

       Narrow (format)

	 J
	 F
	 M
	 A
	 M
	 J
	 J
	 A
	 S
	 O
	 N
	 D

       Wide (stand-alone)

	 January
	 February
	 March
	 April
	 May
	 June
	 July
	 August
	 September
	 October
	 November
	 December

       Abbreviated (stand-alone)

	 Jan
	 Feb
	 Mar
	 Apr
	 May
	 Jun
	 Jul
	 Aug
	 Sep
	 Oct
	 Nov
	 Dec

       Narrow (stand-alone)

	 J
	 F
	 M
	 A
	 M
	 J
	 J
	 A
	 S
	 O
	 N
	 D

   Quarters
       Wide (format)

	 1st quarter
	 2nd quarter
	 3rd quarter
	 4th quarter

       Abbreviated (format)

	 Q1
	 Q2
	 Q3
	 Q4

       Narrow (format)

	 1
	 2
	 3
	 4

       Wide (stand-alone)

	 1st quarter
	 2nd quarter
	 3rd quarter
	 4th quarter

       Abbreviated (stand-alone)

	 Q1
	 Q2
	 Q3
	 Q4

       Narrow (stand-alone)

	 1
	 2
	 3
	 4

   Eras
       Wide

	 Before Christ
	 Anno Domini

       Abbreviated

	 BC
	 AD

       Narrow

	 B
	 A

   Date Formats
       Full

	  2008-02-05T18:30:30 = Tuesday, February 5, 2008
	  1995-12-22T09:05:02 = Friday, December 22, 1995
	 -0010-09-15T04:44:23 = Saturday, September 15, -10

       Long

	  2008-02-05T18:30:30 = February 5, 2008
	  1995-12-22T09:05:02 = December 22, 1995
	 -0010-09-15T04:44:23 = September 15, -10

       Medium

	  2008-02-05T18:30:30 = Feb 5, 2008
	  1995-12-22T09:05:02 = Dec 22, 1995
	 -0010-09-15T04:44:23 = Sep 15, -10

       Short

	  2008-02-05T18:30:30 = 2/5/08
	  1995-12-22T09:05:02 = 12/22/95
	 -0010-09-15T04:44:23 = 9/15/-10

       Default

	  2008-02-05T18:30:30 = Feb 5, 2008
	  1995-12-22T09:05:02 = Dec 22, 1995
	 -0010-09-15T04:44:23 = Sep 15, -10

   Time Formats
       Full

	  2008-02-05T18:30:30 = 6:30:30 PM UTC
	  1995-12-22T09:05:02 = 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = 4:44:23 AM UTC

       Long

	  2008-02-05T18:30:30 = 6:30:30 PM UTC
	  1995-12-22T09:05:02 = 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = 4:44:23 AM UTC

       Medium

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

       Short

	  2008-02-05T18:30:30 = 6:30 PM
	  1995-12-22T09:05:02 = 9:05 AM
	 -0010-09-15T04:44:23 = 4:44 AM

       Default

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

   Datetime Formats
       Full

	  2008-02-05T18:30:30 = Tuesday, February 5, 2008 6:30:30 PM UTC
	  1995-12-22T09:05:02 = Friday, December 22, 1995 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = Saturday, September 15, -10 4:44:23 AM UTC

       Long

	  2008-02-05T18:30:30 = February 5, 2008 6:30:30 PM UTC
	  1995-12-22T09:05:02 = December 22, 1995 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = September 15, -10 4:44:23 AM UTC

       Medium

	  2008-02-05T18:30:30 = Feb 5, 2008 6:30:30 PM
	  1995-12-22T09:05:02 = Dec 22, 1995 9:05:02 AM
	 -0010-09-15T04:44:23 = Sep 15, -10 4:44:23 AM

       Short

	  2008-02-05T18:30:30 = 2/5/08 6:30 PM
	  1995-12-22T09:05:02 = 12/22/95 9:05 AM
	 -0010-09-15T04:44:23 = 9/15/-10 4:44 AM

       Default

	  2008-02-05T18:30:30 = Feb 5, 2008 6:30:30 PM
	  1995-12-22T09:05:02 = Dec 22, 1995 9:05:02 AM
	 -0010-09-15T04:44:23 = Sep 15, -10 4:44:23 AM

   Available Formats
       d (d)

	  2008-02-05T18:30:30 = 5
	  1995-12-22T09:05:02 = 22
	 -0010-09-15T04:44:23 = 15

       EEEd (d EEE)

	  2008-02-05T18:30:30 = 5 Tue
	  1995-12-22T09:05:02 = 22 Fri
	 -0010-09-15T04:44:23 = 15 Sat

       Hm (H:mm)

	  2008-02-05T18:30:30 = 18:30
	  1995-12-22T09:05:02 = 9:05
	 -0010-09-15T04:44:23 = 4:44

       hm (h:mm a)

	  2008-02-05T18:30:30 = 6:30 PM
	  1995-12-22T09:05:02 = 9:05 AM
	 -0010-09-15T04:44:23 = 4:44 AM

       Hms (H:mm:ss)

	  2008-02-05T18:30:30 = 18:30:30
	  1995-12-22T09:05:02 = 9:05:02
	 -0010-09-15T04:44:23 = 4:44:23

       hms (h:mm:ss a)

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

       M (L)

	  2008-02-05T18:30:30 = 2
	  1995-12-22T09:05:02 = 12
	 -0010-09-15T04:44:23 = 9

       Md (M/d)

	  2008-02-05T18:30:30 = 2/5
	  1995-12-22T09:05:02 = 12/22
	 -0010-09-15T04:44:23 = 9/15

       MEd (E, M/d)

	  2008-02-05T18:30:30 = Tue, 2/5
	  1995-12-22T09:05:02 = Fri, 12/22
	 -0010-09-15T04:44:23 = Sat, 9/15

       MMM (LLL)

	  2008-02-05T18:30:30 = Feb
	  1995-12-22T09:05:02 = Dec
	 -0010-09-15T04:44:23 = Sep

       MMMd (MMM d)

	  2008-02-05T18:30:30 = Feb 5
	  1995-12-22T09:05:02 = Dec 22
	 -0010-09-15T04:44:23 = Sep 15

       MMMEd (E, MMM d)

	  2008-02-05T18:30:30 = Tue, Feb 5
	  1995-12-22T09:05:02 = Fri, Dec 22
	 -0010-09-15T04:44:23 = Sat, Sep 15

       MMMMd (MMMM d)

	  2008-02-05T18:30:30 = February 5
	  1995-12-22T09:05:02 = December 22
	 -0010-09-15T04:44:23 = September 15

       MMMMEd (E, MMMM d)

	  2008-02-05T18:30:30 = Tue, February 5
	  1995-12-22T09:05:02 = Fri, December 22
	 -0010-09-15T04:44:23 = Sat, September 15

       ms (mm:ss)

	  2008-02-05T18:30:30 = 30:30
	  1995-12-22T09:05:02 = 05:02
	 -0010-09-15T04:44:23 = 44:23

       y (y)

	  2008-02-05T18:30:30 = 2008
	  1995-12-22T09:05:02 = 1995
	 -0010-09-15T04:44:23 = -10

       yM (M/yyyy)

	  2008-02-05T18:30:30 = 2/2008
	  1995-12-22T09:05:02 = 12/1995
	 -0010-09-15T04:44:23 = 9/-010

       yMEd (EEE, M/d/yyyy)

	  2008-02-05T18:30:30 = Tue, 2/5/2008
	  1995-12-22T09:05:02 = Fri, 12/22/1995
	 -0010-09-15T04:44:23 = Sat, 9/15/-010

       yMMM (MMM y)

	  2008-02-05T18:30:30 = Feb 2008
	  1995-12-22T09:05:02 = Dec 1995
	 -0010-09-15T04:44:23 = Sep -10

       yMMMEd (EEE, MMM d, y)

	  2008-02-05T18:30:30 = Tue, Feb 5, 2008
	  1995-12-22T09:05:02 = Fri, Dec 22, 1995
	 -0010-09-15T04:44:23 = Sat, Sep 15, -10

       yMMMM (MMMM y)

	  2008-02-05T18:30:30 = February 2008
	  1995-12-22T09:05:02 = December 1995
	 -0010-09-15T04:44:23 = September -10

       yQ (Q yyyy)

	  2008-02-05T18:30:30 = 1 2008
	  1995-12-22T09:05:02 = 4 1995
	 -0010-09-15T04:44:23 = 3 -010

       yQQQ (QQQ y)

	  2008-02-05T18:30:30 = Q1 2008
	  1995-12-22T09:05:02 = Q4 1995
	 -0010-09-15T04:44:23 = Q3 -10

   Miscellaneous
       Prefers 24 hour time?

       No

       Local first day of the week

       Sunday

SUPPORT

       See DateTime::Locale.

AUTHOR

       Dave Rolsky <autarch@urth.org>

COPYRIGHT

       Copyright (c) 2008 David Rolsky. All rights reserved. This program is free software; you can redistribute it and/or modify it under the
       same terms as Perl itself.

       This module was generated from data provided by the CLDR project, see the LICENSE.cldr in this distribution for details on the CLDR data's
       license.

perl v5.18.2							    2017-10-06						DateTime::Locale::en_GU(3)
All times are GMT -4. The time now is 07:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy