Sponsored Content
Full Discussion: awk remove first duplicates
Top Forums Shell Programming and Scripting awk remove first duplicates Post 302885752 by sybadm on Tuesday 28th of January 2014 08:52:28 AM
Old 01-28-2014
awk remove first duplicates

Hi All,
I have searched many threads for possible close solution. But I was unable to get simlar scenario.

I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate).
Code:
i/P file
12  NIL ABD LON
11  NIL ABC SIG    <= First duplicate for 3rd column need to be removed
12  NIL ABC AMR
13  NIL ABC AMR
11  NIL ABK AMR

Code:
O/P desired based on 3rd column
12  NIL ABD LON
12  NIL ABC AMR
13  NIL ABC AMR
11  NIL ABK AMR

Many thanks,

Last edited by joeyg; 01-28-2014 at 10:09 AM.. Reason: corrected a spelling error
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

4. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

5. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

6. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

7. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

awk - Remove duplicates during array build

Greetings Experts, Issue: Within awk script, remove the duplicate occurrences that are space (1 single space character) separated Description: I am processing 2 files using awk and during processing, I am building an array and there are duplicates on this; how can I delete the duplicates... (3 Replies)
Discussion started by: chill3chee
3 Replies
SPLAIN(1)						 Perl Programmers Reference Guide						 SPLAIN(1)

NAME
diagnostics - Perl compiler pragma to force verbose warning diagnostics splain - standalone program to do the same thing SYNOPSIS
As a pragma: use diagnostics; use diagnostics -verbose; enable diagnostics; disable diagnostics; Aa a program: perl program 2>diag.out splain [-v] [-p] diag.out DESCRIPTION
The "diagnostics" Pragma This module extends the terse diagnostics normally emitted by both the perl compiler and the perl interpreter, augmenting them with the more explicative and endearing descriptions found in perldiag. Like the other pragmata, it affects the compilation phase of your program rather than merely the execution phase. To use in your program as a pragma, merely invoke use diagnostics; at the start (or near the start) of your program. (Note that this does enable perl's -w flag.) Your whole compilation will then be sub- ject(ed :-) to the enhanced diagnostics. These still go out STDERR. Due to the interaction between runtime and compiletime issues, and because it's probably not a very good idea anyway, you may not use "no diagnostics" to turn them off at compiletime. However, you may control their behaviour at runtime using the disable() and enable() methods to turn them off and on respectively. The -verbose flag first prints out the perldiag introduction before any other diagnostics. The $diagnostics::PRETTY variable can generate nicer escape sequences for pagers. Warnings dispatched from perl itself (or more accurately, those that match descriptions found in perldiag) are only displayed once (no duplicate descriptions). User code generated warnings ala warn() are unaffected, allowing duplicate user messages to be displayed. The splain Program While apparently a whole nuther program, splain is actually nothing more than a link to the (executable) diagnostics.pm module, as well as a link to the diagnostics.pod documentation. The -v flag is like the "use diagnostics -verbose" directive. The -p flag is like the $diag- nostics::PRETTY variable. Since you're post-processing with splain, there's no sense in being able to enable() or disable() processing. Output from splain is directed to STDOUT, unlike the pragma. EXAMPLES
The following file is certain to trigger a few errors at both runtime and compiletime: use diagnostics; print NOWHERE "nothing "; print STDERR " This message should be unadorned. "; warn " This is a user warning"; print " DIAGNOSTIC TESTER: Please enter a <CR> here: "; my $a, $b = scalar <STDIN>; print " "; print $x/$y; If you prefer to run your program first and look at its problem afterwards, do this: perl -w test.pl 2>test.out ./splain < test.out Note that this is not in general possible in shells of more dubious heritage, as the theoretical (perl -w test.pl >/dev/tty) >& test.out ./splain < test.out Because you just moved the existing stdout to somewhere else. If you don't want to modify your source code, but still have on-the-fly warnings, do this: exec 3>&1; perl -w test.pl 2>&1 1>&3 3>&- | splain 1>&2 3>&- Nifty, eh? If you want to control warnings on the fly, do something like this. Make sure you do the "use" first, or you won't be able to get at the enable() or disable() methods. use diagnostics; # checks entire compilation phase print " time for 1st bogus diags: SQUAWKINGS "; print BOGUS1 'nada'; print "done with 1st bogus "; disable diagnostics; # only turns off runtime warnings print " time for 2nd bogus: (squelched) "; print BOGUS2 'nada'; print "done with 2nd bogus "; enable diagnostics; # turns back on runtime warnings print " time for 3rd bogus: SQUAWKINGS "; print BOGUS3 'nada'; print "done with 3rd bogus "; disable diagnostics; print " time for 4th bogus: (squelched) "; print BOGUS4 'nada'; print "done with 4th bogus "; INTERNALS
Diagnostic messages derive from the perldiag.pod file when available at runtime. Otherwise, they may be embedded in the file itself when the splain package is built. See the Makefile for details. If an extant $SIG{__WARN__} handler is discovered, it will continue to be honored, but only after the diagnostics::splainthis() function (the module's $SIG{__WARN__} interceptor) has had its way with your warnings. There is a $diagnostics::DEBUG variable you may set if you're desperately curious what sorts of things are being intercepted. BEGIN { $diagnostics::DEBUG = 1 } BUGS
Not being able to say "no diagnostics" is annoying, but may not be insurmountable. The "-pretty" directive is called too late to affect matters. You have to do this instead, and before you load the module. BEGIN { $diagnostics::PRETTY = 1 } I could start up faster by delaying compilation until it should be needed, but this gets a "panic: top_level" when using the pragma form in Perl 5.001e. While it's true that this documentation is somewhat subserious, if you use a program named splain, you should expect a bit of whimsy. AUTHOR
Tom Christiansen <tchrist@mox.perl.com>, 25 June 1995. perl v5.8.0 2003-02-18 SPLAIN(1)
All times are GMT -4. The time now is 02:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy