Removing duplicate records in a file based on single column
Hi,
I want to remove duplicate records including the first line based on column1. For example
inputfile(filer.txt):
-------------
expected output:
----------------
Is it possible to achieve this using awk command ??
I tried below awk command , it is working but i dont want to give two times file name(filer.txt) in the command. I am allowed to give file name only one time.
Please suggest me how to achieve this.
Thanks in advance
Last edited by Franklin52; 08-22-2011 at 03:57 AM..
Reason: Please use code tags for code and data samples, thank you
Hi all,
I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g.
COL1 COL2 COL3 ............... COL47
1 XX 45 ... (4 Replies)
Hi,
Need to find a duplicate records on the first column,
ANU4501710430989 0000000W20389390
ANU4501710430989 0000000W67065483
ANU4501130050520 0000000W80838713
ANU4501210170685 0000000W69246611... (3 Replies)
Can anyone help me to removing duplicate records from 2 separate files in UNIX?
Please find the sample records for both the files
cat Monday.dat
3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE
3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE
3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
I am a newbie to shell scripting ..
I have a .csv file. It has 1000 some rows and about 7 columns...
but before I insert this data to a table I have to parse it and clean it ..basing on the value of the first column..which a string of phone number type...
example below..
column 1 ... (2 Replies)
Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column.
e.g.
a 1
a 2
a 3
b 1
c 1
gives
b 1
c 1
but requires 11 duplicates before it deletes.
Thanks for the help
Video tutorial on how to use code tags in The UNIX... (11 Replies)
I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1.
https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html
Can someone explain this command please? How are there no... (5 Replies)
Hi,
I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines
Command : sort -t'|' -nuk1 file.txt
Input :
38376KZ|09/25/15|1.057
38376KZ|09/25/15|1.057
02006YB|09/25/15|0.859
12593PS|09/25/15|2.803... (2 Replies)
I have csv file with 30, 40 columns
Pasting just three column for problem description
I want to filter record if column 1 matches CN or DN then,
check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345
and if column 2 contains 6789, 6789... (5 Replies)
Hi Experts,
I have csv file with 30, 40 columns
Pasting just 2 column for problem description.
Need to print error if below combination is not present in file
check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same.
For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
LEARN ABOUT DEBIAN
devel::refcount
Devel::Refcount(3pm) User Contributed Perl Documentation Devel::Refcount(3pm)NAME
"Devel::Refcount" - obtain the REFCNT value of a referent
SYNOPSIS
use Devel::Refcount qw( refcount );
my $anon = [];
print "Anon ARRAY $anon has " . refcount($anon) . " reference
";
my $otherref = $anon;
print "Anon ARRAY $anon now has " . refcount($anon) . " references
";
DESCRIPTION
This module provides a single function which obtains the reference count of the object being pointed to by the passed reference value.
FUNCTIONS
$count = refcount($ref)
Returns the reference count of the object being pointed to by $ref.
COMPARISON WITH SvREFCNT
This function differs from "Devel::Peek::SvREFCNT" in that SvREFCNT() gives the reference count of the SV object itself that it is passed,
whereas refcount() gives the count of the object being pointed to. This allows it to give the count of any referent (i.e. ARRAY, HASH,
CODE, GLOB and Regexp types) as well.
Consider the following example program:
use Devel::Peek qw( SvREFCNT );
use Devel::Refcount qw( refcount );
sub printcount
{
my $name = shift;
printf "%30s has SvREFCNT=%d, refcount=%d
",
$name, SvREFCNT($_[0]), refcount($_[0]);
}
my $var = [];
printcount 'Initially, $var', $var;
my $othervar = $var;
printcount 'Before CODE ref, $var', $var;
printcount '$othervar', $othervar;
my $code = sub { undef $var };
printcount 'After CODE ref, $var', $var;
printcount '$othervar', $othervar;
This produces the output
Initially, $var has SvREFCNT=1, refcount=1
Before CODE ref, $var has SvREFCNT=1, refcount=2
$othervar has SvREFCNT=1, refcount=2
After CODE ref, $var has SvREFCNT=2, refcount=2
$othervar has SvREFCNT=1, refcount=2
Here, we see that SvREFCNT() counts the number of references to the SV object passed in as the scalar value - the $var or $othervar
respectively, whereas refcount() counts the number of reference values that point to the referent object - the anonymous ARRAY in this
case.
Before the CODE reference is constructed, both $var and $othervar have SvREFCNT() of 1, as they exist only in the current lexical pad. The
anonymous ARRAY has a refcount() of 2, because both $var and $othervar store a reference to it.
After the CODE reference is constructed, the $var variable now has an SvREFCNT() of 2, because it also appears in the lexical pad for the
new anonymous CODE block.
PURE-PERL FALLBACK
An XS implementation of this function is provided, and is used by default. If the XS library cannot be loaded, a fallback implementation in
pure perl using the "B" module is used instead. This will behave identically, but is much slower.
Rate pp xs
pp 225985/s -- -66%
xs 669570/s 196% --
SEE ALSO
o Test::Refcount - assert reference counts on objects
AUTHOR
Paul Evans <leonerd@leonerd.org.uk>
perl v5.14.2 2011-11-15 Devel::Refcount(3pm)