If you were to run out of memory, you could use tac file | awk '!($0 in S) {print; S[$0]}' | tac posted by MadeInGermany.
A similar code in shell, with filename in variable FILE:
Code:
nl $FILE |
tee f1 |
sort -k 2 -k 1,1rn |
tee f2 |
uniq --skip-fields=1 |
tee f3 |
sort -k 1,1n |
tee f4 |
sed 's/^.*\t//'
Line numbers are added, then the body is sorted, with the secondary sort being reverse numeric. The GNU uniq allows fields to be skipped, the result sorted in numeric order, thus retaining the original order, after which the line number is stripped. Before stripping, this looks like:
Pipelines are useful for doing large-granularity parallel computing, and the disk is not touched because the pipes are simply buffers (usually 65K). The tee in the above is to allow intermediate results to be seen.
I have run across some uniq versions that keep the most recent version of a duplicate ( Solaris if memory serves ).
This was done on:
Code:
Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian 5.0.8 (lenny, workstation)
bash GNU bash 3.2.39
nl (GNU coreutils) 6.10
sort (GNU coreutils) 6.10
uniq (GNU coreutils) 6.10
sed GNU sed version 4.1.5
Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working.
I've search the forum but nothing comes close.
I have a sample file:
27147140631203RA CCD *
27147140631203RA PPN *
37147140631207RD AAA
47147140631203RD JNA... (12 Replies)
Hello experts,
I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter
sort -u -k 2,2 File.csv > Output.csv
File.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample... (3 Replies)
Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here )
I'm trying to write a scrip that will remove duplicate entries
the script can... (5 Replies)
Hi,
I have a file in the below format.,
test test (10)
to to (25)
see see (45)
and i need the output in the format of
test 10
to 25
see 45
Some one help me? (6 Replies)
Hi All
In unix ,we have a file ,there we have to remove the duplicates by using one specific column.
Can any body tell me the command.
ex:
file1
id,name
1,ww
2,qwq
2,asas
3,asa
4,asas
4,asas
o/p:
1,ww
2,qwq
3,asa (7 Replies)
I have a test file with the following 2 columns:
Col 1 | Col 2
T1 | 1 <= remove
T5 | 1
T4 | 2
T1 | 3
T3 | 3
T4 | 1 <= remove
T1 | 2 <= remove
T3 ... (7 Replies)
i want to remove all the duplictaes in a file.I dont want even a single entry.
For the input data:
12345|12|34
12345|13|23
3456|12|90
15670|12|13
12345|10|14
3456|12|13
i need the below data in one file
15670|12|13
and the below data in another file (9 Replies)
I have an input file abc.txt with info like:
abcd
rateuse
inklite
robet
rateuse
abcd
I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file.
can anyone help me in this :( (4 Replies)
i hav two files like
i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies
LEARN ABOUT DEBIAN
graphics::primitive::insets
Graphics::Primitive::Insets(3pm) User Contributed Perl Documentation Graphics::Primitive::Insets(3pm)NAME
Graphics::Primitive::Insets - Space between things
DESCRIPTION
Graphics::Primitive::Insets represents the amount of space that surrounds something. This object can be used to represent either padding
or margins (in the CSS sense, one being inside the bounding box, the other being outside)
SYNOPSIS
use Graphics::Primitive::Insets;
my $insets = Graphics::Primitive::Insets->new({
top => 5,
bottom => 5,
left => 5,
right => 5
});
METHODS
Constructor
new Creates a new Graphics::Primitive::Insets.
Instance Methods
as_array
Return these insets as an array in the form of top, right, bottom and left.
bottom
Set/Get the inset from the bottom.
equal_to
Determine if these Insets are equal to another.
left
Set/Get the inset from the left.
right
Set/Get the inset from the right.
top Set/Get the inset from the top.
zero
Sets all the insets (top, left, bottom, right) to 0.
AUTHOR
Cory Watson, "<gphat@cpan.org>"
SEE ALSO perl(1)COPYRIGHT & LICENSE
Copyright 2008-2010 by Cory G Watson.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.12.3 2010-08-21 Graphics::Primitive::Insets(3pm)