10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi All,
I have multiple files and i need to segregate unique and duplicates into files.
Eg: /source/ -- path
abc_12092016.csv
abc_11092016.csv
abc_12092016.csv
ID,NAME,NUMBER
1,XYZ,1234
2,SDF,3456
1,XYZ,1234
abc_11092016.csv
4,RTY,7890
6,WER,5678
8,YUI,0987
6,WER,5678
in the... (1 Reply)
Discussion started by: ajayr1982
1 Replies
2. Shell Programming and Scripting
Hello Team,
I need your help on the following:
My input file a.txt is as below:
3330690|373846|108471
3330690|373846|108471
0640829|459725|100001
0640829|459725|100001
3330690|373847|108471
Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies
3. Shell Programming and Scripting
Hello !
I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts...
Could you guys help me removing the duplicate lines of a file, based only on a single "column"?
For example:
M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies
4. UNIX for Dummies Questions & Answers
Background:
I have a file of thousands of potential SSR primers from Batch Primer 3.
I can't use primers that will contain the same sequence ID or sequence as another primer.
I have some basic shell scripting skills, but not enough to handle this.
What you need to know:
I need to remove the... (1 Reply)
Discussion started by: msatseqs
1 Replies
5. Programming
Hey Guys,
I'm using SQuirreL SQL v3.5 GUI to fetch some data that I need for something I'm working on. I'm also using the IBM Informix Driver (*Version 3.5) to connect to the Database.
What I want to do, if it's even possible, is to show all COLUMNS if they contain the word "Email".
So in... (2 Replies)
Discussion started by: mrm5102
2 Replies
6. Shell Programming and Scripting
Is it possible to modify file like this.
1. Remove all the duplicate names in a define column i.e 4th col
2. Count the no.of unique names separated by ";" and print as a 5th col
thanx in advance!!
Q
input
c1 30 3 Eh2
c10 96 3 Frp
c41 396 3 Ua5;Lop;Kol;Kol
c62 2 30 Fmp;Fmp;Fmp
... (5 Replies)
Discussion started by: quincyjones
5 Replies
7. Shell Programming and Scripting
Is it possible to remove redundant names in the 4th column?
input
cqWE 100 200 singapore;singapore
AZO 300 400 brazil;america;germany;ireland;germany
....
....
output
cqWE 100 200 singapore
AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies
8. Shell Programming and Scripting
HOW CAN I SELECT AN UNIQUE STRING FROM A FIELD? ACTUALLY I WANT TO PRINT RECORDS THAT 2ND FIELD OF THAT HAVE ONE CHARACTER AND IT MUST BE "P"
AWK '$2~"" {PRINT $0}' IN > OUTBUT THIS CODE PRINT ALL RECORDS WHICH 2ND FIELDS OF THEM START WITH "P" AND MAY CONTAINS ANOTHER CHARACTER! (1 Reply)
Discussion started by: saeed.soltani
1 Replies
9. Shell Programming and Scripting
I have 84 files with the following names splitseqs.1, spliseqs.2 etc.
and I want to change the .number to a unique filename.
E.g.
change splitseqs.1 into splitseqs.7114_1#24
and
change spliseqs.2 into splitseqs.7067_2#4
So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies
10. Shell Programming and Scripting
I have values in the variable as so the for loop just fetches one by one
params=$'$HEW_SRC_DATABASE_LIB\nprmAttunityUser\nprmAttunityPwd\nprmODBCDataSource\nprmLoadInd\nprmSrc_Lib_ATM\nprmODBCDataSource_ATM'
and i have a grep command like this
ret=`grep \$y $pf`
... (0 Replies)
Discussion started by: vee_789
0 Replies
Array::Unique(3pm) User Contributed Perl Documentation Array::Unique(3pm)
NAME
Array::Unique - Tie-able array that allows only unique values
SYNOPSIS
use Array::Unique;
tie @a, 'Array::Unique';
Now use @a as a regular array.
DESCRIPTION
This package lets you create an array which will allow only one occurrence of any value.
In other words no matter how many times you put in 42 it will keep only the first occurrence and the rest will be dropped.
You use the module via tie and once you tied your array to this module it will behave correctly.
Uniqueness is checked with the 'eq' operator so among other things it is case sensitive.
As a side effect the module does not allow undef as a value in the array.
EXAMPLES
use Array::Unique;
tie @a, 'Array::Unique';
@a = qw(a b c a d e f);
push @a, qw(x b z);
print "@a
"; # a b c d e f x z
DISCUSSION
When you are collecting a list of items and you want to make sure there is only one occurrence of each item, you have several option:
1) using an array and extracting the unique elements later
You might use a regular array to hold this unique set of values and either remove duplicates on each update by that keeping the array
always unique or remove duplicates just before you want to use the uniqueness feature of the array. In either case you might run a
function you call @a = unique_value(@a);
The problem with this approach is that you have to implement the unique_value function (see later) AND you have to make sure you don't
forget to call it. I would say don't rely on remembering this.
There is good discussion about it in the 1st edition of the Perl Cookbook of O'Reilly. I have copied the solutions here, you can see
further discussion in the book.
Extracting Unique Elements from a List (Section 4.6 in the Perl Cookbook 1st ed.)
# Straightforward
%seen = ();
@uniq = ();
foreach $item (@list) [
unless ($seen{$item}) {
# if we get here we have not seen it before
$seen{$item} = 1;
push (@uniq, $item);
}
}
# Faster
%seen = ();
foreach $item (@list) {
push(@uniq, $item) unless $seen{$item}++;
}
# Faster but different
%seen;
foreach $item (@list) {
$seen{$item}++;
}
@uniq = keys %seen;
# Faster and even more different
%seen;
@uniq = grep {! $seen{$_}++} @list;
2) using a hash
Some people use the keys of a hash to keep the items and put an arbitrary value as the values of the hash:
To build such a list:
%unique = map { $_ => 1 } qw( one two one two three four! );
To print it:
print join ", ", sort keys %unique;
To add values to it:
$unique{$_}=1 foreach qw( one after the nine oh nine );
To remove values:
delete @unique{ qw(oh nine) };
To check if a value is there:
$unique{ $value }; # which is why I like to use "1" as my value
(thanks to Gaal Yahas for the above examples)
There are three drawbacks I see:
1) You type more.
2) Your reader might not understand at first why did you use hash and what will be the values.
3) You lose the order.
Usually non of them is critical but when I saw this the 10th time in a code I had to understand with 0 documentation I got frustrated.
3) using Array::Unique
So I decided to write this module because I got frustrated by my lack of understanding what's going on in that code I mentioned.
In addition I thought it might be interesting to write this and then benchmark it.
Additionally it is nice to have your name displayed in bright lights all over CPAN ... or at least in a module.
Array::Unique lets you tie an array to hmmm, itself (?) and makes sure the values of the array are always unique.
Since writing this I am not sure if I really recommend its usage. I would say stick with the hash version and document that the
variable is aggregating a unique list of values.
4) Using real SET
There are modules on CPAN that let you create and maintain SETs. I have not checked any of those but I guess they just as much of an
overkill for this functionality as Unique::Array.
BUGS
use Array::Unique;
tie @a, 'Array::Unique';
@c = @a = qw(a b c a d e f b);
@c will contain the same as @a AND two undefs at the end because
@c you get the same length as the right most list.
TODO
Test:
Change size of the array Elements with false values ('', '0', 0)
splice:
splice @a;
splice @a, 3;
splice @a, -3;
splice @a, 3, 5;
splice @a, 3, -5;
splice @a, -3, 5;
splice @a, -3, -5;
splice @a, ?, ?, @b;
Benchmark speed
Add faster functions that don't check uniqueness so if I know part of the data that comes from a unique source then I can speed up the
process, In short shoot myself in the leg.
Enable optional compare with other functions
Write even better implementations.
AUTHOR
Gabor Szabo <gabor@pti.co.il>
LICENSE
Copyright (C) 2002-2008 Gabor Szabo <gabor@pti.co.il> All rights reserved. http://www.pti.co.il/
You may distribute under the terms of either the GNU General Public License or the Artistic License, as specified in the Perl README file.
No WARRANTY whatsoever.
CREDITS
Thanks for suggestions and bug reports to
Szabo Balazs (dLux)
Shlomo Yona
Gaal Yahas
Jeff 'japhy' Pinyan
Werner Weichselberger
VERSION
Version: 0.08
Date: 2008 June 04
perl v5.10.0 2009-03-06 Array::Unique(3pm)