Sponsored Content
Full Discussion: Grep Unique
Top Forums UNIX for Dummies Questions & Answers Grep Unique Post 302475859 by methyl on Tuesday 30th of November 2010 05:57:37 AM
Old 11-30-2010
A similar way. Only slightly less efficient:

Code:
awk -F: '{print $1}' filename|sort|uniq|tr -d '@'

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep unique occurrences

Hi, How do i grep unique occurrences from a file. I have a log file with multiple occurrences of 'foo' for instance. When i do: grep foo, I get all the lines that contain foo. Is there any way to get only one line for foo? Example file: foo at 09.01am Fri 11 May 2007 foo at 09.13am... (6 Replies)
Discussion started by: mz043
6 Replies

2. UNIX for Dummies Questions & Answers

Getting unique list of numbers using grep

Hi, I am going to fetch a list of numbers that starts with "0032" from a file with a format like the given below: " 0032459999 0032458888 0032457777 0032451111 0032452222 0032453333 0032459999 0032458888 0032457777 0032451111 0032452222 0032453333 " I want to get a unique... (6 Replies)
Discussion started by: tinku
6 Replies

3. Shell Programming and Scripting

grep for unique value

If i want to grep for a value in a file but display only unique value then which option can i use. ex: Values in the file IP <1.2.3.4> value <2> IP <1.2.3.4> value <2> IP <1.2.3.4> value <3> IP <1.2.3.5> value <1> i should get only the unique value (3 Replies)
Discussion started by: vls1210
3 Replies

4. UNIX and Linux Applications

grep file to find unique instances of username

hello - A SystemOut.log file has recurring entries that follow this format: Principal: auth9.nick.al.gov:389/USERNAME Over the course of a day thousands of lines similar to this are produced, with each username represented hundreds of times. I need to create a new file that shows... (4 Replies)
Discussion started by: 1075FJ40
4 Replies

5. Shell Programming and Scripting

grep with date & unique output

alert.log has the entries with ORA-XXXX, .... Mon Sep 24 15:08:09 2012 WARNING: inbound connection timed out (ORA-3136) Mon Sep 24 15:08:09 2012 WARNING: inbound connection timed out (ORA-3136) Mon Sep 24 15:08:09 2012 WARNING: inbound connection timed out (ORA-3136) Mon Sep 24 15:15:01... (4 Replies)
Discussion started by: Daniel Gate
4 Replies

6. UNIX for Dummies Questions & Answers

Grep to find matching patern and return unique values

Request: grep to find given matching patern and return unique values, eliminate the duplicate values I have to retrieve the unique folder on the below file contents like; /app/oracle/build_lib/pkg320.0_20120927 /app/oracle/build_lib/pkg320.0_20121004_prof... (5 Replies)
Discussion started by: Siva SQL
5 Replies

7. Shell Programming and Scripting

How to grep on unique id which has request and response on different lines?

Hi I want to find out those unique uids from the log file which have request and response. The log file format is as follows. This log has other irrelevant lines too but each uid should have request and reponse, I need those uids only 2013-04-03 10:51:01,808 INFO <?xml version="1.0"... (4 Replies)
Discussion started by: random_thoughts
4 Replies

8. Shell Programming and Scripting

Using grep and a parameter file to return unique values

Hello Everyone! I have updated the first post so that my intentions are easier to understand, and also attached sample files (post #18). I have over 500 text files in a directory. Over 1 GB of data. The data in those files is organised in lines: My intention is to return one line per... (23 Replies)
Discussion started by: clippertm
23 Replies

9. UNIX for Dummies Questions & Answers

Grep unique 1st column

Hello, I'm trying to used awk but am new to this. I have a file like this: Bob is a good boy Bob is a strange person Bob is a good dancer Jane can party Jane is a good girl Jane is batty I'd like to get this: Bob is a good boy is a strange person is a good dancer Jane... (4 Replies)
Discussion started by: Billyjo
4 Replies

10. UNIX for Beginners Questions & Answers

Grep or awk a unique and specific word across many fields

Hi there, I have data with similar structure as this: CHR START-SNP END-SNP REF ALT PATIENT1 PATIENT2 PATIENT3 PATIENT4 chr1 69511 69511 A G homo hetero homo hetero chr2 69513 69513 T C . hetero homo hetero chr3 69814 69814 G C . . homo homo chr4 69815 69815 C A hetero . . hetero is... (10 Replies)
Discussion started by: daashti
10 Replies
Array::Unique(3pm)					User Contributed Perl Documentation					Array::Unique(3pm)

NAME
Array::Unique - Tie-able array that allows only unique values SYNOPSIS
use Array::Unique; tie @a, 'Array::Unique'; Now use @a as a regular array. DESCRIPTION
This package lets you create an array which will allow only one occurrence of any value. In other words no matter how many times you put in 42 it will keep only the first occurrence and the rest will be dropped. You use the module via tie and once you tied your array to this module it will behave correctly. Uniqueness is checked with the 'eq' operator so among other things it is case sensitive. As a side effect the module does not allow undef as a value in the array. EXAMPLES
use Array::Unique; tie @a, 'Array::Unique'; @a = qw(a b c a d e f); push @a, qw(x b z); print "@a "; # a b c d e f x z DISCUSSION
When you are collecting a list of items and you want to make sure there is only one occurrence of each item, you have several option: 1) using an array and extracting the unique elements later You might use a regular array to hold this unique set of values and either remove duplicates on each update by that keeping the array always unique or remove duplicates just before you want to use the uniqueness feature of the array. In either case you might run a function you call @a = unique_value(@a); The problem with this approach is that you have to implement the unique_value function (see later) AND you have to make sure you don't forget to call it. I would say don't rely on remembering this. There is good discussion about it in the 1st edition of the Perl Cookbook of O'Reilly. I have copied the solutions here, you can see further discussion in the book. Extracting Unique Elements from a List (Section 4.6 in the Perl Cookbook 1st ed.) # Straightforward %seen = (); @uniq = (); foreach $item (@list) [ unless ($seen{$item}) { # if we get here we have not seen it before $seen{$item} = 1; push (@uniq, $item); } } # Faster %seen = (); foreach $item (@list) { push(@uniq, $item) unless $seen{$item}++; } # Faster but different %seen; foreach $item (@list) { $seen{$item}++; } @uniq = keys %seen; # Faster and even more different %seen; @uniq = grep {! $seen{$_}++} @list; 2) using a hash Some people use the keys of a hash to keep the items and put an arbitrary value as the values of the hash: To build such a list: %unique = map { $_ => 1 } qw( one two one two three four! ); To print it: print join ", ", sort keys %unique; To add values to it: $unique{$_}=1 foreach qw( one after the nine oh nine ); To remove values: delete @unique{ qw(oh nine) }; To check if a value is there: $unique{ $value }; # which is why I like to use "1" as my value (thanks to Gaal Yahas for the above examples) There are three drawbacks I see: 1) You type more. 2) Your reader might not understand at first why did you use hash and what will be the values. 3) You lose the order. Usually non of them is critical but when I saw this the 10th time in a code I had to understand with 0 documentation I got frustrated. 3) using Array::Unique So I decided to write this module because I got frustrated by my lack of understanding what's going on in that code I mentioned. In addition I thought it might be interesting to write this and then benchmark it. Additionally it is nice to have your name displayed in bright lights all over CPAN ... or at least in a module. Array::Unique lets you tie an array to hmmm, itself (?) and makes sure the values of the array are always unique. Since writing this I am not sure if I really recommend its usage. I would say stick with the hash version and document that the variable is aggregating a unique list of values. 4) Using real SET There are modules on CPAN that let you create and maintain SETs. I have not checked any of those but I guess they just as much of an overkill for this functionality as Unique::Array. BUGS
use Array::Unique; tie @a, 'Array::Unique'; @c = @a = qw(a b c a d e f b); @c will contain the same as @a AND two undefs at the end because @c you get the same length as the right most list. TODO
Test: Change size of the array Elements with false values ('', '0', 0) splice: splice @a; splice @a, 3; splice @a, -3; splice @a, 3, 5; splice @a, 3, -5; splice @a, -3, 5; splice @a, -3, -5; splice @a, ?, ?, @b; Benchmark speed Add faster functions that don't check uniqueness so if I know part of the data that comes from a unique source then I can speed up the process, In short shoot myself in the leg. Enable optional compare with other functions Write even better implementations. AUTHOR
Gabor Szabo <gabor@pti.co.il> LICENSE
Copyright (C) 2002-2008 Gabor Szabo <gabor@pti.co.il> All rights reserved. http://www.pti.co.il/ You may distribute under the terms of either the GNU General Public License or the Artistic License, as specified in the Perl README file. No WARRANTY whatsoever. CREDITS
Thanks for suggestions and bug reports to Szabo Balazs (dLux) Shlomo Yona Gaal Yahas Jeff 'japhy' Pinyan Werner Weichselberger VERSION
Version: 0.08 Date: 2008 June 04 perl v5.10.0 2009-03-06 Array::Unique(3pm)
All times are GMT -4. The time now is 06:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy