Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Get the distinct string from file.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Get the distinct string from file.
# 1  
Old 09-30-2011
Get the distinct string from file.
my question is , i want to get the distinct email address from the file which contains:
<Email address>|<file name>
/tmp/test.log
Code:
petercs@yahoo.com|sf1.pdf
sandycs@yahoo.com|sf2.pdf
davidcs@yahoo.com|sf3.pdf
davidcs@yahoo.com|sf4.pdf

i want to get the result like:
Code:
petercs@yahoo.com
sandycs@yahoo.com
davidcs@yahoo.com

Last edited by Franklin52; 09-30-2011 at 03:20 AM.. Reason: Code tags
# 2  
Old 09-30-2011
Code:
$ awk -F\| '{print $1|"uniq"}' infile

# 3  
Old 09-30-2011
try this...
Code:
cat <inputfile> | cut -d "|" -f 1 | uniq

Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
Last edited by Franklin52; 09-30-2011 at 03:52 AM.. Reason: Please use code tags, thank you
# 4  
Old 09-30-2011
Code:
 
$ nawk -F\| '{a[$1]=$1;next}END{for(i in a){print i}}' inputfile
petercs@yahoo.com
davidcs@yahoo.com
sandycs@yahoo.com

---------- Post updated at 01:14 PM ---------- Previous update was at 01:12 PM ----------

@jayan,

uniq always work on sorted output

Code:
 
$ cat test
petercs@yahoo.com|sf1.pdf
sandycs@yahoo.com|sf2.pdf
davidcs@yahoo.com|sf3.pdf
davidcs@yahoo.com|sf4.pdf
sandycs@yahoo.com|sf2.pdf

$ awk -F\| '{print $1|"uniq"}' test
petercs@yahoo.com
sandycs@yahoo.com
davidcs@yahoo.com
sandycs@yahoo.com

$ awk -F\| '{print $1|"sort|uniq"}' test
davidcs@yahoo.com
petercs@yahoo.com
sandycs@yahoo.com

# 5  
Old 09-30-2011
Code:
sed 's/|.*$//' file|sed '$!N;/^\(.*\)\n\1$/!P;D'
petercs@yahoo.com
sandycs@yahoo.com
davidcs@yahoo.com

# 6  
Old 09-30-2011
Code:
 
$ perl -F"\|" -lane 'push @a, $F[0];END{%seen=(); @unique = grep { ! $seen{$_} ++ } @a; print join("\n",@unique)}' test                            
petercs@yahoo.com
sandycs@yahoo.com
davidcs@yahoo.com

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Get distinct value in bash between two string

Sorry for my english i am french So i am receiving from a script this prompt : tabular;critical;mirroring;DG INTlocaldg VOLUME appears tabular;critical;mirroring;DG INTlocaldg VOLUME bh3vm tabular;critical;mirroring;DG INTlocaldg VOLUME dev tabular;critical;mirroring;DG INTlocaldg VOLUME... (3 Replies)
Discussion started by: cterra
3 Replies

2. Shell Programming and Scripting

How to get distinct Tags from an XML file?

Sample XML file: <?xml version="1.0" encoding="UTF-16" ?> <Provider PROVIDER="xx" SCHEMA_VERSION="2.5"> <Institution UNINUM="xxxx" EXTRACT_DATE="2013-12-31" CUSTOMER_ROW_COUNT="1577" LOAN_ROW_COUNT="3322" BOOK_VALUE_DOLLARS="720163381.46"> <Customer CIF="dww213">... (11 Replies)
Discussion started by: Ariean
11 Replies

3. Shell Programming and Scripting

How do a distinct from a file using sort uniq in bash?

I have an output file .dat. From this file i have to do a distinct of the ID using the sort uniq command in bash script. How can i do it? i found : sort -u ${FILEOUT_DAT} but i don't think is my solution because the id isn't specified.. is there other solution? (7 Replies)
Discussion started by: punticci
7 Replies

4. Shell Programming and Scripting

Select distinct rows in a file by last column

Hi, I have the following file: LOG:015608::ERR:2310:map_spsrec:Invalid parameter LOG:015608::ERR:2471:map_dgdrec:Invalid parameter LOG:015608::ERR:2487:map_nnmrec:Invalid number LOG:015608::ERR:2310:map_nmrec:Invalid number LOG:015608::ERR:2438:map_nmrec:Invalid number As a delimiter I... (2 Replies)
Discussion started by: apenkov
2 Replies

5. UNIX for Dummies Questions & Answers

how to get distinct counts in a column of a file

If i have a file sample.txt with more than 10 columns and 11th column as following data. would it be possible to get the distinct counts of values in single shot,Thank you. Y Y N N N P P o Expected Result: Value count Y 2 N 3 P 2 (2 Replies)
Discussion started by: Ariean
2 Replies

6. Shell Programming and Scripting

Select distinct values from a flat file

Hi , I have a similar problem. Please can anyone help me with a shell script or a perl. I have a flat file like this fruit country apple germany apple india banana pakistan banana saudi mango india I want to get a output like fruit country apple ... (7 Replies)
Discussion started by: smalya
7 Replies

7. Shell Programming and Scripting

Getting Distinct values from second field in a file....

Hi I have a pipe delimited file. I am trying to grab the DISTINCT value from the second field. The file is something like: 1233|apple|ron 1234|apple|elephant 1235|egg|man the output I am trying to get from second field is apple,egg (apple coming only once) Thanks simi (4 Replies)
Discussion started by: simi28
4 Replies

8. Shell Programming and Scripting

Loop through only the distinct values in a file

Datafile has the following data seperated by : FIELD1:FIELD2:FIELD3 D1:/opt/9.1.9:Y D2:/opt/10.1.10:Y D3:/opt/9.1.9:Y D4:/opt/8.1.8:Y D5:/opt/8.1.8:Y D6:/opt/9.1.9:Y D7:/opt/9.1.9:Y D8:/opt/10.1.10:Y D9:/opt/9.1.9:Y D10:/opt/10.1.10:Y I want to do some operations only on the distinct... (2 Replies)
Discussion started by: pbekal
2 Replies

9. UNIX for Dummies Questions & Answers

select distinct row from a file

Hi, buddies out there. I have a text file ( only one column ) which I created using vi editor. The file contains duplicate rows and I would like to select distinct rows, how to go on it using unix command: file content = apple apple orange watermelon apple orange Can it be done... (7 Replies)
Discussion started by: merry susana
7 Replies
Login or Register to Ask a Question