Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Select distinct values from a flat file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Select distinct values from a flat file
# 1  
Old 05-18-2010
Select distinct values from a flat file
Hi ,
I have a similar problem.
Please can anyone help me with a shell script or a perl.
I have a flat file like this
Code:
fruit     country
apple     germany
apple     india
banana    pakistan
banana    saudi
mango     india

I want to get a output like
Code:
 
fruit       country
apple     germany
banana    pakistan
mango     india

Is there anyways this can be done?SmilieSmilie

Moderator's Comments:
Mod Comment Use code tags please, ty.
Last edited by zaxxon; 05-19-2010 at 03:43 AM..
# 2  
Old 05-18-2010
Code:
awk '{a[$1]++}a[$1]<2' file

# 3  
Old 05-18-2010
Thanks sir. Please also tell me how this can be sent to a file
# 4  
Old 05-18-2010
Code:
awk '{a[$1]++}a[$1]<2' file > new.file

# 5  
Old 05-19-2010
Thanks Mate for this.
Unforfunately other module have decided to send a comma seperated file instead of space seperated. Please help again .
I have values like this now
Code:
INDIA,mango,2
PAK,mango,1
USA,apple,1
UK,apple,1

Last edited by zaxxon; 05-19-2010 at 03:43 AM.. Reason: code tags
# 6  
Old 05-19-2010
Quote:
Originally Posted by smalya
Thanks Mate for this.
Unforfunately other module have decided to send a comma seperated file instead of space seperated. Please help again .
I have values like this now

INDIA,mango,2
PAK,mango,1
USA,apple,1
UK,apple,1
A little modificatiion to danmero solution...
Code:
awk -F, '{a[$1]++}a[$1]<2' file > new.file

i guess you might need to change $1 to $2...
# 7  
Old 05-19-2010
Code:
awk -F "," '{a[$2]++}a[$2]<2' file2


cheers,
Devaraj Takhellambam
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Select distinct sequences from fasta file and list

Hi How can I extract sequences from a fasta file with respect a certain criteria? The beginning of my file (containing in total more than 1000 sequences) looks like this: >H8V34IS02I59VP SDACNDLTIALLQIAREVRVCNPTFSFRWHPQVKDEVMRECFDCIRQGLG YPSMRNDPILIANCMNWHGHPLEEARQWVHQACMSPCPSTKHGFQPFRMA... (6 Replies)
Discussion started by: Marion MPI
6 Replies

2. Shell Programming and Scripting

Finding distinct characters from flat file

Hi....I need one help.... I'm having a files which is having the data as follows... a b c c d d d e f Now I need to find out distinct characters from this file and the output should be as follows - a b c d e f Can you please help me on this? I'm using KSH script. (18 Replies)
Discussion started by: Krishanu Saha
18 Replies

3. Shell Programming and Scripting

Select distinct rows in a file by last column

Hi, I have the following file: LOG:015608::ERR:2310:map_spsrec:Invalid parameter LOG:015608::ERR:2471:map_dgdrec:Invalid parameter LOG:015608::ERR:2487:map_nnmrec:Invalid number LOG:015608::ERR:2310:map_nmrec:Invalid number LOG:015608::ERR:2438:map_nmrec:Invalid number As a delimiter I... (2 Replies)
Discussion started by: apenkov
2 Replies

4. Shell Programming and Scripting

distinct values of all the fields

I am a beginner to scripting, please help me in this regard. How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number... (2 Replies)
Discussion started by: vukkusila
2 Replies

5. UNIX for Dummies Questions & Answers

distinct values of all the fields

I am a beginner to scripting, please help me in this regard. How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number... (1 Reply)
Discussion started by: vukkusila
1 Replies

6. Shell Programming and Scripting

grep distinct values

this is a little more complex than that. I have a text file and I need to find all the distinct words that appear in a line after the word TABLESPACE when I grep for just the word tablespace, I get: how do i parse this a little better so i have a smaller file to read? This is just an... (4 Replies)
Discussion started by: guessingo
4 Replies

7. UNIX for Dummies Questions & Answers

Select Distinct on multiple fields

How do I create a script that provides a count of distinct values of a particular field in a file utilizing commonly available UNIX commands (sh or awk)? Field1|Field2|Field3|Field4 AAA|BBB|CCC|DDD 111|222|333|777 AAA|EEE|ZZZ|EEE 111|555|333|444 AAA|EEE|CCC|DDD 111|222|555|444 For... (2 Replies)
Discussion started by: Refresher
2 Replies

8. Shell Programming and Scripting

Getting Distinct values from second field in a file....

Hi I have a pipe delimited file. I am trying to grab the DISTINCT value from the second field. The file is something like: 1233|apple|ron 1234|apple|elephant 1235|egg|man the output I am trying to get from second field is apple,egg (apple coming only once) Thanks simi (4 Replies)
Discussion started by: simi28
4 Replies

9. Shell Programming and Scripting

Loop through only the distinct values in a file

Datafile has the following data seperated by : FIELD1:FIELD2:FIELD3 D1:/opt/9.1.9:Y D2:/opt/10.1.10:Y D3:/opt/9.1.9:Y D4:/opt/8.1.8:Y D5:/opt/8.1.8:Y D6:/opt/9.1.9:Y D7:/opt/9.1.9:Y D8:/opt/10.1.10:Y D9:/opt/9.1.9:Y D10:/opt/10.1.10:Y I want to do some operations only on the distinct... (2 Replies)
Discussion started by: pbekal
2 Replies

10. UNIX for Dummies Questions & Answers

select distinct row from a file

Hi, buddies out there. I have a text file ( only one column ) which I created using vi editor. The file contains duplicate rows and I would like to select distinct rows, how to go on it using unix command: file content = apple apple orange watermelon apple orange Can it be done... (7 Replies)
Discussion started by: merry susana
7 Replies
Login or Register to Ask a Question