Filtering the duplicates


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Filtering the duplicates
# 1  
Old 06-27-2012
Filtering the duplicates

Hello,

I want to filter all the duplicates of a record to one place. Sample input and output will give you better idea.

I am new to unix. Can some one help me on this?

Input:
Code:
7488	7389	chr1.fa	chr1.fa
3546	9887	chr5.fa	chr9.fa
7387	7898	chrX.fa	chr3.fa
7488	7389	chr1.fa	chr1.fa
7488	7389	chr21.fa	chr3.fa
7488	7389	chr1.fa	chr1.fa
9899	4577	chrX.fa	chrM.fa
3546	9887	chr5.fa	chr9.fa

My desired output:
Code:
7488	7389	chr1.fa	chr1.fa
7488	7389	chr1.fa	chr1.fa
7488	7389	chr1.fa	chr1.fa
3546	9887	chr5.fa	chr9.fa
3546	9887	chr5.fa	chr9.fa
7387	7898	chrX.fa	chr3.fa
7488	7389	chr21.fa	chr3.fa
9899	4577	chrX.fa	chrM.fa

Thanks in advance,


Moderator's Comments:
Mod Comment Please view this code tag video for how to use code tags when posting code and data.

Last edited by vbe; 06-27-2012 at 12:30 PM.. Reason: code tags please
# 2  
Old 06-27-2012
Code:
cat input | sort -k3,3

# 3  
Old 06-27-2012
Code:
sort <input

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filtering duplicates based on lookup table and rules

please help solving the following. I have access to redhat linux cluster having 32gigs of ram. I have duplicate ids for variable names, in the file 1,2 are duplicates;3,4 and 5 are duplicates;6 and 7 are duplicates. My objective is to use only the first occurrence of these duplicates. Lookup... (4 Replies)
Discussion started by: ritakadm
4 Replies

2. Shell Programming and Scripting

Need help on filtering

Hi experts, I have a file image.csv as below: COMPUTERNAME,23/07/2013,22/07/2013,21/07/2013,20/07/2013,19/07/2013,18/07/2013,17/07/2013 AED03852180,3,3,3,3,3,3,3 AED03852181,3,3,3,3,3,3,1 AED09020382,3,0,3,0,3,3,3 AED09020383,1,3,3,3,2,1,3 AED09020386,3,3,0,3,3,0,3 ... (4 Replies)
Discussion started by: zaq1xsw2
4 Replies

3. Shell Programming and Scripting

PERL "filtering the log file removing the duplicates

Hi folks, I have a log file in the below format and trying to get the output of the unique ones based on mnemonic IN PERL. Could any one please let me know with the code and the logic ? Severity Mnemonic Log Message 7 CLI_SCHEDULER Logfile for scheduled CLI... (3 Replies)
Discussion started by: scriptscript
3 Replies

4. Shell Programming and Scripting

Filtering out duplicates with the highest version number

Hi, I have a huge text file with filenames which which looks like the following ie uniquenumber_version_filename: e.g. 1234_1_xxxx 1234_2_vfvfdbb 343333_1_vfvfdvd 2222222_1_ggggg 55555_1_xxxxxx 55555_2_vrbgbgg 55555_3_grgrbr What I need to do is examine the file, look for... (4 Replies)
Discussion started by: mantis
4 Replies

5. Shell Programming and Scripting

Filtering

Hi I am interested in DNS resolving a set of sites and each time the output is different- $ host www.yahoo.com www.yahoo.com is an alias for fd-fp3.wg1.b.yahoo.com. fd-fp3.wg1.b.yahoo.com is an alias for ds-fp3.wg1.b.yahoo.com. ds-fp3.wg1.b.yahoo.com is an alias for... (1 Reply)
Discussion started by: jamie_123
1 Replies

6. AIX

Need help with filtering

Hi!! I have a bit of a task here and filtering/scripting not my strongest. I have to collect info of approx 1100 hdiskpower.so i have appended all the hdisk into a text file and i need it to run the command lscfg -vl to confirm if the drive is symmetrix. here's what i have so far at... (3 Replies)
Discussion started by: vpundit
3 Replies

7. Shell Programming and Scripting

Filtering a list

Hi all When I run the system command "rpm -qa |grep xmpp" it lists many files like xmpp-3.2.10.20111024-3 xmpp-3.2.10.201110_asd xmpp-3.2.10.201 and I want to uninstall all the rpms by using rpm -e. How can I do this??? NOTE: The number of rpms will vary... (3 Replies)
Discussion started by: Ananthdoss
3 Replies

8. Shell Programming and Scripting

Please help me to do some filtering

I have to grep a pattern. scenario is like :- Suppose "/etc/sec/one" is a string, i need to check if this string contains "one" using any utility something like if /etc/sec/one | grep ; then Thanks in advance Renjesh Raju (3 Replies)
Discussion started by: Renjesh
3 Replies

9. Shell Programming and Scripting

help need in filtering data

Hello Gurus, Please help me out of the problem. I ve a input file as below input clock; input a; //reset all input b; //input comment output c; output d; output e; input f; //output comment I need the output as follows: \\Inputs (1 Reply)
Discussion started by: user_prady
1 Replies

10. Shell Programming and Scripting

Pipe filtering

Hi, The thing is that sometimes I need to filter an arbitrary output given a condition (think on the "filter" function present on some programming languages) from the shell (I use sh/bash). I tried this: #!/bin/sh # Usage: filter command COMMAND=$1 shift 1 xargs "$@" -I "{}" sh -c "if... (3 Replies)
Discussion started by: tokland
3 Replies
Login or Register to Ask a Question