How can I remove those duplicate sequence in UNIX?What command line I should type?

Login or Register to Ask a Question and Join Our Community

How can I remove those duplicate sequence in UNIX?What command line I should type?

Tags

command, shell scripts, unix

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting How can I remove those duplicate sequence in UNIX?What command line I should type?

01-22-2009

Registered User

14, 0

Join Date: Jan 2009

Last Activity: 24 November 2011, 10:50 PM EST

Posts: 14

Thanks Given: 0

Thanked 0 Times in 0 Posts

How can I remove those duplicate sequence in UNIX?What command line I should type?

The input is:
>HWI-EAS382_30FC7AAXX:4:1:1580:1465
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:1062:1640
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:272:629
AAAAAAAAGCTATAGTCTCGTCACACATACTCACAA
>HWI-EAS382_30FC7AAXX:4:1:1033:1135
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:1421:27
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

My desired output is:
>HWI-EAS382_30FC7AAXX:4:1:1580:1465
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:272:629
AAAAAAAAGCTATAGTCTCGTCACACATACTCACAA

What command line I should type to remove those duplicated sequence?
Thanks for all of your advise.

patrick chia

View Public Profile for patrick chia

Find all posts by patrick chia

01-22-2009

Registered User

7,747, 559

Join Date: Feb 2007

Last Activity: 20 April 2020, 11:28 AM EDT

Location: The Netherlands

Posts: 7,747

Thanks Given: 139

Thanked 559 Times in 520 Posts

Something like this?

Code:

awk '/>/{s=$0;next}!a[$0]++{print s;print}' file

Regards

Franklin52

View Public Profile for Franklin52

Find all posts by Franklin52

01-22-2009

Registered User

2,898, 136

Join Date: Mar 2007

Last Activity: 11 July 2016, 2:55 PM EDT

Location: Toronto, Canada

Posts: 2,898

Thanks Given: 0

Thanked 136 Times in 120 Posts

Code:

awk '!x[$0]++' FILE

cfajohnson

View Public Profile for cfajohnson

Find all posts by cfajohnson

01-22-2009

Registered User

14, 0

Join Date: Jan 2009

Last Activity: 24 November 2011, 10:50 PM EST

Posts: 14

Thanks Given: 0

Thanked 0 Times in 0 Posts

Quote:

Originally Posted by cfajohnson

Code:

awk '!x[$0]++' FILE

Hi, fajohnson...
Your command line is worked. But still left all the header of the nucleotide sequence. Do you have better idea that I just remain the first header of those same nucleotide sequence?
My input:
>HWI-EAS382_30FC7AAXX:4:1:631:449
>HWI-EAS382_30FC7AAXX:4:1:93:1407
>HWI-EAS382_30FC7AAXX:4:1:154:1123
>HWI-EAS382_30FC7AAXX:4:1:912:1008
>HWI-EAS382_30FC7AAXX:4:1:57:316
>HWI-EAS382_30FC7AAXX:4:1:1287:1193
>HWI-EAS382_30FC7AAXX:4:1:1451:1559
>HWI-EAS382_30FC7AAXX:4:1:1431:1913
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

My output just want left this:
>HWI-EAS382_30FC7AAXX:4:1:631:449
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

Thanks for your advise. Have a nice day.

patrick chia

View Public Profile for patrick chia

Find all posts by patrick chia

01-23-2009

Registered User

20, 1

Join Date: Jan 2009

Last Activity: 17 March 2009, 3:42 AM EDT

Posts: 20

Thanks Given: 0

Thanked 1 Time in 1 Post

Quote:

Originally Posted by patrick chia

Hi, fajohnson...
Your command line is worked. But still left all the header of the nucleotide sequence. Do you have better idea that I just remain the first header of those same nucleotide sequence?
My input:
>HWI-EAS382_30FC7AAXX:4:1:631:449
>HWI-EAS382_30FC7AAXX:4:1:93:1407
>HWI-EAS382_30FC7AAXX:4:1:154:1123
>HWI-EAS382_30FC7AAXX:4:1:912:1008
>HWI-EAS382_30FC7AAXX:4:1:57:316
>HWI-EAS382_30FC7AAXX:4:1:1287:1193
>HWI-EAS382_30FC7AAXX:4:1:1451:1559
>HWI-EAS382_30FC7AAXX:4:1:1431:1913
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

My output just want left this:
>HWI-EAS382_30FC7AAXX:4:1:631:449
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

Thanks for your advise. Have a nice day.

Try this:

Code:

awk '/^>/{if(hdr=="") hdr=$0}/^[^>]/{x[$0]++;if(x[$0]==1) {print hdr;print} hdr=""}' infile

skar_a

View Public Profile for skar_a

Find all posts by skar_a

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate line starting with a pattern

HI, I have the below input file /* ----------------- cmdsDlyStartFWJ -----------------*/ UNIX_JOB CMDS065J RUN ANY CMDNAME sleep 5 AGENT CMDSHP USER proddata RUN MON,TUE,WED,THU,FRI DELAYSUB 02:00 /* "Triggers daily file watcher jobs" */ ENVAR...

2. Shell Programming and Scripting

Remove duplicate entries from the same line

Hello, I have a file which have several duplicate entries on the same line: File ID source 1 GM GF GM 2 GM GF GM GF GM GF GM GF GM GF 3 GM GF GM SF GM GF GM SF 4 FF FF FF FF 5 FF GM FF ...

3. UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

So I have a bunch of files that look like this >gi|33332323 MMKCRGVIMVVEKVMKRDGRIVPFDESRIRWAVQ--- >gi|45235353 MMKCR----VEKMRDVFFDESIRWAVQ They go on...sequences are much longer but all in two line (fasta) format. I want to remove duplicate pairs of ID(GI) number and sequence. I tried...

4. Shell Programming and Scripting

Remove duplicate line on condition

Hi Ive been scratching over this for some time with no solution. I have a file like this 1 bla bla 1 2 bla bla 2 4 bla bla 3 5 bla bla 1 6 bla bla 1 I want to remove consecutive occurrences of lines like bla bla 1, but the first column may be different. Any ideasss??

5. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from...

6. Shell Programming and Scripting

remove duplicate words in a line

Hi, Please help! I have a file having duplicate words in some line and I want to remove the duplicate words. The order of the words in the output file doesn't matter. INPUT_FILE pink_kite red_pen ball pink_kite ball yellow_flower white no white no cloud nine_pen pink cloud pink nine_pen...

7. Shell Programming and Scripting

How can I calculate the total of nucleotide in Unix?What command line I should type?

For example, if I have the file whose content are: >HWI-EAS382_30FC7AAXX:7:1:927:1368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA >HWI-EAS382_30FC7AAXX:7:1:924:1373 ACGAACTTTAAAGCACCTCTTGGCTCGTATGCCGTC I want my output calculate the total of nucleotide. So my output should look like this:...

8. Shell Programming and Scripting

How to remove those sequence with same amino acid?What command line I should type?

My input is listed as: giNumber RefAminoAcid VarAminoAcid 10190711 P P 10190711 D D 109255248 I A 110349771 A ...

9. UNIX for Dummies Questions & Answers

Remove duplicate entry in one line

Can anyone help me how can i print only the unique entry in a line? MI_AP MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR so that it will on output the one unique entry per line. MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN NT_CAR I can't find the same situation on the knowledge...

10. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate...

Login or Register to Ask a Question