The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
How can I remove those duplicate sequence in UNIX?What command line I should type? How can I remove those duplicate sequence in UNIX?What command line I should type?
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How can I calculate the total of nucleotide in Unix?What command line I should type? patrick chia Shell Programming and Scripting 2 01-22-2009 04:39 AM
How to remove those sequence with same amino acid?What command line I should type? patrick chia Shell Programming and Scripting 4 01-20-2009 09:50 PM
Remove duplicate entry in one line kharen11 UNIX for Dummies Questions & Answers 5 07-05-2007 02:56 PM
Identify duplicate words in a line using command srinivasan_85 UNIX for Dummies Questions & Answers 8 05-01-2007 01:29 AM
Remove Duplicate line Student37 UNIX for Dummies Questions & Answers 1 02-22-2005 03:00 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 01-22-2009
patrick chia patrick chia is offline
Banned
  
 

Join Date: Jan 2009
Posts: 14
How can I remove those duplicate sequence in UNIX?What command line I should type?
The input is:
>HWI-EAS382_30FC7AAXX:4:1:1580:1465
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:1062:1640
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:272:629
AAAAAAAAGCTATAGTCTCGTCACACATACTCACAA
>HWI-EAS382_30FC7AAXX:4:1:1033:1135
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:1421:27
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

My desired output is:
>HWI-EAS382_30FC7AAXX:4:1:1580:1465
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>HWI-EAS382_30FC7AAXX:4:1:272:629
AAAAAAAAGCTATAGTCTCGTCACACATACTCACAA

What command line I should type to remove those duplicated sequence?
Thanks for all of your advise.
  #2 (permalink)  
Old 01-22-2009
Franklin52 Franklin52 is offline Forum Staff  
Moderator
  
 

Join Date: Feb 2007
Posts: 4,293
Something like this?

Code:
awk '/>/{s=$0;next}!a[$0]++{print s;print}' file
Regards
  #3 (permalink)  
Old 01-22-2009
cfajohnson's Avatar
cfajohnson cfajohnson is offline Forum Advisor  
Shell programmer, author
  
 

Join Date: Mar 2007
Location: Toronto, Canada
Posts: 2,361
Code:
awk '!x[$0]++' FILE
  #4 (permalink)  
Old 01-22-2009
patrick chia patrick chia is offline
Banned
  
 

Join Date: Jan 2009
Posts: 14
Quote:
Originally Posted by cfajohnson View Post
Code:
awk '!x[$0]++' FILE
Hi, fajohnson...
Your command line is worked. But still left all the header of the nucleotide sequence. Do you have better idea that I just remain the first header of those same nucleotide sequence?
My input:
>HWI-EAS382_30FC7AAXX:4:1:631:449
>HWI-EAS382_30FC7AAXX:4:1:93:1407
>HWI-EAS382_30FC7AAXX:4:1:154:1123
>HWI-EAS382_30FC7AAXX:4:1:912:1008
>HWI-EAS382_30FC7AAXX:4:1:57:316
>HWI-EAS382_30FC7AAXX:4:1:1287:1193
>HWI-EAS382_30FC7AAXX:4:1:1451:1559
>HWI-EAS382_30FC7AAXX:4:1:1431:1913
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

My output just want left this:
>HWI-EAS382_30FC7AAXX:4:1:631:449
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

Thanks for your advise. Have a nice day.
  #5 (permalink)  
Old 01-23-2009
skar_a skar_a is offline
Registered User
  
 

Join Date: Jan 2009
Posts: 20
Quote:
Originally Posted by patrick chia View Post
Hi, fajohnson...
Your command line is worked. But still left all the header of the nucleotide sequence. Do you have better idea that I just remain the first header of those same nucleotide sequence?
My input:
>HWI-EAS382_30FC7AAXX:4:1:631:449
>HWI-EAS382_30FC7AAXX:4:1:93:1407
>HWI-EAS382_30FC7AAXX:4:1:154:1123
>HWI-EAS382_30FC7AAXX:4:1:912:1008
>HWI-EAS382_30FC7AAXX:4:1:57:316
>HWI-EAS382_30FC7AAXX:4:1:1287:1193
>HWI-EAS382_30FC7AAXX:4:1:1451:1559
>HWI-EAS382_30FC7AAXX:4:1:1431:1913
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

My output just want left this:
>HWI-EAS382_30FC7AAXX:4:1:631:449
TTTCCGCGAACTGCAAAAGACGTTTCGTATGCCGTT

Thanks for your advise. Have a nice day.
Try this:
Code:
awk '/^>/{if(hdr=="") hdr=$0}/^[^>]/{x[$0]++;if(x[$0]==1) {print hdr;print} hdr=""}' infile
Closed Thread

Bookmarks

« Merging fields --- Help me plz | Converting Column values to comma delimted single Row »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 06:49 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0