Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicates based on query and subject fields from blast output file Post 302640513 by pravin27 on Tuesday 15th of May 2012 01:40:56 AM
Old 05-15-2012
AWK ??

Code:
awk '!a[$1$2]++' filename

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicates based on a column in fixed width file

Hi, How to output the duplicate record to another file. We say the record is duplicate based on a column whose position is from 2 and its length is 11 characters. The file is a fixed width file. ex of Record: DTYU12333567opert tjhi kkklTRG9012 The data in bold is the key on which... (1 Reply)
Discussion started by: Qwerty123
1 Replies

2. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

3. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

4. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

5. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

6. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

7. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

8. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

9. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies
Boulder::Blast(3pm)					User Contributed Perl Documentation				       Boulder::Blast(3pm)

NAME
Boulder::Blast - Parse and read BLAST files SYNOPSIS
use Boulder::Blast; # parse from a single file $blast = Boulder::Blast->parse('run3.blast'); # parse and read a set of blast output files $stream = Boulder::Blast->new('run3.blast','run4.blast'); while ($blast = $stream->get) { # do something with $blast object } # parse and read a whole directory of blast runs $stream = Boulder::Blast->new(<*.blast>); while ($blast = $stream->get) { # do something with $blast object } # parse and read from STDIN $stream = Boulder::Blast->new; while ($blast = $stream->get) { # do something with $blast object } # parse and read as a filehandle $stream = Boulder::Blast->newFh(<*.blast>); while ($blast = <$stream>) { # do something with $blast object } # once you have a $blast object, you can get info about it: $query = $blast->Blast_query; @hits = $blast->Blast_hits; foreach $hit (@hits) { $hit_sequence = $hit->Name; # get the ID $significance = $hit->Signif; # get the significance @hsps = $hit->Hsps; # list of HSPs foreach $hsp (@hsps) { $query = $hsp->Query; # query sequence $subject = $hsp->Subject; # subject sequence $signif = $hsp->Signif; # significance of HSP } } DESCRIPTION
The Boulder::Blast class parses the output of the Washington University (WU) or National Cenber for Biotechnology Information (NCBI) series of BLAST programs and turns them into Stone records. You may then use the standard Stone access methods to retrieve information about the BLAST run, or add the information to a Boulder stream. The parser works equally well on the contents of a static file, or on information read dynamically from a filehandle or pipe. METHODS
parse() Method $stone = Boulder::Blast->parse($file_path); $stone = Boulder::Blast->parse($filehandle); The parse() method accepts a path to a file or a filehandle, parses its contents, and returns a Boulder Stone object. The file path may be absolute or relative to the current directgly. The filehandle may be specified as an IO::File object, a FileHandle object, or a reference to a glob ("*FILEHANDLE" notation). If you call parse() without any arguments, it will try to parse the contents of standard input. new() Method $stream = Boulder::Blast->new; $stream = Boulder::Blast->new($file [,@more_files]); $stream = Boulder::Blast->new(*FILEHANDLE); If you wish, you may create the parser first with Boulder::Blast new(), and then invoke the parser object's parse() method as many times as you wish to, producing a Stone object each time. TAGS
The following tags are defined in the parsed Blast Stone object: Information about the program These top-level tags provide information about the version of the BLAST program itself. Blast_program The name of the algorithm used to run the analysis. Possible values include: blastn blastp blastx tblastn tblastx fasta3 fastx3 fasty3 tfasta3 tfastx3 tfasty3 Blast_version This gives the version of the program in whatever form appears on the banner page, e.g. "2.0a19-WashU". Blast_program_date This gives the date at which the program was compiled, if and only if it appears on the banner page. Information about the run These top-level tags give information about the particular run, such as the parameters that were used for the algorithm. Blast_run_date This gives the date and time at which the similarity analysis was run, in the format "Fri Jul 6 09:32:36 1998" Blast_parms This points to a subrecord containing information about the algorithm's runtime parameters. The following subtags are used. Others may be added in the future: Hspmax the value of the -hspmax argument Expectation the value of E Matrix the matrix in use, e.g. BLOSUM62 Ctxfactor the value of the -ctxfactor argument Gapall The value of the -gapall argument Information about the query sequence and subject database Thse top-level tags give information about the query sequence and the database that was searched on. Blast_query The identifier for the search sequence, as defined by the FASTA format. This will be the first set of non-whitespace characters following the ">" character. In other words, the search sequence "name". Blast_query_length The length of the query sequence, in base pairs. Blast_db The Unix filesystem path to the subject database. Blast_db_title The title of the subject database. The search results: the Blast_hits tag. Each BLAST hit is represented by the tag Blast_hits. There may be zero, one, or many such tags. They will be presented in reverse sorted order of significance, i.e. most significant hit first. Each Blast_hits tag is a Stone subrecord containing the following subtags: Name The name/identifier of the sequence that was hit. Length The total length of the sequence that was hit Signif The significance of the hit. If there are multiple HSPs in the hit, this will be the most significant (smallest) value. Identity The percent identity of the hit. If there are multiple HSPs, this will be the one with the highest percent identity. Expect The expectation value for the hit. If there are multiple HSPs, this will be the lowest expectation value in the set. Hsps One or more sub-sub-tags, pointing to a nested record containing information about each high-scoring segment pair (HSP). See the next section for details. The Hsp records: the Hsps tag Each Blast_hit tag will have at least one, and possibly several Hsps tags, each one corresponding to a high-scoring segment pair (HSP). These records contain detailed information about the hit, including the alignments. Tags are as follows: Signif The significance (P value) of this HSP. Bits The number of bits of significance. Expect Expectation value for this HSP. Identity Percent identity. Positives Percent positive matches. Score The Smith-Waterman alignment score. Orientation The word "plus" or "minus". This tag is only present for nucleotide searches, when the reverse complement match may be present. Strand Depending on algorithm used, indicates complementarity of match and possibly the reading frame. This is copied out of the blast report. Possibilities include: "Plus / Minus" "Plus / Plus" -- blastn algorithm "+1 / -2" "+2 / -2" -- blastx, tblastx Query_start Position at which the HSP starts in the query sequence (1-based indexing). Query_end Position at which the HSP stops in the query sequence. Subject_start Position at which the HSP starts in the subject (target) sequence. Subject_end Position at which the HSP stops in the subject (target) sequence. Query, Subject, Alignment These three tags contain strings which, together, create the gapped alignment of the query sequence with the subject sequence. For example, to print the alignment of the first HSP of the first match, you might say: $hsp = $blast->Blast_hits->Hsps; print join(" ",$hsp->Query,$hsp->Alignment,$hsp->Subject)," "; See the bottom of this manual page for an example BLAST run. CAVEATS
This module has been extensively tested with WUBLAST, but very little with NCBI BLAST. It probably will not work with PSI Blast or other variants. The author plans to adapt this module to parse other formats, as well as non-BLAST formats such as the output of Fastn. SEE ALSO
Boulder, Boulder::GenBank AUTHOR
Lincoln Stein <lstein@cshl.org>. Copyright (c) 1998-1999 Cold Spring Harbor Laboratory This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See DISCLAIMER.txt for disclaimers of warranty. EXAMPLE BLASTN RUN
This output was generated by the quickblast.pl program, which is located in the eg/ subdirectory of the Boulder distribution directory. It is a typical blastn (nucleotide->nucleotide) run; however long lines (usually DNA sequences) have been truncated. Also note that per the Boulder protocol, the percent sign (%) is escaped in the usual way. It will be unescaped when reading the stream back in. Blast_run_date=Fri Nov 6 14:40:41 1998 Blast_db_date=2:40 PM EST Nov 6, 1998 Blast_parms={ Hspmax=10 Expectation=10 Matrix=+5,-4 Ctxfactor=2.00 } Blast_program_date=05-Feb-1998 Blast_db= /usr/tmp/quickblast18202aaaa Blast_version=2.0a19-WashU Blast_query=BCD207R Blast_db_title= test.fasta Blast_query_length=332 Blast_program=blastn Blast_hits={ Signif=3.5e-74 Expect=3.5e-74, Name=BCD207R Identity=100%25 Length=332 Hsps={ Subject=GTGCTTTCAAACATTGATGGATTCCTCCCCTTGACATATATATATACTTTGGGTTCCCGCAA... Signif=3.5e-74 Length=332 Bits=249.1 Query_start=1 Subject_end=332 Query=GTGCTTTCAAACATTGATGGATTCCTCCCCTTGACATATATATATACTTTGGGTTCCCGCAA... Positives=100%25 Expect=3.5e-74, Identity=100%25 Query_end=332 Orientation=plus Score=1660 Strand=Plus / Plus Subject_start=1 Alignment=||||||||||||||||||||||||||||||||||||||||||||||||||||||||||... } } = Example BLASTP run Here is the output from a typical blastp (protein->protein) run. Long lines have again been truncated. Blast_run_date=Fri Nov 6 14:37:23 1998 Blast_db_date=2:36 PM EST Nov 6, 1998 Blast_parms={ Hspmax=10 Expectation=10 Matrix=BLOSUM62 Ctxfactor=1.00 } Blast_program_date=05-Feb-1998 Blast_db= /usr/tmp/quickblast18141aaaa Blast_version=2.0a19-WashU Blast_query=YAL004W Blast_db_title= elegans.fasta Blast_query_length=216 Blast_program=blastp Blast_hits={ Signif=0.95 Expect=3.0, Name=C28H8.2 Identity=30%25 Length=51 Hsps={ Subject=HMTVEFHVTSQSW---FGFEDHFHMIIR-AVNDENVGWGVRYLSMAF Signif=0.95 Length=46 Bits=15.8 Query_start=100 Subject_end=49 Query=HLTQD-HGGDLFWGKVLGFTLKFNLNLRLTVNIDQLEWEVLHVSLHF Positives=52%25 Expect=3.0, Identity=30%25 Query_end=145 Orientation=plus Score=45 Subject_start=7 Alignment=H+T + H W GF F++ +R VN + + W V ++S+ F } } Blast_hits={ Signif=0.99 Expect=4.7, Name=ZK896.2 Identity=24%25 Length=340 Hsps={ Subject=FSGKFTTFVLNKDQATLRMSSAEKTAEWNTAFDSRRGFF----TSGNYGL... Signif=0.99 Length=101 Bits=22.9 Query_start=110 Subject_end=243 Query=FWGKVLGFTL-KFNLNLRLTVNIDQLEWEVLHVSLHFWVVEVSTDQTLSVE... Positives=41%25 Expect=4.7, Identity=24%25 Query_end=210 Orientation=plus Score=65 Subject_start=146 Alignment=F GK F L K LR++ EW S + T +... } } = perl v5.10.1 2002-02-04 Boulder::Blast(3pm)
All times are GMT -4. The time now is 07:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy