Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Diff command on two Fastq.gz files Post 302876927 by jim mcnamara on Wednesday 27th of November 2013 08:27:27 AM
Old 11-27-2013
There are VERY, VERY likely NO errors in the merge because gzip, zcat, etc., do a checksum.

A better verification if you honestly do not understand what compression does:
Code:
(gunzip -c file1.gz; gunzip -c file2.gz)> tmp1; gunzip -c bigfile > tmp2
diff tmp1 tmp2
rm tmp1 tmp2

gunzip -c is the same as zcat....
If compression had problems everyone would be doing diffs, and probably not using gzip, gunzip for anything.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

diff 2 files; output diff's to 3rd file

Hello, I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3. For example: file 1 123 1234 123456 file 2 123 2345 23456 file 3 should have 2345 23456 I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies

2. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

3. AIX

diff command

hello i've two files. how i get the diff between the two files to new file. thanks best regards ariec (3 Replies)
Discussion started by: ariec
3 Replies

4. Shell Programming and Scripting

diff command help

Hi all diff file1 file 2 command will give us op of diff between two file. But it aslo give its position and sign "<" or ">". I dont want position and sign in op. Only diff of content should be come as op. Kindly help me for this. Regards Jaydeep (1 Reply)
Discussion started by: jaydeep_sadaria
1 Replies

5. UNIX for Dummies Questions & Answers

Diff command of two files

Hi, I use the diff command to compare two files and append this output to a file. I would like to now not only produce the differences but be able to output the total number of changes made, the number of new files added and the number of files deleted, is there I can do this using the diff... (2 Replies)
Discussion started by: cyberfrog
2 Replies

6. Shell Programming and Scripting

Combining 3 fastq files

Hello, I am working with next-gen short-read sequence data, which we receive in 3 fastq files. These are arranged in 4-line groups for each read: line1: read identifier, beginning, e.g., "@HWI-ST1342..." line2: DNA sequence, for files 1 and 2, 101 characters, for file 3, 7 chars. line3: "+"... (2 Replies)
Discussion started by: ljk
2 Replies

7. Shell Programming and Scripting

Extract length wise sequences from fastq file

I have a fastq file from small RNA sequencing with sequence lengths between 15 - 30. I wanted to filter sequence lengths between 21-25 and write to another fastq file. how can i do that? (4 Replies)
Discussion started by: empyrean
4 Replies

8. Shell Programming and Scripting

One-way diff command?

Hello, I am trying to find the different files between multiple directories in Linux, here is a small assumption of what is inside the directories dir1 dir2 dir3 1.txt 1.txt 1.txt 2.txt 3.txt 3.txt 5.txt 4.txt 5.txt 6.txt 7.txt 8.txt I am using the following... (4 Replies)
Discussion started by: Error404
4 Replies

9. Shell Programming and Scripting

Diff 3 files, but diff only their 2nd column

Guys i have 3 files, but i want to compare and diff only the 2nd column path=`/home/whois/doms` for i in `cat domain.tx` do whois $i| sed -n '/Registry Registrant ID:/,/Registrant Email:/p' > $path/$i.registrant whois $i| sed -n '/Registry Admin ID:/,/Admin Email:/p' > $path/$i.admin... (10 Replies)
Discussion started by: kenshinhimura
10 Replies

10. UNIX for Beginners Questions & Answers

Comparing fastq files and outputting common records

I have two files: File_1: @M04961:22:000000000-B5VGJ:1:1101:9280:7106 1:N:0:86 GGCATGAAAACATACAAACCGTCTTTCCAGAAATTGTTCCAAGTATCGGCAACAGCTTTATCAATACCATGAAAAATATCAACCACACCAGAAGCAGCAT + GGGGGGGGGGGGGGGGGCCGGGGGF,EDFFGEDFG,@DGGCGGEGGG7DCGGGF68CGFFFGGGG@CGDGFFDFEFEFF:30CGAFFDFEFF8CAF;;8F ... (3 Replies)
Discussion started by: Xterra
3 Replies
Bio::SeqIO::fastq(3pm)					User Contributed Perl Documentation				    Bio::SeqIO::fastq(3pm)

NAME
Bio::SeqIO::fastq - fastq sequence input/output stream SYNOPSIS
################## pertains to FASTQ parsing only ################## # grabs the FASTQ parser, specifies the Illumina variant my $in = Bio::SeqIO->new(-format => 'fastq-illumina', -file => 'mydata.fq'); # simple 'fastq' format defaults to 'sanger' variant my $out = Bio::SeqIO->new(-format => 'fastq', -file => '>mydata.fq'); # $seq is a Bio::Seq::Quality object while (my $seq = $in->next_seq) { $out->write_seq($seq); # convert Illumina 1.3 to Sanger format } # for 4x faster parsing, one can do something like this for raw data use Bio::Seq::Quality; # $data is a hash reference containing all arguments to be passed to # the Bio::Seq::Quality constructor while (my $data = $in->next_dataset) { # process $data, such as trim, etc my $seq = Bio::Seq::Quality->new(%$data); # for now, write_seq only accepts Bio::Seq::Quality, but may be modified # to allow raw hash references for speed $out->write_seq($data); } DESCRIPTION
This object can transform Bio::Seq and Bio::Seq::Quality objects to and from FASTQ flat file databases. FASTQ is a file format used frequently at the Sanger Centre and in next-gen sequencing to bundle a FASTA sequence and its quality data. A typical FASTQ entry takes the form: @HCDPQ1D0501 GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT..... +HCDPQ1D0501 !''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65..... where: @ = descriptor, followed by one or more sequence lines + = optional descriptor (if present, must match first one), followed by one or more qual lines When writing FASTQ output the redundant descriptor following the '+' is by default left off to save disk space. If needed, one can set the quality_header() flag in order for this to be printed. FASTQ and Bio::Seq::Quality mapping FASTQ files have sequence and quality data on single line or multiple lines, and the quality values are single-byte encoded. Data are mapped very simply to Bio::Seq::Quality instances: Data Bio::Seq::Quality method ------------------------------------------------------------------------ first non-whitespace chars in descriptor id^ descriptor line desc^ sequence lines seq quality qual* FASTQ variant namespace ^ first nonwhitespace chars are id(), everything else after (to end of line) is in desc() * Converted to PHRED quality scores where applicable ('solexa') FASTQ variants This parser supports all variants of FASTQ, including Illumina v 1.0 and 1.3: variant note ----------------------------------------------------------- sanger original solexa Solexa, Inc.(2004), aka Illumina 1.0 illumina Illumina 1.3 The variant can be specified by passing by either passing the additional -variant parameter to the constructor: my $in = Bio::SeqIO->new(-format => 'fastq', -variant => 'solexa', -file => 'mysol.fq'); or by passing the format and variant together (Bio::SeqIO will now handle this and convert it accordingly to the proper argument): my $in = Bio::SeqIO->new(-format => 'fastq-solexa', -file => 'mysol.fq'); Variants can be converted back and forth from one another; however, due to the difference in scaling for solexa quality reads, converting from 'illumina' or 'sanger' FASTQ to solexa is not recommended. FEEDBACK
Mailing Lists User feedback is an integral part of the evolution of this and other Bioperl modules. Send your comments and suggestions preferably to one of the Bioperl mailing lists. Your participation is much appreciated. bioperl-l@bioperl.org - General discussion http://bioperl.org/wiki/Mailing_lists - About the mailing lists Support Please direct usage questions or support issues to the mailing list: bioperl-l@bioperl.org rather than to the module maintainer directly. Many experienced and reponsive experts will be able look at the problem and quickly address it. Please include a thorough description of the problem with code and data examples if at all possible. Reporting Bugs Report bugs to the Bioperl bug tracking system to help us keep track the bugs and their resolution. Bug reports can be submitted via the web: https://redmine.open-bio.org/projects/bioperl/ AUTHORS - Chris Fields (taken over from Tony Cox) Email: cjfields at bioperl dot org APPENDIX
The rest of the documentation details each of the object methods. Internal methods are usually preceded with a _ Bio::SeqIO interface methods next_seq Title : next_seq Usage : $seq = $stream->next_seq() Function : returns the next sequence in the stream Returns : Bio::Seq::Quality object Args : NONE Status : Stable write_seq Title : write_seq Usage : $stream->write_seq(@seq) Function : writes the $seq object into the stream Returns : 1 for success and 0 for error Args : Bio::Seq::Quality Note : This now conforms to SeqIO spec (module output is same format as next_seq) Status : Stable variant Title : variant Usage : $format = $obj->variant(); Function: Get and set method for the quality sequence variant. This is important for indicating the encoding/decoding to be used for quality data. Current values accepted are: 'sanger' (orginal FASTQ) ASCII encoding from 33-126, PHRED quality score from 0 to 93 'solexa' (aka illumina1.0) ASCII encoding from 59-104, SOLEXA quality score from -5 to 40 'illumina' (aka illumina1.3) ASCII encoding from 64-104, PHRED quality score from 0 to 40 (Derived from the MAQ website): For 'solexa', scores are converted to PHRED qual scores using: $Q = 10 * log(1 + 10 ** (ord($sq) - 64) / 10.0)) / log(10) Returns : string Args : new value, string Plugin-specific methods next_dataset Title : next_dataset Usage : $obj->next_dataset Function : returns a hash reference containing the parsed data Returns : hash reference Args : none Status : Stable write_fastq Title : write_fastq Usage : $stream->write_fastq(@seq) Function: writes the $seq object into the stream Returns : 1 for success and 0 for error Args : Bio::Seq::Quality object Status : Deprecated (delegates to write_seq) write_fasta Title : write_fasta Usage : $stream->write_fasta(@seq) Function: writes the $seq object into the stream Returns : 1 for success and 0 for error Args : Bio::Seq object Note : This method does not currently delegate to Bio::SeqIO::fasta (maybe it should?). Not sure whether we should keep this as a convenience method. Status : Unstable write_qual Title : write_qual Usage : $stream->write_qual(@seq) Function: writes the $seq object into the stream Returns : 1 for success and 0 for error Args : Bio::Seq::Quality object Note : This method does not currently delegate to Bio::SeqIO::qual (maybe it should?). Not sure whether we should keep this as a convenience method. Status : Unstable validate Title : validate Usage : $obj->validate(0) Function : flag for format/qual range validation - default is 1, validate Returns : Bool (0/1) Args : Bool (0/1) Status : Stable (may be moved to interface) quality_header Title : quality_header Usage : $obj->quality_header Function : flag for printing quality header - default is 0, no header Returns : Bool (0/1) Args : Bool (0/1) Status : Unstable (name may change dep. on feedback) perl v5.14.2 2012-03-02 Bio::SeqIO::fastq(3pm)
All times are GMT -4. The time now is 08:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy