Sponsored Content
Top Forums Shell Programming and Scripting Merge CSV files and create a column with the filename from the original file Post 302520900 by fransanchezoria on Monday 9th of May 2011 01:52:33 PM
Old 05-09-2011
Thank you Klashxx!!

That was a step in the right direction. Actually if I run the command with the example files as I posted in the example it works without any problems. I must have oversimplified my data because when I run it on the real thing the result file is huge, 2,3 GB although the source file are less than 100 Kb altogether. I tried to open with openoffice and in the preview window before importing I can see that there is a column with the file name which is great, then some stuff gets scrambled and repeated.

Here is a sample code of one file:

Code:
"Innography URL",Assignee,"Publication Number","Publication Country","Publication Date",Source,Title,Abstract,"Application Number",Citations,"Est. Expiration Date","Family ID","File Date","First Claim","All Claims",Inventors,"First IP Classification","All IPC Classifications","Kind Code","Priority Date","Normalized Assignee","Number of Claims","Number of Backward References","Number of Forward References","Original Assignee",Strength,"Ultimate Parent","US Classification"
"=Hyperlink(""https://app.innography.com/patents/14530190"",""Innography Link"")","Biora Ab",CA2226570,CA,1997-01-30,"CA Patents","Enamel matrix related polypeptide","The invention relates to novel nucleic acid fragments encoding polypeptides which are capable of mediating contact between enamel and cell surface. The invention also relates to expression vectors containing the nucleic acid fragments according to the invention for production of the protein organisms containing said expression vector methods for producing the polypeptide compositions comprising the polypeptides antibodies or antibody fragments recognizing the polypeptides and methods for treating various hard tissue diseases or disorders.",CA22265," ",2006-12-18,27221209,1996-06-26,,,"Wurtz, Tilmann  | Hammarstr, M Lars | Slaby, Ivan  | Cerny, Radim  | Fong, Cheng Dan",C12N01500900,"C12N 15/09|A61K 31/00|A61K 38/00|A61P 1/00|A61P 1/02|C07K 14/435|C07K 14/47|C07K 14/78|C12N 1/15|C12N 1/19|C12N 1/21|C12N 5/10|C12P 21/02",A1,1995-07-13,"Biora Ab",0,0,0,"BIORA AB","0th-10th Percentile",,435317100

As you can see one column contains an "abstract", the file may have all sort of special characters, could that have been the cause why it didnt work?

Thank you again for your help, really appreciated!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

merge two two txt files into one file based on one column

Hi, I have file1.txt and file2.txt and would like to create file3.txt based on one column in UNIX Eg: file1.txt 17328756,0000786623.pdf,0000786623 20115537,0000793892.pdf,0000793892 file2.txt 12521_74_4.zip,0000786623.pdf 12521_15_5.zip,0000793892.pdf Desired Output ... (5 Replies)
Discussion started by: techmoris
5 Replies

2. Shell Programming and Scripting

Merging files to create CSV file

Hi, I have different files of the same type, as: Time: 100 snr: 88 perf: 10 other: 222 Each of these files are created periodically. What I need to do is to merge all of them into one but having the following form: (2 Replies)
Discussion started by: Ravendark
2 Replies

3. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

4. Shell Programming and Scripting

How to create a CSV File by reading fields from separate files

SHELL SCRIPT Hi, I have 3 separate files within a folder. Every File contains data in a single column like File1 contains data mayank sushant dheeraj File2 contains DSA_AT MG_AT FLAT_09 File3 contains data 123123 232323 (2 Replies)
Discussion started by: mayanksargoch
2 Replies

5. Shell Programming and Scripting

create new column for filename

Hi, I created a list with 2 columns. Each line is from a different file. I am getting these with a loop in Perl. I would like to add a 3rd column with the name of the file that the line is coming from. I usually use pr to print the filename but this is not working here ... I was wondering if... (5 Replies)
Discussion started by: danieladna
5 Replies

6. UNIX for Dummies Questions & Answers

How to create a .csv file from 2 different .txt files?

Hi, I need to create a .csv file from information that i have in two different tab delimited .txt file. I just want to select some of the columns of each .txt file and paste them into a .cvs file. My files look like: File 1 transcript_id Seq. Description Seq. Length ... (2 Replies)
Discussion started by: alisrpp
2 Replies

7. Shell Programming and Scripting

Merge different files into the original file

Hello Below is my requirement I have 3 files A1.txt , A2.txt and A3.txt . A2 is dynamically generating file I want the merge of A1,A2 and A3 in A2.txt Could you please help? (3 Replies)
Discussion started by: Pratik4891
3 Replies

8. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

9. UNIX for Dummies Questions & Answers

Merge two csv files using column name

Hi all, I have two separate csv files(comma delimited) file 1 and file 2. File 1 contains PAN,NAME,Salary AAAAA5467D,Raj,50000 AAFAC5467D,Ram,60000 BDCFA5677D,Kumar,90000 File 2 contains PAN,NAME,Dept,Salary ASDFG6756T,Karthik,ABC,450000 QWERT8765Y,JAX,CDR,780000... (5 Replies)
Discussion started by: Nivas
5 Replies

10. Shell Programming and Scripting

I am trying to merge all csv files from source path into 1 file

I am trying to merge all csv files from source path into one single csv file in target. but getting error message: hadoop fs -cat /user/hive/warehouse/stage.db/PK_CLOUD_CHARGE/TCH-charge_*.csv > /user/hive/warehouse/stage.db/PK_CLOUD_CHARGE/final/TCH_pb_charge.csv getting error message:... (0 Replies)
Discussion started by: cplusplus1
0 Replies
Bio::Matrix::PSM::ProtMatrix(3pm)			User Contributed Perl Documentation			 Bio::Matrix::PSM::ProtMatrix(3pm)

NAME
Bio::Matrix::PSM::ProtMatrix - SiteMatrixI implementation, holds a position scoring matrix (or position weight matrix) with log-odds scoring information. SYNOPSIS
use Bio::Matrix::PSM::ProtMatrix; # Create from memory by supplying probability matrix hash both as strings or # arrays where the frequencies Hash entries of the form lN refer to an array # of position-specific log-odds scores for amino acid N. Hash entries of the # form pN represent the position-specific probability of finding amino acid N. my %param = ( 'id' => 'A. thaliana protein atp1', '-e_val' => $score, 'lS' => [ '-2', '3', '-3', '2', '-3', '1', '1', '3' ], 'lF' => [ '-1', '-4', '0', '-5', '0', '-5', '-4', '-4' ], 'lT' => [ '-1', '1', '0', '1', '-2', '-1', '0', '1' ], 'lN' => [ '-3', '-1', '-2', '3', '-5', '5', '-2', '0' ], 'lK' => [ '-2', '0', '-3', '2', '-3', '2', '-3', '-1' ], 'lY' => [ '-2', '-3', '-3', '-4', '-3', '-4', '-4', '-4' ], 'lE' => [ '-3', '4', '-3', '2', '-4', '-2', '-3', '2' ], 'lV' => [ '0', '-2', '1', '-4', '1', '-4', '-1', '-3' ], 'lQ' => [ '-1', '0', '-2', '3', '-4', '1', '-3', '0' ], 'lM' => [ '8', '-3', '8', '-3', '1', '-3', '-3', '-3' ], 'lC' => [ '-2', '-3', '-3', '-4', '-3', '-4', '-3', '-3' ], 'lL' => [ '1', '-3', '1', '-4', '3', '-4', '-2', '-4' ], 'lA' => [ '-2', '1', '-2', '0', '-2', '-2', '2', '2' ], 'lW' => [ '-2', '-4', '-3', '-5', '-4', '-5', '-5', '-5' ], 'lP' => [ '-3', '-2', '-4', '-3', '-1', '-3', '6', '-3' ], 'lH' => [ '-2', '-2', '-3', '-2', '-5', '-2', '-2', '-3' ], 'lD' => [ '-4', '-1', '-3', '1', '-3', '-1', '-3', '4' ], 'lR' => [ '-2', '-1', '-3', '0', '-4', '4', '-4', '-3' ], 'lI' => [ '0', '-3', '0', '-4', '6', '-4', '-2', '-2' ], 'lG' => [ '-4', '-2', '-4', '-2', '-5', '-3', '-1', '-2' ], 'pS' => [ '0', '33', '0', '16', '1', '12', '11', '25' ], 'pF' => [ '0', '0', '2', '0', '3', '0', '0', '0' ], 'pT' => [ '0', '8', '7', '10', '1', '2', '7', '8' ], 'pN' => [ '0', '0', '2', '13', '0', '36', '1', '4' ], 'pK' => [ '0', '5', '0', '13', '1', '15', '0', '2' ], 'pY' => [ '0', '0', '0', '0', '0', '0', '0', '0' ], 'pE' => [ '0', '41', '1', '12', '0', '0', '0', '15' ], 'pV' => [ '0', '3', '9', '0', '2', '0', '3', '1' ], 'pQ' => [ '0', '0', '0', '15', '0', '4', '0', '3' ], 'pM' => [ '100', '0', '66', '0', '2', '0', '0', '0' ], 'pC' => [ '0', '0', '0', '0', '0', '0', '0', '0' ], 'pL' => [ '0', '0', '8', '0', '25', '0', '4', '0' ], 'pA' => [ '0', '10', '1', '9', '2', '0', '22', '16' ], 'pW' => [ '0', '0', '0', '0', '0', '0', '0', '0' ], 'pP' => [ '0', '0', '0', '0', '3', '1', '45', '0' ], 'pH' => [ '0', '0', '0', '0', '0', '0', '1', '0' ], 'pD' => [ '0', '0', '1', '7', '2', '2', '0', '22' ], 'pR' => [ '0', '0', '0', '3', '0', '27', '0', '0' ], 'pI' => [ '0', '0', '3', '0', '59', '1', '2', '3' ], 'pG' => [ '0', '0', '0', '1', '0', '0', '4', '1' ], ); my $matrix = Bio::Matrix::PSM::ProtMatrix( %param ); my $site = Bio::Matrix::PSM::ProtMatrix->new(%param); # Or get it from a file: use Bio::Matrix::PSM::IO; my $psmIO = Bio::Matrix::PSM::IO->new(-file => $file, -format => 'psi-blast'); while (my $psm = $psmIO->next_psm) { #Now we have a Bio::Matrix::PSM::Psm object, # see Bio::Matrix::PSM::PsmI for details #This is a Bio::Matrix::PSM::ProtMatrix object now my $matrix = $psm->matrix; } # Get a simple consensus, where alphabet is: # {A, R, N, D, C, Q, E, G, H, I, L, K, M, F, P, S, T, W, Y, V,} # choosing the highest probability or N if prob is too low my $consensus = $site->consensus; # Retrieving and using regular expressions: my $regexp = $site->regexp; my $count = grep($regexp,$seq); my $count = ($seq=~ s/$regexp/$1/eg); print "Motif $mid is present $count times in this sequence "; DESCRIPTION
ProtMatrix is designed to provide some basic methods when working with position scoring (weight) matrices related to protein sequences. A protein PSM consists of 20 vectors with 20 frequencies (one per amino acid per position). This is the minimum information you should provide to construct a PSM object. The vectors can be provided as strings with frequencies where the frequency is {0..a} and a=1. This is the way MEME compressed representation of a matrix and it is quite useful when working with relational DB. If arrays are provided as an input (references to arrays actually) they can be any number, real or integer (frequency or count). When creating the object the constructor will check for positions that equal 0. If such is found it will increase the count for all positions by one and recalculate the frequency. Potential bug - if you are using frequencies and one of the positions is 0 it will change significantly. However, you should never have frequency that equals 0. Throws an exception if: You mix as an input array and string (for example A matrix is given as array, C - as string). The position vector is (0,0,0,0). One of the probability vectors is shorter than the rest. Summary of the methods I use most frequently (details bellow): iupac - return IUPAC compliant consensus as a string score - Returns the score as a real number IC - information content. Returns a real number id - identifier. Returns a string accession - accession number. Returns a string next_pos - return the sequence probably for each letter, IUPAC symbol, IUPAC probability and simple sequence consenus letter for this position. Rewind at the end. Returns a hash. pos - current position get/set. Returns an integer. regexp - construct a regular expression based on IUPAC consensus. For example AGWV will be [Aa][Gg][AaTt][AaCcGg] width - site width get_string - gets the probability vector for a single base as a string. get_array - gets the probability vector for a single base as an array. get_logs_array - gets the log-odds vector for a single base as an array. New methods, which might be of interest to anyone who wants to store PSM in a relational database without creating an entry for each position is the ability to compress the PSM vector into a string with losing usually less than 1% of the data. this can be done with: my $str=$matrix->get_compressed_freq('A'); or my $str=$matrix->get_compressed_logs('A'); Loading from a database should be done with new, but is not yet implemented. However you can still uncompress such string with: my @arr=Bio::Matrix::PSM::_uncompress_string ($str,1,1); for PSM or my @arr=Bio::Matrix::PSM::_uncompress_string ($str,1000,2); for log odds FEEDBACK
Mailing Lists User feedback is an integral part of the evolution of this and other Bioperl modules. Send your comments and suggestions preferably to one of the Bioperl mailing lists. Your participation is much appreciated. bioperl-l@bioperl.org - General discussion http://bioperl.org/wiki/Mailing_lists - About the mailing lists Support Please direct usage questions or support issues to the mailing list: bioperl-l@bioperl.org rather than to the module maintainer directly. Many experienced and reponsive experts will be able look at the problem and quickly address it. Please include a thorough description of the problem with code and data examples if at all possible. Reporting Bugs Report bugs to the Bioperl bug tracking system to help us keep track the bugs and their resolution. Bug reports can be submitted via the web: https://redmine.open-bio.org/projects/bioperl/ AUTHOR - James Thompson Email tex@biosysadmin.com APPENDIX
new Title : new Usage : my $site = Bio::Matrix::PSM::ProtMatrix->new( %probs, %logs, -IC => $ic, -e_val => $score, -id => $mid -model => \%model ); Function : Creates a new Bio::Matrix::PSM::ProtMatrix object from memory Throws : If inconsistent data for all vectors (all 20 amino acids) is provided, if you mix input types (string vs array) or if a position freq is 0. Example : Returns : Bio::Matrix::PSM::ProtMatrix object Args : Hash references to log-odds scores and probabilities for position-specific scoring info, e-value (optional), information content (optional), id (optional), model for background distribution of proteins (optional). alphabet Title : Returns an array (or array reference if desired) to the alphabet Usage : Function : Returns an array (or array reference) containing all of the allowable characters for this matrix. Throws : Example : Returns : Array or arrary reference. Args : _calculate_consensus Title : _calculate_consensus Usage : Function : Calculates the consensus sequence for this matrix. Throws : Example : Returns : Args : next_pos Title : next_pos Usage : Function : Retrives the next position features: frequencies for all 20 amino acids, log-odds scores for all 20 amino acids at this position, the main (consensus) letter at this position, the probability for the consensus letter to occur at this position and the relative current position as an integer. Throws : Example : Returns : hash (or hash reference) (pA,pR,pN,pD,...,logA,logR,logN,logD,aa,prob,rel) - pN entries represent the probability for amino acid N to be at this position - logN entries represent the log-odds score for having amino acid N at this position - aa is the consensus amino acid - prob is the probability for the consensus amino acid to be at this position - rel is the relative index of the current position (integer) Args : none curpos Title : curpos Usage : Function : Gets/sets the current position. Throws : Example : Returns : Current position (integer). Args : New position (integer). e_val Title : e_val Usage : Function : Gets/sets the e-value Throws : Example : Returns : Args : real number IC Title : IC Usage : Function : Position-specific information content. Throws : Example : Returns : Information content for current position. Args : Information content for current position. accession_number Title : accession_number Usage : Function: accession number, this will be unique id for the ProtMatrix object as well for any other object, inheriting from ProtMatrix. Throws : Example : Returns : New accession number (string) Args : Accession number (string) consensus Title : consensus Usage : Function : Returns the consensus sequence for this PSM. Throws : if supplied with thresold outisde 5..10 range Example : Returns : string Args : (optional) threshold value 5 to 10 (corresponds to 50-100% at each position get_string Title : get_string Usage : Function: Returns given probability vector as a string. Useful if you want to store things in a rel database, where arrays are not first choice Throws : If the argument is outside {A,C,G,T} Example : Returns : string Args : character {A,C,G,T} width Title : width Usage : Function : Returns the length of the site Throws : Example : Returns : number Args : get_array Title : get_array Usage : Function : Returns an array with frequencies for a specified amino acid. Throws : Example : Returns : Array representing frequencies for specified amino acid. Args : Single amino acid (character). get_logs_array Title : get_logs_array Usage : Function : Returns an array with log_odds for a specified base Throws : Example : Returns : Array representing log-odds scores for specified amino acid. Args : Single amino acid (character). id Title : id Usage : Function : Gets/sets the site id Throws : Example : Returns : string Args : string regexp Title : regexp Usage : Function : Returns a case-insensitive regular expression which matches the IUPAC convention. X's in consensus sequence will match anything. Throws : Example : Returns : string Args : Threshold for calculating consensus sequence (number in range 0-100 representing a percentage). Threshold defaults to 20. regexp_array Title : regexp_array Usage : Function : Returns an array of position-specific regular expressions. X's in consensus sequence will match anything. Throws : Example : Returns : Array of position-specific regular expressions. Args : Threshold for calculating consensus sequence (number in range 0-100 representing a percentage). Threshold defaults to 20. Notes : Simply calls regexp method in list context. _compress_array Title : _compress_array Usage : Function : Will compress an array of real signed numbers to a string (ie vector of bytes) -127 to +127 for bi-directional(signed) and 0..255 for unsigned ; Throws : Example : Internal stuff Returns : String Args : array reference, followed by max value and direction (optional, defaults to 1), direction of 1 is unsigned, anything else is signed. _uncompress_string Title : _uncompress_string Usage : Function : Will uncompress a string (vector of bytes) to create an array of real signed numbers (opposite to_compress_array) Throws : Example : Internal stuff Returns : string, followed by max value and direction (optional, defaults to 1), direction of 1 is unsigned, anything else is signed. Args : array get_compressed_freq Title : get_compressed_freq Usage : Function: A method to provide a compressed frequency vector. It uses one byte to code the frequence for one of the probability vectors for one position. Useful for relational database. Improvment of the previous 0..a coding. Throws : Example : my $strA=$self->get_compressed_freq('A'); Returns : String Args : char sequence_match_weight Title : sequence_match_weight Usage : Function : This method will calculate the score of a match, based on the PSM if such is associated with the matrix object. Returns undef if no PSM data is available. Throws : if the length of the sequence is different from the matrix width Example : my $score=$matrix->sequence_match_weight('ACGGATAG'); Returns : Floating point Args : string _to_IUPAC Title : _to_IUPAC Usage : Function: Converts a single position to IUPAC compliant symbol and returns its probability. Currently returns the most likely amino acid/probability combination. Throws : Example : Returns : char, real number representing an amino acid and a probability. Args : real numbers for all 20 amino acids (ordered by alphabet contained in $self->{_alphabet}, minimum probability threshold. _to_cons Title : _to_cons Usage : Function: Converts a single position to simple consensus character and returns its probability. Currently just calls the _to_IUPAC subroutine. Throws : Example : Returns : char, real number Args : real numbers for A,C,G,T (positional) get_all_vectors Title : get_all_vectors Usage : Function : returns all possible sequence vectors to satisfy the PFM under a given threshold Throws : If threshold outside of 0..1 (no sense to do that) Example : my @vectors = $self->get_all_vectors(4); Returns : Array of strings Args : (optional) floating perl v5.14.2 2012-03-02 Bio::Matrix::PSM::ProtMatrix(3pm)
All times are GMT -4. The time now is 07:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy