Sponsored Content
Top Forums Shell Programming and Scripting Merge CSV files and create a column with the filename from the original file Post 302520900 by fransanchezoria on Monday 9th of May 2011 01:52:33 PM
Old 05-09-2011
Thank you Klashxx!!

That was a step in the right direction. Actually if I run the command with the example files as I posted in the example it works without any problems. I must have oversimplified my data because when I run it on the real thing the result file is huge, 2,3 GB although the source file are less than 100 Kb altogether. I tried to open with openoffice and in the preview window before importing I can see that there is a column with the file name which is great, then some stuff gets scrambled and repeated.

Here is a sample code of one file:

Code:
"Innography URL",Assignee,"Publication Number","Publication Country","Publication Date",Source,Title,Abstract,"Application Number",Citations,"Est. Expiration Date","Family ID","File Date","First Claim","All Claims",Inventors,"First IP Classification","All IPC Classifications","Kind Code","Priority Date","Normalized Assignee","Number of Claims","Number of Backward References","Number of Forward References","Original Assignee",Strength,"Ultimate Parent","US Classification"
"=Hyperlink(""https://app.innography.com/patents/14530190"",""Innography Link"")","Biora Ab",CA2226570,CA,1997-01-30,"CA Patents","Enamel matrix related polypeptide","The invention relates to novel nucleic acid fragments encoding polypeptides which are capable of mediating contact between enamel and cell surface. The invention also relates to expression vectors containing the nucleic acid fragments according to the invention for production of the protein organisms containing said expression vector methods for producing the polypeptide compositions comprising the polypeptides antibodies or antibody fragments recognizing the polypeptides and methods for treating various hard tissue diseases or disorders.",CA22265," ",2006-12-18,27221209,1996-06-26,,,"Wurtz, Tilmann  | Hammarstr, M Lars | Slaby, Ivan  | Cerny, Radim  | Fong, Cheng Dan",C12N01500900,"C12N 15/09|A61K 31/00|A61K 38/00|A61P 1/00|A61P 1/02|C07K 14/435|C07K 14/47|C07K 14/78|C12N 1/15|C12N 1/19|C12N 1/21|C12N 5/10|C12P 21/02",A1,1995-07-13,"Biora Ab",0,0,0,"BIORA AB","0th-10th Percentile",,435317100

As you can see one column contains an "abstract", the file may have all sort of special characters, could that have been the cause why it didnt work?

Thank you again for your help, really appreciated!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

merge two two txt files into one file based on one column

Hi, I have file1.txt and file2.txt and would like to create file3.txt based on one column in UNIX Eg: file1.txt 17328756,0000786623.pdf,0000786623 20115537,0000793892.pdf,0000793892 file2.txt 12521_74_4.zip,0000786623.pdf 12521_15_5.zip,0000793892.pdf Desired Output ... (5 Replies)
Discussion started by: techmoris
5 Replies

2. Shell Programming and Scripting

Merging files to create CSV file

Hi, I have different files of the same type, as: Time: 100 snr: 88 perf: 10 other: 222 Each of these files are created periodically. What I need to do is to merge all of them into one but having the following form: (2 Replies)
Discussion started by: Ravendark
2 Replies

3. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

4. Shell Programming and Scripting

How to create a CSV File by reading fields from separate files

SHELL SCRIPT Hi, I have 3 separate files within a folder. Every File contains data in a single column like File1 contains data mayank sushant dheeraj File2 contains DSA_AT MG_AT FLAT_09 File3 contains data 123123 232323 (2 Replies)
Discussion started by: mayanksargoch
2 Replies

5. Shell Programming and Scripting

create new column for filename

Hi, I created a list with 2 columns. Each line is from a different file. I am getting these with a loop in Perl. I would like to add a 3rd column with the name of the file that the line is coming from. I usually use pr to print the filename but this is not working here ... I was wondering if... (5 Replies)
Discussion started by: danieladna
5 Replies

6. UNIX for Dummies Questions & Answers

How to create a .csv file from 2 different .txt files?

Hi, I need to create a .csv file from information that i have in two different tab delimited .txt file. I just want to select some of the columns of each .txt file and paste them into a .cvs file. My files look like: File 1 transcript_id Seq. Description Seq. Length ... (2 Replies)
Discussion started by: alisrpp
2 Replies

7. Shell Programming and Scripting

Merge different files into the original file

Hello Below is my requirement I have 3 files A1.txt , A2.txt and A3.txt . A2 is dynamically generating file I want the merge of A1,A2 and A3 in A2.txt Could you please help? (3 Replies)
Discussion started by: Pratik4891
3 Replies

8. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

9. UNIX for Dummies Questions & Answers

Merge two csv files using column name

Hi all, I have two separate csv files(comma delimited) file 1 and file 2. File 1 contains PAN,NAME,Salary AAAAA5467D,Raj,50000 AAFAC5467D,Ram,60000 BDCFA5677D,Kumar,90000 File 2 contains PAN,NAME,Dept,Salary ASDFG6756T,Karthik,ABC,450000 QWERT8765Y,JAX,CDR,780000... (5 Replies)
Discussion started by: Nivas
5 Replies

10. Shell Programming and Scripting

I am trying to merge all csv files from source path into 1 file

I am trying to merge all csv files from source path into one single csv file in target. but getting error message: hadoop fs -cat /user/hive/warehouse/stage.db/PK_CLOUD_CHARGE/TCH-charge_*.csv > /user/hive/warehouse/stage.db/PK_CLOUD_CHARGE/final/TCH_pb_charge.csv getting error message:... (0 Replies)
Discussion started by: cplusplus1
0 Replies
DIALIGN(1)							    User Manual 							DIALIGN(1)

NAME
dialign2-2 - Multiple alignment program using the segment-to-segment approach SYNOPSIS
dialign2-2 [options] [seq_file] seq_file is the name of the input sequence file; this must be a multiple FASTA file (all sequences in one file). DESCRIPTION
dialign2-2 is a program that constructs alignments from gapfree pairs of similar segments of the sequences. If (possibly) coding nucleic acid sequences are to be aligned, DIALIGN optionally translates the compared `nucleic acid segments' to `peptide segments' according to the genetic code -- without presupposing any of the three possible reading frames, so all combinations of reading frames get checked for significant similarity. By default, DIALIGN creates a single file containing o An alignment of the input sequences in DIALIGN format. o The same alignment in FASTA format. o A sequence tree in PHYLIP format. This tree is constructed by applying the UPGMA clustering method to the DIALIGN similarity scores. It roughly reflects the different degrees of similarity among sequences. For detailed phylogenetic analysis, we recommend the usual methods for phylogenetic reconstruction. The format of the output files is documented in /usr/share/doc/dialign/USER_GUIDE.gz. The FASTA, CLUSTALW and MSF output formats are optionally available (see OPTIONS). OPTIONS
-afc Creates additional output file "*.afc" containing data of all fragments considered for alignment. WARNING: this file can be HUGE! -afc_v Like "-afc" but verbose: fragments are explicitly printed. WARNING: this file can be EVEN BIGGER! -anc Anchored alignment. Requires a file seq_file.anc containing anchor points. -cs If segments are translated, not only the `Watson strand' but also the `Crick strand' is looked at. -cw Additional output file in CLUSTAL W format. -ds `DNA alignment speed up'. Non-translated nucleic acid fragments are taken into account only if they start with at least two matches. Speeds up DNA alignment at the expense of sensitivity. -fa Additional output file in FASTA format. -ff Creates file *.frg containing information about all fragments that are part of the respective optimal pairwise alignmnets plus information about consistency in the multiple alignment. -fn out_file Output files are named out_file.extension. -fop Creates file *.fop containing coordinates of all fragments that are part of the respective pairwise alignments. -fsm Creates file *.fsm containing coordinates of all fragments that are part of the final alignment -iw Overlap weights switched off (by default, overlap weights are used if up to 35 sequences are aligned). This option speeds up the alignment but may lead to reduced alignment quality. -lgs `Long genomic sequences' - combines the following options: -ma, -thr 2, -lmax 30, -smin 8, -nta, -ff, -fop, -ff, -cs, -ds, -pst. -lgs_t Like "-lgs" but with all segment pairs assessed at the peptide level (rather than 'mixed alignments' as with the "-lgs" option). Therefore faster than -lgs but not very sensitive for non-coding regions. -lmax x Maximum fragment length = x (default: x = 40 or x = 120 for `translated' fragments). Shorter x speeds up the program but may affect alignment quality. -lo (Long Output) Additional file *.log with information abut fragments selected for pairwise alignment and about consistency in multi-alignment proceedure. -ma `mixed alignments' consisting of P-fragments and N-fragments if nucleic acid sequences are aligned. -mask Residues not belonging to selected fragments are replaced by `*' characters in output alignment (rather than being printed in lower-case characters) -mat Creates file *mat with substitution counts derived from the fragments that have been selected for alignment. -mat_thr t Like "-mat" but only fragments with weight score > t are considered. -max_link "Maximum linkage" clustering used to construct sequence tree (instead of UPGMA). -min_link "Minimum linkage" clustering used. -mot "Motif" option. -msf Separate output file in MSF format. -n Input sequences are nucleic acid sequences. No translation of fragments. -nt Input sequences are nucleic acid sequences and `nucleic acid segments' are translated to `peptide segments'. -nta `No textual alignment'. Textual alignment suppressed. This option makes sense if other output files are of intrest -- e.g. the fragment files created with -ff, -fop, -fsm or -lo. -o Fast version, resulting alignments may be slightly different. -ow Overlap weights enforced (By default, overlap weights are used only if up to 35 sequences are aligned since calculating overlap weights is time consuming). Warning: overlap weights generally improve alignment quality but the running time increases in the order O(n^4) with the number of sequences. This is why, by default, overlap weights are used only for sequence sets with < 36 sequences. -pst "Print status". Creates and updates a file *.sta with information about the current status of the program run. This option is recommended if large data sets are aligned since it allows the user to estimate the remaining running time. -smin x Minimum similarity value for first residue pair (or codon pair) in fragments. Speeds up protein alignment or alignment of translated DNA fragments at the expense of sensitivity. -stars x Maximum number of `*' characters indicating degree of local similarity among sequences. By default, no stars are used but numbers between 0 and 9, instead. -stdo Results written to standard output. -ta Standard textual alignment printed (overrides suppression of textual alignments in special options, e.g. -lgs). -thr x Threshold T = x. -xfr "Exclude fragments". List of fragments can be specified that are NOT considered for pairwise alignment. General remark: If contradictory options are used, subsequent options override previous ones, e.g.: dialign2-2 -nt -n seq_file runs the program with the "-n" option (no translation!), while dialign2-2 -n -nt seq_file runs it with the "-nt" option (translation!). SEE ALSO
The full documentation is in /usr/share/doc/dialign/. The website of dialign: http://dialign.gobics.de/ DIALIGN2 has been re-implemented in dialign-tx(1). See http://dialign-tx.gobics.de/ ENVIRONMENT VARIABLES
You can create an environment variable `DIALIGN2_DIR' pointing to a directory where the substitution matrices are (see FILES). When installed from the Debian package, it is not necessary to set this environnement variable to run DIALIGN. FILES
DIALIGN2 needs the files tp400_dna, tp400_prot, tp400_trans and BLOSUM. When DIALIGN is installed from the Debian package, they are stored in /usr/share/dialign/. DIALIGN 2 uses the BLOSUM62 amino acid substitution matrix. In the current version, it is NOT possible to replace BLOSUM62 by other similarity matrices. REFERENCE
B. Morgenstern (1999). DIALIGN 2: improvement of the segment-to-segment approach to multiple sequence alignment. Bioinformatics 15, 211 - 218. Public research assisted by DIALIGN should cite this article. AUTHORS
Burkhard Morgenstern <bmorgen@gwdg.de> Author of DIALIGN Said Abdeddaim Author of DIALIGN Charles Plessy <plessy@debian.org> Wrote this manpage COPYRIGHT
DIALIGN was written by Burkhard Morgenstern and Said Abdeddaim at University of Bielefeld (FSPM and International Graduate School in Bioinformatics and Genome Research), GSF (ISG, IBB, MIPS/IBI), North Carolina State University, Universite de Rouen, MPI fuer Biochemie (Martinsried), University of Goettingen, Institute of Microbiology and Genetics. This manual page was adapted from the DIALIGN manual by Charles Plessy <plessy@debian.org> for the Debian system (but may be used by others). Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Lesser General Public License, Version 2.1 any later version published by the Free Software Foundation. On Debian systems, the complete text of the GNU Lesser General Public License can be found in /usr/share/common-licenses/LGPL. Copyright (C) 1999 Burkhard Morgenstern (for DIALIGN) Copyright (C) 2006, 2007, 2008 Charles Plessy (for this manpage) dialign2-2 2.2.1 avril 9, 2006 DIALIGN(1)
All times are GMT -4. The time now is 10:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy