Join multiple files by column with awk Post: 302427249

Sponsored Content

Top Forums Shell Programming and Scripting Join multiple files by column with awk Post 302427249 by macsx82 on Friday 4th of June 2010 10:22:26 AM

06-04-2010

Registered User

Hi Ikki!
Thanks for your reply! I use awk bacause I'm more familiar with its sintax, i've just used sed a couple of time!
My files contain datas from genetic chips, and each file belongs to a person.
Each file has ~360000 lines.
You're right, I have more than one file shorter than the other..at the moment there are 95 shorter files. But they could increase, and yes, in shorter files it's possible that a line starting with cnvi0000004 is followed by a line starting with cnvi0000006 or cnvi0000008...it depends on how many records are missing for this person, but all input file are sorted by the first column.
As I said I've written an R-script that works, but it is extremely slow. In this script I compare a list of "complete" names with another and see if there are differences. Once i found elements that aren't in the short list, i add them in this list in order to have elements with same length. In this way I can merge all column and insert tabs instead of missing datas. I post the R-code:

Code:

#define file path
files_path="/home/###/###/people/"

#read all file names in the directory and save in a vector
only_files <- dir(path=files_path, pattern = "*.in") 
files = paste(files_path,only_files, sep="")

#load files to create the "complete list" I need the first column that contain the name of the record
tot_file <- read.table(files[1], sep="\t", header=TRUE)[c(1,2,3)]
tot_file_noname <- cbind(Chr=tot_file$Chr, Position=tot_file$Position)


for (i in 1:length(files)) { 
#
        xx_file <- read.table(files[i], sep="\t", header=TRUE)[c(1,3,4)]
        xx_file_noname <- cbind(xx_file$Position, xx_file$Log.R.Ratio)

#now I read each file and if i find some mismatch from the complete list 
#I add them in the current xx_file object with value "NaN"

    if (length(xx_file$name) != length(tot_file$name)){
                print('different!')
                mismatch=NULL

                match <- tot_file$name %in% xx_file$name
                                    
                for(i in 1:length(match)){ if (match[i]== FALSE){ mismatch = c(mismatch,i)}}

                missing_snp = NULL
# add missing values
                for (i in mismatch){
                    missing <- data.frame(Position = tot_file[i,]$Position, Log.R.Ratio="NaN")
                    missing_snp <- rbind(missing_snp, missing)
                }

                    xx_file_noname <- rbind(xx_file[,c(2,3)], missing_snp)
    }else{
        print('equals!')        
    }    

    tot_file_noname = cbind(tot_file_noname, xx_file_noname[,2])
}

# write the "big" file
write.table(tot_file_noname, file = "gigante.dat", append = FALSE, quote = FALSE, sep = "\t", eol = "\n", na = "NaN", dec =".", row.names = FALSE, col.names =TRUE)

Now I'm trying to port this in a shell script to have a faster response. My purpose was to avoid files preprocessing if i can, because of the large amount of data stored in each file, if possible.
I tried your command also, and it is running, the only problem is to add manually 1664 columns for the cut command, but I think I can work on It!
Hope I have been clear enough, and greatly appreciate your help!

macsx82

View Public Profile for macsx82

Find all posts by macsx82

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Join 2 files with multiple columns: awk/grep/join?

Hello, My apologies if this has been posted elsewhere, I have had a look at several threads but I am still confused how to use these functions. I have two files, each with 5 columns: File A: (tab-delimited) PDB CHAIN Start End Fragment 1avq A 171 176 awyfan 1avq A 172 177 wyfany 1c7k A 2 7...

2. Shell Programming and Scripting

Join multiple files based on 1 common column

I have n files (for ex:64 files) with one similar column. Is it possible to combine them all based on that column ? file1 ax100 20 30 40 ax200 22 33 44 file2 ax100 10 20 40 ax200 12 13 44 file2 ax100 0 0 4 ax200 2 3 4

3. Shell Programming and Scripting

Join and awk max column

Hi Friends, I have a file1 with 3400 records that are tab separated and I have a file2 with 6220 records. I want to merge both these files. I tried using join file1 and file2 after sorting. But, the records should be (3400*6220 = 21148000). Instead, I get only around 11133567. Is there anything...

4. Shell Programming and Scripting

Awk - join multiple files

Is it possible to join all the files with input1 based on 1st column? input1 a b c d e f input2 a b input3 a e input4 c

5. UNIX for Dummies Questions & Answers

How to use the the join command to join multiple files by a common column

Hi, I have 20 tab delimited text files that have a common column (column 1). The files are named GSM1.txt through GSM20.txt. Each file has 3 columns (2 other columns in addition to the first common column). I want to write a script to join the files by the first common column so that in the...

6. Shell Programming and Scripting

Awk: Multiple Replace In Column From Two Different Files

Master_1.txt 2372,MTS,AP 919821,Airtel,DL 0819,MTS,MUM 919849788001,Airtel,AP 1430,Aircel MP,20 405899143999999,MTS,KRL USSDLIKE,MTS,DEL Master_2.txt 919136,DL 9664,RAJ 919143,KOL 9888,PUN Input File:

7. UNIX for Dummies Questions & Answers

Join with awk different column

hi guys, i need help I need to join file2 to file1 when column 3 in my file1 and column 1 in my file2 in the same string file1 AA|RR|ESKIM RE|DD|RED WE|WW|SUPSS file2 ESKIM|ES SUPSS|SS Output AA|RR|ESKIM|ES RE|DD|RED| WE|WW|SUPSS|SS

8. Shell Programming and Scripting

Join 2nd column of multiple files

Dear All, I have many files formatted like this: file1.txt: 1/2-SBSRNA4 18 A1BG 3 A1BG-AS1 6 A1CF 0 A2LD1 1 A2M 1160 file2.txt 1/2-SBSRNA4 53 A1BG 1 A1BG-AS1 7 A1CF 0 A2LD1 3 A2M 2780

9. Shell Programming and Scripting

Join columns across multiple lines in a Text based on common column using BASH

10. Shell Programming and Scripting

Join, merge, fill NULL the void columns of multiples files like sql "LEFT JOIN" by using awk

Hello, This post is already here but want to do this with another way Merge multiples files with multiples duplicates keys by filling "NULL" the void columns for anothers joinning files file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: 1|123|jojo 1|NULL|bibi...

LEARN ABOUT DEBIAN

xgfmerge

XGFMERGE(1)						      General Commands Manual						       XGFMERGE(1)

NAME

       xgfmerge - merges two Xgridfit program files

SYNOPSIS

       xgfmerge [options] file-a file-b [...]

DESCRIPTION

       Xgfmerge  merges  two or more Xgridfit program files, where file-a is a program generated automatically from an existing font using TTX and
       ttx2xgf, and file-b and any other files in the list contain other programming for the font.  Xgfmerge outputs  the  merged  programming	to
       stdout.	To capture the output in a file, use this syntax:

	    $xgfmerge -o merged-file.xgf file-a.xgf file-b.xgf

       Note: To merge Xgridfit instructions with those already in a font, the best method is now to run Xgridfit in merge-mode (option -m).

OPTIONS

       -c     Look  for  a  <default  type="compile-globals"/>	element  in  each  file after the first in the list. If value="no" then ignore all
	      <default>, <control-value>, <function>, <macro> and <pre-program> elements in the file.

       -h     Display a help message.

       -n     Merge <no-compile> elements from all files. <glyph> elements with duplicate "ps-name" attributes are ignored.

       -o file
	      The file to write the output to. If this option is not used, output is written to stdout.

       -p     When a <pre-program> element in a file other than file-a is available, use it instead of the one from file-a. If this option is  not
	      present,	xgfmerge  merges  the programming in the <pre-program> of file-a with that in the <pre-program> of whichever file is being
	      merged at the moment.  The result is unlikely to be good if a single run of xgfmerge merges the contents	of  more  than	two  <pre-
	      program> elements.

       -s     Sort <glyph> elements in the output file into alphabetical order.

       -v     Verbose output: xgfmerge tells you what it is doing at each step.

       -x     Resolve  XIncludes  before  merging  all	files in the list except for file-a. Any XIncludes from file-a.xgf are stripped out of the
	      files before this operation is performed. If XInclude is used to bring global  elements  (<control-value>,  <function>,  etc.)  into
	      these files, it is probably a good idea to use the -c option as well.

FILES

       /usr/share/xml/xgridfit/util/merge.xsl
	      An XSLT script that performs the merge.

       /usr/share/xml/xgridfit/util/xinclude.xsl
	      An XSLT script that strips out unwanted XIncludes.

       /usr/share/xml/xgridfit/util/sort-glyphs.xsl
	      An XSLT script that sorts glyph elements in a file.

       /usr/share/xml/xgridfit/utils/add-blanks.sed
	      Does some formatting of the output.

SEE ALSO

       xgridfit(1), ttx(1), ttx2xgf(1).

AUTHOR

       Xgfmerge was written by Peter Baker <psb6m@virginia.edu>.

       This manual page was written by Peter Baker

								    2009-12-18							       XGFMERGE(1)