Sponsored Content
Top Forums Shell Programming and Scripting script to merge two files on an index Post 302639971 by LMHmedchem on Sunday 13th of May 2012 10:49:07 PM
Old 05-13-2012
Thanks for the post, that seems to get me allot of the way there.

I modified as I would use it in a bash script,
Code:
#!/usr/bin/bash

INDEX=$1
INDEX_FILE=$2
MERGE_FILE=$3

awk -vIDX=$INDEX -vO="id filePath MDL_NUMBER RI3_1 fw" '
FNR==1 {
   headers=split(O, htxt)
   split("", o)
   for(hd in htxt) p[htxt[hd]]=hd
   for(i=1;i<=NF;i++) {
       if ($i==IDX) keypos=i
       if ($i in p) o[p[$i]]=i
   }
   next;
}
{ for(c in o) {K[$keypos]; OUT[$keypos,c]= $(o[c]) } }
END {
    $0=""
    for(i=1;i<=headers;i++)$i=htxt[i];
    print
    $0=""
    for(key in K) {
    for(i=1;i<=headers;i++)
        if(i in htxt) $i=OUT[key,i]l
    print
    }
}' $INDEX_FILE $MERGE_FILE

run as,
./data_merge_awk.sh MDL_NUMBER index_file merge_file > output_file

The only issue is that I most often use this on tab delimited data. I tried changing the split argument from " " to "/t", but that doesn't do it.

The only downside I can see is having to hard code all of the columns I need to keep for each separate use. Some of the files I use this kind of thing for would have hundreds of columns. I'm not familiar with specifying the field names like you did. Is there a syntax for specifying "all columns but this one", etc?

LMHmedchem
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shell script to merge files

Can anybody help me out with this problem " a shell program that takes one or any number of file names as input; sorts the lines of each file in ascending order and displays the non blank lines of each sorted file and merge them as one combined sorted file. The program generates an error... (1 Reply)
Discussion started by: arya
1 Replies

2. Shell Programming and Scripting

Merge two files in windows using perl script

Hi I want to merge two or more files using perl in windows only(Just like Paste command in Unix script) . How can i do this.Is ther any single command to do this? Thanks Kunal (1 Reply)
Discussion started by: kunal_dixit
1 Replies

3. Filesystems, Disks and Memory

why the inode index of file system starts from 1 unlike array index(0)

why do inode indices starts from 1 unlike array indexes which starts from 0 its a question from "the design of unix operating system" of maurice j bach id be glad if i get to know the answer quickly :) (0 Replies)
Discussion started by: sairamdevotee
0 Replies

4. Shell Programming and Scripting

script needed to merge two files and report differences

Hello, I have two txt files that look like this: db.0.0.0.0: Total number of NS records = 1 db.127.0.0.0: Total number of NS records = 1 Total number of PTR records = 1 db.172.19.0.0: Total number of NS records = 1 Total number of PTR records = 3 db.172.19.59.0: Total... (8 Replies)
Discussion started by: richsark
8 Replies

5. Shell Programming and Scripting

script to merge xml files with options

Hi, I have a very basic knowledge of shell scripting & would like some help with a little problem I have. I sometimes use a program calle phronix & sometimes like to compare its results which are *.xml files. Which is easy enough but a friend wants to avoid typing the path to the files.... (2 Replies)
Discussion started by: ptrbee
2 Replies

6. Shell Programming and Scripting

merge two files via looping script

Hi all, I hope you can help me. I got a file a and a file b File a contains a b c d e f g h File b contains 1 2 3 (8 Replies)
Discussion started by: stinkefisch
8 Replies

7. Shell Programming and Scripting

Sort from start index and end index in line

Hi All, I have a file (FileNames.txt) which contains the following data in it. $ cat FileNames.txt MYFILE17XXX208Sep191307.csv MYFILE19XXX208Sep192124.csv MYFILE20XXX208Sep192418.csv MYFILE22XXX208Sep193234.csv MYFILE21XXX208Sep193018.csv MYFILE24XXX208Sep194053.csv... (5 Replies)
Discussion started by: krish_indus
5 Replies

8. Shell Programming and Scripting

merge two text files of different size on common index

I have two text files. text file 1: ID filePath col1 col2 col3 1 10584588.mol 269.126 190.958 23.237 2 10584549.mol 281.001 200.889 27.7414 3 10584511.mol 408.824 158.316 29.8561 4 10584499.mol 245.632 153.241 25.2815 5 10584459.mol ... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

9. Programming

Merge sort when array starts from zero(0) index???

Hi friends, I have implemented the merge sort algorith in c, before I put forward my question, you please have a look at my code. // The array is sorted, as 1234 #include <stdio.h> #include <stdlib.h> #include <math.h> int A = {4, 3, 2, 1}; void Merge_Sort(int , int, int); void... (0 Replies)
Discussion started by: gabam
0 Replies

10. Shell Programming and Scripting

Merge multiple tab delimited files with index checking

Hello, I have 40 data files where the first three columns are the same (in theory) and the 4th column is different. Here is an example of three files, file 2: A_f0_r179_pred.txt Id Group Name E0 1 V N(,)'1 0.2904 2 V N(,)'2 0.3180 3 V N(,)'3 0.3277 4 V N(,)'4 0.3675 5 V N(,)'5 0.3456 ... (8 Replies)
Discussion started by: LMHmedchem
8 Replies
GIT-MERGE-INDEX(1)						    Git Manual							GIT-MERGE-INDEX(1)

NAME
git-merge-index - Run a merge for files needing merging SYNOPSIS
git merge-index [-o] [-q] <merge-program> (-a | [--] <file>*) DESCRIPTION
This looks up the <file>(s) in the index and, if there are any merge entries, passes the SHA1 hash for those files as arguments 1, 2, 3 (empty argument if no file), and <file> as argument 4. File modes for the three files are passed as arguments 5, 6 and 7. OPTIONS
-- Do not interpret any more arguments as options. -a Run merge against all files in the index that need merging. -o Instead of stopping at the first failed merge, do all of them in one shot - continue with merging even when previous merges returned errors, and only return the error code after all the merges. -q Do not complain about a failed merge program (a merge program failure usually indicates conflicts during the merge). This is for porcelains which might want to emit custom messages. If git merge-index is called with multiple <file>s (or -a) then it processes them in turn only stopping if merge returns a non-zero exit code. Typically this is run with a script calling git's imitation of the merge command from the RCS package. A sample script called git merge-one-file is included in the distribution. ALERT ALERT ALERT! The git "merge object order" is different from the RCS merge program merge object order. In the above ordering, the original is first. But the argument order to the 3-way merge program merge is to have the original in the middle. Don't ask me why. Examples: torvalds@ppc970:~/merge-test> git merge-index cat MM This is MM from the original tree. # original This is modified MM in the branch A. # merge1 This is modified MM in the branch B. # merge2 This is modified MM in the branch B. # current contents or torvalds@ppc970:~/merge-test> git merge-index cat AA MM cat: : No such file or directory This is added AA in the branch A. This is added AA in the branch B. This is added AA in the branch B. fatal: merge program failed where the latter example shows how git merge-index will stop trying to merge once anything has returned an error (i.e., cat returned an error for the AA file, because it didn't exist in the original, and thus git merge-index didn't even try to merge the MM thing). AUTHOR
Written by Linus Torvalds <torvalds@osdl.org[1]> One-shot merge by Petr Baudis <pasky@ucw.cz[2]> DOCUMENTATION
Documentation by David Greaves, Junio C Hamano and the git-list <git@vger.kernel.org[3]>. GIT
Part of the git(1) suite NOTES
1. torvalds@osdl.org mailto:torvalds@osdl.org 2. pasky@ucw.cz mailto:pasky@ucw.cz 3. git@vger.kernel.org mailto:git@vger.kernel.org Git 1.7.1 07/05/2010 GIT-MERGE-INDEX(1)
All times are GMT -4. The time now is 06:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy