04-15-2014
Re:Merge records based on multiple columns
Hi,
I am not an expert shell scripting programmer ,but I have tried few options
1) For loop by reading each row and doing concatenated grep to find the matching column
2) I have also gone through the link (on this website itself )given below and tried to change given solution as per requirement
shell-programming-and-scripting
208027-merge-multiple-lines-same-file-common-key-using-awk.html
but ordering on the basis of 15th column is still an issue and also I am getting double quotes in concatenation of last field
I am ok with any solution being in KSH,BASH or AWK. Perl and any other language could be the last option but not preferable for now.
I am not sure about OS version too as I am out of my office and can't check
let me know if these informations are helpful
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to... (4 Replies)
Discussion started by: oordonez
4 Replies
2. Shell Programming and Scripting
Hi ,
I have the below source
source data
1|2|3|:123:abc|4
1|2|a| | 5
1|2|3|4|:a:s:D.....:n|t
Target data should be
1|2|3|:123:abc|4
1|2|3|:123:abc|4
1|2|a| | 5
1|2|3|4|:a:s:D.....:n|t
1|2|3|4|:a:s:D.....:n|t
1|2|3|4|:a:s:D.....:n|t
1|2|3|4|:a:s:D.....:n|t (3 Replies)
Discussion started by: mora
3 Replies
3. Shell Programming and Scripting
I have two files
1.txt
34, ABC, 7, 8, 0.9
35, CDE, 6.5, -2, 0.01
2.txt
34, ABC, 9, 6, -1.9
35, CDE, 8.5, -2.3, 5.01
So in both files common columns are 1 and 2
so final o/p should look like
34, ABC, 7, 8, 0.9, 9, 6, -1.9
35, CDE, 6.5, -2, 0.01, 8.5, -2.3, 5.01
I tried using... (3 Replies)
Discussion started by: manas_ranjan
3 Replies
4. Shell Programming and Scripting
Hi everyone,
I have a file result.txt with records as following and another file mirna.txt with a list of miRNAs e.g. miR22, miR123, miR13 etc.
Gene Transcript miRNA
Gar Nm_111233 miR22
Gar Nm_123440 miR22
Gar Nm_129939 miR22
Hel Nm_233900 miR13
Hel ... (6 Replies)
Discussion started by: miclow
6 Replies
5. UNIX for Dummies Questions & Answers
Hi all,
I'm looking for a way to merge multiple columns (from one file) into a single column in an output file.
The file I have looks somewhat like this:
@HWI-ST212 1:N:0 AGTCCTACCGGGAGT + @@@DDDDDHHHHHII
@HWI-ST212 1:N:0 CGTTTAAAAATTTCT + @;@B;DDDDH?:F;F... (4 Replies)
Discussion started by: Vnguyen
4 Replies
6. UNIX for Dummies Questions & Answers
Hi all, I've searched the web for a long time trying to figure out how to merge columns from multiple files.
I know paste will append columns like so:
paste file1 file2 file3 file4 file5 ...
But this becomes inconvenient when you want to append a large number of files into a single file.
... (2 Replies)
Discussion started by: torchij
2 Replies
7. Shell Programming and Scripting
Hello and Good day
I have a lot of files with same number of rows and columns.$2 and $3 are the same in all files .
I need to merge $2,$3,$6 from first file and $6 from another files.
File1:
$1 $2 $3 $4 $5 $6... (8 Replies)
Discussion started by: ali.seifaddini
8 Replies
8. Shell Programming and Scripting
011111123444 1234 1 20000
011111123444 1235 1 30000
011111123446 1234 3 40000
011111123447 1234 4 50000
011111123448 1234 3 50000
File2:
011111123444,Rsttponrfgtrgtrkrfrgtrgrer
011111123446,Rsttponrfgtrgtr
011111123447,Rsttponrfgtrguii
011111123448,Rsttponrfgtrgtjiiu
I have 2 files... (4 Replies)
Discussion started by: vinus
4 Replies
9. UNIX for Dummies Questions & Answers
Hi, I am trying to selectively merge two files based on keys reported in the 1st column.
File1:
#file1-header1
file1-header2
111 qwe rtz uio
198 asd fgh jkl
165 yxc
789 poi uzt rew
89 lkj
File2:
#file2-header2
file2-header2
165 ghz nko2 ... (2 Replies)
Discussion started by: dovah
2 Replies
10. UNIX for Beginners Questions & Answers
I will like to merge several files using 'cat', but I observe the output is not consistent. the merge begins at the last line of the first file.
file1.txt:
1234
1234
1234
file2.txt:
aaaa
bbbb
cccc
dddd
cat file1.txt file2.txt > file3.txt
file3.txt:
1234
1234
1234aaaa
bbbb
cccc... (13 Replies)
Discussion started by: geomarine
13 Replies
paste(1) General Commands Manual paste(1)
Name
paste - merge file data
Syntax
paste file1 file2...
paste -dlist file1 file2...
paste -s [-dlist] file1 file2...
Description
In the first two forms, concatenates corresponding lines of the given input files file1, file2, etc. It treats each file as a column or
columns of a table and pastes them together horizontally (parallel merging).
In the last form, the command combines subsequent lines of the input file (serial merging).
In all cases, lines are glued together with the tab character, or with characters from an optionally specified list. Output is to the
standard output, so it can be used as the start of a pipe, or as a filter, if - is used in place of a file name.
Options
- Used in place of any file name, to read a line from the standard input. (There is no prompting).
-dlist Replaces characters of all but last file with nontabs characters (default tab). One or more characters immediately following -d
replace the default tab as the line concatenation character. The list is used circularly, i. e. when exhausted, it is reused. In
parallel merging (i. e. no -s option), the lines from the last file are always terminated with a new-line character, not from the
list. The list may contain the special escape sequences:
(new-line), (tab), \ (backslash), and (empty string, not a null
character). Quoting may be necessary, if characters have special meaning to the shell (for example, to get one backslash, use
-d"\\" ).
Without this option, the new-line characters of each but the last file (or last line in case of the -s option) are replaced by a
tab character. This option allows replacing the tab character by one or more alternate characters (see below).
-s Merges subsequent lines rather than one from each input file. Use tab for concatenation, unless a list is specified with -d
option. Regardless of the list, the very last character of the file is forced to be a new-line.
Examples
ls | paste -d" " -
list directory in one column
ls | paste - - - -
list directory in four columns
paste -s -d"
" file
combine pairs of lines into lines
Diagnostics
line too long
Output lines are restricted to 511 characters.
too many files
Except for -s option, no more than 12 input files may be specified.
See Also
cut(1), grep(1), pr(1)
paste(1)