I'm trying to use the "join" function for more than 1 field. Since it's not possible as it is, I want to take my input files and concatenate the joining fields as 1 field (separated by "|"). I wrote 2 awk script to do and undo it (see below). However I'm new to awk and I'm certain I could do it in a much more efficient way.
I found various topics around the question but often the syntax proposed is a bit of a mystery to me. For instance someone posted this:
what does the trailing '1' mean? what are there 2 separated {} and what distinguish them? finally, where can I find doc on that kind of questions (googling "awk trailing digit" didn't help me much!!)
Here are my scripts, I don't care much about syntax shortcuts, I only care about speed of execution!
any help would be greatly appreciated
to concatenate:
Code:
#!/bin/sh
#
# usage:
# nawk -F$'\t' -v JF=3,5 -f concatene.awk ~/tmp/tmp15
# nawk -F$'\t' -v JF=15,16,17,18 -f concatene.awk split/snp_j > concat
#
# JF stands for "join fields"
BEGIN { FS="\t";OFS="\t" }
{
if (NR==1) { # to do it only once (NR starts at 1)
N=split(JF,JFS,",");
for (i=1;i<=N;i++) { # reverse it
RJFS[JFS[i]] = i;
}
}
LINE="";
for (FIELD_INDEX=1 ; FIELD_INDEX<=N ; FIELD_INDEX++ ) {
LINE=(FIELD_INDEX==1 ? "" : LINE"|")$JFS[FIELD_INDEX];
}
for (FIELD_INDEX=1 ; FIELD_INDEX<=NF ; FIELD_INDEX++ ) {
if (!RJFS[FIELD_INDEX]) {
LINE=LINE"\t"$FIELD_INDEX;
}
}
print LINE;
}
example:
input: a b c d e f
output: c|e a b d f
to "un"concatenate:
Code:
#!/bin/sh
# nawk -F$'\t' -v JF=3,5 -f unconcatene.awk test
BEGIN { FS="\t";OFS="\t" }
{
if (NR==1) { # to do it only once (NR starts at 1)
N=split(JF,JFS,",");
for (i=1;i<=N;i++) { # reverse it
RJFS[JFS[i]] = i;
}
}
N2=split($1,JFS2,"|"); # N=N2
for (i=1;i<=N;i++) { # reverse it
RJFS[JFS[i]] = JFS2[i];
}
SIZE=NF-1+N;
FIELD_INDEX=2;
LINE="";
for (NEW_FIELD_INDEX=1 ; NEW_FIELD_INDEX<=SIZE ; NEW_FIELD_INDEX++ ) {
LINE=LINE(NEW_FIELD_INDEX==1 ? "" : "\t");
if (RJFS[NEW_FIELD_INDEX]) {
LINE=(LINE)RJFS[NEW_FIELD_INDEX];
} else {
LINE=(LINE)$FIELD_INDEX;
FIELD_INDEX++;
}
}
print LINE;
}
I'm working on formatting some attendance data to meet a vendors requirements to upload to their system. With some help on the forums here, I have the data close. But they've since changed what they want.
The vendor wants me to submit three fields to them. Field 1 is the studentid field,... (4 Replies)
Hi,
I have a huge text file. It looks like
abcde bangalo country 12345 lastfield
i want to get first 3 characters from field1 and first 3 characters from field 2 and insert the result as a new field. example the result should be:
abcde bangalo abcban country 12345 lastfield
Please... (4 Replies)
Need a script that manipulates a fixed length file that will compare 2 fields in that file and if they are equal write that line to a new file.
i.e. If fields 87-93 = fields 119-125, then write the entire line to a new file. Do this for every line in the file. After we get only the fields... (1 Reply)
Hi colleagues,
I have a file in this format.
"/cccc/pppp/dddd/ggg/prueba.txt".
ERROR" THE error bbbbbbbbbb finish rows.
"/kkkk/mmmm/hhhh/jjj/ejemplo.txt".
ERROR This is other error rows.I need my file in this format.
"/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows.... (3 Replies)
Hi All,
I want to remove the rows from File1.csv by comparing the columns/fields in the File2.csv. I only need the records whose first column is same and the second column is different for the same record in both files.Here is an example on what I need.
File1.csv:
RAJAK|ACTIVE|1... (2 Replies)
I have a flat file A.txt with field seperate by a pipe
2012/11/13 20:06:11 | 284:hawk pid=014268 opened Locations 12, 13, 14, 15 for /home/hawk_t112/t112/macteam/qt/NET12/full_ddr3_2X_FV_4BD_1.qt/dbFiles/t112.proto|2012/11/14 15:19:26 | still running |norway|norway
2012/11/14 12:53:51 | ... (6 Replies)
The below awk parser works for most data inputs, but I am having trouble with the last one. The problem is in the below rules steps 1 and 2 come from $2 (NC_000013.10:g.20763686_20763687delinsA) and steps 3 and 4 come from $1 (NM_004004.5:c.34_35delGGinsT).
Parse Rules:
The header is... (0 Replies)
Hello All,
I have many zipped XMLs (example file name in tgz formate - file_rec.trx.2016-01-23.000123.exc.85sesdzd45wsds5299c8f2994f7.tgz) looks following and I need to verify two numbers, they are RecordNumber and EnrolData (only sequence number, NOT hole).
for all the records, both should be... (5 Replies)
Need your help in solving this puzzle. Any kind of help will be appreciated and link for any documents to read and learn and to deal with such scenarios would be helpful
Concatenate column1 and column2 of file 1. Then check for the concatenated value in Column1 of File2. If found extract the... (14 Replies)
I have below code inside my awk script
if ( $0 ~ /SVC IN:/ )
{
svc_in=substr( $0,23 , 3);
if (msg_start == 1 && msg_end == 0)
{
msg_arr=$0;
}
}
else if ( $0 ~ /^SVC OUT:/ )
{
svc_out=substr( $0, 9, 3);
if (msg_start == 1 && msg_end == 0)
... (6 Replies)
Discussion started by: bhagya123
6 Replies
LEARN ABOUT DEBIAN
plan9-join
JOIN(1) General Commands Manual JOIN(1)NAME
join - relational database operator
SYNOPSIS
join [ options ] file1 file2
DESCRIPTION
Join forms, on the standard output, a join of the two relations specified by the lines of file1 and file2. If one of the file names is the
standard input is used.
File1 and file2 must be sorted in increasing ASCII collating sequence on the fields on which they are to be joined, normally the first in
each line.
There is one line in the output for each pair of lines in file1 and file2 that have identical join fields. The output line normally con-
sists of the common field, then the rest of the line from file1, then the rest of the line from file2.
Input fields are normally separated spaces or tabs; output fields by space. In this case, multiple separators count as one, and leading
separators are discarded.
The following options are recognized, with POSIX syntax.
-a n In addition to the normal output, produce a line for each unpairable line in file n, where n is 1 or 2.
-v n Like -a, omitting output for paired lines.
-e s Replace empty output fields by string s.
-1 m
-2 m Join on the mth field of file1 or file2.
-jn m Archaic equivalent for -n m.
-ofields
Each output line comprises the designated fields. The comma-separated field designators are either 0, meaning the join field, or
have the form n.m, where n is a file number and m is a field number. Archaic usage allows separate arguments for field designators.
-tc Use character c as the only separator (tab character) on input and output. Every appearance of c in a line is significant.
EXAMPLES
sort /etc/passwd | join -t: -1 1 -a 1 -e "" - bdays
Add birthdays to the /etc/passwd file, leaving unknown birthdays empty. The layout of /adm/users is given in passwd(5); bdays con-
tains sorted lines like
tr : ' ' </etc/passwd | sort -k 3 3 >temp
join -1 3 -2 3 -o 1.1,2.1 temp temp | awk '$1 < $2'
Print all pairs of users with identical userids.
SOURCE
/src/cmd/join.c
SEE ALSO sort(1), comm(1), awk(1)BUGS
With default field separation, the collating sequence is that of sort -b -ky,y; with -t, the sequence is that of sort -tx -ky,y.
One of the files must be randomly accessible.
JOIN(1)
03-22-2010
