awk runs but output is empty


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk runs but output is empty
# 1  
Old 12-19-2015
awk runs but output is empty

The awk below runs, however the output file is 0 bytes. It is basically matching input files that are 21 - 259 records to a file of 11,137,660 records. Basically, what it does is use the input files of which there are 4 to search and match in a large 11,000,000 record file and output the average of all the $7 in the matches. I can not seem to figure out why the file is empty. The search.txt and input.txt will be different each time, but always in the format as the examples. Thank you Smilie.

input
Code:
 
AGRN 
CCDC39  
CCDC40  
CFTR

search
Code:
 
chr1    955543  955763  chr1:955543-955763 AGRN-6|gc=75    1   0 
chr1    955543  955763  chr1:955543-955763 AGRN-6|gc=75    2   2 
chr1    955543  955763  chr1:955543-955763 AGRN-6|gc=75    3   2

Desired output ($4 $5 last row of $6 "bases with an average of" all matching $7
Code:
chr1:955543-955763 AGRN|gc=75 3 bases with an average of 0.7

awk
Code:
  awk '
 NR == FNR {input[$0]; next}
 {
    split($5, a, "-")
    if (a[1] in input) {
         key = $4 OFS $5
         n[key]++
         sum[key] += $7
     }
 }
 END {
     for (key in n) 
         printf "%s %.1f\n", key, sum[key]/n[key]
 }
' search.txt input.txt > output.txt


Last edited by cmccabe; 12-19-2015 at 11:03 AM.. Reason: fixed format
# 2  
Old 12-19-2015
Try reversing the order of input files.

---------- Post updated at 16:16 ---------- Previous update was at 16:14 ----------

And, make sure there are no trailing spaces in input.txt. Or, use $1 as the array index.
# 3  
Old 12-19-2015
The resulting output is 0 bytes, so I tried another awk which does produce output, but not in the desired way and I need some help with the output. Thank you Smilie.

Code:
BEGIN { 
    slen = 0; 
} 
 
# get input file(s) 
ARGIND > 1 { 
    ###printf("input_push: DEBUG %s\n",$0); 
    input[$0]; 
    next; 
} 
 
# get single search list 
{ 
    ###printf("search_push: DEBUG %s\n",$0); 
    search[slen++] = $0; 
    next; 
} 
 
END { 
    # sum up data 
    for (sidx = 0;  sidx < slen;  ++sidx) { 
        sval = search[sidx]; 
        ###printf("search_end: DEBUG %s\n",sval); 
 
        split(sval,sary) 
        split(sary[5],a,"-"); 
        ###printf("search_end: DEBUG sary[5]='%s' a[1]='%s'\n",sary[5],a[1]); 
 
        if (a[1] in input) { 
            key = sary[4] OFS sary[5] 
            n[key]++ 
            sum[key] = sary[7] 
        } 
    } 
 
    for (key in n) 
        printf "%s %.1f\n", key, sum[key]/n[key] 
}

then I invoke it awk -f script.awk search.txt input.txt > output.txt

output.txt is close:
chr1:955543 AGRN-6|gc=75 0.7 and $3 should be 1.3 but I can't seem to find the error

---------- Post updated at 11:27 AM ---------- Previous update was at 09:26 AM ----------

I modified the awk to:

Code:
awk '
 NR == FNR {input[$0]; next}
 {
    split($5, a, "-")
    if (a[1] in input) {
         key = $4 OFS $5
         n[key]++
         sum[key] += $7
     }
 }
 END {
     for (key in n) 
         printf "%s %.1f\n", key, sum[key]/n[key]
 }
' input.txt search.txt > output.txt

output.txt
Code:
chr1:955543 AGRN-6|gc=75 1.3

(closer to the desired). Thank you Smilie

Last edited by cmccabe; 12-19-2015 at 01:28 PM.. Reason: added details
# 4  
Old 12-19-2015
Would this help?
Code:
awk '
 NR == FNR {input[$1]; next}
 {
    split($5, a, "-")
    if (a[1] in input) {
         key = $4 OFS $5
         n[key]++
         sum[key] += $7
         six[key]  = $6
     }
 }
 END {
     for (key in n)
         printf "%s %s bases with an average of %.1f\n", key, six[key], sum[key]/n[key]
 }
' file2 file1
chr1:955543-955763 AGRN-6|gc=75 3 bases with an average of 1.3


Last edited by RudiC; 12-19-2015 at 01:35 PM.. Reason: added result line
This User Gave Thanks to RudiC For This Post:
# 5  
Old 12-19-2015
That works great.... thank you Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to reformat output if input file is empty, but not if file has data in it

The below awk improved bu @MadeInGermany, works great as long as the input file has data in it in the below format: input chrX 25031028 25031925 chrX:25031028-25031925 ARX 631 18 chrX 25031028 25031925 chrX:25031028-25031925 ARX 632 14... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk runs and produces output but with error

When I run the awk below, I get an error message awk -v OFS='\t' '$(NF-1)=="Benign" || ($(NF-2) OFS $(NF-1))=="Likely Benign" {$(NF)=$(NF-2) OFS $(NF-1)} {print $0 }' input awk: cmd. line:1: (FILENAME=VUS FNR=8) fatal: attempt to access field -1 input Chr Start End Ref ... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

This function (decode64) runs on gawk but not on busybox awk

Hello, I'm trying to figure out a way to use a decode64 function in an embedded system who has few utilities, including busybox. Right now have something like this (taken from "google base64-and-base85-encoding-awk-scripts" sorry, I'm not able to post urls yet) _decode64() { &&... (4 Replies)
Discussion started by: chilicuil
4 Replies

4. Shell Programming and Scripting

Script which telnets to a device, runs commands and prints output to a file

I am connecting to a device using telnet, I want my script to perform certain commands : ie- show device , show inventory..etc and write the output it sees from the terminal to a file. this is what I have got : #!/usr/bin/expect -- set running 1 spawn telnet <ip address> expect ... (1 Reply)
Discussion started by: samantha123
1 Replies

5. Shell Programming and Scripting

awk command in script gives error while same awk command at prompt runs fine: Why?

Hello all, Here is what my bash script does: sums number columns, saves the tot in new column, outputs if tot >= threshold val: > cat getnon0file.sh #!/bin/bash this="getnon0file.sh" USAGE=$this" InFile="xyz.38" Min="0.05" # awk '{sum=0; for(n=2; n<=NF; n++){sum+=$n};... (4 Replies)
Discussion started by: catalys
4 Replies

6. Shell Programming and Scripting

Output only non-empty arguments

Hello, I am VERY new to shell scripting here, so please go easy. I have an assignment that requires creating a script using bash shell, outputting all command line arguments that are not empty ones such as " ", and showing total number of arguments. I know how to show the total with $# and all... (6 Replies)
Discussion started by: moderwarfare
6 Replies

7. Shell Programming and Scripting

Script Runs fine but not giving any output

Hi, My script is running with no erros but not giving any output can anyonehelp. #!/bin/ksh . /home/application/bin/application.env OUTFILE=Result.txt PROD_PASSWORD=`${GET_PWD} -f ${PWD_FILE_PATH} -s ${PROD_SERVER} -u ${PROD_USER}` echo "1)To get the book last loaded details " read... (7 Replies)
Discussion started by: jagadish_gaddam
7 Replies

8. Shell Programming and Scripting

command runs, no output

I have a script that searches for specific information from log files. #!/bin/sh sed -n '/*C/,/END/p' /sn/log/OMlog* > crit.out sed -n '/REPT INITIALIZATION/,/err:/p' /sn/log/OMlog* > switchcc.out ./start.awk /sn/log/OMlog* > ARs.out ./end.awk /sn/log/OMlog* > ARe.out cat crit.out... (1 Reply)
Discussion started by: grinds
1 Replies

9. UNIX for Dummies Questions & Answers

The output file is empty. Please Help

Dear all, Currently I writing a ksh script to perform some sql query. I already pipe results in a output file. But when I checked it, the output file is empty. Below is part of the script that I wrote: ------------------------------------------------------------------------ function... (4 Replies)
Discussion started by: balzzz
4 Replies
Login or Register to Ask a Question