Eliminating the final awk in a pipeline that runs for 11 seconds probably isn't going to affect run time much, but, just using standard awk and shell features, you can use a FIFO to get rid of the 2nd awk in your pipeline:
Code:
#!/bin/ksh
# Create a FIFO and create a writer for that FIFO...
FIFO="FIFO.$$"
mkfifo "$FIFO"
echo 0 > "$FIFO"&
awk -v count=7 -v FIFO="$FIFO" '
BEGIN { # Initialize SUM and create a reader for the FIFO...
getline SUM < FIFO
# printf("getline SUM found %s (should be 0)\n", SUM)
# Initialize the pipelines to receive data from this script.
for(i = 1; i <= count; i++) {
proc[i] = "./proc." i ".mawk -F , > " FIFO
# Note that if the "proc.<digit>.mawk" scripts really are
# the same, you only need one script; you just need to
# add a comment so awk sees different strings, for
# example, you could use:
# proc[i] = "./proc.mawk -F, > " FIFO " #" i
# and just have one script instead of count different scripts.
}
}
NR > 1 {# Distribute input lines to the various processing scripts.
mod = NR%count + 1
print $0 | proc[mod]
}
END { # Close pipelines to the other mawk scripts.
for(i = 1; i <= count; i++)
close(proc[i])
# Read results from the other mawk scripts and add them into SUM.
while((getline sum < FIFO) > 0) {
SUM += sum
# printf("read sum(%s) from FIFO\n", sum)
# printf("Intermediate SUM is %s\n", SUM)
}
# Print the total.
print SUM
}' datafile.csv
rm -f "$FIFO"
As always, if you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
Note that I just used print SUM at the end instead of printf("%.0f\n",SUM) because the script calculating the sums in the child mawk scripts you showed us was only summing integer values. If you wanted to sum the floating point values in addition to the integers, and print the sum rounded to the nearest integer, the line in the proc.digit.mawk scripts:
Code:
if ($i ~ /^[0-9]+$/)
would have to be something more like:
Code:
if ($i ~ /^[0-9]+([.][0-9]*)?$|^[0-9]*[.][0-9]+$|^[0-9]+[.][0-9]*$/)
and, note that neither of the above will include any numbers in the output total from fields in your input file that contain any leading or trailing <space> characters.
Although written and tested using the Korn shell and a plain awk, there is nothing here that should cause any problems for any other shell that recognizes Bourne/POSIX shell syntax and nothing that should be a problem for gawk or mawk.
This User Gave Thanks to Don Cragun For This Post:
Good morning all,
This is the file name in question OD_Orders_2019-02-19.csv
I am trying to create a bash script to read into files with yesterdays date on the file name while retaining the rest of the files name. I would like for $y to equal, the name of the file with a formula output with... (2 Replies)
Hi,
I am trying to write a bash script in which I need to pass a external variable to the awk program. I tired using -v but it not accepting the value.
Here is my sample code.
#!/usr/bin/bash
######################################################################################
####... (5 Replies)
I'm trying to get the universities result data into different file, where the $9 contains unversity field and field7,4 & 5 contains the keys to sort the students by marks. How to use uni variable to match against $9 inside awk?
c=0
for uni in `cat /tmp/global_rank| awk -F ',' '{print... (1 Reply)
I want to incorporate the variable in the for statement as a column of my processed file. In the INCORRECT example below, it is $i which corresponds to the i in my for loop:
for i in x86_64 i686; do
awk '{ print $1" "$4" "$5" "$i }'awk $file-$i > processed-$i.log
doneThanks! (3 Replies)
Dear all,
¿How can i insert a variable in a AWK pattern?
I have almost succeeded in solving a puzzle with AWK but now i want to make a script. Let me explain.
cat file.txt | awk 'BEGIN {RS="\\n\\n"} /tux/ { print "\n"$0 }'
I know that this command makes right what i want to do, but mi... (8 Replies)
hi all i am trying to save an awk value into an array in bash:
total=`awk '{sum+=$3} END {print sum}' "$count".txt"`
((count++))
the above statement is in a while loop..
$count is to keep track of file numbers (1.txt,2.txt,3.txt,etc.)
i get the following error:
./lines1:... (1 Reply)
So I have a if statement inside an awk to check if $2 of a awk equals a specific IP but the test fails. So here is what I have.
# !/bin/sh
echo "Enter client ID"
read ID
echo "Enter month (01, 02, 03)"
read month
echo "Enter day (03, 15)"
read day
echo "Enter Year (07, 08)"
read... (6 Replies)
How would I pass awk output to a perl variable?
For example, I want to save the value in the 4th column into the variable called test. My best guess is something as follow, but I am sure this isn't correct.
$test = system("awk '/NUMBER/{print \$4}' $_"); (8 Replies)
i need help with an awk question. i am looking to have an external variable be defined outside of awk but used in awk. so if we have fields $1, $2, $3 so on and so forth, i would like to be able to dictate what field is being printed by something like $. so if i had a counter called test, make it 3... (8 Replies)
I try to get the month (of last save) and the filename into a variable, is this possible ?
something like this :
for month in `ls -la | awk '{print $6}'`
do
if ]
then
a=filename of the matching file
cp $a /Sep
fi
thanks,
Steffen (1 Reply)
09-25-2015
