There aren't 3 numbers in. There are, in fact, millions and millions. The 3 lines that I posted are just a sample from a 9,000,000 line file. What I'm trying to do is use all of a system's CPU to complete a task faster. So, let me explain a little differently.
I have a huge file. Normally, awk would just read from line 1 to the last line, using a single CPU core. This script gives me what I want:
If I run that script on my data file, it works just fine, albeit slowly, as it uses only a single core. So, I decided to instead make 7 copies of the script and use them all at the same time. Those are called proc.1.mawk, proc.2.mawk ... proc.7.mawk. Another mawk script is feeding lines into those scripts. Lines where NR%7 is 0 go to the first proc script, those where NR%7 is 1 go to the second proc script, and so on so that those where NR%7 is 6 go to the last proc script. That mawk script is simply this:
I call that the "distributor". You can see that it (1) calculates NR%7 for each line, (2) figures out which processor script should be handling that line, and (3) pipes the line to that processor script. So, each proc script is doing 1/7th of the total lines. That all works properly and, in fact, shows a huge improvement in processing time.
The problem is simply that each of those proc scripts write their output to stdout. So, I have to do something like this:
That works just fine, but I'd rather have summing occur in the same distributor that passes the lines to the processor scripts. I tried something like this:
That fails with a syntax error.
So, is there a way to get the distributor to receive back the output of the proc.NUM.mawk scripts and sum them together?
I try to get the month (of last save) and the filename into a variable, is this possible ?
something like this :
for month in `ls -la | awk '{print $6}'`
do
if ]
then
a=filename of the matching file
cp $a /Sep
fi
thanks,
Steffen (1 Reply)
i need help with an awk question. i am looking to have an external variable be defined outside of awk but used in awk. so if we have fields $1, $2, $3 so on and so forth, i would like to be able to dictate what field is being printed by something like $. so if i had a counter called test, make it 3... (8 Replies)
How would I pass awk output to a perl variable?
For example, I want to save the value in the 4th column into the variable called test. My best guess is something as follow, but I am sure this isn't correct.
$test = system("awk '/NUMBER/{print \$4}' $_"); (8 Replies)
So I have a if statement inside an awk to check if $2 of a awk equals a specific IP but the test fails. So here is what I have.
# !/bin/sh
echo "Enter client ID"
read ID
echo "Enter month (01, 02, 03)"
read month
echo "Enter day (03, 15)"
read day
echo "Enter Year (07, 08)"
read... (6 Replies)
hi all i am trying to save an awk value into an array in bash:
total=`awk '{sum+=$3} END {print sum}' "$count".txt"`
((count++))
the above statement is in a while loop..
$count is to keep track of file numbers (1.txt,2.txt,3.txt,etc.)
i get the following error:
./lines1:... (1 Reply)
Dear all,
¿How can i insert a variable in a AWK pattern?
I have almost succeeded in solving a puzzle with AWK but now i want to make a script. Let me explain.
cat file.txt | awk 'BEGIN {RS="\\n\\n"} /tux/ { print "\n"$0 }'
I know that this command makes right what i want to do, but mi... (8 Replies)
I want to incorporate the variable in the for statement as a column of my processed file. In the INCORRECT example below, it is $i which corresponds to the i in my for loop:
for i in x86_64 i686; do
awk '{ print $1" "$4" "$5" "$i }'awk $file-$i > processed-$i.log
doneThanks! (3 Replies)
I'm trying to get the universities result data into different file, where the $9 contains unversity field and field7,4 & 5 contains the keys to sort the students by marks. How to use uni variable to match against $9 inside awk?
c=0
for uni in `cat /tmp/global_rank| awk -F ',' '{print... (1 Reply)
Hi,
I am trying to write a bash script in which I need to pass a external variable to the awk program. I tired using -v but it not accepting the value.
Here is my sample code.
#!/usr/bin/bash
######################################################################################
####... (5 Replies)
Good morning all,
This is the file name in question OD_Orders_2019-02-19.csv
I am trying to create a bash script to read into files with yesterdays date on the file name while retaining the rest of the files name. I would like for $y to equal, the name of the file with a formula output with... (2 Replies)
Discussion started by: Ibrahim A
2 Replies
LEARN ABOUT CENTOS
igawk
IGAWK(1) Utility Commands IGAWK(1)NAME
igawk - gawk with include files
SYNOPSIS
igawk [ all gawk options ] -f program-file [ -- ] file ...
igawk [ all gawk options ] [ -- ] program-text file ...
DESCRIPTION
Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).
AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like
@include getopt.awk
in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.
OPTIONS
See gawk(1) for a full description of the AWK language and the options that gawk supports.
EXAMPLES
cat << EOF > test.awk
@include getopt.awk
BEGIN {
while (getopt(ARGC, ARGV, "am:q") != -1)
...
}
EOF
igawk -f test.awk
SEE ALSO gawk(1)
Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.
AUTHOR
Arnold Robbins (arnold@skeeve.com).
Free Software Foundation Nov 3 1999 IGAWK(1)