Sponsored Content
Top Forums Shell Programming and Scripting awk - saving results of external script to variable. Post 302955996 by treesloth on Thursday 24th of September 2015 02:11:44 PM
Old 09-24-2015
awk - saving results of external script to variable.

So, I've been playing with speeding up some analysis we do by using multiple threads of awk (actually, mawk, but code-compatible as far as I use it) on multiple CPU cores. So, I have a big data file and I have several copies of exactly the same processor script, written in mawk. I also have a "distributor" in mawk whose only job is to determine the modulus of a given line's number and use that to send the line to the appropriate processor script. Then, when that's all done, the results of the processor scripts are joined together into a final result. Here's the "distributor" line:

Code:
mawk -F , -v count=6 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

So, lines 6, 12, 18, etc will go to proc.1.mawk, 7, 13, 19 will go to proc.2.mawk, and so on. Those work very well. The problem is in gathering up those results. Currently, each proc.X.mawk script just spits out its answer to stdout, and I have to pipe to another awk to sum those. My question, then: Is there a way to have the "distributor" receive the results from the proc.X.mawk scripts and sum them itself, such as by saving to a variable that can be acted on later?

Thanks in advance.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

store awk results in a variable

I try to get the month (of last save) and the filename into a variable, is this possible ? something like this : for month in `ls -la | awk '{print $6}'` do if ] then a=filename of the matching file cp $a /Sep fi thanks, Steffen (1 Reply)
Discussion started by: forever_49ers
1 Replies

2. Shell Programming and Scripting

awk help (external variable)

i need help with an awk question. i am looking to have an external variable be defined outside of awk but used in awk. so if we have fields $1, $2, $3 so on and so forth, i would like to be able to dictate what field is being printed by something like $. so if i had a counter called test, make it 3... (8 Replies)
Discussion started by: pupp
8 Replies

3. Shell Programming and Scripting

Saving output from awk into a perl variable

How would I pass awk output to a perl variable? For example, I want to save the value in the 4th column into the variable called test. My best guess is something as follow, but I am sure this isn't correct. $test = system("awk '/NUMBER/{print \$4}' $_"); (8 Replies)
Discussion started by: userix
8 Replies

4. Shell Programming and Scripting

If statement in awk with external variable

So I have a if statement inside an awk to check if $2 of a awk equals a specific IP but the test fails. So here is what I have. # !/bin/sh echo "Enter client ID" read ID echo "Enter month (01, 02, 03)" read month echo "Enter day (03, 15)" read day echo "Enter Year (07, 08)" read... (6 Replies)
Discussion started by: doublejz
6 Replies

5. Shell Programming and Scripting

saving awk value in a bash array variable

hi all i am trying to save an awk value into an array in bash: total=`awk '{sum+=$3} END {print sum}' "$count".txt"` ((count++)) the above statement is in a while loop.. $count is to keep track of file numbers (1.txt,2.txt,3.txt,etc.) i get the following error: ./lines1:... (1 Reply)
Discussion started by: npatwardhan
1 Replies

6. Shell Programming and Scripting

Insert external variable in a AWK pattern

Dear all, ¿How can i insert a variable in a AWK pattern? I have almost succeeded in solving a puzzle with AWK but now i want to make a script. Let me explain. cat file.txt | awk 'BEGIN {RS="\\n\\n"} /tux/ { print "\n"$0 }' I know that this command makes right what i want to do, but mi... (8 Replies)
Discussion started by: antuan
8 Replies

7. Shell Programming and Scripting

[awk] - how to insert an external variable

I want to incorporate the variable in the for statement as a column of my processed file. In the INCORRECT example below, it is $i which corresponds to the i in my for loop: for i in x86_64 i686; do awk '{ print $1" "$4" "$5" "$i }'awk $file-$i > processed-$i.log doneThanks! (3 Replies)
Discussion started by: graysky
3 Replies

8. Shell Programming and Scripting

Help needed with awk external variable

I'm trying to get the universities result data into different file, where the $9 contains unversity field and field7,4 & 5 contains the keys to sort the students by marks. How to use uni variable to match against $9 inside awk? c=0 for uni in `cat /tmp/global_rank| awk -F ',' '{print... (1 Reply)
Discussion started by: InduInduIndu
1 Replies

9. Shell Programming and Scripting

Passing external variable to awk

Hi, I am trying to write a bash script in which I need to pass a external variable to the awk program. I tired using -v but it not accepting the value. Here is my sample code. #!/usr/bin/bash ###################################################################################### ####... (5 Replies)
Discussion started by: jpkumar10
5 Replies

10. Shell Programming and Scripting

I want to add a variable for the results from the formula of one variable and results of another var

Good morning all, This is the file name in question OD_Orders_2019-02-19.csv I am trying to create a bash script to read into files with yesterdays date on the file name while retaining the rest of the files name. I would like for $y to equal, the name of the file with a formula output with... (2 Replies)
Discussion started by: Ibrahim A
2 Replies
Tk_CreateClientMessageHandler(3)			       Tk Library Procedures				  Tk_CreateClientMessageHandler(3)

__________________________________________________________________________________________________________________________________________________

NAME
Tk_CreateClientMessageHandler, Tk_DeleteClientMessageHandler - associate procedure callback with ClientMessage type X events SYNOPSIS
#include <tk.h> Tk_CreateClientMessageHandler(proc) Tk_DeleteClientMessageHandler(proc) ARGUMENTS
Tk_ClientMessageProc *proc (in) Procedure to invoke whenever a ClientMessage X event occurs on any display. _________________________________________________________________ DESCRIPTION
Tk_CreateClientMessageHandler arranges for proc to be invoked in the future whenever a ClientMessage X event occurs that is not handled by WM_PROTOCOL. Tk_CreateClientMessageHandler is intended for use by applications which need to watch X ClientMessage events, such as drag and drop applications. The callback to proc will be made by Tk_HandleEvent; this mechanism only works in programs that dispatch events through Tk_HandleEvent (or through other Tk procedures that call Tk_HandleEvent, such as Tk_DoOneEvent or Tk_MainLoop). Proc should have arguments and result that match the type Tk_ClientMessageProc: typedef int Tk_ClientMessageProc( Tk_Window tkwin, XEvent *eventPtr); The tkwin parameter to proc is the Tk window which is associated with this event. EventPtr is a pointer to the X event. Whenever an X ClientMessage event is processed by Tk_HandleEvent, the proc is called if it was not handled as a WM_PROTOCOL. The return value from proc is normally 0. A non-zero return value indicates that the event is not to be handled further; that is, proc has done all processing that is to be allowed for the event. If there are multiple ClientMessage event handlers, each one is called for each event, in the order in which they were established. Tk_DeleteClientMessageHandler may be called to delete a previously-created ClientMessage event handler: it deletes each handler it finds that matches the proc argument. If no such handler exists, then Tk_DeleteClientMessageHandler returns without doing anything. Although Tk supports it, it's probably a bad idea to have more than one callback with the same proc argument. KEYWORDS
bind, callback, event, handler Tk 8.4 Tk_CreateClientMessageHandler(3)
All times are GMT -4. The time now is 12:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy