Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk - saving results of external script to variable.
Top Forums Shell Programming and Scripting awk - saving results of external script to variable. Post 302956014 by treesloth on Thursday 24th of September 2015 04:16:51 PM
Old 09-24-2015
There aren't 3 numbers in. There are, in fact, millions and millions. The 3 lines that I posted are just a sample from a 9,000,000 line file. What I'm trying to do is use all of a system's CPU to complete a task faster. So, let me explain a little differently.

I have a huge file. Normally, awk would just read from line 1 to the last line, using a single CPU core. This script gives me what I want:

Code:
#!/usr/local/bin/mawk -f

    {
    for (i = 1; i <= NF; i++)
        {
        if ($i ~ /^[0-9]+$/)
            {
            SUM += $i
            }
        }
    }
END {
    printf "%.0f\n",  SUM
    }

If I run that script on my data file, it works just fine, albeit slowly, as it uses only a single core. So, I decided to instead make 7 copies of the script and use them all at the same time. Those are called proc.1.mawk, proc.2.mawk ... proc.7.mawk. Another mawk script is feeding lines into those scripts. Lines where NR%7 is 0 go to the first proc script, those where NR%7 is 1 go to the second proc script, and so on so that those where NR%7 is 6 go to the last proc script. That mawk script is simply this:

Code:
awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv

I call that the "distributor". You can see that it (1) calculates NR%7 for each line, (2) figures out which processor script should be handling that line, and (3) pipes the line to that processor script. So, each proc script is doing 1/7th of the total lines. That all works properly and, in fact, shows a huge improvement in processing time.

The problem is simply that each of those proc scripts write their output to stdout. So, I have to do something like this:

Code:
awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; print $0 | proc}' datafile.csv | awk '{SUM += $1} END {print SUM}'

That works just fine, but I'd rather have summing occur in the same distributor that passes the lines to the processor scripts. I tried something like this:

Code:
awk -F , -v count=7 'NR > 1 {mod = NR%count + 1 ; proc = "./proc." mod ".mawk -F ,"; SUM += (print $0 | proc)}' datafile.csv

That fails with a syntax error.

So, is there a way to get the distributor to receive back the output of the proc.NUM.mawk scripts and sum them together?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

store awk results in a variable

I try to get the month (of last save) and the filename into a variable, is this possible ? something like this : for month in `ls -la | awk '{print $6}'` do if ] then a=filename of the matching file cp $a /Sep fi thanks, Steffen (1 Reply)
Discussion started by: forever_49ers
1 Replies

2. Shell Programming and Scripting

awk help (external variable)

i need help with an awk question. i am looking to have an external variable be defined outside of awk but used in awk. so if we have fields $1, $2, $3 so on and so forth, i would like to be able to dictate what field is being printed by something like $. so if i had a counter called test, make it 3... (8 Replies)
Discussion started by: pupp
8 Replies

3. Shell Programming and Scripting

Saving output from awk into a perl variable

How would I pass awk output to a perl variable? For example, I want to save the value in the 4th column into the variable called test. My best guess is something as follow, but I am sure this isn't correct. $test = system("awk '/NUMBER/{print \$4}' $_"); (8 Replies)
Discussion started by: userix
8 Replies

4. Shell Programming and Scripting

If statement in awk with external variable

So I have a if statement inside an awk to check if $2 of a awk equals a specific IP but the test fails. So here is what I have. # !/bin/sh echo "Enter client ID" read ID echo "Enter month (01, 02, 03)" read month echo "Enter day (03, 15)" read day echo "Enter Year (07, 08)" read... (6 Replies)
Discussion started by: doublejz
6 Replies

5. Shell Programming and Scripting

saving awk value in a bash array variable

hi all i am trying to save an awk value into an array in bash: total=`awk '{sum+=$3} END {print sum}' "$count".txt"` ((count++)) the above statement is in a while loop.. $count is to keep track of file numbers (1.txt,2.txt,3.txt,etc.) i get the following error: ./lines1:... (1 Reply)
Discussion started by: npatwardhan
1 Replies

6. Shell Programming and Scripting

Insert external variable in a AWK pattern

Dear all, ¿How can i insert a variable in a AWK pattern? I have almost succeeded in solving a puzzle with AWK but now i want to make a script. Let me explain. cat file.txt | awk 'BEGIN {RS="\\n\\n"} /tux/ { print "\n"$0 }' I know that this command makes right what i want to do, but mi... (8 Replies)
Discussion started by: antuan
8 Replies

7. Shell Programming and Scripting

[awk] - how to insert an external variable

I want to incorporate the variable in the for statement as a column of my processed file. In the INCORRECT example below, it is $i which corresponds to the i in my for loop: for i in x86_64 i686; do awk '{ print $1" "$4" "$5" "$i }'awk $file-$i > processed-$i.log doneThanks! (3 Replies)
Discussion started by: graysky
3 Replies

8. Shell Programming and Scripting

Help needed with awk external variable

I'm trying to get the universities result data into different file, where the $9 contains unversity field and field7,4 & 5 contains the keys to sort the students by marks. How to use uni variable to match against $9 inside awk? c=0 for uni in `cat /tmp/global_rank| awk -F ',' '{print... (1 Reply)
Discussion started by: InduInduIndu
1 Replies

9. Shell Programming and Scripting

Passing external variable to awk

Hi, I am trying to write a bash script in which I need to pass a external variable to the awk program. I tired using -v but it not accepting the value. Here is my sample code. #!/usr/bin/bash ###################################################################################### ####... (5 Replies)
Discussion started by: jpkumar10
5 Replies

10. Shell Programming and Scripting

I want to add a variable for the results from the formula of one variable and results of another var

Good morning all, This is the file name in question OD_Orders_2019-02-19.csv I am trying to create a bash script to read into files with yesterdays date on the file name while retaining the rest of the files name. I would like for $y to equal, the name of the file with a formula output with... (2 Replies)
Discussion started by: Ibrahim A
2 Replies

LEARN ABOUT CENTOS

slabinfo

SLABINFO(5)						     Linux Programmer's Manual						       SLABINFO(5)

NAME

       /proc/slabinfo - kernel slab allocator statistics

SYNOPSIS

       cat /proc/slabinfo

DESCRIPTION

       Frequently  used  objects  in the Linux kernel (buffer heads, inodes, dentries, etc.)  have their own cache.  The file /proc/slabinfo gives
       statistics.  For example:

	   % cat /proc/slabinfo
	   slabinfo - version: 1.1
	   kmem_cache		 60	78    100    2	  2    1
	   blkdev_requests     5120   5120     96  128	128    1
	   mnt_cache		 20	40     96    1	  1    1
	   inode_cache	       7005  14792    480 1598 1849    1
	   dentry_cache        5469   5880    128  183	196    1
	   filp 		726    760     96   19	 19    1
	   buffer_head	      67131  71240     96 1776 1781    1
	   vm_area_struct      1204   1652     64   23	 28    1
	   ...
	   size-8192		  1	17   8192    1	 17    2
	   size-4096		 41	73   4096   41	 73    1
	   ...

       For each slab cache, the cache name, the number of currently active objects, the total number of available objects, the size of each object
       in  bytes,  the	number of pages with at least one active object, the total number of allocated pages, and the number of pages per slab are
       given.

       Note that because of object alignment and slab cache overhead, objects are not normally packed tightly into pages.  Pages with even one in-
       use object are considered in-use and cannot be freed.

       Kernels	compiled with slab cache statistics will also have "(statistics)" in the first line of output, and will have 5 additional columns,
       namely: the high water mark of active objects; the number of times objects have been allocated; the number of times  the  cache	has  grown
       (new  pages  added  to this cache); the number of times the cache has been reaped (unused pages removed from this cache); and the number of
       times there was an error allocating new pages to this cache.  If slab cache statistics are not enabled for this kernel, these columns  will
       not be shown.

       SMP  systems  will  also  have  "(SMP)" in the first line of output, and will have two additional columns for each slab, reporting the slab
       allocation policy for the CPU-local cache (to reduce the need for inter-CPU synchronization when allocating objects from the  cache).   The
       first  column  is  the per-CPU limit: the maximum number of objects that will be cached for each CPU.  The second column is the batchcount:
       the maximum number of free objects in the global cache that will be transferred to the per-CPU cache if it  is  empty,  or  the	number	of
       objects to be returned to the global cache if the per-CPU cache is full.

       If  both  slab  cache  statistics  and SMP are defined, there will be four additional columns, reporting the per-CPU cache statistics.  The
       first two are the per-CPU cache allocation hit and miss counts: the number of times an object was or was not available in the per-CPU cache
       for  allocation.   The  next  two are the per-CPU cache free hit and miss counts: the number of times a freed object could or could not fit
       within the per-CPU cache limit, before flushing objects to the global cache.

       It is possible to tune the SMP per-CPU slab cache limit and batchcount via:

	   echo "cache_name limit batchcount" > /proc/slabinfo

FILES

       <linux/slab.h>

VERSIONS

       /proc/slabinfo exists since Linux 2.1.23.  SMP per-CPU caches exist since Linux 2.4.0-test3.

NOTES

       Since Linux 2.6.16 the file /proc/slabinfo is present only if the CONFIG_SLAB kernel configuration option is enabled.

COLOPHON

       This page is part of release 3.53 of the Linux man-pages project.  A description of the project, and information about reporting bugs,  can
       be found at http://www.kernel.org/doc/man-pages/.

								    2007-09-30							       SLABINFO(5)
All times are GMT -4. The time now is 12:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy