Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Process file every minute

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Process file every minute
# 29  
Old 01-05-2012
Quote:
Originally Posted by James_Owen
Going back to the pipe, now I have a pipe output file as shows below
Code:
prw-r--r--*** 0 Jan* 3 11:00 OutPutFile

Now I am also able to read from the data file and capture the data using the pipe.
You should be reading from the pipe. Your daemon should be writing to the pipe.

Quote:
But this continues process for example if there are 40 records in the pipe then the AWK code loops again and again without stopping and adding to the counts every time.
Then your daemon isn't closing the file after it writes to it.
Quote:
Also if the connection between the pipe and the data file fails, will pipe keep unprocessed data be kept by the pipe until it gets process or will it clear the file?
As explained earlier, if the connection is lost, your daemon ought to freeze until something else starts reading the fifo.

It's also possible the daemon could get killed with SIGPIPE.
Quote:
All I want the script to do is to run for 2 minutes, do calculation for those 2 minutes then sleep for 20 seconds. After 20 seconds start again from the where it was last.
I think you're forgetting that using a fifo means the data gets captured properly without funny timing tricks. This is important because funny timing tricks are all but guaranteed not to work. So I don't understand why the sleep.

Why not just print a total every 2 minutes?

Code:
awk 'BEGIN { INTERVAL=120;        NEXT=strftime("%s")+120; }

    {
        if(strftime("%s") >= NEXT)
        {
           printf( "\nSummary\n" );
           for( x in agcount )
              printf( "%s,%d\n", x, agcount[x] ) | "sort";

           # Clear out old values
           for( x in agcount)
                   delete agcount[x];

           NEXT=strftime("%s")+120;
        }

        gsub( ">", "" );        # strip uneeded junk and make "foo bar" easy to capture
        gsub( " ", "~" );
        gsub( "<", " " );

        for( i = 1; i <= NF; i++ )          # snarf up each name=value pair
        {
            if( split( $(i), a, "=" ) == 2 )
            {
                gsub(  "\"", "", a[2] );
                gsub(  "~", " ", a[2] );
                values[a[1]] = a[2];
            }
        }

        #gcount[values["Gender"]]++;         # collect counts
        #acount[values["Age"]]++;
        agcount[values["Gender"]","values["Age"]]++;

        printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
    }' input-file

Actually... I'm not sure how that code of yours ever worked. You're not incrementing anything inside values, at all, ever.
# 30  
Old 01-07-2012
@Corona688 Thank you a lot.

But I only have one more problem with the strftime function. Smilie

I know strftime is a GNU gawk extension, which I can't use.

Every time I run the script I get this error:
Quote:
Calling undefined function strftime
Is there an alternative to strftime in AWK?

Thank you againSmilie
# 31  
Old 01-07-2012
See if you have systime() in your awk. If yes, replace strftime("%s") with systime()

If not, try this...
Code:
NEXT=strftime("%s")+120;

#replace with

"date +%s"|getline sec;
NEXT=sec+120;

HTH
--ahamed
# 32  
Old 01-09-2012
Thank you ahamed101 Smilie

I have made the changaes as follows:
Code:
awk 'BEGIN { INTERVAL=120;    "date +%s"|getline sec;
    NEXT=sec+120;}

    {
        if(sec >= NEXT)
        {
           printf( "\nSummary\n" );
           for( x in agcount )
              printf( "%s,%d\n", x, agcount[x] ) | "sort";

           # Clear out old values
           for( x in agcount)
                   delete agcount[x];

           NEXT=sec+120;
        }

        gsub( ">", "" );        # strip uneeded junk and make "foo bar" easy to capture
        gsub( " ", "~" );
        gsub( "<", " " );

        for( i = 1; i <= NF; i++ )          # snarf up each name=value pair
        {
            if( split( $(i), a, "=" ) == 2 )
            {
                gsub(  "\"", "", a[2] );
                gsub(  "~", " ", a[2] );
                values[a[1]] = a[2];
            }
        }

        #gcount[values["Gender"]]++;         # collect counts
        #acount[values["Age"]]++;
        agcount[values["Gender"]","values["Age"]]++;

        printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
    }' input-file

However the print out is not printing to the output file, the print out in red will write to the output file, but without the counts. Also unless I am missing something the process only runs for one minute and not two minutes. Smilie
# 33  
Old 01-10-2012
Guys,

One more this, i want to get the current date to be included in the input-file.

I have made this changes in blue to the code

Code:
 
for( x in agcount )
              printf( "%s,%s,%d\n", x, agcount[x] ) | "sort";

  }' "$(date)" > input-file

yet the in the input-file I am getting $(date), as output

Could someone please help with this issue and the other issue too.


thanks Smilie
Last edited by vbe; 01-10-2012 at 04:35 PM.. Reason: dont forget to use code tags
# 34  
Old 01-19-2012
Guy's

I don't want to bump my thread but can I please get help with this problem.

Thanks

James
# 35  
Old 01-20-2012
Quote:
"$(date)"
That looks like Shell syntax but you are writing an awk program. Even though the awk program appears to be running from Shell, the whole program is enclosed in single quote characters which would disable Shell substitution.
I'm no awk expert but comparing with the syntax on the first line, wouldn't it be :
Code:
"date"

But perhaps with the output string somewhere inside the "printf" statement.
Last edited by methyl; 01-20-2012 at 02:21 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Monitoring processes in parallel and process log file after process exits

I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues: 1. Monitor all Processes parallelly.... (5 Replies)
Discussion started by: shunya
5 Replies

2. Shell Programming and Scripting

Grep from file modified one minute ago

Hello, I have a list of files, an example below: -rw-r--r-- 1 smf_oper esg 910773 Jul 6 12:52 am1slc02_ACS_201607061242571_20346.cdr -rw-r--r-- 1 smf_oper esg 995838 Jul 6 12:52 am1slc01_ACS_201607061243125_19895.cdr -rw-r--r-- 1 smf_oper esg 557235 Jul 6 12:52... (5 Replies)
Discussion started by: nms
5 Replies

3. Shell Programming and Scripting

Want to add those that have the same minute in a file

Hi All, Need your help on how i can sum up the values. I have a file that contains the count and the time. I wanted to add up all the first column having the same datestamp. Please see below. INPUT 1721 2015-12-26 00:01 1440 2015-12-26 00:02 1477 2015-12-26 00:02 411 ... (4 Replies)
Discussion started by: ernesto
4 Replies

4. Shell Programming and Scripting

Check log file size every 10 minute. Alert if log not update

How to check log size every 10min. by script (can use crontab) if log size not change with alert "Log not update" Base run on SunOS 5.8 Generic_Virtual sun4u sparc SUNW,SPARC-Enterprise logFiles="log1.log log2.log" logLocation="/usr/home/test/log/" Out put. Tue Jan 31... (3 Replies)
Discussion started by: ooilinlove
3 Replies

5. Shell Programming and Scripting

Take minute per minute from a log awk

Hi, I've been trying to develop a script that performs the parsing of a log every 1 minute and then generating some statistics. I'm fairly new to programming and this is why I come to ask if I can lend a hand. this is my log: xxxx 16/04/2012 17:00:52 - xxxx714 - E234 - Time= 119 ms.... (8 Replies)
Discussion started by: jockx
8 Replies

6. Shell Programming and Scripting

Count of matched pattern occurences by minute and date in a log file

Anyone knows how to use AWK to achieve the following Sun Feb 12 00:41:01-00:41:59 Success:2 Fail:2 Sun Feb 12 00:42:01-00:42:59 Success:1 Fail:2 Sun Feb 12 01:20:01-01:20:59 Success:1 Fail:2 Mon Feb 13 22:41:01-22:41:59 Success:1 Fail:1 log file: Success Success Fail Fail ... (9 Replies)
Discussion started by: timmywong
9 Replies

7. UNIX for Dummies Questions & Answers

When reading a csv file, counter to read 20 lines and wait for minute then read next 20 till end

Hello All, i am a newbie and need some help when reading a csv file in a bourne shell script. I want to read 10 lines, then wait for a minute and then do a reading of another 10 lines and so on in the same way. I want to do this till the end of file. Any inputs are appreciated ... (3 Replies)
Discussion started by: victor.s
3 Replies

8. Shell Programming and Scripting

Ksh Script to get the start minute of n number of process

How can i write a script.? which lists all X process and gets the start minute of each of them. thanks (1 Reply)
Discussion started by: Anteus
1 Replies

9. Shell Programming and Scripting

Determine if file changed within last minute

Is there a simple command to find or check if a file has been modified within the last minute? The mtime parameter from find cmd only allow days. I am trying to avoid timestamp parsing and do some sort of comparison as I'm real beginner at scripts. If this is the only way, any help is greatly... (3 Replies)
Discussion started by: ystee
3 Replies

10. Programming

how can i run a process for a whole minute?

here's the problem, i have two processes that i need to run and both process should be run at a total of 1 minute each. how do i do that? and one more here's what the processes do: the 1st process show the '+" sign infinitely while the 2nd process displays the "-" infinitely. how could i count the... (1 Reply)
Discussion started by: kelogs1347
1 Replies
Login or Register to Ask a Question