You could also create a new file (base the file name on process_id, username, or session_id, and time of day) each time the original program does a write, then have the report program process and delete the file.
Or, "tee" the original output to two files, so that one is a log file, then keep track of the last record number processed in the log file.
Your suggestion of fifo is allowing to create a back up file which will store data and allows me to keep the data and process it which is not what I am after. All I want is to read from one and write to output file.
That sounds like a trivial modification of what I already gave you... If you don't want to process it -- don't.
Code:
while true
do
cat /path/to/output
done > logfile
instead of replacing /path/to/output all the time, output will be appended to logfile.
Unix tee command looks interesting, except one question to be able to use this command I will need to have a output file to stores all my outputs then using the tee command process this file.
Am right or have I got this wrong? Also If I have a file which collects all the data and then process the file every minute using the AWK code, is there a away to delete the data that has been processed then do the next minute data and so on.
Sorry you guy’s you be bothering you all with my issue and taking a lot of your time.
tee doesn't have to write to a file necessarily. What it does, depends what you do with it.
You've had answer after answer thrown at you and keep saying "that doesn't do what I want".
You are not helping clarify what you do want, or we might have been able to give you an answer a week ago. I suspect some of them might actually do what you want already.
If my fifo example does anything -- it's still unclear whether you even tried -- I continue to think it would be a good foundation. It would allow you to capture the data without timing problems. tee cannot be directly used on a file that keeps replacing itself but could be used once it's put together with the fifo, etc, etc.
Once we finally figure out what you actually do want, changing it to fit shouldn't be hard.
Last edited by Corona688; 12-30-2011 at 08:12 PM..
After following this thread for a week my view is unchanged. You cannot capture transient data reliably from transient data files using standard unix tools.
Imho 1.The system design needs attention. In particular the application which is writing these data files. You get a completely different scenario if the application writes files with unique file names and closes each file after processing.
Imho 2. This processing should be in the primary system. Too many times we find complex data processing in Shell tools which should be done in a high level database language.
Throughout this thread we have learnt absolutely nothing about the process which writes these transient data files or the reason for intercepting the files and we have seen no sign of test code, formal testing, and the results of such testing.
Imho 3. This thread is a complete waste of technical resources.
Out put example (Time is current time)
8:30,Male,23,1
8:31,Female,23,1
8:32,Female,30,4
8:33,Male,50,10
I have an AWK code which parse the xml message and does the counts then writes to output file. But the only problem with this is that the data is in temporary file and I need to capture rapidly changing data from this temporary file.
AWK code
Code:
awk '
{
gsub( ">", "" ); # strip uneeded junk and make "foo bar" easy to capture
gsub( " ", "~" );
gsub( "<", " " );
for( i = 1; i <= NF; i++ ) # snarf up each name=value pair
{
if( split( $(i), a, "=" ) == 2 )
{
gsub( "\"", "", a[2] );
gsub( "~", " ", a[2] );
values[a[1]] = a[2];
}
}
#gcount[values["Gender"]]++; # collect counts
#acount[values["Age"]]++;
agcount[values["Gender"]","values["Age"]]++;
printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
}
END {
printf( "\nSummary\n" );
for( x in agcount )
printf( "%s,%d\n", x, agcount[x] ) | "sort";
}
' input-file
If I understood the fifo approach right then this what I did.
Create a backup.tar
Code:
tar -cpf /path/to/backup.tar /path/to/outputfile
Then create outputfile
Code:
mkfifo /path/to/outputfile
After using while do loop process the messages
Code:
While true Do
awk '
{
gsub( ">", "" ); # strip uneeded junk and make "foo bar" easy to capture
gsub( " ", "~" );
gsub( "<", " " );
for( i = 1; i <= NF; i++ ) # snarf up each name=value pair
{
if( split( $(i), a, "=" ) == 2 )
{
gsub( "\"", "", a[2] );
gsub( "~", " ", a[2] );
values[a[1]] = a[2];
}
}
#gcount[values["Gender"]]++; # collect counts
#acount[values["Age"]]++;
agcount[values["Gender"]","values["Age"]]++;
printf( "%s %s %s %s\n", values["NAME"], values["Age"], values["D.O.B"], values["Gender"] );
}
END {
printf( "\nSummary\n" );
for( x in agcount )
printf( "%s,%d\n", x, agcount[x] ) | "sort";
}
' input-file
done
But I am not sure about this
Code:
while true
do
cat /path/to/output > /tmp/$$
echo "Received data from /path/to/output"
# process /tmp/$$ as you please since it won't vanish from under you
done
Should I write the data from the feed to the backup file, then using the AWK code read from backup file then to the output file and sleep 10 seconds? Same process again and again.
My only problem with this is writing to different files which mean I will have all these data in different files which I don’t need to keep.
If I got this process wrong, I will be grateful if you could with the right process.
Thank you once again and Happy New Year
Last edited by James_Owen; 12-31-2011 at 07:56 PM..
I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues:
1. Monitor all Processes parallelly.... (5 Replies)
Hi All,
Need your help on how i can sum up the values. I have a file that contains the count and the time. I wanted to add up all the first column having the same datestamp. Please see below.
INPUT
1721 2015-12-26 00:01
1440 2015-12-26 00:02
1477 2015-12-26 00:02
411 ... (4 Replies)
How to check log size every 10min. by script (can use crontab)
if log size not change with alert "Log not update"
Base run on SunOS 5.8 Generic_Virtual sun4u sparc SUNW,SPARC-Enterprise
logFiles="log1.log log2.log"
logLocation="/usr/home/test/log/"
Out put.
Tue Jan 31... (3 Replies)
Hi, I've been trying to develop a script that performs the parsing of a log every 1 minute and then generating some statistics. I'm fairly new to programming and this is why I come to ask if I can lend a hand.
this is my log:
xxxx 16/04/2012 17:00:52 - xxxx714 - E234 - Time= 119 ms.... (8 Replies)
Anyone knows how to use AWK to achieve the following
Sun Feb 12 00:41:01-00:41:59 Success:2 Fail:2
Sun Feb 12 00:42:01-00:42:59 Success:1 Fail:2
Sun Feb 12 01:20:01-01:20:59 Success:1 Fail:2
Mon Feb 13 22:41:01-22:41:59 Success:1 Fail:1
log file:
Success
Success
Fail
Fail
... (9 Replies)
Hello All,
i am a newbie and need some help when reading a csv file in a bourne shell script. I want to read 10 lines, then wait for a minute and then do a reading of another 10 lines and so on in the same way. I want to do this till the end of file.
Any inputs are appreciated
... (3 Replies)
Is there a simple command to find or check if a file has been modified within the last minute? The mtime parameter from find cmd only allow days.
I am trying to avoid timestamp parsing and do some sort of comparison as I'm real beginner at scripts. If this is the only way, any help is greatly... (3 Replies)
here's the problem, i have two processes that i need to run and both process should be run at a total of 1 minute each. how do i do that? and one more here's what the processes do: the 1st process show the '+" sign infinitely while the 2nd process displays the "-" infinitely. how could i count the... (1 Reply)
12-30-2011
