Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How do you buffer streamed lines?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How do you buffer streamed lines?
# 1  
Old 07-10-2007
How do you buffer streamed lines?
If I am running a cat|awk|sed on a file to get a bunch of numbers between 1 and 500 billion, how can I stay in stream, and still compare "this value" with "the previous value"?

Really I could do this all in Perl, but if I can get it to work in-stream, its gonna be TONS faster... even if I can get it to work in 2 stream calls its still faster...

My problem is that I dont know how to store the line from the stream in a variable in BASH, then grab the next line to compare to the last... and I tried searching for it, but I have no idea what that would be called

Any help would be greatly appreciated, thanks

- jinno
# 2  
Old 07-10-2007
post example input, and what you want to compare between the two lines.

A few lines should suffice.
# 3  
Old 07-10-2007
Input file looks like:

Code:
...
2007-07-04 04:01:48.364|Test123
2007-07-04 04:01:48.970|Test124
2007-07-04 04:01:55.973|Test125
2007-07-04 04:01:56.029|Test126
...

right now I turn the time into msec for comparison (although this unfortunately does not account for days/months/years...

Code:
...
14508364
14508970
14515973
14516029
...

ideally I would like to turn the timestamp into msec, and compare like:

Code:
...
14508364-... = ...
14508970-14508364 = 606 //GOOD because less than X=1000
14515973-14508970 = 7003 //BAD because greater than X
14516029-14515973 = 56 //GOOD because less than X
...

thus I would want the output to be sent to a file like this:

Code:
...

ERR: Line (5510683)
2007-07-04 04:01:55.973|Test125

ERR:
...

# 4  
Old 07-10-2007
so far I have the quasi-accurate millisecond readings from doing this:

Code:
cat myFile.log | awk -F '|' '{ print $1 }' | sed 's/[^0-9]/:/g' | awk -F ':' '{ print ((3600000*$4)+(60000*$5)+(1000*$6)+$7) }'

# 5  
Old 07-10-2007
Also what OS is this?
# 6  
Old 07-10-2007
Quote:
Originally Posted by reborg
Also what OS is this?
I believe it is FC4
# 7  
Old 07-10-2007
Excellent, that means you have gnu date. Completely untested, but should get you started.
Code:
previous=0
while IFS="[.|]" read datetime milli test; do
    milliseconds=$(date -d "$datetime" +%s)$milli
    if [[ $previous -ne 0 ]] ; then
        (( difference = milliseconds - previous ))
    else
        difference=0
    fi
    if [[ $difference -gt 1000 ]] ; then
       # failed so ouput what you will
       echo $test failed > some_other_file
    fi
    previous=$milliseconds
done < file
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Programming

Separate input buffer into multiple lines

Hi All I have a very tricky problem, not able to solve it. Hence asking this question. I have a code portion like this - int parse_msg(in_buf,line1,line2,sccs_line) char in_buf; char line1; char line2; char sccs_line; { .... (void)fprintf(trace_fp,"parse_msg1 in_buf = %s \n",... (0 Replies)
Discussion started by: nsinha
0 Replies

2. Shell Programming and Scripting

Expect_out(buffer) works but it doesn't get all lines

Hello "expect" experts I am new at Expect. I have searched for a little while how to capture multiple lines with Expect and I am almost succeeded on that but I don't get all the lines of a command's output that the script executes on a server. Here is how my script works in a nutshell - ... (6 Replies)
Discussion started by: capacho6666
6 Replies

3. Programming

buffer in C

Hello, size_t write(int fd, const void *buf, size_t count) { static size_t (*write_func)(int, const void *, size_t) = NULL; if (!write_func) write_func = (size_t(*)(int, const void *, size_t)) dlsym(RTLD_NEXT, "write"); char tmp; memcpy(tmp,buf,count); ... (3 Replies)
Discussion started by: chercheur857
3 Replies

4. UNIX for Dummies Questions & Answers

vi next buffer

Hello, I am using vi to edit file vi filea :e fileb and :e# to switch between filea and fileb Now, I'd like to have many files open at the same time and have a way to cycle between them. :bn does not work; when I type it, nothing happens... Is there something to add to the... (1 Reply)
Discussion started by: JCR
1 Replies

5. Shell Programming and Scripting

How to Strip lines off Streamed EDI Output

Attached is a streamed EDI ANSI X12 output where the segment terminator/delimiter is a tilde ~ character. Is it possible to do the following pseudo-code in a unix script (using either sed, awk and/or grep)? Open file StreamedOutput.txt Search for ISA and delete the data up to the tilde ~ char... (7 Replies)
Discussion started by: sapedi
7 Replies

6. Shell Programming and Scripting

Increase the buffer size to read lengthy lines

Hi All, I am trying to read output from a command. The output format is as follows: Thursday 13 Mar 2008 Information This is sample text Friday 14 Mar 2008 Warning This is one more sample text First line contains informtation (date etc) and the 2nd line contains some information. ... (3 Replies)
Discussion started by: ssunda6
3 Replies

7. Shell Programming and Scripting

Using sed buffer

Hi. I have some questions about using sed. I cannot use the hold buffer. For example i want to put first line to the buffer than take second line and append the buffer to the second line.then 3th to the all. It will be like 2->1->3 th lines. Any idea? (1 Reply)
Discussion started by: burakkilic
1 Replies

8. UNIX for Advanced & Expert Users

vm and buffer cache

i have a serious doubht about the assignment of memory in hp-ux system . i read from somewhere that the page allocation in hp-ux is not unified unlike compaq . i wanted to know in hp-ux kernel ,once the pages are assigned for the univarsal buffer cache... (2 Replies)
Discussion started by: vish_shan
2 Replies

9. UNIX for Advanced & Expert Users

About Buffer

Hi, if someone now how can look the last commands has used for last week? (1 Reply)
Discussion started by: Niko
1 Replies
Login or Register to Ask a Question