Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
 |
| Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here. |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Increase the buffer size to read lengthy lines | ssunda6 | Shell Programming and Scripting | 3 | 03-17-2008 04:45 AM |
| Using sed buffer | burakkilic | Shell Programming and Scripting | 1 | 12-14-2007 04:35 PM |
| How to count lines - ignoring blank lines and commented lines | kthatch | UNIX for Dummies Questions & Answers | 6 | 05-25-2007 01:21 AM |
| buffer the output | collins | High Level Programming | 6 | 11-09-2004 02:57 AM |
| About Buffer | Niko | UNIX for Advanced & Expert Users | 1 | 09-18-2001 03:49 AM |
 |
Powered by 
|
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
|
07-10-2007
|
||||
|
||||
|
How do you buffer streamed lines?
If I am running a cat|awk|sed on a file to get a bunch of numbers between 1 and 500 billion, how can I stay in stream, and still compare "this value" with "the previous value"?
Really I could do this all in Perl, but if I can get it to work in-stream, its gonna be TONS faster... even if I can get it to work in 2 stream calls its still faster... My problem is that I dont know how to store the line from the stream in a variable in BASH, then grab the next line to compare to the last... and I tried searching for it, but I have no idea what that would be called Any help would be greatly appreciated, thanks - jinno |
|
07-10-2007
|
||||
|
||||
|
Input file looks like:
Code:
... 2007-07-04 04:01:48.364|Test123 2007-07-04 04:01:48.970|Test124 2007-07-04 04:01:55.973|Test125 2007-07-04 04:01:56.029|Test126 ... Code:
... 14508364 14508970 14515973 14516029 ... Code:
... 14508364-... = ... 14508970-14508364 = 606 //GOOD because less than X=1000 14515973-14508970 = 7003 //BAD because greater than X 14516029-14515973 = 56 //GOOD because less than X ... Code:
... ERR: Line (5510683) 2007-07-04 04:01:55.973|Test125 ERR: ... |
|
07-10-2007
|
||||
|
||||
|
so far I have the quasi-accurate millisecond readings from doing this:
Code:
cat myFile.log | awk -F '|' '{ print $1 }' | sed 's/[^0-9]/:/g' | awk -F ':' '{ print ((3600000*$4)+(60000*$5)+(1000*$6)+$7) }'
|
|
07-19-2007
|
||||
|
||||
|
using an array as a buffer
I dont know if there was a misunderstanding, but I am trying to get the script to buffer the last n lines that were parsed through it... n is a command-line argument
The biggest problem is file size... if I run the while loop that runs the sed, that ends up being n-squared time... which for a ~1GB log file = disaster I wish I could use the solution posted here: Extracting Data From Large Logs but unfortunately I dont have a simple search... instead, I have to first search for a discrepancy and then print out n+/-x lines This is what I have working so far (no buffer implemented): Code:
egrep -n '^[0-9]' $file | \
while IFS="[.|:]" read number datetime min sec milli rest; do
milliseconds=$(date -d "$datetime:$min:$sec" +%s)$milli
line="$number:$datetime:$min:$sec.$milli|$rest"
if [[ $previous -ne 0 ]] ; then
(( difference = milliseconds - previous ))
else
difference=0
fi
if [[ $difference -gt $maxTime ]] ; then
echo $line
fi
previous=$milliseconds
done
|
|
| Bookmarks |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
Hybrid Mode
