Sponsored Content
Full Discussion: Fail tail algorithm
Top Forums Programming Fail tail algorithm Post 90542 by foureightyeast on Wednesday 23rd of November 2005 11:41:12 AM
Old 11-23-2005
Thanks all for replying! I actually wasn't too concerned anymore about losing some of the data, since after several weeks of prototyping and testing under extreme conditions, I only found that we would hit the issue of reading an incomplete line about 1 hour if the rate of writing to the file was around 2 Meg/10 seconds or 1 Meg in 5 seconds by blasting the third party code as hard as I could. So the chances of my hitting this now after all the "workarounds" I put in, is close to non-existent, since, hopefully, this much traffic won't be going through (otherwise, there would be a nice CPU spike from that process).

What I guess I was more interested in, is the algorithm of keeping track of the end of file by polling an already open file buffer every X seconds better than say, another algorithm where the file pointer were closed, the current file position saved off, delay, open again, seek to that position and check what's there, and repeat? I suppose in the main loop I could delay as much as 5 seconds before checking, but I was wondering too what was tail's algorithm exactly? I would like to try to mimic that functionality as close as possible.

I have another inner loop while there is activity where I read lines from the file, format them to the screen until I hit EOF, then I go back to the main polling loop above where I keep waiting until EOF has moved. But in the inner while loop is where things can spike if there are a lot of reads at once and I don't get a chance to come up for "air". To work-around that, I put in another 10 ms delay after so many reads (around an average of 150 lines, I use fgets, so my buffered reading is per line, rather than reading a chunk of fixed-length data with fread()). I've found tinkering with this inner loop to be the hardest to get as optimal as possible. That is, giving delays that keep the CPU spike down cause the code to take longer to catch up. Giving smaller delays that allows the code to catch up to EOF faster gives a spike. Someone, I've noticed when using UNIX tail and top, that tail never shows up on the top list. So I'm curious how tail manages to keep up so nicely without spiking the CPU.

--Chris
 

9 More Discussions You Might Find Interesting

1. Programming

Feedback algorithm

Hi I search an exemple of scheduling Feedback algorithm, or help about how to create one. Thanks (0 Replies)
Discussion started by: messier79
0 Replies

2. Programming

Algorithm problem

Looking for an algorithm to compute the number of days between two given dates I came across a professor's C program located here: http://cr.yp.to/2001-275/struct1.c I was wondering if anyone could tell me where the value 678882 in the line int d = dateday - 678882; comes from and also the... (1 Reply)
Discussion started by: williamf
1 Replies

3. Programming

FTP's algorithm

what algorithm a FTP application uses i mean whn implemented in socket programming..if you could give a little decription (1 Reply)
Discussion started by: toughguy2handle
1 Replies

4. Shell Programming and Scripting

algorithm

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 21444 tomusr 213M 61M sleep 29 10 1:20:46 0.1% java/43 21249 root 93M 44M sleep 29 10 1:07:19 0.2% java/56 is there anyway i can use a command to get the total of the SIZE? 306M (Derive from... (5 Replies)
Discussion started by: filthymonk
5 Replies

5. Programming

Please help me to develop algorithm

Hi guys , in my study book from which I re-learn C is task to generate all possible characters combination from numbers entered by the user. I know this algorithm must use combinatorics to calculate all permutations. Problem is how to implement algortihm. // This program reads the four numbers... (0 Replies)
Discussion started by: solaris_user
0 Replies

6. Programming

Looking for Your Help on dijkstra algorithm

Can you help to adjust the void dijkstra(int s) function to find a path from source to every node so that the minimum cost on that path is maximum. Ex: From 1 to 2 we have 1 - 3 - 4 - 2 , costs(2+3+4+5) From 1 to 2 we have 1 - 5 - 6 - 2 , costs(3+3+4+5) I need the algorithm to choose path 1... (4 Replies)
Discussion started by: ali2011
4 Replies

7. Shell Programming and Scripting

Joining multiple files tail on tail

I have 250 files that have 16 columns each - all numbered as follows stat.1000, stat.1001, stat.1002, stat.1003....stat.1250. I would like to join all 250 of them together tail by tail as follows. For example stat.1000 a b c d e f stat.1001 g h i j k l So that my output... (2 Replies)
Discussion started by: kayak
2 Replies

8. Shell Programming and Scripting

Masking algorithm

I have a requirement of masking few specific fields in the UNIX file. The details are as following- File is fixed length file with each record of 250 charater length. 2 fields needs to be masked – the positions are 21:30 and 110:120 The character by character making needs to be done which... (5 Replies)
Discussion started by: n78298
5 Replies

9. Shell Programming and Scripting

When redirecting tail -F output fail after log rotation?

Redirecting the tail output fails when log rotation happens even though i have used -F. But when i tail and see the output on terminal this does not happen. Note i have also used & to execute this statement in background. Suppose if i want to tail a file /opt/SAMPLE.txt and redirect its output... (1 Reply)
Discussion started by: CN1
1 Replies
TAIL(1) 						    BSD General Commands Manual 						   TAIL(1)

NAME
tail -- display the last part of a file SYNOPSIS
tail [-F | -f | -r] [-b number | -c number | -n number] [file ...] DESCRIPTION
The tail utility displays the contents of file or, by default, its standard input, to the standard output. The display begins at a byte, line or 512-byte block location in the input. Numbers having a leading plus (``+'') sign are relative to the beginning of the input, for example, ``-c +2'' starts the display at the second byte of the input. Numbers having a leading minus (``-'') sign or no explicit sign are relative to the end of the input, for example, ``-n 2'' displays the last two lines of the input. The default starting location is ``-n 10'', or the last 10 lines of the input. The options are as follows: -b number The location is number 512-byte blocks. -c number The location is number bytes. -f The -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the input. The -f option is ignored if the standard input is a pipe, but not if it is a FIFO. -F The -F option implies the -f option, but tail will also check to see if the file being followed has been renamed or rotated. The file is closed and reopened when tail detects that the filename being read from has a new inode number. The -F option is ignored if reading from standard input rather than a file. -n number The location is number lines. -r The -r option causes the input to be displayed in reverse order, by line. Additionally, this option changes the meaning of the -b, -c and -n options. When the -r option is specified, these options specify the number of bytes, lines or 512-byte blocks to display, instead of the bytes, lines or blocks from the beginning or end of the input from which to begin the display. The default for the -r option is to display all of the input. If more than a single file is specified, each file is preceded by a header consisting of the string ``==> XXX <=='' where ``XXX'' is the name of the file. DIAGNOSTICS
The tail utility exits 0 on success, and >0 if an error occurs. SEE ALSO
cat(1), head(1), sed(1) STANDARDS
The tail utility is expected to be a superset of the IEEE Std 1003.2-1992 (``POSIX.2'') specification. In particular, the -F, -b and -r options are extensions to that standard. The historic command line syntax of tail is supported by this implementation. The only difference between this implementation and historic versions of tail, once the command line syntax translation has been done, is that the -b, -c and -n options modify the -r option, i.e. ``-r -c 4'' displays the last 4 characters of the last line of the input, while the historic tail (using the historic syntax ``-4cr'') would ignore the -c option and display the last 4 lines of the input. HISTORY
A tail command appeared in PWB UNIX. BSD
June 6, 1993 BSD
All times are GMT -4. The time now is 11:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy