Shell Programming and Scripting

i have a file that's about 2GB, i have to get the total number of lines in this file every 10 minutes.

the interval is not an issue. i just need the proper, most efficient way to do this.

any ideas?

i got the following from another thread on this site, but:

Code:

awk 'int(100*rand())%5<1' file

but this randomly pulls out 20% of lines in a file. i'm thinking this code can be slightly modified to get what i want?

Code:

wc -l filename

i'd rather not use "wc -l". it is not efficient at all to be used on a file size of 2GB. i'm hoping there's a better way to get the total line number of a file that big.

Code:

awk 'END {print NR}' filename

You do realize, don't you, that in order to count the number of <newline> characters in a file you have to read the entire file?

If lines are being added to the file constantly, but existing data in the file isn't changing, you could save the line count and file size at the time of your last check. Then when the next size check occurs, you could just read the new data and count the added lines and add the results to your previous count. If you're really concerned about efficiency, you probably want to write this in C or C++ rather than a shell script.

Sky Smart - can you cite a verified reference as to why reading a file from record 1 to EOF is NOT the most efficient for carriage control files. rdrtx1's sample code does that and so does wc -l

I'll answer:
No such valid reference exists. You have to count the number of \n characters to get a line count. The only other possibility is for a fixed length record file. In that case you call stat, ls, or some code you have to get the number of bytes (reading file metadata: struct stat st_size) and then do integer division: bytes/recsz.

One other ' reliable' way is to call x=ftell() on the end of a file when you know the file has finished being written and divide x/recsz - again for fixed record length files. This is an even less efficient way to do stat.


NOTHING else exists. In other words: how can you know how many \n characters exist in the file?

when you run a "wc -l" on a file that big, it takes a while to get the total line count. i understand that the entire file must be read in order to get the total lines.

i'm also aware that in UNIX, there are more than one ways to get something done. on some linux systems a "grep -P" will get you want you want a lot faster than any other utility can. on others, the -P option is not available.

overall, i'm more concerned about speed and what the quickest way is to get total line count on a file that big.