Shell Programming and Scripting

Another thing: do you have very limited memory and/or hard disk space? Grepping a file that size should not be a problem on even relatively modest hardware.

Code:

vnix$ dd if=/dev/urandom of=/tmp/randomfile bs=65536 count=65536
^C  # interrupted when I got bored
23196+2 records in
23196+1 records out
1520214016 bytes (1.5 GB) copied, 317.015 s, 4.8 MB/s

vnix$ time grep '^06/01/' /tmp/randomfile 

real    0m51.461s
user    0m1.940s
sys     0m1.768s

This is a basic PATA disk which should be easy to beat if you have SCSI or SATA.

@asth
If it times out, you have another problem - does it time out with an error or just come back to the prompt or do you have to do ctrl+c or something after some minutes of boredom?
It doubt the tools you are using are the problem. Looks like you got a performance issue - something blocking the resources (disks, cpu, memory, whatever).

300-400MB of log is nothing if you simply parse it without layered loops and such stuff, which all examples posted did not - they were not complex.

If i use tail command then does'nt take much time and i get the required output.
I have space issue thats why it is taking time.
But i have to work on these sites only so need to find out a way to divide the file in chunks and then find out the chunk of last 24 hours say.

Thanks

If you cannot read the file from the beginning, there is no way really to know. But based on experimentation you can probably find a value for tail which is likely to cover more than the last 24 hours by a good margin.

Sounds like you ought to be running some sort of rotation script in your nightly cron job to force the log file into smaller chunks. Which platform are you on? Does the application which generates this log support log rotation?