Programming

Hi
This helps.
But a concern here is that i need to put a while loop in place for reading the bulk characters until i come across "\n" character as my aim is to get line by line from the file.

Thanks for the idea.

Regards
Dhana

There are two sets of functions for reading data,

open/read/write/gets/close

that operate on file 'handles', and

fopen/fread/fwrite/fgets/fclose

that operate on FILE * 'streams'.

The big advantage of using the streams is that they are buffered whereas the file handles are not. What this means is that for the nonbuffered functions, every time you call read() it has to go out to the physical disk and read some data.

With the buffered functions, it allocates a block of memory internally (I believe 8kb but I'm not sure) and when you call fread() or fgets() it only hits the disk if there isn't enough data already in the buffer. This is much faster.

By the way, you can increase buffer size with setbuf() and you can use fgets() to get the next line (next occurrence of \n) rather than a fixed number of characters.

To get the fastest possible speed, as mentioned above, you would have to use a big buffer, read a large chunk of file at once and then go through it looking for line ends. This avoids extra copying the data, i.e. it's copied from disk into memory and then out again.

But I'd try just using fgets() first as it probably is fast enough.

or what ever be the mode of opening a file, set it to buffered using setvbuf, that should turn on buffering mode

Steven's book on advanced unix programming has a table showing read performance on files.

Since you are returning lines, somewhere down inside the C++ stdio module is calling something like fgets. It does call read() to fill a buffer. Steven's table show that buffer sizes of 4096 are probably close optimum. There are other examples that show using
struct statvfs.f_frsize - the block size of the filesystem in question will also help.

See man setvbuf.

The other components of speed are the i/o queue request length, on board disk caching
and how "far above" the native read call your code operates. The first two are system related. If you call this low-level read routine directly and parse out you own lines it will probably speed things up - use 4096 or f_frsize as the number of bytes to read:

This is taken from M. Rochkind's book - example:

Code:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>

ssize_t readall(int fd, void *buf, size_t nbyte)
{
         ssize_t nread=0;
         ssize_t n=0;

         memset(buf,0x0, nbyte+1);
         do
         {
                 if ((n = read(fd, &((char *)buf)[nread], nbyte - nread)) == -1)
                 {
                         if (errno == EINTR)
                                 continue;
                         else
                                 return (-1);
                 }
                 if (n == 0)
                                 return nread;
                 nread += n;
         } while (nread < nbyte);
         return nread;
}



void foo()
{
    ssize_t result=0;
	char buf[4200]={0x0};
	FILE *fp=fopen("somefile","r");

	if(fp!=NULL)
	{
		result=readall(fileno(fp), buf, 4096);
		if(result>0)
		{
			printf("%s", buf);
		}
	    if (result== (-1))
	    {
	    	perror("file I/O error");
	    	exit(1);
	    }
	}
	else
             {
		perror("file open error");
                          exit(1);
               }
}