File - reading - Performance improvement - Page 2 - High Level Programming

dhanamurthy · #8 05-23-2008

Hi
This helps.
But a concern here is that i need to put a while loop in place for reading the bulk characters until i come across "\n" character as my aim is to get line by line from the file.

Thanks for the idea.

Regards
Dhana

seanrowens · #9 06-04-2008

There are two sets of functions for reading data,

open/read/write/gets/close

that operate on file 'handles', and

fopen/fread/fwrite/fgets/fclose

that operate on FILE * 'streams'.

The big advantage of using the streams is that they are buffered whereas the file handles are not. What this means is that for the nonbuffered functions, every time you call read() it has to go out to the physical disk and read some data.

With the buffered functions, it allocates a block of memory internally (I believe 8kb but I'm not sure) and when you call fread() or fgets() it only hits the disk if there isn't enough data already in the buffer. This is much faster.

By the way, you can increase buffer size with setbuf() and you can use fgets() to get the next line (next occurrence of \n) rather than a fixed number of characters.

To get the fastest possible speed, as mentioned above, you would have to use a big buffer, read a large chunk of file at once and then go through it looking for line ends. This avoids extra copying the data, i.e. it's copied from disk into memory and then out again.

But I'd try just using fgets() first as it probably is fast enough.

matrixmadhan · #10 06-04-2008

or what ever be the mode of opening a file, set it to buffered using setvbuf, that should turn on buffering mode

jim mcnamara · #11 06-06-2008

Steven's book on advanced unix programming has a table showing read performance on files.

Since you are returning lines, somewhere down inside the C++ stdio module is calling something like fgets. It does call read() to fill a buffer. Steven's table show that buffer sizes of 4096 are probably close optimum. There are other examples that show using
struct statvfs.f_frsize - the block size of the filesystem in question will also help.

See man setvbuf.

The other components of speed are the i/o queue request length, on board disk caching
and how "far above" the native read call your code operates. The first two are system related. If you call this low-level read routine directly and parse out you own lines it will probably speed things up - use 4096 or f_frsize as the number of bytes to read:

This is taken from M. Rochkind's book - example:

Code:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>

ssize_t readall(int fd, void *buf, size_t nbyte)
{
         ssize_t nread=0;
         ssize_t n=0;

         memset(buf,0x0, nbyte+1);
         do
         {
                 if ((n = read(fd, &((char *)buf)[nread], nbyte - nread)) == -1)
                 {
                         if (errno == EINTR)
                                 continue;
                         else
                                 return (-1);
                 }
                 if (n == 0)
                                 return nread;
                 nread += n;
         } while (nread < nbyte);
         return nread;
}



void foo()
{
    ssize_t result=0;
	char buf[4200]={0x0};
	FILE *fp=fopen("somefile","r");

	if(fp!=NULL)
	{
		result=readall(fileno(fp), buf, 4096);
		if(result>0)
		{
			printf("%s", buf);
		}
	    if (result== (-1))
	    {
	    	perror("file I/O error");
	    	exit(1);
	    }
	}
	else
             {
		perror("file open error");
                          exit(1);
               }
}

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
reading from a file and pass as variables and ignore # in the file	konark	Shell Programming and Scripting	4	11-08-2007 12:55 AM
Announcing collectl - new performance linux performance monitor	MarkSeger	News, Links, Events and Announcements	0	10-26-2007 03:14 PM
Reading file names from a file and executing the relative file from shell script	anushilrai	Shell Programming and Scripting	4	03-10-2006 02:25 AM
performance of writing into a file	shriashishpatil	High Level Programming	1	12-22-2005 07:51 AM
File Upload Performance using IE from Windows to AIX via HTTPS	darontan	AIX	1	10-27-2005 11:04 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode