Maximum buffer size for read()


 
Thread Tools Search this Thread
Top Forums Programming Maximum buffer size for read()
# 1  
Old 02-24-2012
Maximum buffer size for read()

Hi friends,
Hope everybody is fine. First have a look at my code, then we will talk about it.


Code:
 
$ cat copy.c
#include <stdio.h>
#define PERMS 0644 /*  RW for owner, R for group, others */
#define BUFSIZE 1
char *progname;
int main(int argc,char * argv[])
{
        int f1, f2, n;
        char buf[BUFSIZE];
        progname = argv[0];
        if(argc != 3)
        printf("Usage %s from to\n", progname);
        f1 = open(argv[1], 0);
        if(f1 == -1)
        {
        printf("Can't open %s\n", argv[1]);
        }
        f2 = creat(argv[2], 777);
        if(f2 == -1)
        {
        printf("Can't create %s\n", argv[2]);
        }
        while((n = read(f1, buf, BUFSIZE)) > 0)
        {
                printf("%c\n", buf[0]);
                if(write(f2, buf, n) != n)
                {
                printf("Write error\n");
                }
        }
        exit(0);
}

As you can see I've defined a macro bufsize 1. The value 1 is used for my buffer. My question is, what is the unit of 1, bytes, kilobytes, megabytes? Because, the file that I am reading is very lage, containing thousands of lines, how come it copies that large file into another file. Initially, I had set bufsize as 512, then i set it to 1, still the program works fine, copying the large file into another file.
Could you please help me here?

Looking forward to your wonderful replies.

Thanks in advance!
# 2  
Old 02-24-2012
read and write both use (int file descriptor, memory buffer, size_t bytes) and they return ssize_t bytes read. You DO NOT WANT a 1 byte buffer for read except in very special cases. Modern disks have large fetch buffers, modern filesystems read and write really big chunks. All because it is a more efficient use of resources.

Example: Compellent SAN storage uses a preferred internal block size of 2 MB. The ufs filesystem has a default buffer size of 1MB.

So if you periodically read 1 byte at a time, after a while the rest of the data that came in with your original read request is flushed from the cache and has to be read again.

Look at it this way, you are asking the system to pitch 99.99999% of every read when you request 1 byte, then twiddle your vitrual thumbs for a while. Then come back for byte #2.
# 3  
Old 02-24-2012
thanks a lot Smilie
# 4  
Old 02-24-2012
I recall a user complaining that a change to the linux kernel reduced the maximum read size to 4 gigabytes. He thought this a large inconvenience for some reason. So whatever the maximum read size is, it's going to be far larger than whatever you're doing...

Of course, it was never safe for him to assume that read() did it all in one go in the first place. Lots of things can happen. You have done correctly by checking read's return value here.
# 5  
Old 02-24-2012
The readall function below is what corona is talking about, example code:
Code:
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/stat.h>
#include <errno.h>

void  check_null( void *x ) 
{ 
	 if( x==NULL) 
	    { perror("Fatal error"); exit(1);}
}
ssize_t readall(int fd, char *buf, size_t bytes)
 {
     ssize_t bytes_read = 0;
     ssize_t n=0;

     do {
         if ((n = read(fd,
                       &buf[bytes_read],
                       bytes - bytes_read)) == -1)
         {
             if (errno == EINTR)  // resume on INTR
                 continue;
             else
                 return -1;
         }
         if (n == 0)
             return bytes_read;
         bytes_read += n;
     } while (bytes_read < bytes);
     return bytes_read;
 }

 // example use:

size_t filesize(const char *fname)
{
	  struct stat st;
	  if(stat(fname, &st) == -1)
	  {
	  	 fprintf(stderr, "cannot stat %s ", fname);
	  	 perror("");
	  	 exit(1);
	  }
	  return st.st_size;
}

int main(int argc, char **argv)
{
	  if(argc>1)
	  {
	  	 int i=1;
	  	 for( ;argv[i]!=NULL; i++)
	  	 {
	  	 	   size_t len=filesize(argv[i]);
	  	     char *buf=malloc(len);
	  	     FILE *in=fopen(argv[i], "r");	
	  	     check_null(buf);
	  	     check_null(in);
	  	     readall(fileno(in), buf, len);
	  	     printf("Read whole file %s: %u bytes\n", argv[i], len);
	  	     free(buf);	  
	  	     fclose(in);	     
	  	 }
	  	 return 0;
	  }
	  return 1;
}

example run:
Code:
> time readall bp.dat plpl 7084231.gpx.big before.txt after.txt
Read whole file bp.dat: 6962739 bytes
Read whole file plpl: 6800010 bytes
Read whole file 7084231.gpx.big: 1728412 bytes
Read whole file before.txt: 1584865 bytes
Read whole file after.txt: 1584892 bytes

real    0m0.362s
user    0m0.001s
sys     0m0.065s

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

[c] How to calculate size of the file from size of the buffer?

Hi, Can I find size of the file from size of the buffer written? nbECRITS = fwrite(strstr(data->buffer, ";") + 1, sizeof(char), (data->buffsize) - LEN_NOM_FIC, fic_sortie); Thank You :) (1 Reply)
Discussion started by: ezee
1 Replies

2. UNIX for Dummies Questions & Answers

Decrease buffer size

Hi, I am using the below command to get the output in a file called "Logs.txt" tail -f filename | egrep -i "cpu | hung " >> Logs.txt The problem is the Logs.txt file gets updated only after the buffer is 8Kb, but i want to update the file immediately and not wait for the buffer to get 8kb. Is... (8 Replies)
Discussion started by: @bhi
8 Replies

3. Programming

Cannot read a file with read(fd, buffer, buffersize) function

# include <stdio.h> # include <fcntl.h> # include <stdlib.h> # include <sys/stat.h> int main(int argc, char *argv) { int fRead, fPadded, padVal; int btRead; int BUFFSIZE = 512; char buff; if (argc != 4) { printf ("Please provide all of the... (3 Replies)
Discussion started by: naranja18she
3 Replies

4. UNIX for Dummies Questions & Answers

Maximum file size ????

Hi All, - block size of 512KB & every address requires 4 bits - The inode structure contains 10 direct pointers, 1 single indirect, 1 double indirect & 1 triple indirect pointer What could be the possible maximum file size for this system Any guess? I am unable to understand the question... (0 Replies)
Discussion started by: preethgideon
0 Replies

5. Shell Programming and Scripting

Increase the buffer size to read lengthy lines

Hi All, I am trying to read output from a command. The output format is as follows: Thursday 13 Mar 2008 Information This is sample text Friday 14 Mar 2008 Warning This is one more sample text First line contains informtation (date etc) and the 2nd line contains some information. ... (3 Replies)
Discussion started by: ssunda6
3 Replies

6. UNIX for Dummies Questions & Answers

Maximum size of a file in unix

What's the maximum file size supported by unix. (3 Replies)
Discussion started by: nagalenoj
3 Replies

7. Red Hat

buffer cache size

hi everyone, can any one help change the buffer cache size in redhat and suse?? this error i got when i installed oracle 10g and it went well and when i try to mount the database using startup cmd it says too many buffer cache parameters (error code : ora-1034) thnq in advance (0 Replies)
Discussion started by: gsr_kashyap
0 Replies

8. AIX

Pipe Buffer Size

Hi:- One of our users is getting an error: "There is no process to read data written to a pipe.” I am trying to find out what the pipe buffer size is currently set to. How do I go about this? Thanks, (0 Replies)
Discussion started by: janet
0 Replies

9. Programming

Maximum File Size

Hi, When i checked for the maximum file size on solaris 5.9 the max file size obtained was only 2147483647 and all the further writes to the file which had reached that max size is not added to that file. even i had registered the signal SIGXFSZ, but the signal was not delivered to the... (5 Replies)
Discussion started by: matrixmadhan
5 Replies

10. Programming

Using fread if the buffer size is not known

Hi... I am trying to read a binary data that have different types of messages of different lengths. I am using fread() but this functions needs the size and count to read the buffer from the file. I think this may cause that the buffer overlaps other messages. Is there an alternative to read... (1 Reply)
Discussion started by: jlrodz
1 Replies
Login or Register to Ask a Question