UNIX for Advanced & Expert Users

Two things

Read over the man page for lseek64 and make sure you are using the correct compiler flags. i do not believe you are.

Secondly, The read system call does guarantee that it will fill a buffer completely. Trying issuing a second read and see if you get your data.

Thanks for all your replies,
but it seems that its not possible to use posix read to read files larger than 2.1 gig.
This is according to the uberpenguin linus
http://git.kernel.org/?p=linux/kerne...iff;h=e28cc715

At somepoint I guess it was possible but the kernel has regressed to only allow smaller chunks being read.

kernelsourcetree/fs/read_write.c
This was rather annoying, it would have been nice if this had been documented outside of the kernelsource and kernelmailinglist.

Wait -- you're trying to read >2gb all at once?? Do you actually need it all at once? If not, this is extremely bad programming practice!

Why not just map in the file? That'll turn it into a memory area with one call, letting you use it like you'd loaded it all at once without actually having to do so, it'll autoload it as you use it(though you can force it to load it all with the MAP_POPULATE flag). See man mmap.

No, it is most definitely *not* an "extremely bad programming practice". I've worked on real-world systems that access tens if not hundreds of gigabytes at a time. In RAM. They *do* exist.

This code:

Code:

/*
 * rw_verify_area doesn't like huge counts. We limit
 * them to something that fits in "int" so that others
 * won't have to do range checks all the time.
 */
#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)

int rw_verify_area(int read_write, struct file *file, loff_t *ppos, size_t count);  

is nothing more nor less than utter abdication of 64-bit processing.

They certainly do exist, hence my asking whether he actually needed it all in RAM at once. Nine out of ten times I see someone slurping an enormous wad of data into RAM, they'd have been better off doing an mmap...

achenle - just becuase you saw somebody do something does not make it good practice.
Even in commerical software.

My company bought a package (C/shell/perl) that had non-terminating loops in it.
Does that mean it was good practice - or good software? We sent it back.

There's a very good reason to read a large amount of data with just one system call: in a contended file system, doing so allows for better scheduling of IO to data blocks that are more likely to be contiguous on disk, thus reducing overhead lost to disk head seeks. And I haven't SEEN it done, I've DONE it. And improved IO throughput on said system by about 20 or 30 percent.

Fortunately for the customer, the OS wasn't Linux, which has punted the ability to do large read and writes.

I guess 2 gig is enough for everybody, right?

Your company buying buggy software is about as relevant as the color of your car.