Programming

This might be a silly question but I was wondering if, in case of huge files (2-3GB), it is more efficient to map the whole file at once, or to map it page by page.

The file has to be processed sequentially from the start to the end.

Thanks.

could you explain more about what do you mean by map the file?

if you want to process all records / lines in the file, pagination won't help (i've never done such a thing). But depends on what you mean by page by page. If you don't have enough memory, and want to split the file in smaller chunks of a few megabytes, process each one, and then combine results, i haven't tried that myself.

instead, my approach would be to reduce the data set to minimal, and then process it (whether this can ve done or not depends on actual data that you have)

Basically I have this file, whose format is standardized and I cannot touch, and I have to (pre)process it as fast as possible. Now, the file is pretty huge (magnitude of GB), and I can either use simple read()/fread() or mapping the file (mmap()) . By processing, I mean that I have to extract statical data, plus constructing some search indexes.

By mapping the file, I meant memory mapping (i.e. mmap) the whole file at once, or the file page by page, or n pages at the time.

Thanks.

You want to map the minimum number of pages into the address space of the user process so as not to cause any swapping and that totally depends on how the program has been coded. I would start with a few pages at a time instead of the whole file...though you may find that mapping the entire file into memory may work very well and from that you can see that it is a matter of trail and error.

Could you be more clear about "how the program has been coded" ?
What do you mean exactly?

As I said in my last post it is a matter of trial and error. There are a few things to note before trying to mmap the file into the address space of the process. Check how much free (physical not virtual) memory is available. If free physical memory is less than the size of entire file then loading the entire file will create swapping which is undesirable. In this case it is better to load a few pages at a time and see the overall health of the system. So it all depends on how you have coded the mmap call i.e. what parameters you are passing to it and most important is the last one which should be a multiple of the pagesize on your system.

If you are doing a straight sequential read through the file, and not bouncing back and forth among records, consider what Steven's "Advanced Programming in the UNIX Environment" says about buffering files and I/O throughput - basically that a buffer in the size of 16K-32K (set with setvbuf() ) provides the best throughtput on the systems that Rago( the current author) tested on a pretty large file.

Consider sequential I/O first, then mapping second. Both have strong points. The reason sequential I/O does well is that most intelligent disk controllers prefetch several large data blocks, so that there is greatly reduced I/O wait times.

Mapping is reall great if your program references, say record #92, then #40000, then back to #91 - an Applications like a sort or maybe a binary search. On systems with huge amounts of memory it is also the fastest possible way to read a file. But - if your mmap starts using virtual memory (ie swap space), then you lose the speed advantage. Swapping overhead is disk I/O by another name.