Hi,

I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)

Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).

The messages that I write into the file, comes all at once from a socket, so
in oder to write it quickly without loosing any of them I plan to do the following:

- Create a list of preallocated pages (3-4 by default but the list grows if needed)
- Write the data that comes from the socket to the preallocated buffer
- Once a page is full, schedule a write with aio_write (AIO - Asyncronous IO).
- On callback schedule another one if any is full.
And so on..

This is the best I could come up with in the writing part, but if any of you have
a better idea, please let me know.

Now the problem comes when I have to read the file, at a later time, and display all
the messages in order to analyze them as fast as possible.

I first thought of mmap'ing the file in order to copy the data only once, from the file
to the kernel cache (if I understood correctly how mmap works internally) and then
accessing it from the application. But I'm not sure this can be done, or convenient, as
the file might be pretty big (2,3 Giga bytes, although I'm not sure about the magnitude).
Beside the kernel could unload the pages and many page faults could occurred.
So I discarded this idea.

I also thought about the opposite of what I do for writing but I'm not sure is a good idea.

The main problem is that I have to decode the messages before displaying them, as they are of different type and variable length. So reading the whole file at once and then decoding them to copy to another memory location seem time consuming to me. As it
requires 4 copies (disk -> kernel -> user space -> user space after decoding).

Anyway, now it's your turn. :-)
Any help would be appreciated.

Thanks.