Quote:
Originally Posted by
otheus
Wow, what a question. Are you re-engineering a database system?
Nope. I'm just trying to write an application as efficient as possible, that needs to dump indexes table, and I'd like to learn as much as possible from this experience.
Quote:
Originally Posted by
otheus
On slightly-less than gigabyte boundaries. Actually, 256 kB blocks also work very well.
Do you mean to execute an fwrite of a 256KB buffer? Currently I have a list where every element (table) is an array of N entry, for a total size of 4KB per array, and I dump every table at once with a single fwrite.
Quote:
Originally Posted by
otheus
If it's in a different thread, what's the point? You can't just free the memory if the other thread still has a lock on it.
I don't think that's answerable unless one really knows your existing software architecture.
Basically one thread (A) indexes the file, while another thread (B) waits for it to finish, in order to use the produced tables (which I used to keep in memory) to process the data in the file. The problem is that the file indexed are huge (~30GB) and produce more than 4GB of data, which I can't keep in memory (limit of 3GB per process) so, at one point or another I have to dump the data produced in a file in order to free the memory.
The other thread (B), based on a flag, either read the tables from the file or the list in memory.
Thanks for your help,
S.