Quote:
Originally Posted by otheus
Wow, what a question. Are you re-engineering a database system?
|
Nope. I'm just trying to write an application as efficient as possible, that needs to dump indexes table, and I'd like to learn as much as possible from this experience.
Quote:
Originally Posted by otheus
On slightly-less than gigabyte boundaries. Actually, 256 kB blocks also work very well.
|
Do you mean to execute an fwrite of a 256KB buffer? Currently I have a list where every element (table) is an array of N entry, for a total size of 4KB per array, and I dump every table at once with a single fwrite.
Quote:
Originally Posted by otheus
If it's in a different thread, what's the point? You can't just free the memory if the other thread still has a lock on it.
I don't think that's answerable unless one really knows your existing software architecture.
|
Basically one thread (A) indexes the file, while another thread (B) waits for it to finish, in order to use the produced tables (which I used to keep in memory) to process the data in the file. The problem is that the file indexed are huge (~30GB) and produce more than 4GB of data, which I can't keep in memory (limit of 3GB per process) so, at one point or another I have to dump the data produced in a file in order to free the memory.
The other thread (B), based on a flag, either read the tables from the file or the list in memory.
Thanks for your help,
S.
