Sponsored Content
Top Forums Programming Best way to dump metadata to file: when and by who? Post 302332051 by emitrax on Wednesday 8th of July 2009 04:25:23 AM
Old 07-08-2009
Quote:
Originally Posted by otheus
Wow, what a question. Are you re-engineering a database system?
Nope. I'm just trying to write an application as efficient as possible, that needs to dump indexes table, and I'd like to learn as much as possible from this experience.

Quote:
Originally Posted by otheus
On slightly-less than gigabyte boundaries. Actually, 256 kB blocks also work very well.
Do you mean to execute an fwrite of a 256KB buffer? Currently I have a list where every element (table) is an array of N entry, for a total size of 4KB per array, and I dump every table at once with a single fwrite.

Quote:
Originally Posted by otheus
If it's in a different thread, what's the point? You can't just free the memory if the other thread still has a lock on it.

I don't think that's answerable unless one really knows your existing software architecture.
Basically one thread (A) indexes the file, while another thread (B) waits for it to finish, in order to use the produced tables (which I used to keep in memory) to process the data in the file. The problem is that the file indexed are huge (~30GB) and produce more than 4GB of data, which I can't keep in memory (limit of 3GB per process) so, at one point or another I have to dump the data produced in a file in order to free the memory.

The other thread (B), based on a flag, either read the tables from the file or the list in memory.

Thanks for your help,
S.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

help, what is the difference between core dump and panic dump?

help, what is the difference between core dump and panic dump? (1 Reply)
Discussion started by: aileen
1 Replies

2. Shell Programming and Scripting

Importing dump file

Hi, I am trying to import 22 .dmp files but facing the problem with the last table file it never ends the import command, only the table is created but the rows of the table don't get imported. This is the problem with only ine table rest 21 tables are being imported properly. Thanks in... (2 Replies)
Discussion started by: anushilrai
2 Replies

3. Shell Programming and Scripting

Dump an array in a file

Hi, I'm wondering if there's a way to dump the content of an array into a specified part of a file. I know that I can redirect the output in a way that the array adds in the text file, this is done with ">>", but doing by this way, puts the array at the end of the file, and I'm asking for some... (3 Replies)
Discussion started by: IMD
3 Replies

4. Programming

How to use a core dump file

Hi All, May be it is a stupid question, but, I would like to know what is the advantage using a core dump file at the moment of debugging using gdb. I know a core dump has information about the state of the application when it crashed, but, what is the difference between debugging using the... (2 Replies)
Discussion started by: lagigliaivan
2 Replies

5. Shell Programming and Scripting

print metadata to jpg

Hi all, I would like to write a scipt that gets gps informatoin from a jpg and print 's it on the lower left corner, In order to get the gps data I have found a tool called jhead. In know that with the help of the imagemagick command convert it is possible to print text on the pictures. ... (11 Replies)
Discussion started by: flash80
11 Replies

6. UNIX for Dummies Questions & Answers

dump display to a file

Hi: I want to dump whatever command I type on the terminal + whatever is the result of that command's execution to one file. Is it possible in unix? Rgds, Indu (3 Replies)
Discussion started by: indu_shr
3 Replies

7. UNIX for Advanced & Expert Users

LVM - restore metadata on other disk

Hi guys, I would like to ask your opinion about my theory, how to fix my broken LVM without risking any data loss. I use Archlinux at home. I just love this distro, even it gives me a lots of work (particularly after system updates). Basic system spec: AMD FX(tm)-6100 Six-Core Processor... (1 Reply)
Discussion started by: lyynxxx
1 Replies

8. Solaris

Solaris 11.2 dump device "kernel without ZFS metadata"

I've never seen this, is it normal for 11.2? Anyway to change it back to dumping metadata or is this simply an overly verbose message I may ignore? kernel without ZFS metadata (4 Replies)
Discussion started by: LittleLebowski
4 Replies

9. UNIX and Linux Applications

About gvfsd-metadata

I need a hint about gvfsd-metadata using mate on bsd. Or dual-core cpu, quad-core cpu ore an old laptop single core, the gvfsd is an obstacle and does not accelerate anything, vice versa, it slows down many processes, coming from gnome. So someone can give me a hint how to wipe it out for good? I... (1 Reply)
Discussion started by: 1in10
1 Replies
pthread_key_create(3T)													    pthread_key_create(3T)

NAME
pthread_key_create(), pthread_key_delete() - create or delete a thread-specific data key SYNOPSIS
PARAMETERS
key This is either a pointer to the location where the new key value will to be returned (create function) or the thread-spe- cific data key to be deleted (delete function). destructor Function to be called to destroy a data value associated with key when the thread terminates. DESCRIPTION
creates a unique thread-specific data key. The key may be used by threads within the process to maintain thread-specific data. The same key is used by all threads, but each thread has its own thread-specific value associated with key. For each thread, the value associated with key persists for the life of the thread. By default, the system allows a process to create up to number of thread-specific data keys. If the process needs more keys, the environ- ment variable can set the number of keys up to a maximum of 16384. If a value outside the range of and 16384 is specified, or if the envi- ronment variable is not set at all, the default value is considered. When a new thread-specific data key is created, each thread will ini- tially have the value NULL associated with the new key. Each time a thread is created, the new thread will have the value NULL for each thread-specific data key that has been created in the process. A thread may use to change the value associated with a thread-specific data key. Note: is an opaque data type. When a thread terminates, it may have non-NULL values associated with some or all of its thread-specific data keys. Typically, these val- ues will be pointers to dynamically allocated memory. If this memory is not released when the thread terminates, memory leaks in the process occur. An optional function may be provided at key creation time to destroy the thread-specific data of a terminating thread. When a thread terminates, the thread-specific data values associated with the thread will be examined. For each key that has a non-NULL thread-specific data value and a destructor function, the destructor function will be called with the thread-specific data value as its sole argument. The order in which destructor functions are called is unspecified. Once all the destructor functions have been called, the thread-specific data values for the terminating thread are examined again. If there are still non-NULL values in which the associated keys have destructor functions, the process of calling destructor functions is repeated. If after iterations of this loop there are still some non-NULL values with associated destructors, the system may stop calling the destructors or continue calling the destructors until there are no non-NULL values. Note: This may result in an infinite loop. If a destructor function is not desired for key, the value NULL may be passed in the destructor parameter. The function deletes a thread-specific data key. The key must have been previously created by The thread-specific data values associated with key are not required to be NULL when this function is called. Using key after it has been deleted results in undefined behavior. If a destructor function is associated with key, it will not be invoked by the function. Once key has been deleted, any function that was associated with key is not called when a thread exits. It is the responsibility of the application to free any application storage for each of the threads using key. The function can be called from a destructor function. RETURN VALUE
If successful, and return zero. Otherwise, an error number is returned to indicate the error (the variable is not set). ERRORS
If any of the following occur, the function returns the corresponding error number: The value specified by key is invalid. The necessary resources to create another thread-specific data key are not available, or the total number of keys per process has exceeded or an invalid value was specified with the environment variable There is insufficient memory available in which to create key. For each of the following conditions, if the condition is detected, the function returns the corresponding error number: The value specified by key is invalid. AUTHOR
and were derived from the IEEE POSIX P1003.1c standard. SEE ALSO
pthread_getspecific(3T), pthread_setspecific(3T). STANDARDS CONFORMANCE
Pthread Library pthread_key_create(3T)
All times are GMT -4. The time now is 07:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy