Best way to dump metadata to file: when and by who? Post: 302329710

Sponsored Content

Top Forums Programming Best way to dump metadata to file: when and by who? Post 302329710 by emitrax on Monday 29th of June 2009 09:05:09 AM

06-29-2009

Registered User

Best way to dump metadata to file: when and by who?

Hi,

my application (actually library) indexes a file of many GB producing tables (arrays of offset and length of the data indexed) for later reuse. The tables produced are pretty big too, so big that I ran out of memory in my process (3GB limit), when indexing more than 8GB of file or so. Although I could fork another process to work around the memory limit size, this would not fix the problem, so I'd like to dump the tables to a file in order to free the memory, and avoid to re-index the same file more than once.

Bear in mind that currently, the tables produced are kept in memory in a single-linked list, shared with another thread that use it to produce another list of filtered data. So I'd rather not change this schema. The other thread only access the list once the whole file has been indexed.

Now, the questions I'm asking myself are:

- When and how it's best time to dump the tables to a file?

Dumping a table as it gets full doesn't sound very efficient to me. Would I keep nothing in memory? The linked list would always be empty? If I decide to keep N tables in memory, and dump every N, how do I avoid making a check for how many tables I have
in memory at every cycle ?

- Who should dump the metadata produced to file? Different thread? Same thread that index the data? I also wouldn't like to produce metadata files when the file processed is less then a giga (small file case), but at the same time I wouldn't want to complex the code of the indexer, that right now is pretty simply: parse, find the data, create an entry table, add it. If the table is full, create another one and add it to the linked list.

- Let's say I figured out (thanks to you) the best way (in my case) to dump the metadata. What policy should I use to load the data in order to let the other thread
filtering the index data without radically changing the way it works now (e.g. through the linked list) ?

One solution that come to my mind, that would avoid a drastical change in my schema is to create a "list manager" that would provide an interface to add and retrieve element from the list. This entity (either a thread or a process) would take care of keeping some data in memory (linked list) and some other in the file.

Please share with me your skill and experience! :-)

Thanks in advance.

Regards,
S.

emitrax

View Public Profile for emitrax

Find all posts by emitrax

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

help, what is the difference between core dump and panic dump?

2. Shell Programming and Scripting

Importing dump file

Hi, I am trying to import 22 .dmp files but facing the problem with the last table file it never ends the import command, only the table is created but the rows of the table don't get imported. This is the problem with only ine table rest 21 tables are being imported properly. Thanks in...

3. Shell Programming and Scripting

Dump an array in a file

Hi, I'm wondering if there's a way to dump the content of an array into a specified part of a file. I know that I can redirect the output in a way that the array adds in the text file, this is done with ">>", but doing by this way, puts the array at the end of the file, and I'm asking for some...

4. Programming

How to use a core dump file

Hi All, May be it is a stupid question, but, I would like to know what is the advantage using a core dump file at the moment of debugging using gdb. I know a core dump has information about the state of the application when it crashed, but, what is the difference between debugging using the...

5. Shell Programming and Scripting

print metadata to jpg

Hi all, I would like to write a scipt that gets gps informatoin from a jpg and print 's it on the lower left corner, In order to get the gps data I have found a tool called jhead. In know that with the help of the imagemagick command convert it is possible to print text on the pictures. ...

6. UNIX for Dummies Questions & Answers

dump display to a file

Hi: I want to dump whatever command I type on the terminal + whatever is the result of that command's execution to one file. Is it possible in unix? Rgds, Indu

7. UNIX for Advanced & Expert Users

LVM - restore metadata on other disk

Hi guys, I would like to ask your opinion about my theory, how to fix my broken LVM without risking any data loss. I use Archlinux at home. I just love this distro, even it gives me a lots of work (particularly after system updates). Basic system spec: AMD FX(tm)-6100 Six-Core Processor...

8. Solaris

Solaris 11.2 dump device "kernel without ZFS metadata"

I've never seen this, is it normal for 11.2? Anyway to change it back to dumping metadata or is this simply an overly verbose message I may ignore? kernel without ZFS metadata

9. UNIX and Linux Applications

About gvfsd-metadata

I need a hint about gvfsd-metadata using mate on bsd. Or dual-core cpu, quad-core cpu ore an old laptop single core, the gvfsd is an obstacle and does not accelerate anything, vice versa, it slows down many processes, coming from gnome. So someone can give me a hint how to wipe it out for good? I...

LEARN ABOUT CENTOS

font::ttf::ttc

Font::TTF::Ttc(3)					User Contributed Perl Documentation					 Font::TTF::Ttc(3)

NAME

       Font::TTF::Ttc - Truetype Collection class

DESCRIPTION

       A TrueType collection is a collection of TrueType fonts in one file in which tables may be shared between different directories. In order
       to support this, the TTC introduces the concept of a table being shared by different TrueType fonts. This begs the question of what should
       happen to the ' PARENT' property of a particular table. It is made to point to the first directory object which refers to it. It is
       therefore up to the application to sort out any confusion.  Confusion only occurs if shared tables require access to non-shared tables.
       This should not happen since the shared tables are dealing with glyph information only and the private tables are dealing with encoding and
       glyph identification. Thus the general direction is from identification to glyph and not the other way around (at least not without
       knowledge of the particular context).

INSTANCE VARIABLES

       The following instance variables are preceded by a space

       fname (P)
	   Filename for this TrueType Collection

       INFILE (P)
	   The filehandle of this collection

       The following instance variable does not start with a space

       directs
	   An array of directories (Font::TTF::Font objects) for each sub-font in the directory

METHODS

   Font::TTF::Ttc->open($fname)
       Opens and reads the given filename as a TrueType Collection. Reading a collection involves reading each of the directories which go to make
       up the collection.

   $c->read
       Reads a Collection by reading all the directories in the collection

   $c->find($direct, $name, $check, $off, $len)
       Hunts around to see if a table with the given characteristics of name, checksum, offset and length has been associated with a directory
       earlier in the list.  Actually on checks the offset since no two tables can share the same offset in a TrueType font, collection or
       otherwise.

   $c->DESTROY
       Closees any opened files by us

BUGS

       No known bugs, but then not ever executed!

AUTHOR

       Martin Hosken Martin_Hosken@sil.org. See Font::TTF::Font for copyright and licensing.

perl v5.16.3							    2011-10-13							 Font::TTF::Ttc(3)