UNIX for Advanced & Expert Users

I'm wondering about the best way to store large amounts of sorted data, sorted by date/time, in a manner to allow fast retrieval of ranges of dates. Deletion isn't necessary.

A database hardly seems ideal, since they're not optimized for sorted data, and allow things I don't need like the deletion of arbitrary records. A text flat file would work fine, except how to seek in one without reading the whole thing? It's theoretically possible assuming the data's sorted, but I don't know a way to do this from shell, so I considered writing my own application to quickly seek within a huge flat file and/or split into different flat files based on date, until I realized I might be reinventing the wheel. Are there any standard tools for this?

You can try sgrep - it does binary searches on huge sorted files very efficiently.

See sourceforge or sgrep homepage:
Sgrep - Home page

We use it for searching giant xml data files. In looking at the homepage it has added a bunch of features, since our earlier version, but we don't seem to need them.

I suppose it depends on the volume of data. If you are storing everything into a single file, then many shell commands will read the whole file, (grep for example) but if you have a record of the line number, you could use

Code:

sed -n ${i}p $filename

It's a bit of a heavyweight , but it will read you line ${i}

An alternate might be to split the records so that you have one file per day and then you reduce the volume read by tools such as grep, assuming you know the day.

You could then compress the older files then use

Code:

zcat $filename|grep $mysearch

although there is, of course the overhead of the zcat. Perhaps if you found a logic that said "Anything over 30 days is compressed because it is rarely used" then you could build a script that just looks for the file and if only the compressed file exists, uses the above.


Robin
Liverpool/Blackburn
UK


Do let us know how you get on or if there are more questions.

Exactly the problem. It's going to be big, eventually, and I don't want to read the entire file. I don't want to use a real database either, since the data's intended to be sorted, which would add a lot of overhead there too.

On the other hand it's certainly possible to be intelligent about the way you read a flat file. Things like GNU tac and GNU tail use seeking when possible to get approximately where they want and fine-tune from there, instead of reading all 3 gigs like a moron until they find the 10 lines they need.

Yes, I thought of that. One file per month might be more appropriate if I don't want to generate massive numbers of tiny tiny files, but anyway.

The point is, I'm looking for a smart tool to store in and retrieve from such a tree. Beginning to think there is no such thing, might have to build one myself...

All depends on whether your sorted data is fixed or variable length records. Direct access to a specific fixed length record is an easy nut to crack. Quickly accessing variable length records in a flat file is a whole different game. Which is it?

Variable length. I don't see a simple way to do it in shell with fixed-length either, though, since a shell can't seek.

---------- Post updated at 03:29 PM ---------- Previous update was at 02:35 PM ----------

I've got an application partly written for this now, so, solved, I guess.

sqllite ? Berkley Database?