UNIX for Dummies Questions & Answers

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES (because I need those 6 variables for EVERYDAY). This is not cool. This is VERY not cool. Since the tar files aren't even compressed, I'm convinced there's a way to get the data without even extracting, even if everybody refuses to give up the info...

BUT... as a compromise, is it possible for me to extract ONLY those 6 variables I need and not EVERYTHING, only just to throw it away later? I'm basically wasting 2 weeks on this

(Disclaimer: I'm aware that this could've been solved BEFOREHAND, but my bosses don't have that kind of foresight..., and I'm not getting paid by the hour just to watch the program run for 2 weeks)

1. Extract the all data you need just one time, and store it in a file.
2. If you run 14 simultaneous processes it will take just one day.
3. The format of tar is an established standard. Nobody is hiding it from you.
google for 'gnu coreutils' and download the source code. In the tarball is the source for tar itself.

The problem is this:
tar stands for 'tape archive'. To get to file #99 in a tar file you have to actually completely read though files 1 -> 98 first. There is no file metadata like on a hard drive that allows direct access. The same applies once you get to the file you want. You have to read thru the file itself until you get to a given record. You cannot "fast forward" to a given record or a given file.

Modify the program to run in as many separate processes as is possible on your system without overloading it. Each process opens one tar file, gets one data set, writes the "variables" to disk or even better to a database table.

So, you now have the six values for every day. And in one small file.

Do you know how to extract a given file? What command are you using to get the grib file from a tarball?

Is the only contents of each tar archive the 8 GRIB files, or are there archive files to ignore ? I'm fishing for whether your script extracts files which you do not need to process.

Hi,
I have downloaded the data from IPCC global weather data. How to extract the data for El Salvador region from the whole set of data and how to open the data or convert the data into a normal text file data or some other format.