Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

massive tarred grib files totally unacceptable

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers massive tarred grib files totally unacceptable
Prev   Next
# 1  
Old 11-09-2010
massive tarred grib files totally unacceptable
Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES (because I need those 6 variables for EVERYDAY). This is not cool. This is VERY not cool. Since the tar files aren't even compressed, I'm convinced there's a way to get the data without even extracting, even if everybody refuses to give up the info...

BUT... as a compromise, is it possible for me to extract ONLY those 6 variables I need and not EVERYTHING, only just to throw it away later? I'm basically wasting 2 weeks on this

(Disclaimer: I'm aware that this could've been solved BEFOREHAND, but my bosses don't have that kind of foresight..., and I'm not getting paid by the hour just to watch the program run for 2 weeks)
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Attach filename to wc results on massive number of files

Hello, I have massive number of big files that needed to be counted for the total number of lines (> 100x millions) each. I want the file name attached to the count results so that they are aligned nicely matching name and counts. I could do each file at a time, which will take hours to finish,... (8 Replies)
Discussion started by: yifangt
8 Replies

2. Shell Programming and Scripting

Totally new to shell scripting

lllll (34 Replies)
Discussion started by: risarose87
34 Replies

3. UNIX for Dummies Questions & Answers

Help with untarring multiple files from tarred directories and subdirectories

Hi, I want to untar all log files from following tarred directory hierarchy Log_files.tar.gz/subject*.tar.gz/project*/*.log It means there are subject1.tar.gz to subject9.tar.gz and in those tarred subect directories there are project1 - project5 directories and in those directories there... (2 Replies)
Discussion started by: rv_trojan
2 Replies

4. UNIX for Dummies Questions & Answers

Tarred files not being put on tape

Hi I normally backup to tape using the following command in Solaris 9 on SUN Enterprise 3500 server: tar cvf /dev/rmt/0n data but after a power cut, the server went down without being shutdown since the UPS failed. Now when I issue the tar command I get the root (/) being full looks like it... (6 Replies)
Discussion started by: rahmantanko
6 Replies

5. Shell Programming and Scripting

bash script for selecting grib files with cdo

hi, i am new with scripts and have a little problem. i have a file with dates in the form YYYYMMDD, in average about 40 days per year. now i shall cut those days with cdo selday from monthly files. the script shall look up the days, forward them to the cdo operator and write the new grib... (1 Reply)
Discussion started by: jumkae
1 Replies

6. UNIX for Dummies Questions & Answers

Totally brand new to CRON

hi there i am very new to the world of CRON. i would like to know from below the ground up how i can learn abouot cron. where the best tutorials lie. i have an assignment that requires i set up a cron job to download a price file. am i in control of where the price file is downloaded to? (eg to... (2 Replies)
Discussion started by: buzzby
2 Replies

7. AIX

Totally messed up

I have moved the etc/passwd file and we are now unable to get in to Unix - any suggestions? (2 Replies)
Discussion started by: vbagwell
2 Replies

8. UNIX for Dummies Questions & Answers

totally new to unix

I'm trying to write a script, named "worfo" which will read a file in and return, on the screen, the number of words in the file. Also, I am looking to do several extras with this script. I need to allow it to accept the option "-n" which will use all non-alpha characters to delimit words. Also,... (2 Replies)
Discussion started by: adawg1283
2 Replies

9. UNIX for Dummies Questions & Answers

Totally new unix user

Hi, ive just been given a HP UX c180 to play with, along with a few external hd`s, can anyone recommend me a good place to start learning unix please as i havent got a clue what im doing when it comes to unix. Either web links or book recomendations would be great. I would like to set it up with... (1 Reply)
Discussion started by: Leviathan40
1 Replies
Login or Register to Ask a Question

LEARN ABOUT CENTOS

bf_tar

BF_TAR(1)						    Bogofilter Reference Manual 						 BF_TAR(1)

NAME

       bf_tar - shell script to write a tar file of a bogofilter directory to stdout

SYNOPSIS

       bf_tar [-r] [-R] bogofilter_directory

DESCRIPTION

       bf_tar bundles a bogofilter working directory in tar format and copies it to standard output (your console, or where you redirect it, see
       EXAMPLES below).

OPTIONS

       The -r option causes bf_tar to remove inactive log files after the archive has been written successfully. The default is to leave log
       files.

       The -R option causes bf_tar to remove inactive log files before the archive is written. This may reduce chances that the resulting archive
       is recoverable should it become damaged. The archive may be smaller though. The default is to leave log files.

EXIT STATUS

       The script exits with status code 0 if everything went well, and nonzero if it encountered trouble.

EXAMPLES

       o   bf_tar ~/.bogofilter > outfile.tar

	   Writes a standard .tar file containing the essential files from ~/.bogofilter to outfile.tar.

       o   bf_tar ~/.bogofilter | gzip -9 -c > outfile.tar.gz

	   Writes a gzipped .tar.gz file containing the essential files from ~/.bogofilter to outfile.tar.gz.

       o   bf_tar `pwd`/mydirectory > outfile.tar

	   Prepend $(pwd)/ or `pwd`/ if you want to specify an absolute path instead of a relative path.

NOTES

       This script is meant for use with Berkeley DB based bogofilter versions.

       This script requires a SUSv2 compliant pax utility.

       This script expects a SUSv2 compliant shell. Solaris systems should have the SUNWxcu4 package installed (when bogofilter is configured) so
       that /usr/xpg4/bin/sh can be used.

AUTHORS

       Matthias Andree

Bogofilter							    10/22/2012								 BF_TAR(1)