Sponsored Content
Top Forums UNIX for Dummies Questions & Answers massive tarred grib files totally unacceptable Post 302470003 by sammysoil on Tuesday 9th of November 2010 01:02:27 AM
Old 11-09-2010
massive tarred grib files totally unacceptable

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES (because I need those 6 variables for EVERYDAY). This is not cool. This is VERY not cool. Since the tar files aren't even compressed, I'm convinced there's a way to get the data without even extracting, even if everybody refuses to give up the info...

BUT... as a compromise, is it possible for me to extract ONLY those 6 variables I need and not EVERYTHING, only just to throw it away later? I'm basically wasting 2 weeks on this

(Disclaimer: I'm aware that this could've been solved BEFOREHAND, but my bosses don't have that kind of foresight..., and I'm not getting paid by the hour just to watch the program run for 2 weeks)
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Totally new unix user

Hi, ive just been given a HP UX c180 to play with, along with a few external hd`s, can anyone recommend me a good place to start learning unix please as i havent got a clue what im doing when it comes to unix. Either web links or book recomendations would be great. I would like to set it up with... (1 Reply)
Discussion started by: Leviathan40
1 Replies

2. UNIX for Dummies Questions & Answers

totally new to unix

I'm trying to write a script, named "worfo" which will read a file in and return, on the screen, the number of words in the file. Also, I am looking to do several extras with this script. I need to allow it to accept the option "-n" which will use all non-alpha characters to delimit words. Also,... (2 Replies)
Discussion started by: adawg1283
2 Replies

3. AIX

Totally messed up

I have moved the etc/passwd file and we are now unable to get in to Unix - any suggestions? (2 Replies)
Discussion started by: vbagwell
2 Replies

4. UNIX for Dummies Questions & Answers

Totally brand new to CRON

hi there i am very new to the world of CRON. i would like to know from below the ground up how i can learn abouot cron. where the best tutorials lie. i have an assignment that requires i set up a cron job to download a price file. am i in control of where the price file is downloaded to? (eg to... (2 Replies)
Discussion started by: buzzby
2 Replies

5. Shell Programming and Scripting

bash script for selecting grib files with cdo

hi, i am new with scripts and have a little problem. i have a file with dates in the form YYYYMMDD, in average about 40 days per year. now i shall cut those days with cdo selday from monthly files. the script shall look up the days, forward them to the cdo operator and write the new grib... (1 Reply)
Discussion started by: jumkae
1 Replies

6. UNIX for Dummies Questions & Answers

Tarred files not being put on tape

Hi I normally backup to tape using the following command in Solaris 9 on SUN Enterprise 3500 server: tar cvf /dev/rmt/0n data but after a power cut, the server went down without being shutdown since the UPS failed. Now when I issue the tar command I get the root (/) being full looks like it... (6 Replies)
Discussion started by: rahmantanko
6 Replies

7. UNIX for Dummies Questions & Answers

Help with untarring multiple files from tarred directories and subdirectories

Hi, I want to untar all log files from following tarred directory hierarchy Log_files.tar.gz/subject*.tar.gz/project*/*.log It means there are subject1.tar.gz to subject9.tar.gz and in those tarred subect directories there are project1 - project5 directories and in those directories there... (2 Replies)
Discussion started by: rv_trojan
2 Replies

8. Shell Programming and Scripting

Totally new to shell scripting

lllll (34 Replies)
Discussion started by: risarose87
34 Replies

9. Shell Programming and Scripting

Attach filename to wc results on massive number of files

Hello, I have massive number of big files that needed to be counted for the total number of lines (> 100x millions) each. I want the file name attached to the count results so that they are aligned nicely matching name and counts. I could do each file at a time, which will take hours to finish,... (8 Replies)
Discussion started by: yifangt
8 Replies
TAR(1)							      General Commands Manual							    TAR(1)

NAME
tar - tape archiver SYNOPSIS
tar [ key ] [ name ... ] DESCRIPTION
Tar saves and restores files on magtape. Its actions are controlled by the key argument. The key is a string of characters containing at most one function letter and possibly one or more function modifiers. Other arguments to the command are file or directory names specify- ing which files are to be dumped or restored. In all cases, appearance of a directory name refers to the files and (recursively) subdirec- tories of that directory. The function portion of the key is specified by one of the following letters: r The named files are written on the end of the tape. The c function implies this. x The named files are extracted from the tape. If the named file matches a directory whose contents had been written onto the tape, this directory is (recursively) extracted. The owner, modification time, and mode are restored (if possible). If no file argument is given, the entire content of the tape is extracted. Note that if multiple entries specifying the same file are on the tape, the last one overwrites all earlier. t The names of the specified files are listed each time they occur on the tape. If no file argument is given, all of the names on the tape are listed. u The named files are added to the tape if either they are not already there or have been modified since last put on the tape. c Create a new tape; writing begins on the beginning of the tape instead of after the last file. This command implies r. The following characters may be used in addition to the letter which selects the function desired. 0,...,7 This modifier selects the drive on which the tape is mounted. The default is 1. v Normally tar does its work silently. The v (verbose) option causes it to type the name of each file it treats preceded by the function letter. With the t function, v gives more information about the tape entries than just the name. w causes tar to print the action to be taken followed by file name, then wait for user confirmation. If a word beginning with `y' is given, the action is performed. Any other input means don't do it. f causes tar to use the next argument as the name of the archive instead of /dev/mt?. If the name of the file is `-', tar writes to standard output or reads from standard input, whichever is appropriate. Thus, tar can be used as the head or tail of a filter chain Tar can also be used to move hierarchies with the command cd fromdir; tar cf - . | (cd todir; tar xf -) b causes tar to use the next argument as the blocking factor for tape records. The default is 1, the maximum is 20. This option should only be used with raw magnetic tape archives (See f above). The block size is determined automatically when reading tapes (key letters `x' and `t'). l tells tar to complain if it cannot resolve all of the links to the files dumped. If this is not specified, no error messages are printed. m tells tar to not restore the modification times. The mod time will be the time of extraction. FILES
/dev/mt? /tmp/tar* DIAGNOSTICS
Complaints about bad key characters and tape read/write errors. Complaints if enough memory is not available to hold the link tables. BUGS
There is no way to ask for the n-th occurrence of a file. Tape errors are handled ungracefully. The u option can be slow. The b option should not be used with archives that are going to be updated. The current magtape driver cannot backspace raw magtape. If the archive is on a disk file the b option should not be used at all, as updating an archive stored in this manner can destroy it. The current limit on file name length is 100 characters. TAR(1)
All times are GMT -4. The time now is 09:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy