Sponsored Content
Top Forums UNIX for Dummies Questions & Answers massive tarred grib files totally unacceptable Post 302470058 by jim mcnamara on Tuesday 9th of November 2010 05:26:22 AM
Old 11-09-2010
1. Extract the all data you need just one time, and store it in a file.
2. If you run 14 simultaneous processes it will take just one day.
3. The format of tar is an established standard. Nobody is hiding it from you.
google for 'gnu coreutils' and download the source code. In the tarball is the source for tar itself.

The problem is this:
tar stands for 'tape archive'. To get to file #99 in a tar file you have to actually completely read though files 1 -> 98 first. There is no file metadata like on a hard drive that allows direct access. The same applies once you get to the file you want. You have to read thru the file itself until you get to a given record. You cannot "fast forward" to a given record or a given file.

Modify the program to run in as many separate processes as is possible on your system without overloading it. Each process opens one tar file, gets one data set, writes the "variables" to disk or even better to a database table.

So, you now have the six values for every day. And in one small file.

Do you know how to extract a given file? What command are you using to get the grib file from a tarball?
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Totally new unix user

Hi, ive just been given a HP UX c180 to play with, along with a few external hd`s, can anyone recommend me a good place to start learning unix please as i havent got a clue what im doing when it comes to unix. Either web links or book recomendations would be great. I would like to set it up with... (1 Reply)
Discussion started by: Leviathan40
1 Replies

2. UNIX for Dummies Questions & Answers

totally new to unix

I'm trying to write a script, named "worfo" which will read a file in and return, on the screen, the number of words in the file. Also, I am looking to do several extras with this script. I need to allow it to accept the option "-n" which will use all non-alpha characters to delimit words. Also,... (2 Replies)
Discussion started by: adawg1283
2 Replies

3. AIX

Totally messed up

I have moved the etc/passwd file and we are now unable to get in to Unix - any suggestions? (2 Replies)
Discussion started by: vbagwell
2 Replies

4. UNIX for Dummies Questions & Answers

Totally brand new to CRON

hi there i am very new to the world of CRON. i would like to know from below the ground up how i can learn abouot cron. where the best tutorials lie. i have an assignment that requires i set up a cron job to download a price file. am i in control of where the price file is downloaded to? (eg to... (2 Replies)
Discussion started by: buzzby
2 Replies

5. Shell Programming and Scripting

bash script for selecting grib files with cdo

hi, i am new with scripts and have a little problem. i have a file with dates in the form YYYYMMDD, in average about 40 days per year. now i shall cut those days with cdo selday from monthly files. the script shall look up the days, forward them to the cdo operator and write the new grib... (1 Reply)
Discussion started by: jumkae
1 Replies

6. UNIX for Dummies Questions & Answers

Tarred files not being put on tape

Hi I normally backup to tape using the following command in Solaris 9 on SUN Enterprise 3500 server: tar cvf /dev/rmt/0n data but after a power cut, the server went down without being shutdown since the UPS failed. Now when I issue the tar command I get the root (/) being full looks like it... (6 Replies)
Discussion started by: rahmantanko
6 Replies

7. UNIX for Dummies Questions & Answers

Help with untarring multiple files from tarred directories and subdirectories

Hi, I want to untar all log files from following tarred directory hierarchy Log_files.tar.gz/subject*.tar.gz/project*/*.log It means there are subject1.tar.gz to subject9.tar.gz and in those tarred subect directories there are project1 - project5 directories and in those directories there... (2 Replies)
Discussion started by: rv_trojan
2 Replies

8. Shell Programming and Scripting

Totally new to shell scripting

lllll (34 Replies)
Discussion started by: risarose87
34 Replies

9. Shell Programming and Scripting

Attach filename to wc results on massive number of files

Hello, I have massive number of big files that needed to be counted for the total number of lines (> 100x millions) each. I want the file name attached to the count results so that they are aligned nicely matching name and counts. I could do each file at a time, which will take hours to finish,... (8 Replies)
Discussion started by: yifangt
8 Replies
PRISTINE-TAR(1) 						   pristine-tar 						   PRISTINE-TAR(1)

NAME
pristine-tar - regenerate pristine tarballs SYNOPSIS
pristine-tar [-vdk] gendelta tarball delta pristine-tar [-vdk] gentar delta tarball pristine-tar [-vdk] [-m message] commit tarball [upstream] pristine-tar [-vdk] checkout tarball pristine-tar [-vdk] list DESCRIPTION
pristine-tar can regenerate an exact copy of a pristine upstream tarball using only a small binary delta file and the contents of the tarball, which are typically kept in an upstream branch in version control. The delta file is designed to be checked into version control along-side the upstream branch, thus allowing Debian packages to be built entirely using sources in version control, without the need to keep copies of upstream tarballs. pristine-tar supports compressed tarballs, calling out to pristine-gz(1), pristine-bz2(1), and pristine-xz(1) to produce the pristine gzip, bzip2, and xz files. COMMANDS
pristine-tar gendelta tarball delta This takes the specified upstream tarball, and generates a small binary delta file that can later be used by pristine-tar gentar to recreate the tarball. If the delta filename is "-", it is written to standard output. pristine-tar gentar delta tarball This takes the specified delta file, and the files in the current directory, which must have identical content to those in the upstream tarball, and uses these to regenerate the pristine upstream tarball. If the delta filename is "-", it is read from standard input. pristine-tar commit tarball [upstream] pristine-tar commit generates a pristine-tar delta file for the specified tarball, and commits it to version control. The pristine-tar checkout command can later be used to recreate the original tarball based only on the information stored in version control. The upstream parameter specifies the tag or branch that contains the same content that is present in the tarball. This defaults to "refs/heads/upstream", or if there's no such branch, any branch matching "upstream". The name of the tree it points to will be recorded for later use by pristine-tar checkout. Note that the content does not need to be 100% identical to the content of the tarball, but if it is not, additional space will be used in the delta file. The delta files are stored in a branch named "pristine-tar", with filenames corresponding to the input tarball, with ".delta" appended. This branch is created or updated as needed to add each new delta. pristine-tar checkout tarball This regenerates a copy of the specified tarball using information previously saved in version control by pristine-tar commit. pristine-tar list This lists tarballs that pristine-tar is able to checkout from version control. OPTIONS
-v --verbose Verbose mode, show each command that is run. -d --debug Debug mode. -k --keep Don't clean up the temporary directory on exit. -m message --message=message Use this option to specify a custom commit message to pristine-tar commit. EXAMPLES
Suppose you maintain the hello package, in a git repository. You have just created a tarball of the release, hello-1.0.tar.gz, which you will upload to a "forge" site. You want to ensure that, if the "forge" loses the tarball, you can always recreate exactly that same tarball. And you'd prefer not to keep copies of tarballs for every release, as that could use a lot of disk space when hello gets the background mp3s and user-contributed levels you are planning for version 2.0. The solution is to use pristine-tar to commit a delta file that efficiently stores enough information to reproduce the tarball later. cd hello git tag -s 1.0 pristine-tar commit ../hello-1.0.tar.gz 1.0 Remember to tell git to push both the pristine-tar branch, and your tag: git push --all --tags Now it is a year later. The worst has come to pass; the "forge" lost all its data, you deleted the tarballs to make room for bug report emails, and you want to regenerate them. Happily, the git repository is still available. git clone git://github.com/joeyh/hello.git cd hello pristine-tar checkout ../hello-1.0.tar.gz LIMITATIONS
Only tarballs, gzipped tarballs, bzip2ed tarballs, and xzed tarballs are currently supported. Currently only the git revision control system is supported by the "checkout" and "commit" commands. It's ok if the working copy is not clean or has uncommitted changes, or has changes staged in the index; none of that will be touched by "checkout" or "commit". ENVIRONMENT
TMPDIR Specifies a location to place temporary files, other than the default. AUTHOR
Joey Hess <joeyh@debian.org> Licensed under the GPL, version 2 or above. perl v5.14.2 2013-06-01 PRISTINE-TAR(1)
All times are GMT -4. The time now is 07:43 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy