Sponsored Content
Full Discussion: Pipelining with tar
Top Forums UNIX for Dummies Questions & Answers Pipelining with tar Post 302465512 by huntreilly25 on Friday 22nd of October 2010 03:45:57 PM
Old 10-22-2010
Alright here are my updated needs...

I need to gather all the directories and files (using tar) from one machine's usr/local/ directory, compress the file, then using ssh, transfer it to another machine, uncompress it, and extract it into that machine's opt/ directory.

So as far as I can tell the first half will remain the same...
tar cf - usr/local/ | gzip -c - | #now I am assuming I will be using cd to change to the local machine... and then continuing with the last half of the code (note: this is assuming that I am already connected to the remote machine through ssh)

Last edited by huntreilly25; 10-22-2010 at 04:52 PM.. Reason: update
 

10 More Discussions You Might Find Interesting

1. Programming

Pipelining Processes

I am doing a program that will calculate the 199th fibonacci number using pipelines and multiple processes for each calculation. I have figured everything out except why none of the processes before the last one never execute the findfib() function; Our teacher gave us the pipeline code to... (3 Replies)
Discussion started by: Trivialnight
3 Replies

2. UNIX for Advanced & Expert Users

Does tar do crc checking on a tape or tar file?

Trying to answer a question about whether tar table-of-contents is a good tool for verifying tape data. (1 Reply)
Discussion started by: tjlst15
1 Replies

3. Shell Programming and Scripting

Pipelining

My problem is more a question of how to do it more elegantly than how to do it at all. The problem: I have a pipeline which has to write to the screen AND to a logfile: proc1 | tee -a <logfile> What makes things difficult is i also need the return code of proc1. But proc1 | tee -a... (6 Replies)
Discussion started by: bakunin
6 Replies

4. UNIX for Advanced & Expert Users

Tar utility (untar a .tar file) on VxWorks

Hi All Can someone pls guide me if there any utility to compress file on windows & uncompress on vxworks I tried as - - compressed some folders on windows ... i created .tar ( to maintain directory structure ) and compressed to .gz format. - on VxWorks i have uncompressed it to .tar... (1 Reply)
Discussion started by: uday_01
1 Replies

5. UNIX for Advanced & Expert Users

How to create a Tar of multiple Files in Unix and FTP the tar to Windows.

Hi, On my Unix Server in my directory, I have 70 files distributed in the following directories (which have several other files too). These files include C Source Files, Shell Script Source Files, Binary Files, Object Files. a) /usr/users/oracle/bin b) /usr/users/oracle... (1 Reply)
Discussion started by: marconi
1 Replies

6. UNIX for Dummies Questions & Answers

tar -cvf test.tar `find . -mtime -1 -type f` only tar 1 file

Hi all, 4 files are returned when i issue 'find . -mtime -1 -type f -ls'. ./ora_475244.aud ./ora_671958.aud ./ora_934052.aud ./ora_934050.aud However, when I issued the below command: tar -cvf test.tar `find . -mtime -1 -type f`, the tar file only contains the 1st file -... (2 Replies)
Discussion started by: ahSher
2 Replies

7. Shell Programming and Scripting

Bash pipelining

Hello, There is a symbolic link in a folder. I would like to read destination of this link and get base name of pointed file. Let's say, there is symlink : symlink -> ../file.txt. I can do that what I want by this script: readlink symlink | while read var do echo `basename $var` done but... (1 Reply)
Discussion started by: scdmb
1 Replies

8. Shell Programming and Scripting

tar command to explore multiple layers of tar and tar.gz files

Hi all, I have a tar file and inside that tar file is a folder with additional tar.gz files. What I want to do is look inside the first tar file and then find the second tar file I'm looking for, look inside that tar.gz file to find a certain directory. I'm encountering issues by trying to... (1 Reply)
Discussion started by: bashnewbee
1 Replies

9. UNIX for Dummies Questions & Answers

Do I need to extract the entire tar file to confirm the tar folder is fine?

I would like to confirm my file.tar is been tar-ed correctly before I remove them. But I have very limited disc space to untar it. Can I just do the listing instead of actual extract it? Can I say confirm folder integrity if the listing is sucessful without problem? tar tvf file1.tar ... (1 Reply)
Discussion started by: vivien_chu
1 Replies

10. AIX

Tar - pre-checking before making the Tar file

Coming from this thread, just wondering if there is an option to check if the Tar of the files/directory will be without any file-errors without actually making the tar. Scenario: Let's say you have a directory of 20GB, but you don't have the space to make Tar file at the moment, and you want... (14 Replies)
Discussion started by: filosophizer
14 Replies
RDUP-BACKUPS(7) 						       rdup							   RDUP-BACKUPS(7)

NAME
rdup-backups - introduction into making backups with rdup INTRODUCTION
rdup is a simple program that prints out a list of files and directories that are changed changed on a filesystem. It is more sophisticated than for instance find, because rdup will find files that are removed or directories that are renamed. A long time ago rdup included a bunch of shell and Perl scripts that implemented a backup policy. These could be used in a pipeline to per- form a backup. Currently rdup consists out of three basic utilities: rdup With rdup you create the file list on which later programs in the pipeline can work. The default output format also includes the files' content. rdup can be seen as a tar replacement in this respect, but rdup also allows for all kinds of transformations of the content (encryption, compression, reversal), see the -P switch in rdup(1) for more information. rdup-tr With rdup-tr you can transform the files rdup delivers to you. You can create tar, cpio or pax files. You can encrypt pathnames. rdup-tr is filter that reads from standard input and writes to standard output. See rdup-tr(1) for more information. With rdup and rdup-tr you can create an encrypted archive which is put in a directory structure that is also encrypted. rdup-up With rdup-up you can update an existing directory structure with the updates as described by rdup. rdup-up reads rdup input and will create the files, symbolic links, hard links and directories (and sockets, pipes and devices) in the file system. See rdup-up(1) for more information. So the general backup pipeline for rdup will look something like this: create filelist | transform | update filesystem ( rdup | rdup-tr | rdup-up ) Note 1: The same sequence is used for restoring. In both cases you want to move files from location A to B. The only difference is that the transformation is reversed when you restore. Note 2: The use of rdup-tr is optional. BACKUPS AND RESTORES
For rdup there is no difference between backups and restores. If you think about this for a minute you understand why. Making a backup means copying a list of files somewhere else. Restoring files is copying a list of files back to the place they came from. Same difference. So rdup can be used for both, if you did any transformation with rdup during the backup you just need to reverse those operations during the restore. BACKUPS
It is always best to backup to another medium, be it a different local harddisk or a NFS/CIFS mounted filesystem. You can also use ssh to store file on a remote server, ala rsync (although not as network efficient). If you backup to a local disk you can just as well use rsync or plain old tar, but if you store your files at somebody else's disk you will need encryption. This is where you go beyond rsync and rdup comes in. Rsync cannot do per-file encryption, sure you can encrypt the network traffic with ssh, but at the remote side your files are kept in plain view. If you implement remote backups, the easy route is to allow root access on the backup medium. If the backup runs without root access the created files will not have their original ownership. For NFS this can be achieved by using no_root_squash, for ssh you could enable PermitRootLogin. Note that this may be a security risk. SNAPSHOT BACKUPS
We need a little help here in the form of the rdup-simple script. Keep in mind that the following scripts can also be run remotely with the help of ssh. The following script implements the algorithm of rdup-simple. #!/bin/bash # some tmp files are saved in ~/.rdup. This directory must exist DIR=/home # what to backup BACKUP=/vol/backup TODAY=$(date +%Y%m/%d) LIST=~/.rdup/list-$HOSTNAME STAMP=~/.rdup/timestamp-$HOSTNAME # for remote backup, this has to run on the remote host! BUGBUG RET=$? case $RET in 2|*) echo Error >&2 exit 1 ;; 1) # full dump, remove file-list and time-stamp file rm $LIST $STAMP ;; 0) # inc dump # do nothing here ;; esac # this is the place where you want to modify the command line # right now, nothing is translated we just use 'cat' rdup -N $STAMP -Pcat $LIST $DIR | rdup-up $BACKUP/$HOSTNAME/$TODAY # or do a remote backup #rdup -N $STAMP -Pcat $LIST $DIR | ssh root@remotehost # rdup-up $BACKUP/$HOSTNAME/$TODAY LOCAL BACKUPS
With rdup-simple you can easily create backups. Backing up my home directory to a backup directory: rdup-simple ~ /vol/backup/$HOSTNAME This will create a backup in /vol/backup/$HOSTNAME/200705/15. So each day will have its own directory. Multiple sources are allowed, so: rdup-simple ~ /etc/ /var/lib /vol/backup/$HOSTNAME Will backup your home directory, /etc and /var/lib to the backup location. Also if you need to compress your backup, simple add a '-z' switch: rdup-simple -z ~ /etc/ /var/lib /vol/backup/$HOSTNAME REMOTE BACKUPS
For a remote backup to work, both the sending machine and the receiving machine must have rdup installed. The currently implemented proto- col is ssh. Dumping my homedir to the remote server: rdup-simple ~ ssh://miekg@remote/vol/backup/$HOSTNAME The syntax is almost identical, only the destination starts with the magic string 'ssh://'. Compression and encryption are just as easily enabled as with a local backup, just add '-z' and/or a '-k keyfile' argument: rdup-simple -z -k 'secret-file' ~ ssh://miekg@remote/vol/backup/$HOSTNAME Remember though, that because of these advanced features (compression, encryption, etc, ...) the network transfer can never be as efficient as rsync. ALSO SEE
rdup(1), rdup-tr(1), rdup-up(1) and http://www.miek.nl/projects/rdup/ 1.1.x 15 Dec 2008 RDUP-BACKUPS(7)
All times are GMT -4. The time now is 09:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy