Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp
# 8  
Old 04-09-2014
This problem's been coming up for a while, so here's a generic solution. It takes the command you want to run as arguments, running it once per chunk feeding it file data via STDIN, and substitutes @FNAME@ with an 8-digit incrementing sequence.

Code:
#!/usr/bin/perl

# Die screaming instead of silently if child fails
$SIG{PIPE} = sub { die("SIGPIPE"); };

# $block is the read size, 1 meg is probably OK.
# $chunk is the output size.  With a chunk of 2 and block of 1024*1024,
# it writes 2 meg files.
my $block=1024*1024, $chunk=2;

my $running=1, $ccount=0, @l=@ARGV;

while($running) {
        my $n, $fname=sprintf("%08d", $ccount++);
        my $fr=\$fname;

        # Use given arguments as a command, with @FNAME@ substituted
        # for an incrementing number like 00000001
        open(OUT, "|-",
                map { my $v=$_; $v =~ s/\@FNAME\@/${$fr}/; $v } @l
        );

        for($n=0; $n<$chunk; $n++)
        {
                my $bytes=read(STDIN, $data, $block);
                defined($bytes) || die ("Error reading STDIN");
                if($bytes < 1) { # Ran out of data, stop reading
                        $running=0;
                        last;
                }

                print OUT $data;
        }

        close(OUT);
}

printf(STDERR "Wrote %d chunks of %d bytes\n", $ccount, $block*$chunk);

Code:
$ chmod +x pipesplit.pl
$ tar cvzf - /backup | openssl aes-256-cbc -salt -k "password" |
        ./pipesplit.pl curl -u user:pass ftp://hostname/path/@FNAME@ -T -

100 2048k    0     0    0 2048k      0  8368k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8367k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8335k --:--:-- --:--:-- --:--:-- 8325k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8389k --:--:-- --:--:-- --:--:-- 8427k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5    0     0    0     5      0     21 --:--:-- --:--:-- --:--:--    21
Wrote 17 chunks of 2097152 bytes

$

Adjust the value of chunk to taste, it's the number of megabytes.
# 9  
Old 04-09-2014
Thank you very much for answering this tough question. This will be very useful to me!
You are great! Smilie
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Extract tar archive on remote server in another directory

HI All Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything is working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but when im... (9 Replies)
Discussion started by: BACya
9 Replies

2. Shell Programming and Scripting

Extract tar archive on remote server in another directory

HI Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but i have to ... (1 Reply)
Discussion started by: BACya
1 Replies

3. Shell Programming and Scripting

FTP a File to a Remote Server

Hi, I'm fairly new to Linux and need help with a script to ftp a .csv file to a remote server. I can ftp the .csv file manually from my server to the remote server in the "/" location. When I execute the script I receive the following message Could not create file. Here's what I have for... (5 Replies)
Discussion started by: Computergal2104
5 Replies

4. Shell Programming and Scripting

How to wait for tar to finish before sending the archive through ftp?

Hey all, I want to automate tarring a directory then using ftp to transfer the files over. I was able to put the commands together but what I'm noticing is that only the very first file is being tarred and then transferred. tar cvpf new.backup sourceAbove is the command I'm using which works... (4 Replies)
Discussion started by: Keepcase
4 Replies

5. UNIX for Advanced & Expert Users

ftp into remote hosts

Hi all, Have used ftp to transfer files from remote host to localhost. I was wondering how can I ftp into remote hosts. for example from a unix box, connect to an external server and then ftp that file into mainframe ? I would like to avoid pulling it to unix box and then ftping to mainframe.... (1 Reply)
Discussion started by: kodermanna
1 Replies

6. UNIX for Dummies Questions & Answers

To ftp file on a remote server

I want to send some files in .gz format from my desktop to a remote server. What will be the procedure for that thanks (5 Replies)
Discussion started by: supercops
5 Replies

7. UNIX for Advanced & Expert Users

remote ftp login without password

HI all, I need to post some files on to a clients machine and they said we can ftp without username and password. I do the same as a command line it works ftp <hostname>. but when I do that through a script it asks for user name and pasword. Can any one help me how to do a file ftp. ... (4 Replies)
Discussion started by: umathurumella
4 Replies

8. Solaris

backup through tar command on remote tape

Hello Everybody I have two servers, name A & B. I need to take a backup of one directory(/girish) on serverA. But my tape drive is in serverB through tar command. But when I run the following command it doesn't take the backup. Could any one correct my command to take a backup tar cvf - ... (0 Replies)
Discussion started by: girish.batra
0 Replies

9. UNIX for Dummies Questions & Answers

FTP - remote access

Hi All, I want to ftp a file from Windows 2000 to Unix. The file is present in a remote server which has been mapped to one of my drives. I am not able to connect to that directory. I am getting an error saying ?Invalid command 550 //hbxs02/shared No such file or directory. The... (0 Replies)
Discussion started by: shashi_kiran_v
0 Replies

10. UNIX for Dummies Questions & Answers

Remote execute a file via ftp

How can I execute a script on a unix server via ftp from a Windows machine?? Can't use cron/at to schedule the execution and don't want to open up a telnet session just to do it. I want to be able to kick it off after I send the script over on a nightly basis. Reason is script parameter changes... (4 Replies)
Discussion started by: giannicello
4 Replies
Login or Register to Ask a Question