Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp
Top Forums UNIX for Dummies Questions & Answers Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp Post 302896851 by Corona688 on Wednesday 9th of April 2014 01:51:51 PM
Old 04-09-2014
This problem's been coming up for a while, so here's a generic solution. It takes the command you want to run as arguments, running it once per chunk feeding it file data via STDIN, and substitutes @FNAME@ with an 8-digit incrementing sequence.

Code:
#!/usr/bin/perl

# Die screaming instead of silently if child fails
$SIG{PIPE} = sub { die("SIGPIPE"); };

# $block is the read size, 1 meg is probably OK.
# $chunk is the output size.  With a chunk of 2 and block of 1024*1024,
# it writes 2 meg files.
my $block=1024*1024, $chunk=2;

my $running=1, $ccount=0, @l=@ARGV;

while($running) {
        my $n, $fname=sprintf("%08d", $ccount++);
        my $fr=\$fname;

        # Use given arguments as a command, with @FNAME@ substituted
        # for an incrementing number like 00000001
        open(OUT, "|-",
                map { my $v=$_; $v =~ s/\@FNAME\@/${$fr}/; $v } @l
        );

        for($n=0; $n<$chunk; $n++)
        {
                my $bytes=read(STDIN, $data, $block);
                defined($bytes) || die ("Error reading STDIN");
                if($bytes < 1) { # Ran out of data, stop reading
                        $running=0;
                        last;
                }

                print OUT $data;
        }

        close(OUT);
}

printf(STDERR "Wrote %d chunks of %d bytes\n", $ccount, $block*$chunk);

Code:
$ chmod +x pipesplit.pl
$ tar cvzf - /backup | openssl aes-256-cbc -salt -k "password" |
        ./pipesplit.pl curl -u user:pass ftp://hostname/path/@FNAME@ -T -

100 2048k    0     0    0 2048k      0  8368k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8367k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8335k --:--:-- --:--:-- --:--:-- 8325k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8389k --:--:-- --:--:-- --:--:-- 8427k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5    0     0    0     5      0     21 --:--:-- --:--:-- --:--:--    21
Wrote 17 chunks of 2097152 bytes

$

Adjust the value of chunk to taste, it's the number of megabytes.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remote execute a file via ftp

How can I execute a script on a unix server via ftp from a Windows machine?? Can't use cron/at to schedule the execution and don't want to open up a telnet session just to do it. I want to be able to kick it off after I send the script over on a nightly basis. Reason is script parameter changes... (4 Replies)
Discussion started by: giannicello
4 Replies

2. UNIX for Dummies Questions & Answers

FTP - remote access

Hi All, I want to ftp a file from Windows 2000 to Unix. The file is present in a remote server which has been mapped to one of my drives. I am not able to connect to that directory. I am getting an error saying ?Invalid command 550 //hbxs02/shared No such file or directory. The... (0 Replies)
Discussion started by: shashi_kiran_v
0 Replies

3. Solaris

backup through tar command on remote tape

Hello Everybody I have two servers, name A & B. I need to take a backup of one directory(/girish) on serverA. But my tape drive is in serverB through tar command. But when I run the following command it doesn't take the backup. Could any one correct my command to take a backup tar cvf - ... (0 Replies)
Discussion started by: girish.batra
0 Replies

4. UNIX for Advanced & Expert Users

remote ftp login without password

HI all, I need to post some files on to a clients machine and they said we can ftp without username and password. I do the same as a command line it works ftp <hostname>. but when I do that through a script it asks for user name and pasword. Can any one help me how to do a file ftp. ... (4 Replies)
Discussion started by: umathurumella
4 Replies

5. UNIX for Dummies Questions & Answers

To ftp file on a remote server

I want to send some files in .gz format from my desktop to a remote server. What will be the procedure for that thanks (5 Replies)
Discussion started by: supercops
5 Replies

6. UNIX for Advanced & Expert Users

ftp into remote hosts

Hi all, Have used ftp to transfer files from remote host to localhost. I was wondering how can I ftp into remote hosts. for example from a unix box, connect to an external server and then ftp that file into mainframe ? I would like to avoid pulling it to unix box and then ftping to mainframe.... (1 Reply)
Discussion started by: kodermanna
1 Replies

7. Shell Programming and Scripting

How to wait for tar to finish before sending the archive through ftp?

Hey all, I want to automate tarring a directory then using ftp to transfer the files over. I was able to put the commands together but what I'm noticing is that only the very first file is being tarred and then transferred. tar cvpf new.backup sourceAbove is the command I'm using which works... (4 Replies)
Discussion started by: Keepcase
4 Replies

8. Shell Programming and Scripting

FTP a File to a Remote Server

Hi, I'm fairly new to Linux and need help with a script to ftp a .csv file to a remote server. I can ftp the .csv file manually from my server to the remote server in the "/" location. When I execute the script I receive the following message Could not create file. Here's what I have for... (5 Replies)
Discussion started by: Computergal2104
5 Replies

9. Shell Programming and Scripting

Extract tar archive on remote server in another directory

HI Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but i have to ... (1 Reply)
Discussion started by: BACya
1 Replies

10. UNIX for Dummies Questions & Answers

Extract tar archive on remote server in another directory

HI All Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything is working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but when im... (9 Replies)
Discussion started by: BACya
9 Replies

LEARN ABOUT DEBIAN

backup-manager-upload

BACKUP-MANAGER-UPLOAD(8)				       backup-manager-upload					  BACKUP-MANAGER-UPLOAD(8)

NAME

       backup-manager-upload - Multiprotocol uploader for backup-manager.

SYNOPSIS

       backup-manager-upload [options] date

DESCRIPTION

       backup-manager-upload will upload all the archives generated on the given date to the specified host with either ftp or scp.  It's also
       possible to use this program for uploading data to an Amazon S3 account.  Some metadates are available like "today" or "yesterday".

REQUIRED ARGS

       --mode=transfer-mode
	   Select the transfer mode to use : ftp, scp, or s3.

       --host=hostname1,hostname2,...,hostnameN
	   Select a list of remote hosts to connect to.

       --user=username
	   Select the user to use for connection.

OPTIONAL ARGS

       --password=password
	   Select the ftp user's password (only needed for ftp transfers).

       --key=path_to_private_key
	   Select the ssh private key file to use when opening the ssh session for scp transfer.  Obviously, this is only needed for scp transfer
	   mode.  If you don't specify a key file, the user's default private key will be used.

       --directory=directory
	   Select the location on the remote host where files will be uploaded.  Default is /backup/uploads.

       --bucket=bucket
	   Sets the bucket name for the Amazon S3 service backup into.

       --root=directory
	   Select the local directory where files are.	Default is /var/archives

       --gpg-recipient=gpg-recipient
	   Select the gpg public key for encryptiong the archives when uploading with the method ssh-gpg. This can be a short or long key id or a
	   descriptive name. The precise syntax is described in the gpg man page.

       --list
	   Just list the files to upload.

       --ftp-purge
	   Purge the remote directory before uploading files in FTP mode.

       --s3-purge
	   Purge the remote directory before uploading files in FTP mode.

       --ssh-purge
	   Purge the remote directory before uploading files in SSH mode.

       --verbose
	   Flag to enable verbose mode.

       date
	   Date pattern to select some files to upload, can be a valid date (YYYYMMDD) or 'today' or 'yesterday'.

ERROR CODES

       If something goes wrong during an upload, backup-manager-upload will exit with a non null value. In such a case every error messages are
       sent to STDERR.

       Here are the possible error codes:

       bad command line (wrong arguments) : 10
       FTP transfer failure : 20
       SCP transfer failure : 21
       S3 transfer failure : 22
       Unknown upload method: 23

SEE ALSO

       backup-manager(3)

AUTHORS

       Alexis Sukrieh - main code and design

       Brad Dixon - Amazon S3 upload method

       Jan Metzger - ssh-gpg upload method

perl v5.14.2							    2012-05-09						  BACKUP-MANAGER-UPLOAD(8)
All times are GMT -4. The time now is 08:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy