Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp
Top Forums UNIX for Dummies Questions & Answers Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp Post 302896851 by Corona688 on Wednesday 9th of April 2014 01:51:51 PM
Old 04-09-2014
This problem's been coming up for a while, so here's a generic solution. It takes the command you want to run as arguments, running it once per chunk feeding it file data via STDIN, and substitutes @FNAME@ with an 8-digit incrementing sequence.

Code:
#!/usr/bin/perl

# Die screaming instead of silently if child fails
$SIG{PIPE} = sub { die("SIGPIPE"); };

# $block is the read size, 1 meg is probably OK.
# $chunk is the output size.  With a chunk of 2 and block of 1024*1024,
# it writes 2 meg files.
my $block=1024*1024, $chunk=2;

my $running=1, $ccount=0, @l=@ARGV;

while($running) {
        my $n, $fname=sprintf("%08d", $ccount++);
        my $fr=\$fname;

        # Use given arguments as a command, with @FNAME@ substituted
        # for an incrementing number like 00000001
        open(OUT, "|-",
                map { my $v=$_; $v =~ s/\@FNAME\@/${$fr}/; $v } @l
        );

        for($n=0; $n<$chunk; $n++)
        {
                my $bytes=read(STDIN, $data, $block);
                defined($bytes) || die ("Error reading STDIN");
                if($bytes < 1) { # Ran out of data, stop reading
                        $running=0;
                        last;
                }

                print OUT $data;
        }

        close(OUT);
}

printf(STDERR "Wrote %d chunks of %d bytes\n", $ccount, $block*$chunk);

Code:
$ chmod +x pipesplit.pl
$ tar cvzf - /backup | openssl aes-256-cbc -salt -k "password" |
        ./pipesplit.pl curl -u user:pass ftp://hostname/path/@FNAME@ -T -

100 2048k    0     0    0 2048k      0  8368k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8367k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8335k --:--:-- --:--:-- --:--:-- 8325k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8389k --:--:-- --:--:-- --:--:-- 8427k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5    0     0    0     5      0     21 --:--:-- --:--:-- --:--:--    21
Wrote 17 chunks of 2097152 bytes

$

Adjust the value of chunk to taste, it's the number of megabytes.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remote execute a file via ftp

How can I execute a script on a unix server via ftp from a Windows machine?? Can't use cron/at to schedule the execution and don't want to open up a telnet session just to do it. I want to be able to kick it off after I send the script over on a nightly basis. Reason is script parameter changes... (4 Replies)
Discussion started by: giannicello
4 Replies

2. UNIX for Dummies Questions & Answers

FTP - remote access

Hi All, I want to ftp a file from Windows 2000 to Unix. The file is present in a remote server which has been mapped to one of my drives. I am not able to connect to that directory. I am getting an error saying ?Invalid command 550 //hbxs02/shared No such file or directory. The... (0 Replies)
Discussion started by: shashi_kiran_v
0 Replies

3. Solaris

backup through tar command on remote tape

Hello Everybody I have two servers, name A & B. I need to take a backup of one directory(/girish) on serverA. But my tape drive is in serverB through tar command. But when I run the following command it doesn't take the backup. Could any one correct my command to take a backup tar cvf - ... (0 Replies)
Discussion started by: girish.batra
0 Replies

4. UNIX for Advanced & Expert Users

remote ftp login without password

HI all, I need to post some files on to a clients machine and they said we can ftp without username and password. I do the same as a command line it works ftp <hostname>. but when I do that through a script it asks for user name and pasword. Can any one help me how to do a file ftp. ... (4 Replies)
Discussion started by: umathurumella
4 Replies

5. UNIX for Dummies Questions & Answers

To ftp file on a remote server

I want to send some files in .gz format from my desktop to a remote server. What will be the procedure for that thanks (5 Replies)
Discussion started by: supercops
5 Replies

6. UNIX for Advanced & Expert Users

ftp into remote hosts

Hi all, Have used ftp to transfer files from remote host to localhost. I was wondering how can I ftp into remote hosts. for example from a unix box, connect to an external server and then ftp that file into mainframe ? I would like to avoid pulling it to unix box and then ftping to mainframe.... (1 Reply)
Discussion started by: kodermanna
1 Replies

7. Shell Programming and Scripting

How to wait for tar to finish before sending the archive through ftp?

Hey all, I want to automate tarring a directory then using ftp to transfer the files over. I was able to put the commands together but what I'm noticing is that only the very first file is being tarred and then transferred. tar cvpf new.backup sourceAbove is the command I'm using which works... (4 Replies)
Discussion started by: Keepcase
4 Replies

8. Shell Programming and Scripting

FTP a File to a Remote Server

Hi, I'm fairly new to Linux and need help with a script to ftp a .csv file to a remote server. I can ftp the .csv file manually from my server to the remote server in the "/" location. When I execute the script I receive the following message Could not create file. Here's what I have for... (5 Replies)
Discussion started by: Computergal2104
5 Replies

9. Shell Programming and Scripting

Extract tar archive on remote server in another directory

HI Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but i have to ... (1 Reply)
Discussion started by: BACya
1 Replies

10. UNIX for Dummies Questions & Answers

Extract tar archive on remote server in another directory

HI All Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything is working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but when im... (9 Replies)
Discussion started by: BACya
9 Replies

LEARN ABOUT DEBIAN

bup

bup(1)							      General Commands Manual							    bup(1)

NAME

       bup - Backup program using rolling checksums and git file formats

SYNOPSIS

       bup [global options...] <command> [options...]

DESCRIPTION

       bup is a program for making backups of your files using the git file format.

       Unlike  git(1)  itself, bup is optimized for handling huge data sets including individual very large files (such a virtual machine images).
       However, once a backup set is created, it can still be accessed using git tools.

       The individual bup subcommands appear in their own man pages.

GLOBAL OPTIONS

       --version
	      print bup's version number.  Equivalent to bup-version(1)

       -d, --bup-dir=BUP_DIR
	      use the given BUP_DIR parameter as the bup repository location, instead of reading it from  the  $BUP_DIR  environment  variable	or
	      using the default ~/.bup location.

COMMONLY USED SUBCOMMANDS

       bup-fsck(1)
	      Check backup sets for damage and add redundancy information

       bup-ftp(1)
	      Browse backup sets using an ftp-like client

       bup-fuse(1)
	      Mount your backup sets as a filesystem

       bup-help(1)
	      Print detailed help for the given command

       bup-index(1)
	      Create or display the index of files to back up

       bup-on(1)
	      Backup a remote machine to the local one

       bup-restore(1)
	      Extract files from a backup set

       bup-save(1)
	      Save files into a backup set (note: run "bup index" first)

       bup-web(1)
	      Launch a web server to examine backup sets

RARELY USED SUBCOMMANDS

       bup-damage(1)
	      Deliberately destroy data

       bup-drecurse(1)
	      Recursively list files in your filesystem

       bup-init(1)
	      Initialize a bup repository

       bup-join(1)
	      Retrieve a file backed up using bup-split(1)

       bup-ls(1)
	      Browse the files in your backup sets

       bup-margin(1)
	      Determine how close your bup repository is to armageddon

       bup-memtest(1)
	      Test bup memory usage statistics

       bup-midx(1)
	      Index objects to speed up future backups

       bup-newliner(1)
	      Make sure progress messages don't overlap with output

       bup-random(1)
	      Generate a stream of random output

       bup-server(1)
	      The server side of the bup client-server relationship

       bup-split(1)
	      Split a single file into its own backup set

       bup-tick(1)
	      Wait for up to one second.

       bup-version(1)
	      Report the version number of your copy of bup.

SEE ALSO

       git(1) and the README file from the bup distribution.

       The home of bup is at <http://github.com/apenwarr/bup/>.

AUTHORS

       Avery Pennarun <apenwarr@gmail.com>.

Bup unknown-																    bup(1)
All times are GMT -4. The time now is 07:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy