Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp Post 302896851 by Corona688 on Wednesday 9th of April 2014 01:51:51 PM
Old 04-09-2014
This problem's been coming up for a while, so here's a generic solution. It takes the command you want to run as arguments, running it once per chunk feeding it file data via STDIN, and substitutes @FNAME@ with an 8-digit incrementing sequence.

Code:
#!/usr/bin/perl

# Die screaming instead of silently if child fails
$SIG{PIPE} = sub { die("SIGPIPE"); };

# $block is the read size, 1 meg is probably OK.
# $chunk is the output size.  With a chunk of 2 and block of 1024*1024,
# it writes 2 meg files.
my $block=1024*1024, $chunk=2;

my $running=1, $ccount=0, @l=@ARGV;

while($running) {
        my $n, $fname=sprintf("%08d", $ccount++);
        my $fr=\$fname;

        # Use given arguments as a command, with @FNAME@ substituted
        # for an incrementing number like 00000001
        open(OUT, "|-",
                map { my $v=$_; $v =~ s/\@FNAME\@/${$fr}/; $v } @l
        );

        for($n=0; $n<$chunk; $n++)
        {
                my $bytes=read(STDIN, $data, $block);
                defined($bytes) || die ("Error reading STDIN");
                if($bytes < 1) { # Ran out of data, stop reading
                        $running=0;
                        last;
                }

                print OUT $data;
        }

        close(OUT);
}

printf(STDERR "Wrote %d chunks of %d bytes\n", $ccount, $block*$chunk);

Code:
$ chmod +x pipesplit.pl
$ tar cvzf - /backup | openssl aes-256-cbc -salt -k "password" |
        ./pipesplit.pl curl -u user:pass ftp://hostname/path/@FNAME@ -T -

100 2048k    0     0    0 2048k      0  8368k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8367k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8335k --:--:-- --:--:-- --:--:-- 8325k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8389k --:--:-- --:--:-- --:--:-- 8427k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5    0     0    0     5      0     21 --:--:-- --:--:-- --:--:--    21
Wrote 17 chunks of 2097152 bytes

$

Adjust the value of chunk to taste, it's the number of megabytes.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remote execute a file via ftp

How can I execute a script on a unix server via ftp from a Windows machine?? Can't use cron/at to schedule the execution and don't want to open up a telnet session just to do it. I want to be able to kick it off after I send the script over on a nightly basis. Reason is script parameter changes... (4 Replies)
Discussion started by: giannicello
4 Replies

2. UNIX for Dummies Questions & Answers

FTP - remote access

Hi All, I want to ftp a file from Windows 2000 to Unix. The file is present in a remote server which has been mapped to one of my drives. I am not able to connect to that directory. I am getting an error saying ?Invalid command 550 //hbxs02/shared No such file or directory. The... (0 Replies)
Discussion started by: shashi_kiran_v
0 Replies

3. Solaris

backup through tar command on remote tape

Hello Everybody I have two servers, name A & B. I need to take a backup of one directory(/girish) on serverA. But my tape drive is in serverB through tar command. But when I run the following command it doesn't take the backup. Could any one correct my command to take a backup tar cvf - ... (0 Replies)
Discussion started by: girish.batra
0 Replies

4. UNIX for Advanced & Expert Users

remote ftp login without password

HI all, I need to post some files on to a clients machine and they said we can ftp without username and password. I do the same as a command line it works ftp <hostname>. but when I do that through a script it asks for user name and pasword. Can any one help me how to do a file ftp. ... (4 Replies)
Discussion started by: umathurumella
4 Replies

5. UNIX for Dummies Questions & Answers

To ftp file on a remote server

I want to send some files in .gz format from my desktop to a remote server. What will be the procedure for that thanks (5 Replies)
Discussion started by: supercops
5 Replies

6. UNIX for Advanced & Expert Users

ftp into remote hosts

Hi all, Have used ftp to transfer files from remote host to localhost. I was wondering how can I ftp into remote hosts. for example from a unix box, connect to an external server and then ftp that file into mainframe ? I would like to avoid pulling it to unix box and then ftping to mainframe.... (1 Reply)
Discussion started by: kodermanna
1 Replies

7. Shell Programming and Scripting

How to wait for tar to finish before sending the archive through ftp?

Hey all, I want to automate tarring a directory then using ftp to transfer the files over. I was able to put the commands together but what I'm noticing is that only the very first file is being tarred and then transferred. tar cvpf new.backup sourceAbove is the command I'm using which works... (4 Replies)
Discussion started by: Keepcase
4 Replies

8. Shell Programming and Scripting

FTP a File to a Remote Server

Hi, I'm fairly new to Linux and need help with a script to ftp a .csv file to a remote server. I can ftp the .csv file manually from my server to the remote server in the "/" location. When I execute the script I receive the following message Could not create file. Here's what I have for... (5 Replies)
Discussion started by: Computergal2104
5 Replies

9. Shell Programming and Scripting

Extract tar archive on remote server in another directory

HI Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but i have to ... (1 Reply)
Discussion started by: BACya
1 Replies

10. UNIX for Dummies Questions & Answers

Extract tar archive on remote server in another directory

HI All Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything is working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but when im... (9 Replies)
Discussion started by: BACya
9 Replies
FTPFS(4)						     Kernel Interfaces Manual							  FTPFS(4)

NAME
ftpfs - file transfer protocol (FTP) file system SYNOPSIS
ftpfs [ -/dq ] [ -m mountpoint ] [ -a password ] system DESCRIPTION
Ftpfs dials the TCP file transfer protocol (FTP) port, 21, on system and mounts itself (see bind(2)) on mountpoint (default /n/ftp) to pro- vide access to files on the remote machine. If required by the remote machine, ftpfs will prompt for a user name and password. The user names ftp and anonymous conventionally offer guest/read-only access to machines. Anonymous FTP may be called without user interaction by using the -a option and specifying the password. By default the file seen at the mount point is the user's remote home directory. The option -/ forces the mount point to correspond to the remote root. To avoid seeing startup messages from the server use option -q. To see all messages from the server use option -d. To terminate the connection, unmount (see bind(1)) the mount point. EXAMPLE
You want anonymous FTP access to the system export.lcs.mit.edu. The first import(4) command is only necessary if your machine does not have access to the desired system, but another, called gateway in this example, does. import gateway /net ftpfs -a yourname@yourmachine export.lcs.mit.edu SOURCE
/sys/src/cmd/ftpfs SEE ALSO
bind(2) BUGS
Symbolic links on remote Unix systems will always have mode 0777 and a length of 8. After connecting to a TOPS-20 system, the mount point will contain only one directory, usually /n/ftp/PS:<ANONYMOUS>. However, walking to any valid directory on that machine will succeed and cause that directory entry to appear under the mount point. Ftpfs caches files and directories. A directory will fall from the cache after 5 quiescent minutes or if the local user changes the direc- tory by writing or removing a file. Otherwise, remote changes to the directory that occur after the directory has been cached might not be immediately visible. There is no way to issue the appropriate commands to handle special synthetic FTP file types such as directories that automatically return a tar of their contents. FTPFS(4)
All times are GMT -4. The time now is 05:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy