Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Remote FTP Backup -Tar archive+ encrypt+ split to a remote ftp Post 302896851 by Corona688 on Wednesday 9th of April 2014 01:51:51 PM
Old 04-09-2014
This problem's been coming up for a while, so here's a generic solution. It takes the command you want to run as arguments, running it once per chunk feeding it file data via STDIN, and substitutes @FNAME@ with an 8-digit incrementing sequence.

Code:
#!/usr/bin/perl

# Die screaming instead of silently if child fails
$SIG{PIPE} = sub { die("SIGPIPE"); };

# $block is the read size, 1 meg is probably OK.
# $chunk is the output size.  With a chunk of 2 and block of 1024*1024,
# it writes 2 meg files.
my $block=1024*1024, $chunk=2;

my $running=1, $ccount=0, @l=@ARGV;

while($running) {
        my $n, $fname=sprintf("%08d", $ccount++);
        my $fr=\$fname;

        # Use given arguments as a command, with @FNAME@ substituted
        # for an incrementing number like 00000001
        open(OUT, "|-",
                map { my $v=$_; $v =~ s/\@FNAME\@/${$fr}/; $v } @l
        );

        for($n=0; $n<$chunk; $n++)
        {
                my $bytes=read(STDIN, $data, $block);
                defined($bytes) || die ("Error reading STDIN");
                if($bytes < 1) { # Ran out of data, stop reading
                        $running=0;
                        last;
                }

                print OUT $data;
        }

        close(OUT);
}

printf(STDERR "Wrote %d chunks of %d bytes\n", $ccount, $block*$chunk);

Code:
$ chmod +x pipesplit.pl
$ tar cvzf - /backup | openssl aes-256-cbc -salt -k "password" |
        ./pipesplit.pl curl -u user:pass ftp://hostname/path/@FNAME@ -T -

100 2048k    0     0    0 2048k      0  8368k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8367k --:--:-- --:--:-- --:--:-- 8393k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8335k --:--:-- --:--:-- --:--:-- 8325k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
...
                                 Dload  Upload   Total   Spent    Left  Speed
100 2048k    0     0    0 2048k      0  8389k --:--:-- --:--:-- --:--:-- 8427k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5    0     0    0     5      0     21 --:--:-- --:--:-- --:--:--    21
Wrote 17 chunks of 2097152 bytes

$

Adjust the value of chunk to taste, it's the number of megabytes.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remote execute a file via ftp

How can I execute a script on a unix server via ftp from a Windows machine?? Can't use cron/at to schedule the execution and don't want to open up a telnet session just to do it. I want to be able to kick it off after I send the script over on a nightly basis. Reason is script parameter changes... (4 Replies)
Discussion started by: giannicello
4 Replies

2. UNIX for Dummies Questions & Answers

FTP - remote access

Hi All, I want to ftp a file from Windows 2000 to Unix. The file is present in a remote server which has been mapped to one of my drives. I am not able to connect to that directory. I am getting an error saying ?Invalid command 550 //hbxs02/shared No such file or directory. The... (0 Replies)
Discussion started by: shashi_kiran_v
0 Replies

3. Solaris

backup through tar command on remote tape

Hello Everybody I have two servers, name A & B. I need to take a backup of one directory(/girish) on serverA. But my tape drive is in serverB through tar command. But when I run the following command it doesn't take the backup. Could any one correct my command to take a backup tar cvf - ... (0 Replies)
Discussion started by: girish.batra
0 Replies

4. UNIX for Advanced & Expert Users

remote ftp login without password

HI all, I need to post some files on to a clients machine and they said we can ftp without username and password. I do the same as a command line it works ftp <hostname>. but when I do that through a script it asks for user name and pasword. Can any one help me how to do a file ftp. ... (4 Replies)
Discussion started by: umathurumella
4 Replies

5. UNIX for Dummies Questions & Answers

To ftp file on a remote server

I want to send some files in .gz format from my desktop to a remote server. What will be the procedure for that thanks (5 Replies)
Discussion started by: supercops
5 Replies

6. UNIX for Advanced & Expert Users

ftp into remote hosts

Hi all, Have used ftp to transfer files from remote host to localhost. I was wondering how can I ftp into remote hosts. for example from a unix box, connect to an external server and then ftp that file into mainframe ? I would like to avoid pulling it to unix box and then ftping to mainframe.... (1 Reply)
Discussion started by: kodermanna
1 Replies

7. Shell Programming and Scripting

How to wait for tar to finish before sending the archive through ftp?

Hey all, I want to automate tarring a directory then using ftp to transfer the files over. I was able to put the commands together but what I'm noticing is that only the very first file is being tarred and then transferred. tar cvpf new.backup sourceAbove is the command I'm using which works... (4 Replies)
Discussion started by: Keepcase
4 Replies

8. Shell Programming and Scripting

FTP a File to a Remote Server

Hi, I'm fairly new to Linux and need help with a script to ftp a .csv file to a remote server. I can ftp the .csv file manually from my server to the remote server in the "/" location. When I execute the script I receive the following message Could not create file. Here's what I have for... (5 Replies)
Discussion started by: Computergal2104
5 Replies

9. Shell Programming and Scripting

Extract tar archive on remote server in another directory

HI Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but i have to ... (1 Reply)
Discussion started by: BACya
1 Replies

10. UNIX for Dummies Questions & Answers

Extract tar archive on remote server in another directory

HI All Please suggest how to untar archive on remote sever. When im trying use regular command without any flags everything is working fine: $( ssh <user>@<server> -n '. ~/.profile >/dev/null 2>&1 ; cd /path_1 ; copiedIVR_name=`ls -tr | tail -1` ; tar xvf $copiedIVR_name ' ) but when im... (9 Replies)
Discussion started by: BACya
9 Replies
pack_fopen_chunk(3alleg4)					  Allegro manual					 pack_fopen_chunk(3alleg4)

NAME
pack_fopen_chunk - Opens a sub-chunk of a file. Allegro game programming library. SYNOPSIS
#include <allegro.h> PACKFILE *pack_fopen_chunk(PACKFILE *f, int pack); DESCRIPTION
Opens a sub-chunk of a file. Chunks are primarily intended for use by the datafile code, but they may also be useful for your own file rou- tines. A chunk provides a logical view of part of a file, which can be compressed as an individual entity and will automatically insert and check length counts to prevent reading past the end of the chunk. The PACKFILE parameter is a previously opened file, and `pack' is a bool- ean parameter which will turn compression on for the sub-chunk if it is non-zero. Example: PACKFILE *output = pack_fopen("out.raw", "w!"); ... /* Create a sub-chunk with compression. */ output = pack_fopen_chunk(output, 1); if (!output) abort_on_error("Error saving data!"); /* Write some data to the sub-chunk. */ ... /* Close the sub-chunk, recovering parent file. */ output = pack_fclose_chunk(output); The data written to the chunk will be prefixed with two length counts (32-bit, a.k.a. big-endian). For uncompressed chunks these will both be set to the size of the data in the chunk. For compressed chunks (created by setting the `pack' flag), the first length will be the raw size of the chunk, and the second will be the negative size of the uncompressed data. To read the chunk, use the following code: PACKFILE *input = pack_fopen("out.raw", "rp"); ... input = pack_fopen_chunk(input, 1); /* Read data from the sub-chunk and close it. */ ... input = pack_fclose_chunk(input); This sequence will read the length counts created when the chunk was written, and automatically decompress the contents of the chunk if it was compressed. The length will also be used to prevent reading past the end of the chunk (Allegro will return EOF if you attempt this), and to automatically skip past any unread chunk data when you call pack_fclose_chunk(). Chunks can be nested inside each other by making repeated calls to pack_fopen_chunk(). When writing a file, the compression status is inherited from the parent file, so you only need to set the pack flag if the parent is not compressed but you want to pack the chunk data. If the parent file is already open in packed mode, setting the pack flag will result in data being compressed twice: once as it is written to the chunk, and again as the chunk passes it on to the parent file. RETURN VALUE
Returns a pointer to the sub-chunked PACKFILE, or NULL if there was some error (eg. you are using a custom PACKFILE vtable). SEE ALSO
pack_fclose_chunk(3alleg4), pack_fopen(3alleg4) Allegro version 4.4.2 pack_fopen_chunk(3alleg4)
All times are GMT -4. The time now is 01:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy