Sponsored Content
Full Discussion: FTP a huge Size file
Top Forums Shell Programming and Scripting FTP a huge Size file Post 302548222 by Chubler_XL on Wednesday 17th of August 2011 10:28:54 PM
Old 08-17-2011
Quote:
Originally Posted by Naga06
Dear All,
I cannot compress / split it as the originality of the file is very important and cannot take even 0.1% risk. Bcoz when the system crashes, using this file I'll be able to restore the entire system.
Why not do a md5sum of the file before you send it and check it on the receving end:

Code:
source_box $ md5sum mybigfile.tar
81836bd568b30ec974bff32af98458d1 *mybigfile.tar
source_box $ gzip mybigfile.tar
source_box $ rsync ./mybigfile.tar.gz dest_box:mybigfile.tar.gz
 
dest_box$ gzip -d mybigfile.tar.gz
dest_box$ $ md5sum mybigfile.tar
81836bd568b30ec974bff32af98458d1 *mybigfile.tar

This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

ftp file size

how to compare file size which has been received through ftp get from a remote location with local copy available any clue regards (5 Replies)
Discussion started by: sathiya
5 Replies

2. UNIX for Dummies Questions & Answers

ftp hangs on file size = 0

I have an ftp process which runs every 10 minutes between Unix and an NT box. Normally it works, but when the script tries to get a file from NT that has a length of 0, the ftp process hangs (as if it is still waiting for the end of the file). This is the script... (3 Replies)
Discussion started by: mheinrich
3 Replies

3. Shell Programming and Scripting

How to compare file size after ftp?

Is possible if I want to campare file size on source and destination after ftp transfer? If anybody know, please explain to me. (1 Reply)
Discussion started by: icemania
1 Replies

4. UNIX for Advanced & Expert Users

Need to synchronize filesystems with huge size

Hello, I am using IXOS 3rd party utility to maintain invoices (images) in my client's SAP system. We have a DR (Disaster Recovery) setup & for that I need to synchronize my /ixos filesystem (with size more than 50 GB) between two servers say x182 & x050. Initially I thought to tar the... (4 Replies)
Discussion started by: vishal_ranjan
4 Replies

5. Shell Programming and Scripting

Checking the size of a file after FTP

Hi I am doing a FTP process through which I am copying a file from my local server to Remote server. After this I want to check the size of the file Below is my program: LOCALDIR=/batch/ediprocess REMOTESERVER=test.appl.com REMOTEPATH=batch/ftpTest LOGIN=px PASSWORD=abcd ftp -n... (3 Replies)
Discussion started by: shanth_chandra
3 Replies

6. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

7. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

8. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

9. AIX

FTP huge file transfer

Hi, I need to transfer 2000 files from one host to another.. I modified /etc/security/limits to -1 and ulimit -f, ulimit -s, ulimit -a.. Even then only 700 files are transferred. Could You please help me to sort out this issue.. I think some configuration related to memory is... (3 Replies)
Discussion started by: Priya Amaresh
3 Replies

10. UNIX for Advanced & Expert Users

FTP zero size file

Hi AM using unix Aix Ksh I need to clarify regarding sending the zero size file to another server. VAL=ftp.sh -c put souce_file Dest_file $1 $2 $3 $4 $5 $VAL 2 > $ERR When am Sending the Zero Size File getting alerts netout write returned Zero. But i would like to know $VAL... (4 Replies)
Discussion started by: Venkatesh1
4 Replies
RDUP-BACKUPS(7) 						       rdup							   RDUP-BACKUPS(7)

NAME
rdup-backups - introduction into making backups with rdup INTRODUCTION
rdup is a simple program that prints out a list of files and directories that are changed changed on a filesystem. It is more sophisticated than for instance find, because rdup will find files that are removed or directories that are renamed. A long time ago rdup included a bunch of shell and Perl scripts that implemented a backup policy. These could be used in a pipeline to per- form a backup. Currently rdup consists out of three basic utilities: rdup With rdup you create the file list on which later programs in the pipeline can work. The default output format also includes the files' content. rdup can be seen as a tar replacement in this respect, but rdup also allows for all kinds of transformations of the content (encryption, compression, reversal), see the -P switch in rdup(1) for more information. rdup-tr With rdup-tr you can transform the files rdup delivers to you. You can create tar, cpio or pax files. You can encrypt pathnames. rdup-tr is filter that reads from standard input and writes to standard output. See rdup-tr(1) for more information. With rdup and rdup-tr you can create an encrypted archive which is put in a directory structure that is also encrypted. rdup-up With rdup-up you can update an existing directory structure with the updates as described by rdup. rdup-up reads rdup input and will create the files, symbolic links, hard links and directories (and sockets, pipes and devices) in the file system. See rdup-up(1) for more information. So the general backup pipeline for rdup will look something like this: create filelist | transform | update filesystem ( rdup | rdup-tr | rdup-up ) Note 1: The same sequence is used for restoring. In both cases you want to move files from location A to B. The only difference is that the transformation is reversed when you restore. Note 2: The use of rdup-tr is optional. BACKUPS AND RESTORES
For rdup there is no difference between backups and restores. If you think about this for a minute you understand why. Making a backup means copying a list of files somewhere else. Restoring files is copying a list of files back to the place they came from. Same difference. So rdup can be used for both, if you did any transformation with rdup during the backup you just need to reverse those operations during the restore. BACKUPS
It is always best to backup to another medium, be it a different local harddisk or a NFS/CIFS mounted filesystem. You can also use ssh to store file on a remote server, ala rsync (although not as network efficient). If you backup to a local disk you can just as well use rsync or plain old tar, but if you store your files at somebody else's disk you will need encryption. This is where you go beyond rsync and rdup comes in. Rsync cannot do per-file encryption, sure you can encrypt the network traffic with ssh, but at the remote side your files are kept in plain view. If you implement remote backups, the easy route is to allow root access on the backup medium. If the backup runs without root access the created files will not have their original ownership. For NFS this can be achieved by using no_root_squash, for ssh you could enable PermitRootLogin. Note that this may be a security risk. SNAPSHOT BACKUPS
We need a little help here in the form of the rdup-simple script. Keep in mind that the following scripts can also be run remotely with the help of ssh. The following script implements the algorithm of rdup-simple. #!/bin/bash # some tmp files are saved in ~/.rdup. This directory must exist DIR=/home # what to backup BACKUP=/vol/backup TODAY=$(date +%Y%m/%d) LIST=~/.rdup/list-$HOSTNAME STAMP=~/.rdup/timestamp-$HOSTNAME # for remote backup, this has to run on the remote host! BUGBUG RET=$? case $RET in 2|*) echo Error >&2 exit 1 ;; 1) # full dump, remove file-list and time-stamp file rm $LIST $STAMP ;; 0) # inc dump # do nothing here ;; esac # this is the place where you want to modify the command line # right now, nothing is translated we just use 'cat' rdup -N $STAMP -Pcat $LIST $DIR | rdup-up $BACKUP/$HOSTNAME/$TODAY # or do a remote backup #rdup -N $STAMP -Pcat $LIST $DIR | ssh root@remotehost # rdup-up $BACKUP/$HOSTNAME/$TODAY LOCAL BACKUPS
With rdup-simple you can easily create backups. Backing up my home directory to a backup directory: rdup-simple ~ /vol/backup/$HOSTNAME This will create a backup in /vol/backup/$HOSTNAME/200705/15. So each day will have its own directory. Multiple sources are allowed, so: rdup-simple ~ /etc/ /var/lib /vol/backup/$HOSTNAME Will backup your home directory, /etc and /var/lib to the backup location. Also if you need to compress your backup, simple add a '-z' switch: rdup-simple -z ~ /etc/ /var/lib /vol/backup/$HOSTNAME REMOTE BACKUPS
For a remote backup to work, both the sending machine and the receiving machine must have rdup installed. The currently implemented proto- col is ssh. Dumping my homedir to the remote server: rdup-simple ~ ssh://miekg@remote/vol/backup/$HOSTNAME The syntax is almost identical, only the destination starts with the magic string 'ssh://'. Compression and encryption are just as easily enabled as with a local backup, just add '-z' and/or a '-k keyfile' argument: rdup-simple -z -k 'secret-file' ~ ssh://miekg@remote/vol/backup/$HOSTNAME Remember though, that because of these advanced features (compression, encryption, etc, ...) the network transfer can never be as efficient as rsync. ALSO SEE
rdup(1), rdup-tr(1), rdup-up(1) and http://www.miek.nl/projects/rdup/ 1.1.x 15 Dec 2008 RDUP-BACKUPS(7)
All times are GMT -4. The time now is 09:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy