Visit Our UNIX and Linux User Community


Trying to copy a TON of files with a script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Trying to copy a TON of files with a script
# 1  
Old 11-05-2009
Trying to copy a TON of files with a script

Ok, here's the problem. I need to copy about 200 GB from an old FreeBSD (5.2.1) server to a new Debian server. The old server's fastest port is the ethernet port. I set up an NFS server on the new machine so I can just copy the files over using regular commands/scripts etc. Thing is, I probably shouldn't copy dozens of GB of files using the 'cp' command. The last time I tried that it crashed the server.

My plan is to write a script that will process a list of files copying each file from the list one at a time (while making a separate list of files successfully copied that way if anything goes wrong, I can difference the two lists and pick up where it left off). More importantly, I can copy all files ahead of time and then generate a smaller list of files modified since the original copy so that on migration day I have a minimal number of files to copy minimizing downtime.

The main problem (that I can see) with this is, any list I generate will contain both files and folders, but the folders don't need to be copied, only recreated with identical owners and permissions. Unless there is a flag for the 'cp' command that copies a folder without it's contents (not that I know of), I need to find some other way to test each path to see what it is, then if it is a directory, somehow find the permissions and owner and duplicate it on the new server.

If anyone has any advice, can see any other problems or knows of a better way I might do this, I'd appreciate the help. Thanks!
# 2  
Old 11-05-2009
sounds to me like you want to use rsync
# 3  
Old 11-05-2009
wow, that's exactly what I want. I'm always trying to reinvent the wheel it seems

I installed rsync on my Debian box (the FreeBSD already had it) and then ran this from the BSD box:

rsync -r -H -p -o -g -t -v -e ssh /users4 root@myserver.domain:/

Everything seems to be working great. And I can't believe it actually compares parts of files. That will make transferring the large mail files so much easier. I'll have the server migrated in less than an hour during lunch with this tool Smilie
# 4  
Old 11-06-2009
Yes, rsync is wonderful.

Now that you know that, take a look at Rsnapshot, which uses rsync to create backup sets. I found the docs a little confusing so I wrote Using Rsnapshot

Mac OS X uses a scheme just like this for their Time Machine backups. They have a few tricks rsnapshot lacks, but the basic idea is the same and it's definitely a convenient scheme - thanks to rsync!
# 5  
Old 11-06-2009
Ok, that's even more awesome! I will definitely bookmark that and take a look once the new machine is up and running. Incremental backups would be wonderful.

Previous Thread | Next Thread
Test Your Knowledge in Computers #455
Difficulty: Medium
Java was originally developed at Oracle starting in December 1990.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to copy files from on folder to another

I am trying to copy files with specific date and name to another folder. I am very new to shell scripting so i am finding it hard to do that. see the sample code i have written below. srcdir="/media/ubuntu/CA52057F5205720D/Users/st4r8_000/Desktop/office work/26 nov"... (13 Replies)
Discussion started by: Aqeel Abbas
13 Replies

2. Shell Programming and Scripting

Error on script to copy files

Hi I have the following script to copy files from one directory to another: #!/bin/sh touch -mt 201304240000 /var/tmp/ref1 touch -mt 201305152359 /var/tmp/ref2 find /moneta_collected02/in_psl -type f \( -newer /var/tmp/ref1 -a ! -newer /var/tmp/ref2 \) > file_lst cp -pr $(< file_lst)... (6 Replies)
Discussion started by: fretagi
6 Replies

3. UNIX for Dummies Questions & Answers

Script to search and copy files

HI everyone, I been to this site before for help and found my answers on other threads now I am posting my own :). I have a list of file names with out extensions on an txt file. I need a way for the script to search on the server for each file name and copy the files over to a new directory.... (12 Replies)
Discussion started by: sergiol
12 Replies

4. UNIX for Dummies Questions & Answers

Script to copy files from a certain date

I need to copy files from a directory that has a lot of files in it. However I only want to copy them from a certain date. My thoughts so far are to use ls -l and to pipe this into awk and print out tokens 6 (month)and 7 (day). $ ls -l -rw-r--r-- 1 prodqual tst 681883 Jun 12... (2 Replies)
Discussion started by: millsy5
2 Replies

5. Shell Programming and Scripting

Copy Files with script

Hello, I have written a script to copy files from one partion to another. Not sure if this is correct. #!/bin/sh CDR_SOURCE=/storage/archive/logmgmt/result/billing/ CDR_DEST=/storage4/archive/logmgmt/result/billing/ cp $CDR_SOURCE $CDR_DEST; exit 0 The CDR_SOURCE folder has... (5 Replies)
Discussion started by: Siddheshk
5 Replies

6. Shell Programming and Scripting

bash script to copy files

hey everyone, new here i have arch setup and i am using smbnetfs to mount some windows shares in /mnt/smbnet what i want to do is copy files from my home dir to a dir in /mnt/smbnet but i also need it to remove files if i have deleted them from my home dir seems that cp would be the... (8 Replies)
Discussion started by: dodgefan67
8 Replies

7. Shell Programming and Scripting

Script to Poll Directory and Copy files

Hi all, I'm looking for a script to poll a specified directory and copy new files to another location. The script should only copy new files so, I based on mtime I guess? Can anyone point me in the right direction of a script which could do this? My scripting skills aren't too bad, but... (1 Reply)
Discussion started by: JayC89
1 Replies

8. Shell Programming and Scripting

Need a script to copy files and check

Hi all, I need a script in ksh: 1: Copy files from directory (A) to directory (B) 2: Check if files that will be copied in directory (A) have never been yet copied to (B) 3: Never copy the last created file of directory (A) This script will run on crontab. Thanks in advance for your... (1 Reply)
Discussion started by: Camaro
1 Replies

9. Shell Programming and Scripting

copy similar files only both at different locations using script.

Hello, Here is the situation.............. # pwd /opt/123 # cat index.txt abc-monitor/homedir/public_html/index.php abc-monitor/homedir/public_html/test/index.php abc-monitor/homedir/public_html/test1/index.php # cp index.txt index.home # cat /root/x (1 Reply)
Discussion started by: fed.linuxgossip
1 Replies

10. Shell Programming and Scripting

copy files without password in script

1. I wrote a ftp code to copy a file from unix box to another unix box. But it has clear text password in the code. Due to security constraints, clear text password in the code is not allowed. Please let me know the ways to write the code without prompting for password. 2. I tried writing scp... (0 Replies)
Discussion started by: jwala
0 Replies

Featured Tech Videos