Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Trying to copy a TON of files with a script
Top Forums Shell Programming and Scripting Trying to copy a TON of files with a script Post 302368822 by davidstvz on Thursday 5th of November 2009 06:03:22 PM
Old 11-05-2009
Trying to copy a TON of files with a script
Ok, here's the problem. I need to copy about 200 GB from an old FreeBSD (5.2.1) server to a new Debian server. The old server's fastest port is the ethernet port. I set up an NFS server on the new machine so I can just copy the files over using regular commands/scripts etc. Thing is, I probably shouldn't copy dozens of GB of files using the 'cp' command. The last time I tried that it crashed the server.

My plan is to write a script that will process a list of files copying each file from the list one at a time (while making a separate list of files successfully copied that way if anything goes wrong, I can difference the two lists and pick up where it left off). More importantly, I can copy all files ahead of time and then generate a smaller list of files modified since the original copy so that on migration day I have a minimal number of files to copy minimizing downtime.

The main problem (that I can see) with this is, any list I generate will contain both files and folders, but the folders don't need to be copied, only recreated with identical owners and permissions. Unless there is a flag for the 'cp' command that copies a folder without it's contents (not that I know of), I need to find some other way to test each path to see what it is, then if it is a directory, somehow find the permissions and owner and duplicate it on the new server.

If anyone has any advice, can see any other problems or knows of a better way I might do this, I'd appreciate the help. Thanks!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

copy files without password in script

1. I wrote a ftp code to copy a file from unix box to another unix box. But it has clear text password in the code. Due to security constraints, clear text password in the code is not allowed. Please let me know the ways to write the code without prompting for password. 2. I tried writing scp... (0 Replies)
Discussion started by: jwala
0 Replies

2. Shell Programming and Scripting

copy similar files only both at different locations using script.

Hello, Here is the situation.............. # pwd /opt/123 # cat index.txt abc-monitor/homedir/public_html/index.php abc-monitor/homedir/public_html/test/index.php abc-monitor/homedir/public_html/test1/index.php # cp index.txt index.home # cat /root/x (1 Reply)
Discussion started by: fed.linuxgossip
1 Replies

3. Shell Programming and Scripting

Need a script to copy files and check

Hi all, I need a script in ksh: 1: Copy files from directory (A) to directory (B) 2: Check if files that will be copied in directory (A) have never been yet copied to (B) 3: Never copy the last created file of directory (A) This script will run on crontab. Thanks in advance for your... (1 Reply)
Discussion started by: Camaro
1 Replies

4. Shell Programming and Scripting

Script to Poll Directory and Copy files

Hi all, I'm looking for a script to poll a specified directory and copy new files to another location. The script should only copy new files so, I based on mtime I guess? Can anyone point me in the right direction of a script which could do this? My scripting skills aren't too bad, but... (1 Reply)
Discussion started by: JayC89
1 Replies

5. Shell Programming and Scripting

bash script to copy files

hey everyone, new here i have arch setup and i am using smbnetfs to mount some windows shares in /mnt/smbnet what i want to do is copy files from my home dir to a dir in /mnt/smbnet but i also need it to remove files if i have deleted them from my home dir seems that cp would be the... (8 Replies)
Discussion started by: dodgefan67
8 Replies

6. Shell Programming and Scripting

Copy Files with script

Hello, I have written a script to copy files from one partion to another. Not sure if this is correct. #!/bin/sh CDR_SOURCE=/storage/archive/logmgmt/result/billing/ CDR_DEST=/storage4/archive/logmgmt/result/billing/ cp $CDR_SOURCE $CDR_DEST; exit 0 The CDR_SOURCE folder has... (5 Replies)
Discussion started by: Siddheshk
5 Replies

7. UNIX for Dummies Questions & Answers

Script to copy files from a certain date

I need to copy files from a directory that has a lot of files in it. However I only want to copy them from a certain date. My thoughts so far are to use ls -l and to pipe this into awk and print out tokens 6 (month)and 7 (day). $ ls -l -rw-r--r-- 1 prodqual tst 681883 Jun 12... (2 Replies)
Discussion started by: millsy5
2 Replies

8. UNIX for Dummies Questions & Answers

Script to search and copy files

HI everyone, I been to this site before for help and found my answers on other threads now I am posting my own :). I have a list of file names with out extensions on an txt file. I need a way for the script to search on the server for each file name and copy the files over to a new directory.... (12 Replies)
Discussion started by: sergiol
12 Replies

9. Shell Programming and Scripting

Error on script to copy files

Hi I have the following script to copy files from one directory to another: #!/bin/sh touch -mt 201304240000 /var/tmp/ref1 touch -mt 201305152359 /var/tmp/ref2 find /moneta_collected02/in_psl -type f \( -newer /var/tmp/ref1 -a ! -newer /var/tmp/ref2 \) > file_lst cp -pr $(< file_lst)... (6 Replies)
Discussion started by: fretagi
6 Replies

10. Shell Programming and Scripting

Shell script to copy files from on folder to another

I am trying to copy files with specific date and name to another folder. I am very new to shell scripting so i am finding it hard to do that. see the sample code i have written below. srcdir="/media/ubuntu/CA52057F5205720D/Users/st4r8_000/Desktop/office work/26 nov"... (13 Replies)
Discussion started by: Aqeel Abbas
13 Replies

LEARN ABOUT BSD

cp

CP(1)							      General Commands Manual							     CP(1)

NAME

       cp - copy

SYNOPSIS

       cp [ -ip ] file1 file2

       cp [ -ipr ] file ... directory

DESCRIPTION

       File1  is copied onto file2.  By default, the mode and owner of file2 are preserved if it already existed; otherwise the mode of the source
       file modified by the current umask(2) is used.  The -p option causes cp to attempt to preserve (duplicate) in its copies  the  modification
       times and modes of the source files, ignoring the present umask.

       In the second form, one or more files are copied into the directory with their original file-names.

       Cp refuses to copy a file onto itself.

       If  the	-i option is specified, cp will prompt the user with the name of the file whenever the copy will cause an old file to be overwrit-
       ten. An answer of 'y' will cause cp to continue. Any other answer will prevent it from overwriting the file.

       If the -r option is specified and any of the source files are directories, cp copies each subtree rooted at that name;  in  this  case  the
       destination must be a directory.

SEE ALSO

       cat(1), mv(1), rcp(1C)

4th Berkeley Distribution					   June 8, 1985 							     CP(1)
All times are GMT -4. The time now is 04:46 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy