11-05-2009
Trying to copy a TON of files with a script
Ok, here's the problem. I need to copy about 200 GB from an old FreeBSD (5.2.1) server to a new Debian server. The old server's fastest port is the ethernet port. I set up an NFS server on the new machine so I can just copy the files over using regular commands/scripts etc. Thing is, I probably shouldn't copy dozens of GB of files using the 'cp' command. The last time I tried that it crashed the server.
My plan is to write a script that will process a list of files copying each file from the list one at a time (while making a separate list of files successfully copied that way if anything goes wrong, I can difference the two lists and pick up where it left off). More importantly, I can copy all files ahead of time and then generate a smaller list of files modified since the original copy so that on migration day I have a minimal number of files to copy minimizing downtime.
The main problem (that I can see) with this is, any list I generate will contain both files and folders, but the folders don't need to be copied, only recreated with identical owners and permissions. Unless there is a flag for the 'cp' command that copies a folder without it's contents (not that I know of), I need to find some other way to test each path to see what it is, then if it is a directory, somehow find the permissions and owner and duplicate it on the new server.
If anyone has any advice, can see any other problems or knows of a better way I might do this, I'd appreciate the help. Thanks!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
1. I wrote a ftp code to copy a file from unix box to another unix box. But it has clear text password in the code. Due to security constraints, clear text password in the code is not allowed. Please let me know the ways to write the code without prompting for password.
2. I tried writing scp... (0 Replies)
Discussion started by: jwala
0 Replies
2. Shell Programming and Scripting
Hello,
Here is the situation..............
# pwd
/opt/123
# cat index.txt
abc-monitor/homedir/public_html/index.php
abc-monitor/homedir/public_html/test/index.php
abc-monitor/homedir/public_html/test1/index.php
# cp index.txt index.home
# cat /root/x (1 Reply)
Discussion started by: fed.linuxgossip
1 Replies
3. Shell Programming and Scripting
Hi all,
I need a script in ksh:
1: Copy files from directory (A) to directory (B)
2: Check if files that will be copied in directory (A) have never been yet copied to (B)
3: Never copy the last created file of directory (A)
This script will run on crontab.
Thanks in advance for your... (1 Reply)
Discussion started by: Camaro
1 Replies
4. Shell Programming and Scripting
Hi all, I'm looking for a script to poll a specified directory and copy new files to another location.
The script should only copy new files so, I based on mtime I guess?
Can anyone point me in the right direction of a script which could do this?
My scripting skills aren't too bad, but... (1 Reply)
Discussion started by: JayC89
1 Replies
5. Shell Programming and Scripting
hey everyone, new here
i have arch setup and i am using smbnetfs to mount some windows shares in /mnt/smbnet
what i want to do is copy files from my home dir to a dir in /mnt/smbnet but i also need it to remove files if i have deleted them from my home dir
seems that cp would be the... (8 Replies)
Discussion started by: dodgefan67
8 Replies
6. Shell Programming and Scripting
Hello,
I have written a script to copy files from one partion to another. Not sure if this is correct.
#!/bin/sh
CDR_SOURCE=/storage/archive/logmgmt/result/billing/
CDR_DEST=/storage4/archive/logmgmt/result/billing/
cp $CDR_SOURCE $CDR_DEST;
exit 0
The CDR_SOURCE folder has... (5 Replies)
Discussion started by: Siddheshk
5 Replies
7. UNIX for Dummies Questions & Answers
I need to copy files from a directory that has a lot of files in it. However I only want to copy them from a certain date. My thoughts so far are to use ls -l and to pipe this into awk and print out tokens 6 (month)and 7 (day).
$ ls -l
-rw-r--r-- 1 prodqual tst 681883 Jun 12... (2 Replies)
Discussion started by: millsy5
2 Replies
8. UNIX for Dummies Questions & Answers
HI everyone,
I been to this site before for help and found my answers on other threads now I am posting my own :).
I have a list of file names with out extensions on an txt file. I need a way for the script to search on the server for each file name and copy the files over to a new directory.... (12 Replies)
Discussion started by: sergiol
12 Replies
9. Shell Programming and Scripting
Hi
I have the following script to copy files from one directory to another:
#!/bin/sh
touch -mt 201304240000 /var/tmp/ref1
touch -mt 201305152359 /var/tmp/ref2
find /moneta_collected02/in_psl -type f \( -newer /var/tmp/ref1 -a ! -newer /var/tmp/ref2 \) > file_lst
cp -pr $(< file_lst)... (6 Replies)
Discussion started by: fretagi
6 Replies
10. Shell Programming and Scripting
I am trying to copy files with specific date and name to another folder. I am very new to shell scripting so i am finding it hard to do that. see the sample code i have written below.
srcdir="/media/ubuntu/CA52057F5205720D/Users/st4r8_000/Desktop/office work/26 nov"... (13 Replies)
Discussion started by: Aqeel Abbas
13 Replies
LEARN ABOUT REDHAT
rpc.nfsd
rpc.nfsd(8) System Manager's Manual rpc.nfsd(8)
NAME
rpc.nfsd - NFS server process
SYNOPSIS
/usr/sbin/rpc.nfsd [-p port] nproc
DESCRIPTION
The rpc.nfsd program implements the user level part of the NFS service. The main functionality is handled by the nfsd.o kernel module; the
user space program merely starts the specified number of kernel threads.
The rpc.mountd server provides an ancially service needed to satisfy mount requests by NFS clients.
OPTIONS
-p port
specify a diferent port to listen on for NFS requests. By default, rpc.nfsd will listen on port 2049.
nproc specify the number of NFS server threads. By default, just one thread is started. However, for optimum performance several threads
should be used. The actual figure depends on the number of and the work load created by the NFS clients, but a useful starting point
is 8 threads. Effects of modifying that number can be checked using the nfsstat(8) program.
SEE ALSO
rpc.mountd(8), exportfs(8), rpc.rquotad(8), nfsstat(8).
AUTHOR
Olaf Kirch, Bill Hawes, H. J. Lu, G. Allan Morris III, and a host of others.
31 May 1999 rpc.nfsd(8)