Sponsored Content
Top Forums Shell Programming and Scripting cp to copy only non-corrupt files Post 302584913 by LMHmedchem on Monday 26th of December 2011 01:34:19 PM
Old 12-26-2011
Quote:
Originally Posted by methyl
What Operating System did you use to format this disc?
The format was done under windows XP 32-bit. This is a multi boot box, but this data drive is primarily used for windows data. It does have a second NTFS partition that I share with linux installations in other boot partitions. Check disk never ran on that partition.

Quote:
Originally Posted by methyl
Can we assume that this new disc is formatted NTFS rather than basic FAT? If not, it will not be able to deal with large files.
Yes it was NTFS

Quote:
Originally Posted by methyl
How did you format the disc? Did you run chkdisk.exe on the new disc before using it?
Normally I create partitions using EASEUS partition master (v9.1). I believe that this does a quick format by default. After the drive started acting up, I reverted to creating and formatting partitions with windows disk manager. It is an adequate tool if all you need to do is to create or delete partitions. I tried both long an quick formats. One curious things is that after the drive started acting up, I deleted the partition in windows, but I could not quick format a new partition after creating one. I got an error the the format failed. If I did a long format, it would finish and the drive was usable. That suggests bad sectors that a quick format can't work around, but the WDLD tool doesn't find bad sectors, nor does HDtune free.

I did not run checkdisk on the drive before using it. Is that a standard practice? I guess it makes sense, I try to test most of my other components. I have run the WDLD tool on some of my new drives before but I can't remember if I did it this time.

Quote:
Originally Posted by methyl
I too am amazed that you have the equipment for a low-level disc format. You will have needed to enter all bad sectors manually.
A hardware forum suggested to me that a low level formats may fix issues, especially if there was a problem in the partition tables or MBR. It would also work around bad sectors if possible. I just used a software tool called HDD Low Level Format Tool (HDDGURU: Laptop and Desktop Hard Disk Drives, Tests, Software, Firmware, Tools, Data Recovery, HDD Repair). I don't know that this does much of anything different then the windows long format, but it does remove the MBR. I had to activate the drive in windows after running the tool. It also took like 10 hours to run.

Quote:
Originally Posted by methyl
Because you have posted on unix.com , we must assume that unix is involved somewhere in this process.
I have cygwin installed and make significant use of it, so I do allot of things in bash. I copied the files from the bad source disk to the replacement drive using,

cp -Rfp sourceDir/ destinationDir/ >& sourceDir_copylog.txt

This has been my standard procedure for quite a while. It is much faster than using any windows tool and it doesn't quit if it suddenly runs into a file it can't copy. The redirected stderr and stdout gives a record of any files that couldn't be copied. This box also has ubuntu, cent, scientific, and suse installed, so those are available if there are some native linux techniques to try.

Quote:
Originally Posted by methyl
Does the source disc belong to the system on which you are trying to do the copy? If not, where did it come from? What is the format of the source disc and what Operating System and software wrote the files on the disc?
The source disk is a windows NTFS drive of the same make and model (WDCB 1TB 6gb/s). Partitions on the source disk were created and formatted using EASEUS.

Quote:
Originally Posted by methyl
What proof do you have that the source disc is corrupt? What did you type when trying to copy the files? What error messages do you get?
I run rsync every night to backup the data drive to a backup drive in the same box. About a month ago, I noticed the rsync wasn't finishing and it seemed like the issue was with the destination drive. I ran the WDLD tool on it and it failed the short test. I ran the long test and it said there were bad sectors that it tired to fix. The tool failed trying to fix the bad sectors, so I RMAd the drive. I have an external backup of the same drive, so when the replacement arrived, I used cp to restore my internal backup from my external. The internal backup drive has worked well since.

Recently I noticed that there were some issues with the primary data drive, so I ran WDLD, found errors, and RMAd that drive as well. This was the same bad sectors error that I got on the backup drive.

This new problem arose when trying to copy data onto the replacement for the backup drive when it arrived. I did the same file copy with cp -Rfp as before.

Quote:
Originally Posted by methyl
How big is the largest file (especially if bigger than 2 Gb)? A detailed hardware and software inventory would help. I wonder if you are fitting modern disc drives to an old computer?
I'm not sure how bit the largest file is. I have some linux iso files stored on this drive, and those are pushing 5GB. Those are probably the largest thing I have.

The hardware is as follows,
PSU: CORSAIR CMPSU-750TX 750W
MOBO: GA-EP45T-DS3R f3 BIOS
CPU: Q9550
RAM: 2x2GB DDR3 OCZ3RPR13334GK 1333MHz, 6-6-6-20, 1.75v
GPU: EVGA 896-P3-1257-AR GeForce GTX260 Core 216
SSD-OS1: OCZ VertexII 60GB, WinXP-32bit sp3
HDD-OS2: VelociRaptor 150GB, Ubuntu 10.10 64-bit, CentOS 5.1 64-bit, Suse 12.1 64-bit
HDD-Data: 1TB Western Digital Caviar Black
HDD-Backup: 1TB Western Digital Caviar Black (10GB pagefile partition)

As far as software, I'm not sure what is relevant, but I have cygwin installed for the gnu compilers, Zone Alarm ISS, java JRE, eclipse, MS office, Adobe CS, a bunch of chemistry and statistics tools, and various system tools.

I wouldn't call this an old computer by any means, but it is certainly not the most current hardware either. I found on another forum that there is an issue with using the western digital 6GB/s SATA III drives on a SATA II controller. These are all supposed to be back compatible, but apparently you need to add jumper to restrict the drive to 3GB/s. Others have reported bad sectors popping up over time without the jumper. It would have been nice for WD to advertise that a bit better. I am confident that is what was causing the bad sectors to pop up in the first place. But I put a jumper on the replacement drive that is acting up now and that didn't help.

At this point, I am able to use all of the other drives in this box, so I am inclined to think that the problem is not with the SATA controller. I have also run memtest 86+ and it didn't find anything wrong with the memory. I will run a long prime95 later today to see if my system is generally stable.

At this point, I have the data drive with the bad sectors unplugged and I'm waiting for an RMA of the replacement drive.

Corona688, this is the replacement that I did an RMA on, I still have the original bad sector drive that I'm trying to get the data off of. If I plug that drive it, it works and I can open files and such. I don't know what data on it is affected by the bad sectors, so I've left it un plugged for now. The rest of the computer seems to work fine. I can boot in to the OS, run apps, etc, and the other drives in the box aren't triggering checkdisk.

My plan is to load the bad sector hdd and the RMA replacement drive into a new computer I have and use ubuntu live linux to do dd_rescue to try to recover the data. Then I will boot windows in the new computer and see if the new drive is stable. If it is, I will put it into the suspect machine and see what happens. Hopefully the replacement drive I got was just bad and that is all there is too it.

Did I answer everyone's questions? Sorry for the delay in response, it has been a busy weekend.

LMHmedchem

Last edited by LMHmedchem; 12-26-2011 at 02:39 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

corrupt or lost data

Thank you livin Free for all your help. We removed a lot of spool files and report files. Which should have freed up some space. But now I think a major problem we have is we have lost or corrupt files which are preventing us from coming up correctly. Can we load or can you copy us a directory... (1 Reply)
Discussion started by: NOT A CLUE
1 Replies

2. UNIX for Dummies Questions & Answers

I got a corrupt /etc/inittab file....what next?

Hi guys, For some reason a client has given us a Sun Netra T1 with Solaris 8 to administer for them. That's always good business. However, the other day we rebooted the machine and to our amazement, after doing the preliminary hardware tests, we got an error messgae saying that /etc/inittab was... (3 Replies)
Discussion started by: Ivo
3 Replies

3. HP-UX

Corrupt Member File

I have been fine adding/removing printers up until this week. Now when I go to add a new remote printer I get "corrupted member file". I go to /etc/lp/member and the byte count on the new printer name is 0. I VI the file and put /dev/null in to make it the correct size and it all looks fine and... (2 Replies)
Discussion started by: astout
2 Replies

4. Solaris

How to corrupt a superblock?

I need to corrupt a superblock of a mounted device in a soalris m/c and check recovery from an alternate superblock. How can this be done? (2 Replies)
Discussion started by: sujathan
2 Replies

5. Solaris

How to safely copy full filesystems with large files (10Gb files)

Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS. I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies

6. Shell Programming and Scripting

how to copy files followed by list of names of all the files in /etc?

....... (2 Replies)
Discussion started by: pcbuilder
2 Replies

7. HP-UX

corrupt disk

Hallo Friends, I have application X running on hpux 11.11 and oracle 9i release 2. I recently had a hardware failure on disk /dev/dsk/c2t0d0 Below is the systemlog file : root@a7dmc:/var/adm/syslog > /opt/resmon/bin/resdata -R 155713541 -r /storage/events/enclosures/gazemon/0_1_1_0.0.0... (11 Replies)
Discussion started by: kekanap
11 Replies

8. Red Hat

Unable to copy files due to many files in directory

I have directory that has some billion file inside , i tried copy some files for specific date but it's always did not respond for long time and did not give any result.. i tried everything with find command and also with xargs.. even this command find . -mtime -2 -print | xargs ls -d did not... (2 Replies)
Discussion started by: before4
2 Replies

9. Solaris

Solaris 10 GIF files corrupt during unzip

Problem occurs on one Solaris build. Every time we unzip the Jan CPU, there are several patches that error out (appears to be related to the GIF files). When we unzip the CPU on another Solaris build to a network storage area, we can execute without issue on the original machine. Any ideas? ... (1 Reply)
Discussion started by: grahamr72
1 Replies

10. UNIX for Dummies Questions & Answers

Copy files from one drive to another, keeping most recently modified files

Hi all, I am a bit of a beginner with shell scripting.. What I want to do is merge two drives, for example moving all data from X to Y. If a file in X doesn't exist in Y, it will be moved there. If a file in X also exists in Y, the most recently modified file will be moved to (or kept) in... (5 Replies)
Discussion started by: apocolapse
5 Replies
All times are GMT -4. The time now is 05:05 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy