data corruption with ftp transfer Post: 39019

Sponsored Content

Top Forums UNIX for Advanced & Expert Users data corruption with ftp transfer Post 39019 by malcom on Monday 4th of August 2003 07:38:34 AM

08-04-2003

Registered User

data corruption - THE ANSWER / SOLUTION

Hi all,

today I received the answer from our development related to the data corruption problem.
The solution is XFS, our used XFS has a bug, but see below the detailed answer :

--- snip ---

We have now found the cause of the data corruption.
This is original bug from XFS.

The special timing of the delete operation of a block and
flush operation from the memory leads to the data corruption.

To be prepared for next I/O operation, XFS pre-allocates the data
on the memory larger than what is actually needed.
Sometimes also unnecessary block are removed from pre-allocated data
on the memory to efficiently use the memory resources.
This happens when memory resource are fully utilized.

When this "pre-allocation" and "removal of unnecessary data"
occurs with certain timing, next data to be processed is written
in the area where just removed from the pre-allocation which shouldn't
happen. Since the data is written into the wrong place, the block
in the memory which should contain the data does not contain the
actual data which causes the data corruption when it is flushed to

Following conditions are needed to have the bug to aprear:

- Memory is fully loaded
- I/O is done via NFS
- Large file is being written continuously and concurrently
(magnitude of GB)

--- snap ---

may be someone has a similar problem, so thanks a lot to all and have a nice day.

Regards
Alex

malcom

View Public Profile for malcom

Find all posts by malcom

10 More Discussions You Might Find Interesting

1. AIX

FTP - Data Transfer Limitations.

Hi, I have to transfer data from our production site to DR site(another city). I am using FTP for transfering data. But I am unable to get the same data transfer rate on AIX machines, one I am geting on windows machines. I want to know, is there any constraint on data transfer using FTP on AIX...

2. Solaris

ftp transfer

Hi ....I'm trying to ftp 2 pkgs SUNWqlc & SUNWqlcx since they were accidentally deleted from my servers. For one reaseon or the other my server is not loading the cdrom, so I mounted the cdrom on a differernt server & ftp's to it. I tried to get this pkg using the bin & I get an error message...

3. Shell Programming and Scripting

ftp zip files corruption

Hi I have a zip file that I am ftping to another server. After I have ztped the file it is corrupt and cannot be opened by winzip. Can anyone throw any light on to why this would happen. I have no problems with sending the individual files, but when zipped into one file it becomes...

4. UNIX for Dummies Questions & Answers

Keep alive packet for control session in ftp data transfer

I'm new to UNIX. We are using SunOS5.9. In order to maintain the control session active during large file tranfers in ftp, i like to send keep alive packets for every 10 seconds.I like to know the steps to be done to achieve this. Please help me guys.Thanks in advance.

5. UNIX for Advanced & Expert Users

rsync data corruption

Dear Peoples I am using the following command to transfer the files inbetween two servers, i am using this command in a crontab for doing it in every 1 hour on all days. rsync --stats --archive --verbose --compress --force --rsh=ssh --exclude-from=/root/cfg/mkt_scn.exclude...

6. Solaris

Data corruption

I have a solaris 5.6 on which oracle is installed. we have an alert file alert_net1.log now whenever any datacorruption happens we get the file id and block id in the above file. through this file and block id , we try to find out which table is corrupted and then try to...

7. UNIX for Advanced & Expert Users

File system testing for Data corruption

Hi, could any one tell is there any test-suite or any idea How to do data corruption validation testing, means there is no any data corruption ? Regards Manish

8. Shell Programming and Scripting

Script for verification of Data corruption

Hi, I am from File system back ground and doing File system testing on Linux, I need script that scirpt that create the different multiple types of .txt, device file and then copy to mount point /mnt/ dir and then verify the created files and copied files, if created files and copied files are...

9. Shell Programming and Scripting

Using SFTP and FTP to transfer data from One Remote Server To Another

HI I need to write a script in 415univ server which should go to 534unix server and move the files from there to windows server. I am not able to get it bcoz sftp prompt is not allowing ftp command. Can some one plz help me Thanks in advance

10. UNIX for Beginners Questions & Answers

How to see the status of all the ftp put & get files logs and curent ftp transfer status ?

How to see the status of all the ftp put & get files logs and curent ftp transfer status if any active ftp running in the background ?

LEARN ABOUT CENTOS

uuid

UUID(3) 							    Libuuid API 							   UUID(3)

NAME

       uuid - DCE compatible Universally Unique Identifier library

SYNOPSIS

       #include <uuid.h>

DESCRIPTION

       The UUID library is used to generate unique identifiers for objects that may be accessible beyond the local system.  This library generates
       UUIDs compatible with those created by the Open Software Foundation (OSF) Distributed Computing Environment (DCE) utility uuidgen.

       The UUIDs generated by this library can be reasonably expected to be unique within a system, and unique across all systems.  They could	be
       used, for instance, to generate unique HTTP cookies across multiple web servers without communication between the servers, and without fear
       of a name clash.

CONFORMING TO

       OSF DCE 1.1

AUTHOR

       Theodore Y. Ts'o

AVAILABILITY

       libuuid is part of the util-linux package since version 2.15.1 and is available from ftp://ftp.kernel.org/pub/linux/utils/util-linux/.

SEE ALSO

       uuid_clear(3), uuid_compare(3), uuid_copy(3), uuid_generate(3), uuid_is_null(3), uuid_parse(3), uuid_time(3), uuid_unparse(3)

util-linux							     May 2009								   UUID(3)