How to safely copy full filesystems with large files (10Gb files)


 
Thread Tools Search this Thread
Operating Systems Solaris How to safely copy full filesystems with large files (10Gb files)
# 8  
Old 09-27-2010
Copying sparse files is frequently a problem. The ability to replicate sparse files holes-and-all is often not merely system-specific but filesystem-specific too. I wrote a sparsecat utility which turns space full of NULLs into sparse holes but it does it brute force -- it doesn't know where the holes originally were, it just checks for sectors full of zeroes. It's not guaranteed in any sense either.
# 9  
Old 09-27-2010
FWIW, on Linux systems (not sure about Solaris, sorry), you can detect sparse files by comparing the output of these two commands:

Code:
du -s -B1 --apparent-size sparse-file

and:

Code:
du -s -B1 sparse-file

This User Gave Thanks to Neo For This Post:
# 10  
Old 09-27-2010
For twenty gigs worth of space, the difference should be apparent enough between du -h and ls -lh.
# 11  
Old 09-27-2010
Quote:
Originally Posted by dragonov7
I have tried to do it using:
Code:
# cd /u01
# find . -depth -print | cpio -pdumv /u02

Instead of using cpio which isn't sparse file aware, use ufsdump/ufsrestore to backup your directories. Make also sure you backup stable data by either locking the filesystem (lockfs) or creating a snapshot of it (fssnap).
This User Gave Thanks to jlliagre For This Post:
# 12  
Old 09-27-2010
But... is the "df" command able to detect sparse files???, it should detect them, should it?. I am kind of confused here... if I create a 5Gb sparse file on a filesystem, lets say /u01, then
Code:
du -sh sparse-file

should say that the file is 5Gb... then if I copy the same file to another filesystem, /u02, and do the
Code:
du -sh sparse-file

, the result should be the same. Now, lets say in both filesystems: /u01 and /u02 I have only this file, the df -h command will report the same as du...

What I mean is, it doesn't matter that a sparse file is 10 Gb big and only 1 Gb of it is filled with usefull data, the df or ls -lh or du -sh commands have to return that the file is 10 Gb big.... right?!?!
# 13  
Old 09-27-2010
A 5GB sparse file may take up only 500MB for instance. If you copy the file with a program that does not take sparse files into account, its copy will take up 5GB on the target file system.
This User Gave Thanks to Scrutinizer For This Post:
# 14  
Old 09-27-2010
I found the infamous 20 Gb sparse file!!! :-), you were right guys!, now I'll follow jlliagre's advice on using ufsdump/ufsrestore to do this job... I'll try:
Code:
ufsdump 0f - /dev/rdsk/c0t0d0s0 | (cd /u02; ufsrestore xf -)

and see how it goes...

Thank you to everyone for your time and help!!!

PS: I used
Code:
ls -lh filename

to see the file size (it showed 20 Gb) and
Code:
du -h filename

to see the real/actual size it is consuming (it showed 96 Kb).

Last edited by dragonov7; 09-27-2010 at 06:50 PM.. Reason: Added the PS to show others how I found the sparse file
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Safely Remove Files with Special Chars

Hey Guys, I'm swamped writing code for the forums: Could someone write a script or command line to safely delete files with special chars in filenames from a directory: Example: -rw-r--r-- 1 root root 148 Apr 30 23:00 ?xA?? -rw-r--r-- 1 root root 148... (8 Replies)
Discussion started by: Neo
8 Replies

2. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies

3. Red Hat

Can all files under /tmp be safely removed

I wanted to know whether all files under /tmp can be safely removed. I guess that /tmp may also have temporary files for applications currently being worked on, so at the most those applications may just shut down. I hope that my question is clear whether all files under /tmp can be safely... (5 Replies)
Discussion started by: RHCE
5 Replies

4. Shell Programming and Scripting

Copy down remote files and rename them to include the server name with full path

I need to pull down a good bit of files for another support team for an upgrade project. I have a server.list with all of the server names. I need to do two parts: FIRST: I have this example, but it does not list the server name in front of each line. #! /bin/bash for server in $(<... (10 Replies)
Discussion started by: asnatlas
10 Replies

5. Red Hat

Advice regarding filesystems handling large number of files

Hi All, I have a CentOS operating system installed. I work with really huge number of files which are not only huge in number but some of them really huge in size. Minimum number of files could be 1 million to 2 million in one directory itself. Some of the files are even several Gigabytes in... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies

6. Shell Programming and Scripting

How to safely rm/mv files/directory

Hi all, Am writing a script that does a rm/mv if a file exist, however, in one scenario, one of the variables which is supposed to a variable for a directory is undefined/blank so instead of the variable resolving to /tmp/logfile.dmp, it resolves instead to / so the rm translates to a rm /... (2 Replies)
Discussion started by: newbie_01
2 Replies

7. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234 (7 Replies)
Discussion started by: ad23
7 Replies

8. UNIX for Dummies Questions & Answers

Finding hidden files under mounted filesystems

I have never heard of this before but someone at work here says there is a command to find files that are under currently mounted filesystems. Does anyone know what this command is and is it available on HP-UX? (3 Replies)
Discussion started by: keelba
3 Replies

9. Answers to Frequently Asked Questions

32 or 64 bit filesystems/files/OS's/CPU's

Determining if an OS is using 32 or 64 bits This may be our number one question. Sadly I have not been able to find a definitive answer for Linux. If you have a Linux solution, please post it in our Linux forum. I will edit this post to include it. 32 / 64 bit (Solaris) hp-ux ... (0 Replies)
Discussion started by: Perderabo
0 Replies
Login or Register to Ask a Question