locate holes in a sparse file.


 
Thread Tools Search this Thread
Top Forums Programming locate holes in a sparse file.
# 1  
Old 04-25-2007
locate holes in a sparse file.

Lets say my program reads one block (4096 bytes) at a time from a file. It needs to report whether that block is a hole.

The program currently uses read() to read 1 block into a 4KB buffer.

Now I know that even if that block is a hole, it will be replaced by a sequence of NULL bytes '\0' when read into a buffer. And there is no way to detect if a particular block is a hole.

So for every block thats read in I will need to check if it contains consecutive stretches of zeros, and if so then report it as a hole.

Can anyone think of a way to do this ???

(yes I know that incase that block was not a hole but actually had sequence of zeros stored, my program will incorrectly report it as a hole)

Thanks Smilie
# 2  
Old 04-25-2007
static char zeroes_4k[4096];

int its_a_hole=0;
char buffer[sizeof(zeroes_4k);

i=read(fd,buffer,sizeof(buffer));

if (i==sizeof(buffer))
{
if (!memcmp(buffer,zeroes_4k,sizeof(zeroes_4k))
{
its_a_hole=1;
}
}
# 3  
Old 04-25-2007
sparse files can have less than a block of 0x0 characters. especially uncopied (or unrestored) files.

The easiest way: copy the file and see a difference in stat results for st_size between file_uncopied vs file_copied.

When you copy a sparse file (if it not alrerady a copy or a restore) it becomes larger.
Note that sysadmins detest these files because of what they do to disk space when restored from tape. Are you playing with zum?
# 4  
Old 04-26-2007
st_size will not change when a sparse file is copied. st_blocks is what will change. You can stat a file and see if st_blocks is large enough to cover st_size. If not, then the file is sparse.
# 5  
Old 04-26-2007
MY bad - it is just st_blocks that is affected.
# 6  
Old 04-26-2007
Thanks for your reply guys !

Yes if st_blocks is not as large as st_size that means the file is sparse. And I can even calculate the number of holes it has. But my problem was to know exactly which blocks are holes.

Thanks porter !!! is there a particular reason why you have declared the char array as static ?

Thanks again everyone.. ! Smilie
# 7  
Old 04-26-2007
Quote:
Originally Posted by tantric
(yes I know that incase that block was not a hole but actually had sequence of zeros stored, my program will incorrectly report it as a hole)
I think that you can tell the difference if you are willing to destroy the source file. Copy the file block-by-block starting from the end and truncate the source file as you go. stat the file after truncation to see if you freed a real block (that is, did st_blocks decrease?). This will be implementation dependant... posix tells us
Quote:
The unit for the st_blocks member of the stat structure is not defined within IEEE Std 1003.1-2001. In some implementations it is 512 bytes. It may differ on a file system basis. There is no correlation between values of the st_blocks and st_blksize, and the f_bsize (from <sys/statvfs.h>) structure members.

Traditionally, some implementations defined the multiplier for st_blocks in <sys/param.h> as the symbol DEV_BSIZE.
So some experimentation may be needed to how this works on your system.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Locate the column names with their values in the file and the printing the same in the other file

I have text file in Linux with two rows : first row conmtain the column nam and the second row contain its value .I nned to fetch few columns first and then redirect the data of those colum in the another file. Any ideas?? (1 Reply)
Discussion started by: Anamica
1 Replies

2. Programming

C Data Structure to represent a Sparse Array

Which data structure will be most appropriate to represent a sparse array? (1 Reply)
Discussion started by: rupeshkp728
1 Replies

3. Shell Programming and Scripting

using find but avoiding sparse files

I am no Unix administrator...I live in windows land. I wrote a script to find files of certain names and process them but was later advised to avoid checking sparse files since it would use up a lot of resources and the files I was looking for were not there. How do I avoid doing the find on... (3 Replies)
Discussion started by: shellFun
3 Replies

4. Shell Programming and Scripting

Convert a matrix to sparse representation

Hi All, I have a matrix stored in a file matrix.mtx and looks like this: 1 0.5 0.33 0.25 0 0.33 0.25 0.2 0 0 0 0.16 0 0 0 0.14 I want to convert this matrix to its sparse representation like the one give below (sparse_matrix.mtx). This means that above matrix has been converted to its... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

5. Shell Programming and Scripting

File creating in another path.. application unable to locate

I am submitting a concurrent program (of HOST tyme) from Oracle apps screen, The MAIN shell program submits another program, (child) which is also a Shell program. The child writes data to log file. Now the main program, read the log and do some calculations and sends the data to user through... (1 Reply)
Discussion started by: Pradeep Garine
1 Replies

6. UNIX for Advanced & Expert Users

Locate text in file then remove and replace

I'm trying to locate a block of text in a file, remove it and then replace with a new block. I can find the first line number that the text starts on using grep -n. I then need to locate the ending line by searching for the string "}" that follows the line I found. Here's the steps I need to... (1 Reply)
Discussion started by: lchandle
1 Replies

7. Shell Programming and Scripting

Locate file and path

Hi, I am writing a script to manage my server a bit better, but want to make it so if a path changes I dont need to update the script. I was thinking of doing something like if then echo -e "psa/admin/sbin located " else echo "no luck chump" fi but would need to... (2 Replies)
Discussion started by: foz
2 Replies

8. UNIX for Dummies Questions & Answers

URGENT! How to copy with holes?

I need to write a program that reads in data from a file with holes and copy the file. the cp command copies the file but the holes use disk blocks then; but when this progrm copies the file the new file should not have hose extra blocks. the file should be copied with holes. PLZZ help! (2 Replies)
Discussion started by: kominbhai
2 Replies

9. Shell Programming and Scripting

How to copy file and locate in new folder?

Hi All, Please advise me how to make a copy of file from a list and store in one particular location? For example , I have aaa.txt which contains as below, But, those *usg files might be randomly store in different location.... > cat aaa.txt adc.usg dfdjkf.usg ugjfk.usg And I want... (3 Replies)
Discussion started by: cedrichiu
3 Replies

10. UNIX for Advanced & Expert Users

sparse files

what are sparse files? (2 Replies)
Discussion started by: areef4u
2 Replies
Login or Register to Ask a Question