size of a directory

collins · #1 (**permalink**) 10-13-2004

hai friends

I need a program to find the size of a directory.. When i tried to get the size, it always gives the default space allocated for it. How can i findout the exact size of a directory using a c program

Thanks in advance
Collins

AbEnd · #2 (**permalink**) 10-14-2004

What do you mean by "exact size of a directory"?

AbEnd · #3 (**permalink**) 10-14-2004

Or du(1) and fts(3).

AbEnd · #4 (**permalink**) 10-14-2004

YEah, right about fts... Wasn't sure if this could be standard or not. But AFAIK, Linux has it too.

But there is no significant overhead with du... This definitively is I/O bound. The biggest problem is that this is prolly only there on UNIX systems. And now that I think about it... du/fts will probably be optimized to run much faster than whatever could be coded trivially... Heh, what overhead? Not to mention du/fts will probably get right the traversing of the hierarchy while it is being modified. Modularity.

AbEnd · #5 (**permalink**) 10-15-2004

Quote:

Originally posted by Driver
> This definitively is I/O bound.

Irrelevant. Invoking an external program adds the time needed to create a process, execute the du program, communicate back the results and destroy the process. Whether this is going to be a problem or not depends on the application; And a portable directory traverser, once written, will compile even using only, say, DOS/Windows libraries - with minimal changes (many libraries provide the exact same functions prefixed with a ``_'' character, and/or declared in a header called direct.h instead of dirent.h)

If you mean that there is a portability issue with du, I agree with that. But spawning the sub-process will take almost no time compared to reading the directories. The overhead will never be noticeable.

Quote:

> And now that I think about it... du/fts will probably be optimized to
> run much faster than whatever could be coded trivially... Heh, what
> overhead? Not to mention du/fts will probably get right the
> traversing of the hierarchy while it is being modified. Modularity.

In think that you are dreaming this up in order to support your assertion that ``du'' is a superior solution

If you can explain how a program can ``get right the traversing of the hierarchy while it is being modified'' (and what you actually mean by that), I'm all ears. As to whether it will be ``optimized to run much faster'', well, there's not a lot of room for any relevant differences. In part because it is ``I/O bound'', as you said earlier.

1) AFAIK, it keeps opened FDs to the parent directories as it does the traversal. If some directories are moved, it will always use the right parent dir. There might be other stuff they did in fts as IIRC, there were security issues related with that as rm -r, chmod -R, find, etc, are often using fts and could be used as root on user files (ex, what happens if root does rm -rf ~user/stuff and user moves ~/stuff/a/b/c to /tmp while rm is traversing it? if using chdir ".." without checking if the parent dir is the same, it could end up traversing files starting at /. If not using "..", it will not traverse all the files in ~user/stuff because the path have changed. Keeping the open'd FD requires keeping them in dynamic data structures and this is a waste of time to code again).

2) ... Dude... It Is BECAUSE this is I/O bound that adding some CPU overhead to reduce the I/O will give a faster results. You see for yourself; they have a lot of code in that library and this is prolly not to make things slower (although a lot of it could be for security): http://www.freebsd.org/cgi/cvsweb.cg...libc/gen/fts.c

AbEnd · #6 (**permalink**) 10-15-2004

Quote:

Originally posted by Driver
> if using chdir ".." without checking if the parent dir is the same, it could end up traversing files starting at /.
> If not using "..", it will not traverse all the files in ~user/stuff because the path have changed.

If you really want to use two chdir() calls for every sub directory, then I agree that ``du'' will be faster. As soon as you call opendir(), the contents of the directory are likely going to be slurped into a buffer for subsequent use anyway. There is no way you can lose this data even if you chdir() into a sub directory and back before reading from the buffer. However, while this is an implementation detail, what you're getting at here does not matter for the program we are talking about.

> Keeping the open'd FD requires keeping them in dynamic data structures and this is a waste of time to code again).

We do not wish to delete files. We want to stat() them and add up the sizes. The results are never guaranteed to be accurate, either, no matter at which time you look at them.

It can still be much more accurate using a du that uses fts. Ex, it should be harder to make the program sum up everything in / erroneously.

Quote:

> 2) ... Dude... It Is BECAUSE this is I/O bound that adding some CPU overhead to reduce the I/O will give a faster results.

Well then, ``dude'', and why would fts be able to ``add some CPU overhead to reduce the I/O'' any more than a straightforward ad-hoc implementation?

Heh. That's something that puzzles me about most textbook algorithms too, dude. But this is definitively possible to reduce I/O while adding some CPU overhead; think of the tree/hash used databases....

Quote:

> You see for yourself; they have a lot of code in that library and this is prolly not to make things slower (although a lot of it could be for
> security): http://www.freebsd.org/cgi/cvsweb.c.../libc/gen/fts.c

Do you really believe that ``they'' make the only operating system that matters? No other system but various 4.4BSD derivatives and possibly glibc (but they probably rolled their own compatible implementation) will even posses this file to begin with.

Most fts should be similar, if not better (especially considering that this one is very old and really free (so it could have been forked a few times)). Just consider this one a proof of concept if you want...

AbEnd · #7 (**permalink**) 10-15-2004

Quote:

Originally posted by Driver
> Heh. That's something that puzzles me about most textbook algorithms too, dude. But this is definitively possible to reduce I/O
> while adding some CPU overhead; think of the tree/hash used databases...

I'm not ``puzzled'' about the concept of trading processing time versus I/O; I merely asked you to back up your claim that it is actually possible and meaningful to do so, in this particular case.

I pasted link to FreeBSD's fts. It does it.

Quote:

> Most fts should be similar, if not better (especially considering that this one is very old and really free (so it could have been forked a few
> times)).

Most fts ... I thought we had already established that fts is a BSD thing

Most OSes will use something similar for rm/chmod/du/find, etc. I call these things fts. At the very least, their du must be implemented correctly.

Quote:

> Just consider this one a proof of concept if you want...

Proof of WHAT concept? It calls opendir(), readdir() and stat() as well, just like your own implementation would. You have yet to show what it is that's supposed to make this code less I/O intensive and thus faster than a straightforward manual implementation here! And keep in mind that an ad-hoc solution can also save all the bookkeeping work; It can stat() a file and throw it away, without all the ``bloat'' found in fts.c.

fts is bloated, but it is faster and secure. See source code.

Quote:

Anyway. If you don't mind, I will leave this fruitless discussion in favor of something more productive.

Sure.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
size of directory	redbeard_06	Shell Programming and Scripting	19	08-08-2008 05:08 AM
Size of Directory	tantric	High Level Programming	1	06-28-2007 03:43 PM
size of directory with ls -l	pascalbout	AIX	2	02-21-2006 03:21 PM
size of a directory?	yls177	UNIX for Dummies Questions & Answers	2	11-25-2002 09:44 AM
Size of a directory	minazk	Shell Programming and Scripting	9	10-01-2002 07:03 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by