Pipe usage error while visiting directories


 
Thread Tools Search this Thread
Top Forums Programming Pipe usage error while visiting directories
# 1  
Old 03-26-2019
Pipe usage error while visiting directories

To begin with FYI, I really struggled with the question before asking to simplify as much as I can around 1 hour and to increase the code's readability I neglect error checks.

I intend to communicate parent and child using PIPE. In the following program I do traverse given path and its subdirectories and calculate total size of what are included in the current directory individually. Upon completion, I write each directory's PID, size and path into `COMMON_FIFO` which will be read in `main()` eventually. My questions are following that
  • Can data corruption occur while writing `FIFO` cuz of its atomic size since the main process wait for all children to be reaped. Doesn't it? If yes, how can I solve the problem?
  • Is the written information(each directory's PID, size and path) guaranteed post-order since each directory is gone into orderly? If no, how can it be solved?

Code:
    char const * COMMON_FIFO = "FIFO_TEST";
    
    int walk_dir_and_calculate_sizes(char * path) {
    	opendir() ..
    	int totalRegularSizes = 0;
    	while ((entry = readdir) != NULL) {
    		char const *name = entry->d_name;
    		COMBINE path and name for example path is a/, name is b.txt(a/b.txt) or another DIRECTORY c(a/c)
    		IF (entry->d_type is DIRECTORY) {
    			if (name is "." or name is "..")
    			continue;
    
    			childPid = fork();
    			if (childPid == 0) {		//child
    				walk_dir_and_calculate_sizes(COMBINED_PATH);
    				_exit(0);
    			}
    
    			if (childPid > 0) {			// parent
    				continue;
    				// continue traversing if remaining dirs or files exist
    			}
    		}
    		ELSE {	// regular file like .txt, .pdf etc.
    			totalRegularSizes += another_func_giving_size_of_file(COMBINED_PATH);
    		}
    	}
    
    	closedir(..);
    	while (wait(&status) > 0);
    
    	RESTORE COMBINED_PATH to directory path, it is ok
    
    	int fd = OPEN(COMMON_FIFO, O_WRONLY);
    	char folderSizeInformation[100] = Current PID, directory size(= totalRegularSizes), directory name combinations
    	WRITE(fd, folderSizeInformation, strlen(folderSizeInformation)
    	CLOSE(fd)
    
    	return 123456 not important I think;
    
    }
    
    int main(char ** argv) {
    	mkfifo(COMMON_FIFO, 0644);
        OPEN(COMMON_FIFO, O_RDONLY|O_NONBLOCK);
    
        walk_dir_and_calculate_sizes(argv[1]);
    
        PRINT FIFO ON SCREEN
    
        exit(0);
        
    }


Let's modify the prior code's `walk_dir_and_calculate_sizes` function in which each directory's process calculates total sizes what the directory includes(regular files) to `WITH_PIPE_walk_dir_and_calculate_sizes` in which I intend to transmit each subdirectory's calculated size to its parent by using PIPE. For example, Directorey A includes B which includes C. Invidially `A's size is 10kb, B's is 5kb, C's 1kb`. In walk_dir_and_calculate_sizes gives result in `A's size is 10kb, B's is 5kb, C's 1kb` however in WITH_PIPE_walk_dir_and_calculate_sizes it yields `A's size is 16kb, B's is 6kb, C's 1kb` since C doesn't have child(subdirectory).


Code:
    char const * COMMON_FIFO = "FIFO_TEST";
    
    int walk_dir_and_calculate_sizes(char * path) {
    
    	/* PIPE */
    	int pfd[2];
    
    	opendir() ..
    	int totalRegularSizes = 0;
    	while ((entry = readdir) != NULL) {
    		char const *name = entry->d_name;
    		COMBINE path and name for example path is a/, name is b.txt(a/b.txt) or another DIRECTORY c(a/c)
    		IF (entry->d_type is DIRECTORY) {
    			if (name is "." or name is "..")
    			continue;
    
    			/* PIPE */
    			pipe(pfd);
    
    
    			childPid = fork();
    			if (childPid == 0) {		//child
    				walk_dir_and_calculate_sizes(COMBINED_PATH);
    				_exit(0);
    			}
    
    			if (childPid > 0) {			// parent
    				continue;
    				// continue traversing if remaining dirs or files exist
    			}
    		}
    		ELSE {	// regular file like .txt, .pdf etc.
    			totalRegularSizes += another_func_giving_size_of_file(COMBINED_PATH);
    		}
    	}
    
    	closedir(..);
    
    	/* PIPE COMMUNICATE READ */
    	while (wait(&status) > 0) {
    		close(pfd[1]);
    		int readVal = -1;
    		READ(pfd[0], &readVal, sizeof(readVal));
    		totalRegularSizes += readVal
    	}
    
    	RESTORE COMBINED_PATH to directory path, it is ok
    
    	int fd = OPEN(COMMON_FIFO, O_WRONLY);
    	char folderSizeInformation[100] = Current PID, directory size(= totalRegularSizes), directory name combinations
    	WRITE(fd, folderSizeInformation, strlen(folderSizeInformation)
    	close(fd)
    
    
    	/* PIPE COMMUNICATE WRITE */
    	// Since it is recursive function the deepest directory doesn't have any child(subdirectory)
    	// it omits wait() function and comes here
    
    	close(pfd[0]);
    	WRITE(pfd[1], &totalRegularSizes, sizeof(totalRegularSizes));
    
    
    	return 123456 not important I think;
    
    }

But in `WITH_PIPE_walk_dir_and_calculate_sizes` function I get *Bad File Descriptor* error while closing read end before writing to pipe. I'm really in dilemma. Why is my idea wrong? How can my intent be achieved Of course I'm not experienced guy, I can have another mistakes or oversights, please inform me.

Thanks a lot.

@Edit, by the way, I overlooked the point that I'm doing only one pipe as thinking one directory has one subdirectory BUT a directory can of course have more than one subdirectory. So I think we need more pipes to a parent.

Last edited by beginnerboy; 03-26-2019 at 01:49 PM..
# 2  
Old 03-26-2019
If it's saying bad file descriptor it probably means it. Without seeing your actual code, I can't tell why you're closing a bad file descriptor, you should print the FD's to stderr when you open a pipe, and print them to stderr again when you try and close it to see what's going on. But I have some further comments.

fork() is pointless. Disks don't multithread. Forcing it to read 19 directories at once will make your disk run 19 times slower. You already benefit from the caching and read-ahead built into the OS, too.

Second, there's a system function for what you want to do, ftw() It operates depth-first, so every time you see a new second-level folder, you'll know everything afterwards will be within that folder until it leaves.

Code:
#include <ftw.h>
#include <sys/stat.h>
#include <stdio.h>
#include <string.h>

struct FTW {
        int base;
        int level;
};

struct result_t {
        char name[256];
        long int size;
} result[64];

int last_result=0;

int ftw_callback(const char *fpath, const struct stat *sb,
        int typeflag, struct FTW *ftwbuf);

int main(void) {
        int n;
        const char *ROOT="./";
        strcpy(result[0].name, ROOT);
        ftw(ROOT, ftw_callback, 8);

        for(n=0; n<=last_result; n++)
                printf("%s\t%ld bytes\n", result[n].name, result[n].size);
}

int ftw_callback(const char *fpath, const struct stat *sb,
        int typeflag, struct FTW *ftwbuf) {

        // Found a new second-level folder
        if((typeflag == FTW_D) && (ftwbuf->level == 1)) {
                last_result++;
                strcpy(result[last_result].name, fpath);
        }

        if(typeflag == FTW_F)
        {
                int res=last_result;
                // Special case for level-1 files, those are in ROOT
                if(ftwbuf->level == 1) res=0;
                result[res].size += sb->st_size;
        }

        return(0);
}


Last edited by Corona688; 03-26-2019 at 03:18 PM..
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 03-26-2019
If you really wanted to do IPC between processes just to count directories, though, shared memory beats pipes IMO. mmap() an anonymous segment, and each fork()ed process will have access to it. Give each child a unique index to mess with so they don't stomp on each other, wait() for each child to quit, and tada.
This User Gave Thanks to Corona688 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Capture error before pipe

Hi, I have a script that runs a tar command to standard out then pipes to a gzip: tar cfE - * | gzip -c > OUT.gz At the moment, even if the tar fails (e.g. because of lack of disk space), the gzip still runs successfully. Is there a way to make the whole line exit with a non-zero error... (6 Replies)
Discussion started by: Catullus
6 Replies

2. UNIX for Dummies Questions & Answers

broken pipe error

I'm new to scripting, and this forum has been invaluable in helping me out. I'm hoping I can get some personal help now though. I have a korn script that takes a list of servers and either telnets or sshs into it (only some are set up for ssh). What I'm doing now is trying to telnet first, and... (10 Replies)
Discussion started by: aimeet
10 Replies

3. Shell Programming and Scripting

listing file date and time without usage of pipe

Hi , I am using below code to list the 6th,7th and 8th field of the file ls -lrt test | awk '{print $6,$7,$8}' output: Nov 21 19:34 Now the problem here is that I want to do it without the usage of pipes as its now allowed in my production environment Please let me know... (6 Replies)
Discussion started by: harish612
6 Replies

4. Shell Programming and Scripting

Broken Pipe error

Hello while doing sftp over server "A" , i am getting a broken pipe error i.e cat: write error: Broken pipe what does that mean? please let me know if you want any other info on this.. (3 Replies)
Discussion started by: urfrnddpk
3 Replies

5. Programming

Pipe error

hi guys, o have a big error in this program but i cant solve someone ?! #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <string.h> int main(int argc, char *argv){ int cont = 2, posicao; char geraArquivo= "|cat>>", espaco=" "; char nomeArquivo, comando,... (11 Replies)
Discussion started by: beandj
11 Replies

6. Programming

Broken Pipe error

All, I am using the below code The C code : if ((fp2=fopen(szout_fname,"r"))==NULL) { sprintf(stream_ptr1,"cat %s | sort -t, -rn -k 11,11 | awk -F\",\" '{ \ if ( \$3 ==\"%s\" ) {print... (0 Replies)
Discussion started by: arunkumar_mca
0 Replies

7. AIX

How to monitor the IBM AIX server for I/O usage,memory usage,CPU usage,network..?

How to monitor the IBM AIX server for I/O usage, memory usage, CPU usage, network usage, storage usage? (3 Replies)
Discussion started by: laknar
3 Replies

8. UNIX for Dummies Questions & Answers

du - Disk Usage for only files and NOT directories.

Hello, Could any one help me how to find the Disk Usage for all the files in the running directory and the sub directories without the disk usage of the directory. I mean to say, i need only the file names without the size of the directories. See, i used this command du -a .|sort... (3 Replies)
Discussion started by: RRVARMA
3 Replies

9. AIX

broken pipe error

Hi, I am working on AIX 5.3 . I have client-server program which is in ProC.while sending packet to server i am getting error as broken pipe and program exiting. please help?/? (1 Reply)
Discussion started by: ajaysahoo
1 Replies

10. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies
Login or Register to Ask a Question