URGENT Help required regarding the use of FORK system call


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users URGENT Help required regarding the use of FORK system call
# 1  
Error URGENT Help required regarding the use of FORK system call

I desperately wanted one of the UNIX Gurus to help me resolve my problem asap(I have to deliver the code to the client by Monday 08-oct).

I have a file with around 5 million records (50 lakhs). Now my original process was taking around 30 hours to read the complete file, process each and every record and write it to another file. we do a lot of calculations for each and every record so it takes that time.

Now I planned to implement PARALLEL processing in my program. So I am dividing the complete input file into 5 chunks (each of 1 million) and sending every chunk to every child process to process. Now every child process will process its own chunk and write it to its own temporary file. Finally in the parent process I am planning to merge all the temporary files together. By doing this I believe I shall save a lot of processing time.

What I am basically interested to know is that what are the side effects of using FORK in the C programs? Are there any SYSTEM level impacts by using FORK? Is there any system call to merge multiple files into ONE? I am interested in knowing what happens if a Child is killed? How can I reprocess the child that is killed? How do I ensure that there are no ZOMBIES or INFANT process created?

Can some one briefly advise how I can proceed with my Logic. I have already written the logic but I want to cross check if there is something I am missing in my logic.

Thanks,
Kumar
# 2  
Well, a few general comments...cracking a program into 5 subprocesses like this makes sense only if you have 5 or more cpu's available.

To have children a process must never fork(). To have no zombies, a process must issue a wait() each time a child dies. A process can catch SIGCHLD to be notified of the death of a child.

I would try to drive my processing time per record down and use buffered i/o. Make sure that you don't fork any processes per record.

Finally, if your program sucks, the client will not give you any brownie points for the fact that it was on time. And if it's a few days late but perfect, he will forget the lateness eventually. Don't sacrifice the quality for the timetable, it's not worth it.
# 3  
Having said that..... it is still not clear how the proposed algorithm by the poster will greatly speed up the processing. As stated, having 5 parallel tasks on the same platform with one CPU does not offer very many enhancements over one task on the the platform with one CPU.

Prior to diving into coding and programming, it would be wise to develop an architecture and/or processing algorithm that works. I don't think was have got to that point yet, have we?

For example:

What is the CPU? Is the system CPU constrained?

How much memory is on the platform? Is the system memory (and/or swap) contrained?

Questions like these need to be addressed before looking at the system calls. It is quite possible the system is simply memory constrained and trashing due to swap problems...... (or simply needs more memory).
# 4  
The previous guru's are quite correct.

I might even suggest that you forget about fork(), wait(),
SIGCHLD, etc. and just launch 5 different instances of the
same program with say a file name as an argument...


#!/bin/ksh

myprog file1 > /tmp/file1.$$ &
kid1pid=$!
myprog file2 > /tmp/file2.$$ &
kid2pid=$!
....
myprog file5 > /tmp/file3.$$ &
kid5pid=$!

# at this point you wait on the kids
wait

# here you can assemble the files...
Cwd=`pwd`
cd /tmp
for i in `ls file?.*`
do
cat $i >> $Cwd/newfile.$$
done
rm -f file?.*
cd $Cwd

exit 0



...so... it ain't pretty, it ain't slick or elegant, hell, it ain't even C
but it is simple and will work by Monday Smilie
# 5  
Thanks a lot rwb1959!!

Your suggestion looks great. I shall better try this rather than using fork. I have already implemented the fork mechanism but I started to be a bit worried after looking at your responses.

Anyway thanks a lot for the help.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #365
Difficulty: Medium
The original code for vi was written by Bill Joy in 1976.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help: how to call fork() in shell script? New to linux

Hi, I'm writing a shell script where I want to call fork(). However I wrote like this "var=fork()" in c style and got this error: "syntax error near unexpected token `(' " How could I call fork() in shell script? Thanks in advance. Duplicate Post - Continue Here - Please Do Not Cross Post... (0 Replies)
Discussion started by: Xiaoya
0 Replies

2. UNIX for Advanced & Expert Users

Doubt with fork() system call

Hi I wrote a simple fork program to illustrate the fork() system cal. here it is #include<stdio.h> #include<sys/stat.h> #include<sys/types.h> main() { int flag; flag=fork(); if(flag==0) { printf("Child \n"); printf("Process id= %d\n",getpid()); ... (3 Replies)
Discussion started by: badsha6642
3 Replies

3. Programming

Problem with execution of fork system call if i use \n

hi all, i tried the following source codes: fork1.c: main() { printf("demo of fork\n"); fork(); printf("hello"); } output: demo of fork hello hello fork2.c: main() (3 Replies)
Discussion started by: pnirmala
3 Replies

4. Shell Programming and Scripting

fork system call and \n

hi, i tried the following source codes: fork1.c: main() { printf("demo of fork\n"); fork(); printf("hello"); } output: demo of fork hello hello fork2.c: main() { printf("demo of fork"); (0 Replies)
Discussion started by: pnirmala
0 Replies

5. Homework & Coursework Questions

fork system call understanding

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: i have a problem in understanding the behaviour of fork . i understood fork as to create a new process and... (4 Replies)
Discussion started by: MrUser
4 Replies

6. UNIX for Dummies Questions & Answers

fork system call

Hi folks, I want to know how this below program works? #include <stdio.h> int main() { printf("A\n"); fork(); printf("B\n"); fork(); fork(); printf("D\n"); fork(); printf("C\n"); } This is just for example. How this type of programs where fork is used many places, how the... (1 Reply)
Discussion started by: u_peerless
1 Replies

7. Programming

Help required with using system() call

Hi, I try to write a C program which lists the output of a paticular command with all the available options (a to z) for the command in the directory of execution. This program will generate the output if the option exists for the particular command else it will display some message saying... (9 Replies)
Discussion started by: ramkrix
9 Replies

8. UNIX for Dummies Questions & Answers

fork() system call

Can anyone explain me what really happens when a system call fork() is called ? I like to know what happens internally. Thanks in Advance. - Arun (1 Reply)
Discussion started by: arunviswanath
1 Replies

9. Programming

Fork() system call time?

One more question. How can i calculate the time that system needs to make fork() system call? I need to make it with times function but i really don't know how. :( (2 Replies)
Discussion started by: davidoff
2 Replies

10. Programming

Need urgent help with fork()

Hy! I must wrote some code with fork() command. The thing is that i have a while statement which count till 10. I must wrote a program that one child has only one parent. So one parent has only one child and one child has only one parent. Can you please help me with these code. int main()... (2 Replies)
Discussion started by: davidoff
2 Replies

Featured Tech Videos