Visit Our UNIX and Linux User Community


Multiple process write to same log file at the same time


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multiple process write to same log file at the same time
# 1  
Old 09-25-2010
Multiple process write to same log file at the same time

If we have 3 process to write to same log file at the same time like below. will it cause the data outdated because the multiple process writing same time? It this a safe way to keep the log for multiple process?

p1 >> test.log &;
p2 >> test.log &;
p3 >> test.log &


Thanks,
# 2  
Old 09-27-2010
any suggestions?

Thanks
# 3  
Old 09-27-2010
Can you detail what the processes p? are written in ? What O/S are they running on, and are they script, or do you have code ?
For example in script, linux has a "lockfile" command that could help in this instance.
# 4  
Old 09-27-2010
Thanks, My question is a general question without special user case.

Maybe we can create a example to insert data 1-10000 in process to log and run 3 processes at the same time .

As you point , lockfile in linux could lock the file for exclusive writing, so I assume if we don't use lockfile or similar command, the above situation will cause the race condition or outdated data.

If I am wrong in above, please let me know.

Thanks again for your comments.
# 5  
Old 09-27-2010
lock file concept is outdated - not maintainable or fool proof. Check for some mutex/synchronization mechanims.
# 6  
Old 09-27-2010
The example will not cause a race condition, at worst it will corrupt the log file. However, If the processes write the data in single write statements (most do, either by using single "write" API calls or by using streamed file I/O which will buffer output), and each log line is less than 4kb (streamed I/O is), and the log file is local, then most modern UNIX/Linux hosts will not intermingle bytes within separate log lines. This is only a rule of thumb however, so I would not rely on this to work 100% across different O/S.

From your example in the first post it seems you launching the processes from the shell and are separate processes, and therefore you cannot use mutex's as matrixmadhan has suggested. Is this correct, or are you in control of the code, and could you implement threads (as mutexes are very useful and would solve this issue). Otherwise, Im afraid that I am in disagreement with matrixmadhan's suggestion that lock files are outdated - they are still a very useful tool which most daemons and other applications rely on (for example the whole System V "init" script architecture on most UNIX/Linux implementations still use this mechanism in creating "pid" files in /var/run, etc).

So I would say that if you are creating a simple application or script that calls multiple processes and isnt 100% mission critical/bullet proof, then "lockfile" would probably be a useful tool for you. If, however, you are developing a mission critical service that has to 100% guarantee data integrity, that you should look into more advanced synchronization techniques and/or data repository services.

I hope this helps.

Previous Thread | Next Thread
Test Your Knowledge in Computers #719
Difficulty: Medium
Alan Minsky wrote the book Artificial Neural Networks, attacking the work of Frank Rosenblatt, which became the foundational work in the analysis of artificial intelligence applications in machine learning.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Subprocess.popen() should write to log without waiting for process to complete

Here is my test code process = sp.Popen( + , bufsize=1, universal_newlines=True, stdout=sp.PIPE, stderr=sp.STDOUT, cwd=src_home) output, _ =... (2 Replies)
Discussion started by: ezee
2 Replies

2. Shell Programming and Scripting

Ksh: How to write the log file simultaneously when .sql file executes by UNIX process

Hello Team- we would like to implement an approach which has to write the log file simultaneously when .sql file is executing by Unix process. At present,it is writing the log file once the process is completed. I've tested the current process with the below approaches and none of them... (1 Reply)
Discussion started by: Hima_B
1 Replies

3. Shell Programming and Scripting

Batch write multiple usb in same time

Hi there, I would write a bash script to format then create a range of folder (folder name is number) to a usb stick. for numbers in $(seq -w 001 999) do pause "Press any key to start" mkfs.vfat -F32 /dev/sdc1 mount /dev/sdc1 /media/usb mkdir /media/$numbers umount /dev/sdc1... (8 Replies)
Discussion started by: wayneliao38
8 Replies

4. UNIX for Advanced & Expert Users

Multiple processes write to log file at the same time

If we have 3 process to write to same log file at the same time like below. will it cause the data outdated because the multiple process writing same time? It this a safe way to keep the log for multiple process? p1 >> test.log &; p2 >> test.log &; p3 >> test.log & Thanks, (1 Reply)
Discussion started by: casttree
1 Replies

5. UNIX for Advanced & Expert Users

when a process fails to write to /dev/log

Hi , when a process fails to write to /dev/log ? (1 Reply)
Discussion started by: Gopi Krishna P
1 Replies

6. Shell Programming and Scripting

Write process output to a file

When I run <ls -l> to get a list of all the files, I want the displayed result to be written to a text file. Is there a way to do that? (1 Reply)
Discussion started by: kn.naresh
1 Replies

7. Shell Programming and Scripting

how to run multiple process at the same time

Hello guys, Look what im doing: I need to run a process from a SERVER1 to SERVER2, SERVER3 and SERVER4. The shell of the process is in each SERVER (2 to 4) So from SERVER1 i do: for i in SERVER2 SERVER3 SERVER4 do rsh $i ' ./process.sh ' done The problem is: each process.sh... (2 Replies)
Discussion started by: lestat_ecuador
2 Replies

8. UNIX for Dummies Questions & Answers

Script to Extract time from log files and write to a excel

Can someone help me with writing a unix script for following requirement 1) I have a log file in which we have start time and end time (format: hh:mm:ss) Example: starting script on Thu Jun 5 20:50:52 --------- Thu Jun 5 21:55:33 - Script Completed 2) I want to extract... (4 Replies)
Discussion started by: santosham
4 Replies

9. Shell Programming and Scripting

Script to Extract time from log files and write to a excel

Can someone help me with writing a unix script for following requirement 1) I have a log file in which we have start time and end time (format: hh:mm:ss) Example: starting script on Thu Jun 5 20:50:52 Thu Jun 5 21:55:33 - Script Completed 2) I want to extract start time and end time of... (0 Replies)
Discussion started by: santosham
0 Replies

10. UNIX for Advanced & Expert Users

Script to Extract time from log files and write to a excel

Can someone help me with writing a unix script for following requirement 1) I have a log file in which we have start time and end time (format: hh:mm:ss) Example: starting script on Thu Jun 5 20:50:52 Thu Jun 5 21:55:33 - Script Completed 2) I want to extract start time and end time of... (0 Replies)
Discussion started by: santosham
0 Replies

Featured Tech Videos