Sponsored Content
Top Forums Programming Writing fast and efficiently - how ? Post 22408 by Perderabo on Monday 3rd of June 2002 12:11:13 PM
Old 06-03-2002
You don't give us many details. But if this this like a log file, I would simply use the write(2) system call. If many processes open a file in append mode and then issue write's to it (via the system call directly), the writes will be atomic. Each record will be appended to the file in the order that the writes occur. No additional locking is needed. And the file will be buffered in the buffer cache. The syncer and/or the unix write-behind policy will actually write to disk. This is a very simple solution with low overhead and should be very portable.

I don't understand exactly how you're using shared memory. This is also a viable solution. Most database packages use shared memory and the database vendor compete heavily on performance. But they typically have access to secret os facilities designed just for them.

I have never worked much with threads. But what I have read suggests that maybe they would be an option. The theory goes that when processes lean too much on ipc facilities, they should simply become threads and talk via global data.

Ultimately you should probably try several approaches and benchmark them to find the fastest. And sometimes, if there is a lot of processing to do, there is a lot of processing to do. You can't always wave a magic wand and render intense processing trivial.
 

9 More Discussions You Might Find Interesting

1. IP Networking

how to use PING command efficiently

Do anyone telle me please how to use PING command to verify connection (TCP/IP) between serveurs. thanks (1 Reply)
Discussion started by: hoang
1 Replies

2. Filesystems, Disks and Memory

Writing fast and efficiently - how ?

I have a lot of processes all of which need to write quite a lot of data to the filesystem ( to a single file). This is managed today in the following way : all the processes write the data to a shared memory block, which is manged by a process that empties it to a file, thus allowing more... (1 Reply)
Discussion started by: Seeker
1 Replies

3. Shell Programming and Scripting

Using xapply efficiently?

Hi all, Were currently using xapply to run multiple ssh instances that then calls a script that returns the PID of a webserver process. Currently we have like 30 xapply statements in a script call checkit which checks various webserver processes on various unix/linux boxes. My question... (0 Replies)
Discussion started by: bdsffl
0 Replies

4. UNIX Desktop Questions & Answers

how to search files efficiently using patterns

hi friens, :) if i need to find files with extension .c++,.C++,.cpp,.Cpp,.CPp,.cPP,.CpP,.cpP,.c,.C wat is the pattern for finding them :confused: (2 Replies)
Discussion started by: arunsubbhian
2 Replies

5. Shell Programming and Scripting

How to parse a string efficiently

I am new to the boards and to shell programming and have a requirement to name new files received with a unique sequence number. I need to look at a particular file pattern that exists and then to increment a sequence by 1 and write the new file. Example of file names and sequence # ... (4 Replies)
Discussion started by: sandiego_coder
4 Replies

6. Shell Programming and Scripting

Parse and delete lines efficiently

Hi I have a set of options in the form of key value in a file. Need to find a particular value of 'a' and delete all lines till the next 'a' keyword . Ex : a bbb c ddd e fff g hhh a sss c ggg e xxx f sss a ddd d sss r sss g hhh (5 Replies)
Discussion started by: TDUser
5 Replies

7. UNIX for Dummies Questions & Answers

Efficiently Repeat Text

Hi, Often when I use echo statements in scripts I echo a line of #'s above and below. For example: echo ##### echo hello world echo ##### However, I generally have a series of about 75 #'s. For example: echo #(x 75) echo hello world echo #(X 75) While this helps to delineate... (7 Replies)
Discussion started by: msb65
7 Replies

8. Shell Programming and Scripting

Getting remote variables more efficiently

Hello all, I have a script that has to get variables remotely. Rather than having the script login to the remote server 3 separate times, is there a faster way to get each variable? ##Server comes from input or list## CHKINSTALL=`ssh server "swlist | grep -i program" | grep -v... (2 Replies)
Discussion started by: LinuxRacr
2 Replies

9. Shell Programming and Scripting

Purging 2000+ directories efficiently

Hi I have a requirement wherein i need to purge some directories. I have more than 2000 directories where i need to keep data for 10 days and delete the rest. What i am looking for is an efficient way to achieve this. There are four mount points from where i need to delete the files. ... (3 Replies)
Discussion started by: Apoorvbarwa
3 Replies
Lock(3pm)						User Contributed Perl Documentation						 Lock(3pm)

NAME
DB_File::Lock - Locking with flock wrapper for DB_File SYNOPSIS
use DB_File::Lock; use Fcntl qw(:flock O_RDWR O_CREAT); $locking = "read"; $locking = "write"; $locking = { mode => "read", nonblocking => 0, lockfile_name => "/path/to/shared.lock", lockfile_mode => 0600, }; [$X =] tie %hash, 'DB_File::Lock', $filename, $flags, $mode, $DB_HASH, $locking; [$X =] tie %hash, 'DB_File::Lock', $filename, $flags, $mode, $DB_BTREE, $locking; [$X =] tie @array, 'DB_File::Lock', $filename, $flags, $mode, $DB_RECNO, $locking; # or place the DB_File arguments inside a list reference: [$X =] tie %hash, 'DB_File::Lock', [$filename, $flags, $mode, $DB_HASH], $locking; ...use the same way as DB_File for the rest of the interface... DESCRIPTION
This module provides a wrapper for the DB_File module, adding locking. When you need locking, simply use this module in place of DB_File and add an extra argument onto the tie command specifying if the file should be locked for reading or writing. The alternative is to write code like: open(LOCK, "<$db_filename.lock") or die; flock(LOCK, LOCK_SH) or die; tie(%db_hash, 'DB_File', $db_filename, O_RDONLY, 0600, $DB_HASH) or die; ... then read the database ... untie(%db_hash); close(LOCK); This module lets you write tie(%db_hash, 'DB_File::Lock', $db_filename, O_RDONLY, 0600, $DB_HASH, 'read') or die; ... then read the database ... untie(%db_hash); This is better for two reasons:(1) Less cumbersome to write.(2) A fatal exception in the code working on the database which does not lead to process termination will probably not close the lockfile and therefore cause a dropped lock. USAGE DETAILS
Tie to the database file by adding an additional locking argument to the list of arguments to be passed through to DB_File, such as: tie(%db_hash, 'DB_File::Lock', $db_filename, O_RDONLY, 0600, $DB_HASH, 'read'); or enclose the arguments for DB_File in a list reference: tie(%db_hash, 'DB_File::Lock', [$db_filename, O_RDONLY, 0600, $DB_HASH], 'read'); The filename used for the lockfile defaults to "$filename.lock" (the filename of the DB_File with ".lock" appended). Using a lockfile separate from the database file is recommended because it prevents weird interactions with the underlying database file library The additional locking argument added to the tie call can be:(1) "read" -- acquires a shared lock for reading(2) "write" -- acquires an exclusive lock for writing(3) A hash with the following keys (all optional except for the "mode"): mode the locking mode, "read" or "write". lockfile_name specifies the name of the lockfile to use. Default is "$filename.lock". This is useful for locking multiple resources with the same lockfiles. nonblocking determines if the flock call on the lockfile should block waiting for a lock, or if it should return failure if a lock can not be immediately attained. If "nonblocking" is set and a lock can not be attained, the tie command will fail. Currently, I'm not sure how to differentiate this between a failure form the DB_File layer. lockfile_mode determines the mode for the sysopen call in opening the lockfile. The default mode will be formulated to allow anyone that can read or write the DB_File permission to read and write the lockfile. (This is because some systems may require that one have write access to a file to lock it for reading, I understand.) The umask will be prevented from applying to this mode. Note: One may import the same values from DB_File::Lock as one may import from DB_File. GOOD LOCKING ETIQUETTE
To avoid locking problems, realize that it is critical that you release the lock as soon as possible. See the lock as a "hot potato", something that you must work with and get rid of as quickly as possible. See the sections of code where you have a lock as "critical" sections. Make sure that you call "untie" as soon as possible. It is often better to write: # open database file with lock # work with database # lots of processing not related to database # work with database # close database and release lock as: # open database file with lock # work with database # close database and release lock # lots of processing not related to database # open database file with lock # work with database # close database and release lock Also realize that when acquiring two locks at the same time, a deadlock situation can be caused. You can enter a deadlock situation if two processes simultaneously try to acquire locks on two separate databases. Each has locked only one of the databases, and cannot continue without locking the second. Yet this will never be freed because it is locked by the other process. If your processes all ask for their DB files in the same order, this situation cannot occur. OTHER LOCKING MODULES
There are three locking wrappers for DB_File in CPAN right now. Each one implements locking differently and has different goals in mind. It is therefore worth knowing the difference, so that you can pick the right one for your application. Here are the three locking wrappers: Tie::DB_Lock -- DB_File wrapper which creates copies of the database file for read access, so that you have kind of a multiversioning concurrent read system. However, updates are still serial. Use for databases where reads may be lengthy and consistency problems may occur. Tie::DB_LockFile -- DB_File wrapper that has the ability to lock and unlock the database while it is being used. Avoids the tie-before- flock problem by simply re-tie-ing the database when you get or drop a lock. Because of the flexibility in dropping and re-acquiring the lock in the middle of a session, this can be massaged into a system that will work with long updates and/or reads if the application follows the hints in the POD documentation. DB_File::Lock (this module) -- extremely lightweight DB_File wrapper that simply flocks a lockfile before tie-ing the database and drops the lock after the untie. Allows one to use the same lockfile for multiple databases to avoid deadlock problems, if desired. Use for databases where updates are reads are quick and simple flock locking semantics are enough. (This text duplicated in the POD documentation, by the way.) AUTHOR
David Harris <dharris@drh.net> Helpful insight from Stas Bekman <stas@stason.org> SEE ALSO
DB_File(3). perl v5.10.0 2009-07-23 Lock(3pm)
All times are GMT -4. The time now is 07:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy