Sponsored Content
Special Forums UNIX and Linux Applications Infrastructure Monitoring Possible performance improvement (Bash and flat file) Post 302419566 by Corona688 on Friday 7th of May 2010 02:01:06 PM
Old 05-07-2010
Quote:
Originally Posted by prafulnama
The catch is for certain metrics, it requires the last 5 values that it observed so I store those in a flat file and every time a new value is retrieved from the web page, that along with the stored values are used to compare against the threshold. Basically, I am doing everything sequentially so 2 loops, one to read in the IP and the next to do the web page download, threshold check, etc. Every time a new IP is added or a new metric needs to be monitored, the time taken to loop back to a machine increases. I wanted to see if there was a way to improve this?
It would help to see the actual code.
Quote:
Intuitively, I feel, because all historical values are stored in a single flat file, something like multi processing would not work since, a process would have that file locked. Any ideas?
Most systems don't do that kind of locking unless you explicitly ask for it. But having two processes simultaneously read the same file handle wouldn't be a great idea, they might each get half a line or somesuch. If you're just reading flat files line by line, you could try a 'reader' script that reads everything for them and parcels them out individually. That'd have some extra overhead for the extra process and its pipes, but would let more than one reader operate at once.

I'll need to see your actual code to help you here, I think, at least some of it. What needs to be optimized depends not just on what you're doing, but how you're doing it. If you're new to shell scripting there's some trivial design mistakes that could be causing slowdowns... excessive use of pipes and/or backticks is particularly bad. If you've got pipe chains on almost every line, there's probably much room for improvement. In my early scripting days I wrote a linewrapper in BASH that fed everything through about 9 sub-processes, it ended up processing at 10 kilobytes per second!

Last edited by Corona688; 05-07-2010 at 03:02 PM.. Reason: fix inexplicable doublepost
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with Flat Files Please!! BASH (New User)

Hello All, I am brand new to the UNIX world and so far and very intrigued and enjoy scripting. This is just a new language for me. I would really like assistance with the below request. Any help would be greatly appreciated! I want to create a flat file in Vi that has a header field and... (0 Replies)
Discussion started by: cyberjax21
0 Replies

2. Programming

File - reading - Performance improvement

Hi All I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file. Here is the logic. int main() { string u_line; string Char_List; string u_file; int line_pos; string temp_form_u_file; ... (10 Replies)
Discussion started by: dhanamurthy
10 Replies

3. Shell Programming and Scripting

Any improvement possible in this script

Hi! Thank you for the help yesterday This is the finished product There is one more thing I would like to do to it but I’m not to certain On how to proceed I would like to log all output to a log in order to Be able to roll back This script is meant to be used in repairing a... (4 Replies)
Discussion started by: Ex-Capsa
4 Replies

4. UNIX for Advanced & Expert Users

linux os improvement

can anyone help to share the knowledge on linux os improvement? 1) os account - use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home? 2) user account activity - how to log os user activity share the idea and what tools can do that...thx (5 Replies)
Discussion started by: goodbid
5 Replies

5. Shell Programming and Scripting

Display-performance in terminal, bash or python?

Heyas I've been working on my project TUI (Text User Interface) for quite some time now, its a hobby project, so nothing i sit in front of 8hrs/day. Since the only 'real' programming language i knw is Visual Basic, based upon early steps with MS-Batch files. When i 'joined' linux 3 years ago,... (7 Replies)
Discussion started by: sea
7 Replies

6. Shell Programming and Scripting

[BASH] Performance question - Script to STDOUT

Hello Coders Some time ago i was asking about python and bash performances, and i was told i could post the regarding code, and someone would kindly help to make it faster (if possible). If you have noted, i'm on the way to finalize, finish, stable TUI - Text(ual) User Interface. It is a... (6 Replies)
Discussion started by: sea
6 Replies

7. Shell Programming and Scripting

Performance improvement in grep

Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script. Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies

8. Shell Programming and Scripting

Bash - array loop performance

Hi, another little question... "sn" is an array whose elements can vary from about 55,000 to about 150,000 elements. Each element consists of an integer between 0-255, eg: ${sn} contain the value: 103 . For a decrypt-procedure I need scroll all the elements 4 or 5 times. Here is an example of... (15 Replies)
Discussion started by: math4
15 Replies

9. OS X (Apple)

Create a bash array from a flat file of whitespaces only.

Hi guys and gals... MacBook Pro. OSX 10.13.2, default bash terminal. I have a flat file 1920 bytes in size of whitespaces only. I need to put every single whitespace character into a bash array cell. Below are two methods that work, but both are seriously ugly. The first one requires that I... (7 Replies)
Discussion started by: wisecracker
7 Replies

10. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
maildir(5)                                                      File Formats Manual                                                     maildir(5)

NAME
maildir - directory for incoming mail messages INTRODUCTION
maildir is a structure for directories of incoming mail messages. It solves the reliability problems that plague mbox files and mh fold- ers. RELIABILITY ISSUES
A machine may crash while it is delivering a message. For both mbox files and mh folders this means that the message will be silently truncated. Even worse: for mbox format, if the message is truncated in the middle of a line, it will be silently joined to the next mes- sage. The mail transport agent will try again later to deliver the message, but it is unacceptable that a corrupted message should show up at all. In maildir, every message is guaranteed complete upon delivery. A machine may have two programs simultaneously delivering mail to the same user. The mbox and mh formats require the programs to update a single central file. If the programs do not use some locking mechanism, the central file will be corrupted. There are several mbox and mh locking mechanisms, none of which work portably and reliably. In contrast, in maildir, no locks are ever necessary. Different delivery processes never touch the same file. A user may try to delete messages from his mailbox at the same moment that the machine delivers a new message. For mbox and mh formats, the user's mail-reading program must know what locking mechanism the mail-delivery programs use. In contrast, in maildir, any delivered message can be safely updated or deleted by a mail-reading program. Many sites use Sun's Network Failure System (NFS), presumably because the operating system vendor does not offer anything else. NFS exac- erbates all of the above problems. Some NFS implementations don't provide any reliable locking mechanism. With mbox and mh formats, if two machines deliver mail to the same user, or if a user reads mail anywhere except the delivery machine, the user's mail is at risk. maildir works without trouble over NFS. THE MAILDIR STRUCTURE
A directory in maildir format has three subdirectories, all on the same filesystem: tmp, new, and cur. Each file in new is a newly delivered mail message. The modification time of the file is the delivery date of the message. The message is delivered without an extra UUCP-style From_ line, without any >From quoting, and without an extra blank line at the end. The message is normally in RFC 822 format, starting with a Return-Path line and a Delivered-To line, but it could contain arbitrary binary data. It might not even end with a newline. Files in cur are just like files in new. The big difference is that files in cur are no longer new mail: they have been seen by the user's mail-reading program. HOW A MESSAGE IS DELIVERED
The tmp directory is used to ensure reliable delivery, as discussed here. A program delivers a mail message in six steps. First, it chdir()s to the maildir directory. Second, it stat()s the name tmp/time.pid.host, where time is the number of seconds since the beginning of 1970 GMT, pid is the program's process ID, and host is the host name. Third, if stat() returned anything other than ENOENT, the program sleeps for two seconds, updates time, and tries the stat() again, a limited number of times. Fourth, the program creates tmp/time.pid.host. Fifth, the program NFS-writes the message to the file. Sixth, the program link()s the file to new/time.pid.host. At that instant the message has been successfully delivered. The delivery program is required to start a 24-hour timer before creating tmp/time.pid.host, and to abort the delivery if the timer expires. Upon error, timeout, or normal completion, the delivery program may attempt to unlink() tmp/time.pid.host. NFS-writing means (1) as usual, checking the number of bytes returned from each write() call; (2) calling fsync() and checking its return value; (3) calling close() and checking its return value. (Standard NFS implementations handle fsync() incorrectly but make up for it by abusing close().) HOW A MESSAGE IS READ
A mail reader operates as follows. It looks through the new directory for new messages. Say there is a new message, new/unique. The reader may freely display the contents of new/unique, delete new/unique, or rename new/unique as cur/unique:info. See http://pobox.com/~djb/proto/maildir.html for the meaning of info. The reader is also expected to look through the tmp directory and to clean up any old files found there. A file in tmp may be safely removed if it has not been accessed in 36 hours. It is a good idea for readers to skip all filenames in new and cur starting with a dot. Other than this, readers should not attempt to parse filenames. ENVIRONMENT VARIABLES
Mail readers supporting maildir use the MAILDIR environment variable as the name of the user's primary mail directory. SEE ALSO
mbox(5), qmail-local(8) maildir(5)
All times are GMT -4. The time now is 04:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy