05-07-2010
Quote:
Originally Posted by
prafulnama
The catch is for certain metrics, it requires the last 5 values that it observed so I store those in a flat file and every time a new value is retrieved from the web page, that along with the stored values are used to compare against the threshold. Basically, I am doing everything sequentially so 2 loops, one to read in the IP and the next to do the web page download, threshold check, etc. Every time a new IP is added or a new metric needs to be monitored, the time taken to loop back to a machine increases. I wanted to see if there was a way to improve this?
It would help to see the actual code.
Quote:
Intuitively, I feel, because all historical values are stored in a single flat file, something like multi processing would not work since, a process would have that file locked. Any ideas?
Most systems don't do that kind of locking unless you explicitly ask for it. But having two processes simultaneously read the same file handle wouldn't be a great idea, they might each get half a line or somesuch. If you're just reading flat files line by line, you could try a 'reader' script that reads everything for them and parcels them out individually. That'd have some extra overhead for the extra process and its pipes, but would let more than one reader operate at once.
I'll need to see your actual code to help you here, I think, at least some of it. What needs to be optimized depends not just on what you're doing, but how you're doing it. If you're new to shell scripting there's some trivial design mistakes that could be causing slowdowns... excessive use of pipes and/or backticks is particularly bad. If you've got pipe chains on almost every line, there's probably much room for improvement. In my early scripting days I wrote a linewrapper in BASH that fed everything through about 9 sub-processes, it ended up processing at 10 kilobytes per second!
Last edited by Corona688; 05-07-2010 at 03:02 PM..
Reason: fix inexplicable doublepost
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello All,
I am brand new to the UNIX world and so far and very intrigued and enjoy scripting. This is just a new language for me. I would really like assistance with the below request. Any help would be greatly appreciated!
I want to create a flat file in Vi that has a header field and... (0 Replies)
Discussion started by: cyberjax21
0 Replies
2. Programming
Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.
Here is the logic.
int main()
{
string u_line;
string Char_List;
string u_file;
int line_pos;
string temp_form_u_file;
... (10 Replies)
Discussion started by: dhanamurthy
10 Replies
3. Shell Programming and Scripting
Hi!
Thank you for the help yesterday
This is the finished product
There is one more thing I would like to do to it but I’m not to certain
On how to proceed I would like to log all output to a log in order to
Be able to roll back
This script is meant to be used in repairing a... (4 Replies)
Discussion started by: Ex-Capsa
4 Replies
4. UNIX for Advanced & Expert Users
can anyone help to share the knowledge on linux os improvement?
1) os account
- use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home?
2) user account activity
- how to log os user activity
share the idea and what tools can do that...thx (5 Replies)
Discussion started by: goodbid
5 Replies
5. Shell Programming and Scripting
Heyas
I've been working on my project TUI (Text User Interface) for quite some time now, its a hobby project, so nothing i sit in front of 8hrs/day.
Since the only 'real' programming language i knw is Visual Basic, based upon early steps with MS-Batch files. When i 'joined' linux 3 years ago,... (7 Replies)
Discussion started by: sea
7 Replies
6. Shell Programming and Scripting
Hello Coders
Some time ago i was asking about python and bash performances, and i was told i could post the regarding code, and someone would kindly help to make it faster (if possible).
If you have noted, i'm on the way to finalize, finish, stable TUI - Text(ual) User Interface.
It is a... (6 Replies)
Discussion started by: sea
6 Replies
7. Shell Programming and Scripting
Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script.
Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies
8. Shell Programming and Scripting
Hi,
another little question...
"sn" is an array whose elements can vary from about 55,000 to about 150,000 elements. Each element consists of an integer between 0-255, eg: ${sn} contain the value: 103 . For a decrypt-procedure I need scroll all the elements 4 or 5 times. Here is an example of... (15 Replies)
Discussion started by: math4
15 Replies
9. OS X (Apple)
Hi guys and gals...
MacBook Pro.
OSX 10.13.2, default bash terminal.
I have a flat file 1920 bytes in size of whitespaces only. I need to put every single whitespace character into a bash array cell.
Below are two methods that work, but both are seriously ugly.
The first one requires that I... (7 Replies)
Discussion started by: wisecracker
7 Replies
10. Shell Programming and Scripting
Hello,
For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
LEARN ABOUT MOJAVE
page_util_quote
page_util_quote(n) Parser generator tools page_util_quote(n)
__________________________________________________________________________________________________________________________________________________
NAME
page_util_quote - page character quoting utilities
SYNOPSIS
package require page::util::quote ?0.1?
package require snit
::page::util::quote::unquote char
::page::util::quote::quote'tcl char
::page::util::quote::quote'tclstr char
::page::util::quote::quote'tclcom char
_________________________________________________________________
DESCRIPTION
This package provides a few utility commands to convert characters into various forms.
API
::page::util::quote::unquote char
A character, as stored in an abstract syntax tree by a PEG processor (See the packages grammar::peg::interpreter, grammar::me, and
their relations), i.e. in some quoted form, is converted into the equivalent Tcl character. The character is returned as the result
of the command.
::page::util::quote::quote'tcl char
This command takes a Tcl character (internal representation) and converts it into a string which is accepted by the Tcl parser, will
regenerate the character in question and is 7bit ASCII. The string is returned as the result of this command.
::page::util::quote::quote'tclstr char
This command takes a Tcl character (internal representation) and converts it into a string which is accepted by the Tcl parser and
will generate a human readable representation of the character in question. The string is returned as the result of this command.
The string does not use any unprintable characters. It may use backslash-quoting. High UTF characters are quoted to avoid problems
with the still prevalent ascii terminals. It is assumed that the string will be used in a double-quoted environment.
::page::util::quote::quote'tclcom char
This command takes a Tcl character (internal representation) and converts it into a string which is accepted by the Tcl parser when
used within a Tcl comment. The string is returned as the result of this command.
BUGS, IDEAS, FEEDBACK
This document, will undoubtedly contain bugs and other problems. Please report such in the category page of the Tcllib SF Trackers
[http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have.
KEYWORDS
page, parser generator, quoting, text processing
CATEGORY
Page Parser Generator
COPYRIGHT
Copyright (c) 2007 Andreas Kupries <andreas_kupries@users.sourceforge.net>
page 1.0 page_util_quote(n)