05-07-2010
Quote:
Originally Posted by
prafulnama
The catch is for certain metrics, it requires the last 5 values that it observed so I store those in a flat file and every time a new value is retrieved from the web page, that along with the stored values are used to compare against the threshold. Basically, I am doing everything sequentially so 2 loops, one to read in the IP and the next to do the web page download, threshold check, etc. Every time a new IP is added or a new metric needs to be monitored, the time taken to loop back to a machine increases. I wanted to see if there was a way to improve this?
It would help to see the actual code.
Quote:
Intuitively, I feel, because all historical values are stored in a single flat file, something like multi processing would not work since, a process would have that file locked. Any ideas?
Most systems don't do that kind of locking unless you explicitly ask for it. But having two processes simultaneously read the same file handle wouldn't be a great idea, they might each get half a line or somesuch. If you're just reading flat files line by line, you could try a 'reader' script that reads everything for them and parcels them out individually. That'd have some extra overhead for the extra process and its pipes, but would let more than one reader operate at once.
I'll need to see your actual code to help you here, I think, at least some of it. What needs to be optimized depends not just on what you're doing, but how you're doing it. If you're new to shell scripting there's some trivial design mistakes that could be causing slowdowns... excessive use of pipes and/or backticks is particularly bad. If you've got pipe chains on almost every line, there's probably much room for improvement. In my early scripting days I wrote a linewrapper in BASH that fed everything through about 9 sub-processes, it ended up processing at 10 kilobytes per second!
Last edited by Corona688; 05-07-2010 at 03:02 PM..
Reason: fix inexplicable doublepost
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello All,
I am brand new to the UNIX world and so far and very intrigued and enjoy scripting. This is just a new language for me. I would really like assistance with the below request. Any help would be greatly appreciated!
I want to create a flat file in Vi that has a header field and... (0 Replies)
Discussion started by: cyberjax21
0 Replies
2. Programming
Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.
Here is the logic.
int main()
{
string u_line;
string Char_List;
string u_file;
int line_pos;
string temp_form_u_file;
... (10 Replies)
Discussion started by: dhanamurthy
10 Replies
3. Shell Programming and Scripting
Hi!
Thank you for the help yesterday
This is the finished product
There is one more thing I would like to do to it but I’m not to certain
On how to proceed I would like to log all output to a log in order to
Be able to roll back
This script is meant to be used in repairing a... (4 Replies)
Discussion started by: Ex-Capsa
4 Replies
4. UNIX for Advanced & Expert Users
can anyone help to share the knowledge on linux os improvement?
1) os account
- use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home?
2) user account activity
- how to log os user activity
share the idea and what tools can do that...thx (5 Replies)
Discussion started by: goodbid
5 Replies
5. Shell Programming and Scripting
Heyas
I've been working on my project TUI (Text User Interface) for quite some time now, its a hobby project, so nothing i sit in front of 8hrs/day.
Since the only 'real' programming language i knw is Visual Basic, based upon early steps with MS-Batch files. When i 'joined' linux 3 years ago,... (7 Replies)
Discussion started by: sea
7 Replies
6. Shell Programming and Scripting
Hello Coders
Some time ago i was asking about python and bash performances, and i was told i could post the regarding code, and someone would kindly help to make it faster (if possible).
If you have noted, i'm on the way to finalize, finish, stable TUI - Text(ual) User Interface.
It is a... (6 Replies)
Discussion started by: sea
6 Replies
7. Shell Programming and Scripting
Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script.
Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies
8. Shell Programming and Scripting
Hi,
another little question...
"sn" is an array whose elements can vary from about 55,000 to about 150,000 elements. Each element consists of an integer between 0-255, eg: ${sn} contain the value: 103 . For a decrypt-procedure I need scroll all the elements 4 or 5 times. Here is an example of... (15 Replies)
Discussion started by: math4
15 Replies
9. OS X (Apple)
Hi guys and gals...
MacBook Pro.
OSX 10.13.2, default bash terminal.
I have a flat file 1920 bytes in size of whitespaces only. I need to put every single whitespace character into a bash array cell.
Below are two methods that work, but both are seriously ugly.
The first one requires that I... (7 Replies)
Discussion started by: wisecracker
7 Replies
10. Shell Programming and Scripting
Hello,
For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
LEARN ABOUT CENTOS
pmdabash
PMDABASH(1) General Commands Manual PMDABASH(1)
NAME
pmdabash - Bourne-Again SHell trace performance metrics domain agent
SYNOPSIS
$PCP_PMDAS_DIR/bash/pmdabash [-C] [-d domain] [-l logfile] [-I interval] [-t timeout] [-U username] configfile
DESCRIPTION
pmdabash is an experimental Performance Metrics Domain Agent (PMDA) which exports "xtrace" events from a traced bash(1) process. This
includes the command execution information that would usually be sent to standard error with the set -x option to the shell.
Event metrics are exported showing each command executed, the function name and line number in the script, and a timestamp. Additionally,
the process identifier for the shell and its parent process are exported.
This requires bash version 4 or later.
A brief description of the pmdabash command line options follows:
-d It is absolutely crucial that the performance metrics domain number specified here is unique and consistent. That is, domain should
be different for every PMDA on the one host, and the same domain number should be used for the same PMDA on all hosts.
-l Location of the log file. By default, a log file named bash.log is written in the current directory of pmcd(1) when pmdabash is
started, i.e. $PCP_LOG_DIR/pmcd. If the log file cannot be created or is not writable, output is written to the standard error
instead.
-s Amount of time (in seconds) between subsequent evaluations of the shell trace file descriptor(s). The default is 2 seconds.
-m Maximum amount of memory to be allowed for each event queue (one per traced process). The default is 2 megabytes.
-U User account under which to run the agent. The default is the unprivileged "pcp" account in current versions of PCP, but in older
versions the superuser account ("root") was used by default.
INSTALLATION
In order for a host to export the names, help text and values for the bash performance metrics, do the following as root:
# cd $PCP_PMDAS_DIR/bash
# ./Install
As soon as an instrumented shell script (see INSTRUMENTATION selection below) is run, with tracing enabled, new metric values will appear -
no further setup of the agent is required.
If you want to undo the installation, do the following as root:
# cd $PCP_PMDAS_DIR/bash
# ./Remove
pmdabash is launched by pmcd(1) and should never be executed directly. The Install and Remove scripts notify pmcd(1) when the agent is
installed or removed.
INSTRUMENTATION
In order to allow the flow of event data between a bash(1) script and pmdabash, the script should take the following actions:
#!/bin/sh
source $PCP_DIR/etc/pcp.sh
pcp_trace on $@ # enable tracing
echo "awoke, $count"
pcp_trace off # disable tracing
The tracing can be enabled and disabled any number of times by the script. On successful installation of the agent, several metrics will
be available:
$ pminfo bash
bash.xtrace.numclients
bash.xtrace.maxmem
bash.xtrace.queuemem
bash.xtrace.count
bash.xtrace.records
bash.xtrace.parameters.pid
bash.xtrace.parameters.parent
bash.xtrace.parameters.lineno
bash.xtrace.parameters.function
bash.xtrace.parameters.command
When an instrumented script is running, the generation of event records can be verified using the pmevent(1) command, as follows:
$ pmevent -t 1 -x '' bash.xtrace.records
host: localhost
samples: all
bash.xtrace.records["4538 ./test-trace.sh 1 2 3"]: 5 event records
10:00:05.000 --- event record [0] flags 0x19 (point,id,parent) ---
bash.xtrace.parameters.pid 4538
bash.xtrace.parameters.parent 4432
bash.xtrace.parameters.lineno 43
bash.xtrace.parameters.command "true"
10:00:05.000 --- event record [1] flags 0x19 (point,id,parent) ---
bash.xtrace.parameters.pid 4538
bash.xtrace.parameters.parent 4432
bash.xtrace.parameters.lineno 45
bash.xtrace.parameters.command "(( count++ ))"
10:00:05.000 --- event record [2] flags 0x19 (point,id,parent) ---
bash.xtrace.parameters.pid 4538
bash.xtrace.parameters.parent 4432
bash.xtrace.parameters.lineno 46
bash.xtrace.parameters.command "echo 'awoke, 3'"
10:00:05.000 --- event record [3] flags 0x19 (point,id,parent) ---
bash.xtrace.parameters.pid 4538
bash.xtrace.parameters.parent 4432
bash.xtrace.parameters.lineno 47
bash.xtrace.parameters.command "tired 2"
10:00:05.000 --- event record [4] flags 0x19 (point,id,parent) ---
bash.xtrace.parameters.pid 4538
bash.xtrace.parameters.parent 4432
bash.xtrace.parameters.lineno 38
bash.xtrace.parameters.function "tired"
bash.xtrace.parameters.command "sleep 2"
FILES
$PCP_PMCDCONF_PATH
command line options used to launch pmdabash
$PCP_PMDAS_DIR/bash/help
default help text file for the bash metrics
$PCP_PMDAS_DIR/bash/Install
installation script for the pmdabash agent
$PCP_PMDAS_DIR/bash/Remove
undo installation script for pmdabash
$PCP_LOG_DIR/pmcd/bash.log
default log file for error messages and other information from pmdabash
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the
file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configura-
tion file, as described in pcp.conf(5).
SEE ALSO
bash(1), pmevent(1) and pmcd(1).
Performance Co-Pilot PCP PMDABASH(1)