05-07-2010
Possible performance improvement (Bash and flat file)
Hello,
I am pretty new to shell scripts and I recently wrote one that seems to do what it should but I am exploring the possibility of improving its performance and would appreciate some help. Here is what it does - Its meant to monitor a bunch of systems (reads in IPs one at a time from a flat file). For each IP, it fetches a set of web pages, parses them to extract certain numbers, compares them against defined thresholds and alerts if the metric falls outside the threshold range. The catch is for certain metrics, it requires the last 5 values that it observed so I store those in a flat file and every time a new value is retrieved from the web page, that along with the stored values are used to compare against the threshold. Basically, I am doing everything sequentially so 2 loops, one to read in the IP and the next to do the web page download, threshold check, etc. Every time a new IP is added or a new metric needs to be monitored, the time taken to loop back to a machine increases. I wanted to see if there was a way to improve this? Intuitively, I feel, because all historical values are stored in a single flat file, something like multi processing would not work since, a process would have that file locked. Any ideas?????
Thanks,
-p
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello All,
I am brand new to the UNIX world and so far and very intrigued and enjoy scripting. This is just a new language for me. I would really like assistance with the below request. Any help would be greatly appreciated!
I want to create a flat file in Vi that has a header field and... (0 Replies)
Discussion started by: cyberjax21
0 Replies
2. Programming
Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.
Here is the logic.
int main()
{
string u_line;
string Char_List;
string u_file;
int line_pos;
string temp_form_u_file;
... (10 Replies)
Discussion started by: dhanamurthy
10 Replies
3. Shell Programming and Scripting
Hi!
Thank you for the help yesterday
This is the finished product
There is one more thing I would like to do to it but I’m not to certain
On how to proceed I would like to log all output to a log in order to
Be able to roll back
This script is meant to be used in repairing a... (4 Replies)
Discussion started by: Ex-Capsa
4 Replies
4. UNIX for Advanced & Expert Users
can anyone help to share the knowledge on linux os improvement?
1) os account
- use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home?
2) user account activity
- how to log os user activity
share the idea and what tools can do that...thx (5 Replies)
Discussion started by: goodbid
5 Replies
5. Shell Programming and Scripting
Heyas
I've been working on my project TUI (Text User Interface) for quite some time now, its a hobby project, so nothing i sit in front of 8hrs/day.
Since the only 'real' programming language i knw is Visual Basic, based upon early steps with MS-Batch files. When i 'joined' linux 3 years ago,... (7 Replies)
Discussion started by: sea
7 Replies
6. Shell Programming and Scripting
Hello Coders
Some time ago i was asking about python and bash performances, and i was told i could post the regarding code, and someone would kindly help to make it faster (if possible).
If you have noted, i'm on the way to finalize, finish, stable TUI - Text(ual) User Interface.
It is a... (6 Replies)
Discussion started by: sea
6 Replies
7. Shell Programming and Scripting
Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script.
Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies
8. Shell Programming and Scripting
Hi,
another little question...
"sn" is an array whose elements can vary from about 55,000 to about 150,000 elements. Each element consists of an integer between 0-255, eg: ${sn} contain the value: 103 . For a decrypt-procedure I need scroll all the elements 4 or 5 times. Here is an example of... (15 Replies)
Discussion started by: math4
15 Replies
9. OS X (Apple)
Hi guys and gals...
MacBook Pro.
OSX 10.13.2, default bash terminal.
I have a flat file 1920 bytes in size of whitespaces only. I need to put every single whitespace character into a bash array cell.
Below are two methods that work, but both are seriously ugly.
The first one requires that I... (7 Replies)
Discussion started by: wisecracker
7 Replies
10. Shell Programming and Scripting
Hello,
For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
LEARN ABOUT DEBIAN
sysprofile
SYSPROFILE(8) System Manager's Manual SYSPROFILE(8)
NAME
sysprofile - modular centralized shell configuration
DESCRIPTION
sysprofile is a generic approach to configure shell settings in a modular and centralized way mostly aimed at avoiding work for lazy sysad-
mins. It has only been tested to work with the bash shell.
It basically consists of the small /etc/sysprofile shell script which invokes other small shell scripts having a .bash suffix which are
contained in the /etc/sysprofile.d/ directory. The system administrator can drop in any script he wants without any naming convention
other than that the scripts need to have a .bash suffix to enable automagic sourcing by /etc/sysprofile.
This mechanism is set up by inserting a small shell routine into /etc/profile for login shells and optionally into /etc/bashrc and/or
/etc/bash.bashrc for non-login shells from where the actual /etc/sysprofile script is invoked:
if [ -f /etc/sysprofile ]; then
. /etc/sysprofile
fi
For using "sysprofile" under X11, one can source it in a similar way from /etc/X11/Xsession or your X display manager's Xsession file to
provide the same shell environment as under the console in X11. See the example files in /usr/share/doc/sysprofile/ for illustration.
For usage of terminal emulators with a non-login bash shell under X11, take care to enable sysprofile via /etc/bash.bashrc. If not set
this way, your terminal emulators won't come up with the environment defined by the scripts in /etc/sysprofile.d/.
Users not wanting /etc/sysprofile to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by
simply creating an empty file called $HOME/.nosysprofile in the user's home directory using e.g. the touch(1) command.
Any single configuration file in /etc/sysprofile.d/ can be overridden by any user by creating a private $HOME/.sysprofile.d/ directory
which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to
match exactly the system's default /etc/sysprofile.d/ configuration files. Empty versions of these files contained in the $HOME/.syspro-
file.d/ directory automatically disable sourcing of the system wide version.
Naturally, users can add and include their own private script inventions to be automagically executed by /etc/sysprofile at login time.
OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves.
SEE ALSO
The README files and configuration examples contained in /etc/sysprofile.d/ and the manual pages bash(1), xdm(1x), xdm.options(5), and
wdm(1x). Recommended further reading is everything related with shell programming.
If you need a similar mechanism for executing code at logout time check out the related package syslogout(8) which is a very close compan-
ion to sysprofile.
BUGS
sysprofile in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack
than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better
becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we
take patches... ;-)
AUTHOR
sysprofile was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use
it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into
something more worthwhile than it currently is.
SYSPROFILE(8)