Sponsored Content
Top Forums Shell Programming and Scripting Daily script taking increasingly longer each day Post 302993012 by Threeze on Sunday 5th of March 2017 11:14:42 AM
Old 03-05-2017
Network Daily script taking increasingly longer each day

Hello,

I was wondering if anyone had an idea why a recurring script might take slightly longer each day.

The intent of the script is to run on a dedicated VM and connect to several thousand remote servers to assay their operating status by running a series of commands and storing the results locally. This script runs once daily.

On the VM, the script is run in parallel (using GNU parallel) to increase the rate that the target servers are checked. The VM runs as many parallel connections as it can handle (maxed out resources) which equates to roughly 100 connections at any given time.
The first part of the script initiates an SSH connection and passes about 30 commands to the target server (basic commands such as top, cat ..., grep ..., etc). Results are stored in a local file.
The second part of the script initiates an SCP request and copies zero to many (max 30) gzip files to the local server based on their filename timestamp -- these files represent logs of the activity on the servers, where more active servers have larger/more gzipped log files.

I've set a limit in GNU parallel of 140 seconds. If steps 1 and 2 combined take longer than 140 seconds then end the job and move on. This number is high as the process shouldn't take longer than 100 seconds.

So far I've covered how the data is pulled. The data is stored in many txt files on the local VM, each named with the ip address and date of creation -- this is necessary for later aggregation and analysis by our monitoring tool. I've also had to hash the files into directories based on their IP structure.. i.e.:
ip 1.2.3.4 would be in folder 1.2/1.2.3/1.2.3.4_sshresult_date.txt
This was done because the monitoring/aggregation tool had a difficult time reading from a single folder with such a large number of files. Once our monitoring tool reads the .txt files it also deletes them to prepare for the next day's run.

This is a big issue for me as when this process was first put in, it took 5 hours to complete. In the past 2 months that number has climbed to 8.5 hours, with no changes in the script. I've added some extra logging to the SSH and SCP components of the script and each day I can see that the average SSH time increases by 0.5 to 1 second. The same goes for the SCP execution.


What I've tried:
1) Reinitializing the monitoring/aggregating tool (which sits activtely on the VM as well) in case it was causing a memory leak or file locking issues.
2) Rebooting the server occasionally to clear memory in case there existed a general memory leak from any source.

Some possible explanations that I've thought of or have been suggested to me:
1) inodes on the local VM may be out of wack due to the large number of files created and deleted each day. I've never had to deal with inode management so I'm not sure if this is plausible, or how to deal with it if it is the issue.
2) Something I thought was perhaps the process of connecting each day by SSH and SCP to a remote server is producing some effect on that remote server which is actually causing it to respond more slowly each day -- this would be worst case scenario. I don't see how initiating an SSH&SCP session once each per day could affect the target server, but could this be possible assuming the target server could be considered 'old/fragile'?

Sorry for the long post, I just wanted to make sure I gave enough detail. Please let me know if you have any questions and I'll do my best to answer.

Thanks for any help provided! I'm relatively new to large scale bash scripting and I want this thing to run efficiently.
 

9 More Discussions You Might Find Interesting

1. SCO

Telnet connection to Sco Unixware from Windows 2000 taking longer !!!

Telnet connection to SCO Unixware server taking longer to return the Login prompt !!! Thanx in advance for any suggestions or help in this regard . Cheers KBiswas (9 Replies)
Discussion started by: kbiswas
9 Replies

2. UNIX for Dummies Questions & Answers

Need a script to do daily backups

So I have a set of directories and files that I need to backup from /directory1/ to /directory2/ each night. I have some UNIX/SSH knowledge but don't assume I know a whole lot b/c I would hate to screw something up. Here's the knowledge I have: I can access my server via SSH and can navigate to... (10 Replies)
Discussion started by: JPigford
10 Replies

3. Shell Programming and Scripting

Write a shell script to find whether the first day of the month is a working day

Hi , I am relatively new to unix... Can u pls help me out to find out if the first day of the month is a working day ie from (Monday to Friday)...using Date and If clause in Korn shell.. This is very urgent. Thanks for ur help... (7 Replies)
Discussion started by: phani
7 Replies

4. Shell Programming and Scripting

Script to find previous month last day minus one day timestamp

Hi All, I need to find the previous month last day minus one day, using shell script. Can you guys help me to do this. My Requirment is as below: Input for me will be 2000909(YYYYMM) I need the previous months last day minus 1 day timestamp. That is i need 2000908 months last day minus ... (3 Replies)
Discussion started by: girish.raos
3 Replies

5. UNIX for Dummies Questions & Answers

Run a .sh script daily

Hi, I juat wondering how can you set it up so that .sh files will execute automatically once a day. from google I've got use crontab but when I type this into my session it say I am not allowed to use this programme. Any other ways to achieve what I'm looking for? thanks (1 Reply)
Discussion started by: blackieconnors
1 Replies

6. Shell Programming and Scripting

Script to check if last modified day is previous day

Hi, I would like to write a script that checks if a file ('counter') was modified the previous day, if so erase its contents and write 00000000 into it. For e.g. if the file 'counter' was last modified at 11.30pm on 24th May and the script runs at 12.15am of 25th May, it should erase it's... (1 Reply)
Discussion started by: hegdepras
1 Replies

7. Shell Programming and Scripting

Script for daily use

I have a clear case command for example. ct lsprivate -co this displays the list of checked out files. and i have many views where i work daily I need a script which can run daily at our specified time. setting each and every view i have and list the check outs i have in them. and consolidate... (10 Replies)
Discussion started by: Syed Imran
10 Replies

8. Solaris

Copy via samba on vmware workstation with Solaris taking much longer than usual

i have a vmware workstation with solaris 10 installed on this. i copying 2 gb data via samba from windows to this vmware workstation. copy speed is 24 kb/sec. how i can speed up this copy process ? (7 Replies)
Discussion started by: rehantayyab82
7 Replies

9. UNIX for Dummies Questions & Answers

Script is not longer working in the crontab

This is the crontab it is supossed to be running everyday but it didnt 5 0 * * * /export/app/CO/opge/scr/Informe_parametros_colombia.ksh >/dev/null 2>&1 Inside the above script connects to a database and extract data to a flat file, manually i run the script at about 2 a.m. and Works OK,... (6 Replies)
Discussion started by: alexcol
6 Replies
ns_sched(3aolserver)					    AOLserver Built-In Commands 				      ns_sched(3aolserver)

__________________________________________________________________________________________________________________________________________________

NAME
ns_after, ns_cancel, ns_pause, ns_resume, ns_schedule_daily, ns_schedule_proc, ns_schedule_weekly, ns_unschedule_proc - commands SYNOPSIS
ns_after seconds {script | procname ?args?} ns_cancel id ns_pause id ns_resume id ns_schedule_daily ?-thread? ?-once? hour minute {script | procname ?args?} ns_schedule_proc ?-thread? ?-once? interval {script | procname ?args?} ns_schedule_weekly ?-thread? ?-once? day hour minute {script | procname ?args?} ns_unschedule_proc id _________________________________________________________________ DESCRIPTION
ns_after run the specified script or procedure after the specified number of seconds ns_after returns an id which can be used with the ns_pause, ns_cancel and ns_resume apis. ns_cancel stops the scheduled running of the id returned by an ns_after returns 1 if unscheduled 0 if the script of procedure couldn't be unscheduled ns_pause pauses the scheduled running of the id returned by an ns_after returns 1 if paused, 0 if the script of procedure couldn't be paused ns_resume resumes the scheduled running of the id returned by an ns_after returns 1 if resumed, 0 if the script of procedure couldn't be resumed ns_schedule_daily ns_schedule_daily runs the specified Tcl script or procedure (procname) once a day at the time specified by hour and minute. The hour can be from 0 to 23, and the minute can be from 0 to 59. Specify -thread if you want a thread created to run the procedure. This will allow the scheduler to continue with other scheduled procedures. Specifying -thread is appropriate in situations where the script will not return immediately, such as when the script performs network activity. Specify -once if you want the script to run only one time. The default is that the script will be re-scheduled after each time it is run. ns_schedule_daily returns an id number for the scheduled procedure that is needed to stop the scheduled procedure with ns_unsched- ule_proc. ns_schedule_proc ns_schedule_proc runs the specified Tcl script or procedure (procname) at an interval specified by interval. The interval is the number of seconds between runs of the script. Specify -thread if you want a thread created to run the procedure. This will allow the scheduler to continue with other scheduled procedures. Specifying -thread is appropriate in situations where the script will not return immediately, such as when the script performs network activity. Specify -once if you want the script to run only one time. The default is that the script will be re-scheduled after each time it is run. ns_schedule_proc returns an id number for the scheduled procedure that is needed to stop the scheduled procedure with ns_unsched- ule_proc. ns_schedule_weekly ns_schedule_weekly runs the specified Tcl script or procedure (procname) once a week on the day specified by day and the time speci- fied by hour and minute. The day can be from 0 to 6, where 0 represents Sunday. The hour can be from 0 to 23, and the minute can be from 0 to 59. Specify -thread if you want a thread created to run the procedure. This will allow the scheduler to continue with other scheduled procedures. Specifying -thread is appropriate in situations where the script will not return immediately, such as when the script performs network activity. Specify -once if you want the script to run only one time. The default is that the script will be re-scheduled after each time it is run. ns_schedule_weekly returns an id number for the scheduled procedure that is needed to stop the scheduled procedure with ns_unsched- ule_proc. ns_unschedule_proc id ns_unschedule_proc stops a scheduled procedure from executing anymore. The scheduled procedure to be stopped is identified by its id, which was returned by the ns_schedule* function that was used to schedule the procedure. EXAMPLES
ns_after ns_cancel ns_pause ns_resume This example illustrates a web interface used to manage jobs. Depending on the action provided a job can be created, cancelled, paused or resumed. set action [ns_queryget action] set job [ns_queryget job] switch $action { create { set job [ns_after 10 [ns_queryget script]] ns_puts "Job created with id: $job" } cancel { if {[ns_cancel $job]} { ns_puts "Job $job cancelled" } else { ns_puts "Job $job not cancelled" } } pause { if {[ns_pause $job]} { ns_puts "Job $job paused" } else { ns_puts "Job $job not paused } } resume { if {[ns_resume $job]} { ns_puts "Job $job resumed" } else { ns_puts "Job $job couldn't be resumed" } } default { ns_puts "Invalid action $action" } } ns_schedule_daily This example defines a script called rolllog that uses ns_accesslog to roll the access log to a file with an extension containing the current date. The ns_schedule_daily function is used to execute the rolllog script on a daily basis. # Script to roll and rcp log file to host "grinder" proc rolllog {} { set suffix [ns_strftime "%y-%m-%d"] set new [ns_accesslog file].$suffix ns_accesslog roll $new exec rcp $new grinder:/logs/[file tail $new] } # Schedule "rolllog" to run at 3:30 am each morning ns_schedule_daily -thread 3 30 rolllog ns_schedule_proc proc dosomething blah { ns_log Notice "proc with arg '$blah'" } ns_schedule_proc 10 dosomething $arg1 SEE ALSO
KEYWORDS
schedule pause resume unschedule cancel after AOLserver 4.0 ns_sched(3aolserver)
All times are GMT -4. The time now is 05:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy