Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: My script seems to eat too much memory
Top Forums Shell Programming and Scripting My script seems to eat too much memory Post 302962915 by anominous on Sunday 20th of December 2015 10:34:54 PM
Old 12-20-2015
I believe I fixed the memory problems now. I grab PIDs of background processes, so I can wait if too many of them are running. A https cURL needs like 7 MB on my machine. When hundreds of them are running, they can consume gigabytes.

Before the background job code:
Code:
# wait if too many crawl jobs running
if [ ${#crawl_jobs[@]} -ge $max_crawl_jobs ]; then
  wait "${crawl_jobs[$crawl_index]}"
  unset -v crawl_jobs[$((crawl_index++))]
fi

# wait if too many download processes
if [ $max_downloads -gt 0 ]; then
  while [ $(ps -fu $USER | grep curl.*4cdn | wc -l) -ge $max_downloads ]; do
    sleep 1
  done
fi

After the background job code:
Code:
# save current crawl job's process ID in an array
crawl_jobs+=("$!")

It's a pity that, seemingly, we have a wait command but no tool to count all background jobs.

Your debug code seems like a good idea. I will have a closer look into this, thanks!
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

bash/perl script that will use up all memory

Hi, Is there any known trick that will consume all of my memory and force my machine to swap? I'm scripting a monitor in perl that will alert me when available memory is dangerously low and system is swapping, and I would like to simulate the scenario. I am clueless how to make my system lack... (4 Replies)
Discussion started by: marcpascual
4 Replies

2. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

3. Shell Programming and Scripting

memory leak in igawk script

Hi, We are having igawk statement calling gawk function and processing a master file, each gawk function have rules and action. We are using associative array to keep the output of each rule and finally printing the associative array. We are seeing the memory is growing incrementally in TOP... (1 Reply)
Discussion started by: satvelucbe
1 Replies

4. Shell Programming and Scripting

Memory check script

Hi all, OS: Solaris 10 I'm trying to put together a script to check memory utilisation. The command I'm running is: ps -ef -o pmem -o pid -o rss -o vsz -o args -o user | grep 2 | grep -v "VSZ COMMAND" | sort | tail -10 | sort -r And the output looks something like this: ... (9 Replies)
Discussion started by: d-train
9 Replies

5. Shell Programming and Scripting

Memory limit on php in .sh script

Hi, I have .sh script running php script with "php". When I run php script from web browser, it is running without errors. When I run it from .sh script, I am getting "memory exhausted". It seems to me that running php from .sh does not respect php.ini or have its own setting. So, how can I set... (0 Replies)
Discussion started by: LukasB
0 Replies

6. AIX

Script to monitor CPU , Memory

Hello. I am using AIX 6 and did lot of searches on google for this script and found so many results. Just thought to ask you being the real good place for centralised answer: a) Could you tell me the aix script, which may tell me If the CPU load is above e.g. 60% ? b) Could you tell me the AIX... (2 Replies)
Discussion started by: panchpan
2 Replies

7. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

8. Shell Programming and Scripting

Help with implementing available memory status script

hi , i want write the script which automatically send an alert mail to my mail id when there is low memory available. things which i am able to implement -: i got the output of current memory status into one file . Than i break down the required coloumn and again send it in another file. My... (1 Reply)
Discussion started by: abhinav dixit
1 Replies

9. Shell Programming and Scripting

Shell script for %computational memory & % non computational memory

Dear, How to calculate %computational memory and %non computational memory from AIX server. What command used to find out %computational memory and % non computational memory except topas. Regards Nowshath (1 Reply)
Discussion started by: Nowshath
1 Replies

LEARN ABOUT DEBIAN

queuedefs

queuedefs(4)							   File Formats 						      queuedefs(4)

NAME

       queuedefs - queue description file for at, batch, and cron

SYNOPSIS

       /etc/cron.d/queuedefs

DESCRIPTION

       The queuedefs file describes the characteristics of the queues managed by cron(1M). Each non-comment line in this file describes one queue.
       The format of the lines are as follows:

	      q.[njobj][nicen][nwaitw]

       The fields in this line are:

       q	The name of the queue. a is the default queue for jobs started by at(1); b is the default queue for jobs  started  by  batch  (see
		at(1)); c is the default queue for jobs run from a crontab(1) file.

       njob	The  maximum  number of jobs that can be run simultaneously in that queue; if more than njob jobs are ready to run, only the first
		njob jobs will be run, and the others will be run as jobs that are currently running terminate.  The default value is  100.

       nice	The nice(1) value to give to all jobs in that queue that are not run with a user ID of super-user.  The default value is 2.

       nwait	The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in  that  job's
		queue, or because the system-wide limit of jobs executing has been reached.  The default value is 60.

       Lines beginning with # are comments, and are ignored.

EXAMPLES

       Example 1: A sample file.

       #
       #
       a.4j1n
       b.2j2n90w

       This  file  specifies that the a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value
       of 1.  As no nwait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds before  trying
       again to run it.

       The  b  queue, for batch(1) jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2.  If a job
       cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it. All	other  queues  can
       have up to 100 jobs running simultaneously; they will be run with a nice value of 2, and if a job cannot be run because too many other jobs
       are running cron will wait 60 seconds before trying again to run it.

FILES

       /etc/cron.d/queuedefs	       queue description file for  at, batch, and  cron.

SEE ALSO

       at(1), crontab(1), nice(1), cron(1M)

SunOS 5.10							    1 Mar 1994							      queuedefs(4)
All times are GMT -4. The time now is 02:40 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy