Sponsored Content
Top Forums Shell Programming and Scripting My script seems to eat too much memory Post 302962915 by anominous on Sunday 20th of December 2015 10:34:54 PM
Old 12-20-2015
I believe I fixed the memory problems now. I grab PIDs of background processes, so I can wait if too many of them are running. A https cURL needs like 7 MB on my machine. When hundreds of them are running, they can consume gigabytes.

Before the background job code:
Code:
# wait if too many crawl jobs running
if [ ${#crawl_jobs[@]} -ge $max_crawl_jobs ]; then
  wait "${crawl_jobs[$crawl_index]}"
  unset -v crawl_jobs[$((crawl_index++))]
fi

# wait if too many download processes
if [ $max_downloads -gt 0 ]; then
  while [ $(ps -fu $USER | grep curl.*4cdn | wc -l) -ge $max_downloads ]; do
    sleep 1
  done
fi

After the background job code:
Code:
# save current crawl job's process ID in an array
crawl_jobs+=("$!")

It's a pity that, seemingly, we have a wait command but no tool to count all background jobs.

Your debug code seems like a good idea. I will have a closer look into this, thanks!
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

bash/perl script that will use up all memory

Hi, Is there any known trick that will consume all of my memory and force my machine to swap? I'm scripting a monitor in perl that will alert me when available memory is dangerously low and system is swapping, and I would like to simulate the scenario. I am clueless how to make my system lack... (4 Replies)
Discussion started by: marcpascual
4 Replies

2. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

3. Shell Programming and Scripting

memory leak in igawk script

Hi, We are having igawk statement calling gawk function and processing a master file, each gawk function have rules and action. We are using associative array to keep the output of each rule and finally printing the associative array. We are seeing the memory is growing incrementally in TOP... (1 Reply)
Discussion started by: satvelucbe
1 Replies

4. Shell Programming and Scripting

Memory check script

Hi all, OS: Solaris 10 I'm trying to put together a script to check memory utilisation. The command I'm running is: ps -ef -o pmem -o pid -o rss -o vsz -o args -o user | grep 2 | grep -v "VSZ COMMAND" | sort | tail -10 | sort -r And the output looks something like this: ... (9 Replies)
Discussion started by: d-train
9 Replies

5. Shell Programming and Scripting

Memory limit on php in .sh script

Hi, I have .sh script running php script with "php". When I run php script from web browser, it is running without errors. When I run it from .sh script, I am getting "memory exhausted". It seems to me that running php from .sh does not respect php.ini or have its own setting. So, how can I set... (0 Replies)
Discussion started by: LukasB
0 Replies

6. AIX

Script to monitor CPU , Memory

Hello. I am using AIX 6 and did lot of searches on google for this script and found so many results. Just thought to ask you being the real good place for centralised answer: a) Could you tell me the aix script, which may tell me If the CPU load is above e.g. 60% ? b) Could you tell me the AIX... (2 Replies)
Discussion started by: panchpan
2 Replies

7. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

8. Shell Programming and Scripting

Help with implementing available memory status script

hi , i want write the script which automatically send an alert mail to my mail id when there is low memory available. things which i am able to implement -: i got the output of current memory status into one file . Than i break down the required coloumn and again send it in another file. My... (1 Reply)
Discussion started by: abhinav dixit
1 Replies

9. Shell Programming and Scripting

Shell script for %computational memory & % non computational memory

Dear, How to calculate %computational memory and %non computational memory from AIX server. What command used to find out %computational memory and % non computational memory except topas. Regards Nowshath (1 Reply)
Discussion started by: Nowshath
1 Replies
wait(1)                                                            User Commands                                                           wait(1)

NAME
wait - await process completion SYNOPSIS
/bin/sh wait [pid...] /bin/jsh /bin/ksh /usr/xpg4/bin/sh wait [pid...] wait [ % jobid...] /bin/csh wait DESCRIPTION
The shell itself executes wait, without creating a new process. If you get the error message cannot fork,too many processes, try using the wait command to clean up your background processes. If this doesn't help, the system process table is probably full or you have too many active foreground processes. There is a limit to the number of process IDs associated with your login, and to the number the system can keep track of. Not all the processes of a pipeline with three or more stages are children of the shell, and thus cannot be waited for. /bin/sh, /bin/jsh Wait for your background process whose process ID is pid and report its termination status. If pid is omitted, all your shell's currently active background processes are waited for and the return code will be 0. The wait utility accepts a job identifier, when Job Control is enabled (jsh), and the argument, jobid, is preceded by a percent sign (%). If pid is not an active process ID, the wait utility will return immediately and the return code will be 0. csh Wait for your background processes. ksh When an asynchronous list is started by the shell, the process ID of the last command in each element of the asynchronous list becomes known in the current shell execution environment. If the wait utility is invoked with no operands, it will wait until all process IDs known to the invoking shell have terminated and exit with an exit status of 0. If one or more pid or jobid operands are specified that represent known process IDs (or jobids), the wait utility will wait until all of them have terminated. If one or more pid or jobid operands are specified that represent unknown process IDs (or jobids), wait will treat them as if they were known process IDs (or jobids) that exited with exit status 127. The exit status returned by the wait utility will be the exit status of the process requested by the last pid or jobid operand. The known process IDs are applicable only for invocations of wait in the current shell execution environment. OPERANDS
The following operands are supported: One of the following: pid The unsigned decimal integer process ID of a command, for which the utility is to wait for the termination. jobid A job control job ID that identifies a background process group to be waited for. The job control job ID notation is applicable only for invocations of wait in the current shell execution environment, and only on systems supporting the job control option. USAGE
On most implementations, wait is a shell built-in. If it is called in a subshell or separate utility execution environment, such as one of the following, (wait) nohup wait ... find . -exec wait ... ; it will return immediately because there will be no known process IDs to wait for in those environments. EXAMPLES
Example 1: Using A Script To Identify The Termination Signal Although the exact value used when a process is terminated by a signal is unspecified, if it is known that a signal terminated a process, a script can still reliably figure out which signal is using kill, as shown by the following (/bin/ksh and /usr/xpg4/bin/sh): sleep 1000& pid=$! kill -kill $pid wait $pid echo $pid was terminated by a SIG$(kill -l $(($?-128))) signal. Example 2: Returning The Exit Status Of A Process If the following sequence of commands is run in less than 31 seconds (/bin/ksh and /usr/xpg4/bin/sh): sleep 257 | sleep 31 & jobs -l %% then either of the following commands will return the exit status of the second sleep in the pipeline: wait <pid of sleep 31> wait %% ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of wait: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
csh(1), jobs(1), ksh(1), sh(1), attributes(5), environ(5), standards(5) SunOS 5.10 12 Dec 1997 wait(1)
All times are GMT -4. The time now is 04:00 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy