Efficient UNIX Memory management for Running MapReduce Jobs. Post: 302960718

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Efficient UNIX Memory management for Running MapReduce Jobs. Post 302960718 by ketankirange on Thursday 19th of November 2015 02:13:06 AM

11-19-2015

Registered User

Code:

#-> free -m
             total       used       free     shared    buffers     cached
Mem:         15951      15215        736          0        452       7288
-/+ buffers/cache:       7474       8476
Swap:         4027          0       4027

top command output is here

Code:

#-> top
top - 02:07:52 up 1 day, 12:05,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 210 total,   1 running, 209 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.4%us,  2.9%sy,  0.0%ni, 85.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16334648k total, 15580888k used,   753760k free,   463220k buffers
Swap:  4124664k total,        0k used,  4124664k free,  7462940k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16435 root      20   0 2618m  54m 4816 S 31.9  0.3  25:35.09 python
 8578 root       0 -20  102m  10m 2020 S  4.3  0.1  20:01.54 perfd
13209 cloudera  20   0 2912m 604m  83m S  4.3  3.8  53:26.85 java
13289 cloudera  20   0 2699m 398m  19m S  3.3  2.5  22:47.03 java
13307 cloudera  20   0 2724m 460m  34m S  2.7  2.9  14:55.97 java
16014 yarn      20   0 1095m 255m  19m S  2.3  1.6  18:11.50 java
19155 hdfs      20   0  969m 259m  19m S  1.7  1.6   8:18.92 java
15695 mapred    20   0  927m 240m  19m S  1.3  1.5   4:51.68 java
16798 oozie     20   0 2687m 379m  22m S  1.3  2.4  22:13.72 java
15793 yarn      20   0  976m 247m  20m S  1.0  1.6   5:31.36 java
19297 hdfs      20   0  898m 213m  19m S  1.0  1.3   3:23.79 java
22860 cloudera  20   0 4885m 1.6g  22m S  1.0 10.0  31:58.05 java
  925 root      20   0 20644 1500 1100 R  0.7  0.0   0:00.07 top
19217 hdfs      20   0  952m 225m  19m S  0.7  1.4   2:49.87 java
 8474 root       0 -20 39240 6496 2492 S  0.3  0.0   7:57.41 scopeux
 8735 root      20   0 1775m  12m 6464 S  0.3  0.1   0:42.67 ovcd
 9442 root      20   0  625m  14m 9300 S  0.3  0.1   0:22.67 coda
13240 cloudera  20   0 2285m 195m  18m S  0.3  1.2   3:12.26 java

cAn you please suggest what can i do to tune this

Last edited by Don Cragun; 11-19-2015 at 03:16 AM.. Reason: Add CODE tags.

ketankirange

View Public Profile for ketankirange

Find all posts by ketankirange

5 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

exit unix, but says running jobs

hi, i give a exit to the system, but it says that i have running jobs... so i do a ps and it displays two lines, one is a -ksh and the other is the ps which i am issuing... then i give a who -uH, find my pts.. then do a grep... still the same..... whats wrong..

2. UNIX for Advanced & Expert Users

unix memory management

i am looking for the books or web-sites which explains the unix memory management in detail. do you know any useful material?

3. UNIX for Dummies Questions & Answers

Unix Batch command, and running jobs in queues.

Hello all, I have a quick question. I work in a computational science laboratory, and we recently got a few mac pros to do molecular optimizations on. However, on our normal supercomputers, there are queue systems, mainly PBS. Anyway, the macs obviously don't have PBS, but I've read about...

4. UNIX for Dummies Questions & Answers

at jobs not running

im doing at now <return> touch ~/Desktop/file.txt <ctrl d> and file is never created though it says job 7 at Wed Mar 17 15:15:55 2010 why?

5. Shell Programming and Scripting

Memory management

Hello all. I have a script that uses two arrays in the beginning. Saves certain values that i am extracting from df -h command. array1 and array2 where i is from 0 to 9. It then goes on and saves the values of the arrays into variables. for i 0 to 9 , tmp= array2 // I am no writing the...

LEARN ABOUT OPENSOLARIS

scsetup

scsetup(1M)						  System Administration Commands					       scsetup(1M)

NAME

       scsetup - interactive cluster configuration tool

SYNOPSIS

       scsetup [-f logfile]

DESCRIPTION

       Note -

	 Beginning  with  the Sun Cluster 3.2 release, Sun Cluster software includes an object-oriented command set. Although Sun Cluster software
	 still supports the original command set, Sun Cluster procedural documentation uses only the object-oriented command set. For more  infor-
	 mation about the object-oriented command set, see the Intro(1CL) man page.

       The  scsetup  command  provides the following configuration capabilities, depending on what state the cluster is in when you issue the com-
       mand:

	   o	  When you run the scsetup command at post-installation time, the command performs initial setup tasks, such as configuring quorum
		  devices  and resetting the installmode property. If you did not use automatic quorum configuration when you created the cluster,
		  run the scsetup command immediately after the cluster is installed. Ensure that all nodes have joined the cluster before you run
		  the scsetup command and reset the installmode property.

		  If  you used automatic quorum configuration when you created the cluster, you do not need to run the scsetup command after clus-
		  ter installation. The automatic quorum configuration feature also resets the installmode property of the cluster.

	   o	  When you run the command during normal cluster operation, the scsetup command provides a menu-driven utility. You can  use  this
		  utility to perform most ongoing cluster-administration tasks.

	   o	  When you issue the command from a node that is in noncluster mode, the scsetup utility provides a menu-driven utility for chang-
		  ing and displaying the private IP address range. You must reboot all nodes into noncluster mode before you start  this  form	of
		  the scsetup utility.

       You can issue the scsetup command from any node in the cluster.

       You can use this command only in the global zone.

OPTIONS

       The following options are supported:

       -f logfile	   Specifies  the  name  of a log file to which commands can be logged. If you specify this option, most command sets that
			   the scsetup utility generates are run and logged, or only logged, depending on user responses.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes.

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWsczu			   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Evolving			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       Intro(1CL), cltelemetryattribute(1CL), cldevicegroup(1CL),  clnode(1CL),  clquorum(1CL),  clreslogicalhostname(1CL),  clresourcegroup(1CL),
       clresourcetype(1CL), clressharedaddress(1CL), cluster(1CL),

Sun Cluster 3.2 						    18 Jul 2006 						       scsetup(1M)