Efficient UNIX Memory management for Running MapReduce Jobs. Post: 302960573

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Efficient UNIX Memory management for Running MapReduce Jobs. Post 302960573 by ketankirange on Tuesday 17th of November 2015 05:15:06 AM

11-17-2015

Registered User

Efficient UNIX Memory management for Running MapReduce Jobs.

We are trying to set up a single node cloudera hadoop cluster having 16 GB as RAM on linux machine. We are setting up 5.4.2 version.

Now when we check statistics post the installation and run the top command we find that only 1 -2 GB is available. when we trigger map reduce sample job - no memory is allocated to the job and so job doesnt run.

Can you please let us know what should we do so that more memory is available

Analysis of the top command

cloudera takes 4-5 GB.

mysql 6 GB.(external database to store the metastore) other services 2-3 GB. thus contributing to 13 GB out of 16

ketankirange

View Public Profile for ketankirange

Find all posts by ketankirange

5 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

exit unix, but says running jobs

hi, i give a exit to the system, but it says that i have running jobs... so i do a ps and it displays two lines, one is a -ksh and the other is the ps which i am issuing... then i give a who -uH, find my pts.. then do a grep... still the same..... whats wrong..

2. UNIX for Advanced & Expert Users

unix memory management

i am looking for the books or web-sites which explains the unix memory management in detail. do you know any useful material?

3. UNIX for Dummies Questions & Answers

Unix Batch command, and running jobs in queues.

Hello all, I have a quick question. I work in a computational science laboratory, and we recently got a few mac pros to do molecular optimizations on. However, on our normal supercomputers, there are queue systems, mainly PBS. Anyway, the macs obviously don't have PBS, but I've read about...

4. UNIX for Dummies Questions & Answers

at jobs not running

im doing at now <return> touch ~/Desktop/file.txt <ctrl d> and file is never created though it says job 7 at Wed Mar 17 15:15:55 2010 why?

5. Shell Programming and Scripting

Memory management

Hello all. I have a script that uses two arrays in the beginning. Saves certain values that i am extracting from df -h command. array1 and array2 where i is from 0 to 9. It then goes on and saves the values of the arrays into variables. for i 0 to 9 , tmp= array2 // I am no writing the...

LEARN ABOUT OPENDARWIN

qrun

qrun(8B)								PBS								  qrun(8B)

NAME

       qrun - run a pbs batch job

SYNOPSIS

       qrun [-a] [-H host] job_identifier ...

DESCRIPTION

       The  qrun  command is used to force a batch server to initiate the execution of a batch job.  The job is run regardless of scheduling posi-
       tion, resource requirements, or state.

       In order to execute qrun, the user must have PBS Operation or Manager privilege.

OPTIONS

       -a	      Specifies the job(s) is to be run asynchronously.

       -H host	      Specifies the host within the cluster on which the job(s) are to be run.	The host argument is the name of a host that is  a
		      member  of  the  cluster	of hosts managed by the server.  If the option is not specified, the server will select the "worst
		      possible" host on which to execute the job.

OPERANDS

       The qrun command accepts one or more job_identifier operands of the form:
	   sequence_number[.server_name][@server]

STANDARD ERROR

       The qrun command will write a diagnostic message to standard error for each error occurrence.

EXIT STATUS

       Upon successful processing of all the operands presented to the qrun command, the exit status will be a value of zero.

       If the qrun command fails to process any operand, the command exits with a value greater than zero.

SEE ALSO

       pbs_server(8B), pbs_sched(8B), qmgr(1B)

Local																	  qrun(8B)