Visit Our UNIX and Linux User Community


memory free up using 'find'


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users memory free up using 'find'
# 1  
Old 10-31-2002
memory free up using 'find'

Hi,

I am facing an interesting aspect of find command...

to be clear, we are running a small web server with oracle 8i database and Oralce9iAS on Sun E250 with Solaris 2.6

Over a period of time, the free memory ( displayed in 'top' utility ) drops down.. we could relate this to dedicated oracle client connections, as not releasing the allocated memory..

by fluke, one of the team member ran the find command and thats it.. the free memory ( displayed in 'top' utility ) starts growing... We could get nearly 100Mb with that simple find command..

Can somebody throw some light on this ??

In other words, is it something being flushed from kernel buffer ?

Thanks in Advance..
# 2  
Old 10-31-2002
Solaris tries to cache stuff in memory on the off chance that you might someday. If you need memory it just discards that stuff and hands you the memory. find needs memory. It is recursive. I suppose its memory requirements could grow and shrink as it runs. If your page-outs are zero, you have enough memory. The stat "free memory" meant more in the old days.
# 3  
Old 10-31-2002
Thanks.. it was really useful.
# 4  
Old 11-01-2002
Well, I'm glad you found it useful... unfortunately, I was wrong. I am rereading the memory management section of Solaris Internals Core kernel Architecture and more is sinking in this time. I will need more time to finish my reading and I will post again on Saturday.

But I know enough now to say that page-outs also no longer have their original meaning. You can be in memory trouble on Solaris with no page-outs.
# 5  
Old 11-02-2002
OK, I think I got it now. I'm going to list every source that I used to research this. For more info you may want to read them yourself.

Originally, Unix used a structure called a buffer cache for all file i/o (Bach, 1986 38-58) (Leffler, et al, 1989, 208-213). HP-UX continues to use a buffer cache (Sauers and Weygant, 2000 189-190) but it can dynamically change in size. This is the sort of Unix system that I'm used to. On this kind of Unix system, a page-out is almost always due to paging activity causing by a shortage of free memory. The only exception that I know of is that memory mapped files do all i/o via the paging subsystem (Vahalia, 1996 438-440) and a program can invoke msync() to force the data to disk (Stevens, 1992, 411).

I have always used page-outs to see if a system is paging. No problem there, that's practically the definition of "paging". It may be that I really should be looking at "scan rate". There are two reasons for this: first, when the scanner first starts to run, if it is running a the slowest possible speed, it may take longer that one minute for the first page to be stolen is identified (Mauro and McDougall, 2001 180-182). Also, if the page has not been modified since it was paged-in, it will not be paged-out. Instead it is simply placed on the free list (Bach 1986, 296) (Leffler, et al, 1989, 154). So for a traditional unix system, scan-rate may be the more reliable metric. The only time that you are scanning pages is if you are low on memory.

At some point a new way of doing things emerged. The earliest reference to it that I can find is System V Release 4 (Goodheart and Cox, 1994, 283-287). Solaris has migrated to it (Mauro and McDougall, 167-190, 589-599). Linux also uses this model (Mosberger and Eranian, 2002, 23-25).

With the new way, the buffer cache has shrunk in size. It is only used for metadata. The data from files is stored in pages. All of memory becomes a "buffer cache". Sync operations cause page outs. After memory is mostly full, we need the page stealer to free up pages so new data can be read.

With the new way it is normal to be out of memory. Mauro and McDougall say (page 171) "The free list is generally very small, since most pages that are no longer used by a process or the kernel still keep their vnode/offset information intact." And on page 594: "Have you ever noticed that when you boot your machine there is a lot of free memory, and as the machine is used, memory continues to fall to zero, then just hangs there? Here is why: the file system is using all available memory to cache the reads and writes to each file--and it's completely normal."

This is not say that everything is cool. Sun is having problems with the new approach. On page 596: "If your system seems slow while file I/O is going on, it's because your applications are being paged in and out as a direct result of the file system activity." They go to say that Sun invented a kludge to help this situation called "priority paging" in Solaris 7. You can run this in 2.6 if you get patch 105181-3. And for 2.5.1, you need patch 103640-25. Another book (Musumeci and Loukides, 2002) says on page 116 " The effect is generally excellent. Desktop systems and on-line transaction processing environments tend to feel much more responsive...as much as a 300% performance increase."

I'm just scratching the surface here, but I'm not going to keep typing. Buy some of the books I listed below. And if you're running Solaris, look up "priority paging" on Sun's website.

(Bach 1986) The Design of the Unix Operating System by Maurice J. Bach

(Leffler, et. al., 1989) The Design and Implementation of the 4.3BSD Unix Operating System by Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels, and John S. Quarterman

(Sauers and Weygant, 2000) HP-UX Tuning and Performance: Concepts, Tools, and Methods by Robert F. Sauers and Peter S. Weygant

(Vahalia, 1996) Unix Internals, The New Frontiers by Uresh Vahalia

(Stevens, 1992) Advanced Programming in the Unix Environment by W. Richard Stevens

(Goodheart and Cox, 1994) The Magic Garden Explained: The Internals of Unix System V Release 4, An Open Systems Design by Berny Goodheart and James Cox

(Mauro and McDougall, 2001) Solaris Internals Core Kernel Architecture by Jim Mauro and Richard McDougall

(Mossberg and Eranian, 2002) IA-64 Linux Kernel Design and Implementation by David Mosberger and Stephane Eranian

(Musumeci and Loukides, 2002) System Performance Tuning, 2nd Edition by Gian-Paolo D. Musumeci and Mike Loukides
# 6  
Old 11-03-2002
Its like I always say.

Ask a question, get a research paper!

Thanks for the great explanation Perderabo! Smilie

Smilie
# 7  
Old 11-03-2002
Here are some links that may be handy:

Performance and Tuning on Solaris 2.6, 7 and 8
This mentions priority paging and has links to patches needed to run priority paging on 2.5.1 and 2.6.

Priority Paging - Frequently Asked Questions
More info on priority paging and a link to download the memstat utility which can display the additional memory statistics added to the kernel in Solaris 7. Starting with Solaris 8, "vmstat -p" will display these stats.

Priority Paging Is Not Needed With the New Solaris 8 Caching Architecture
So don't do that. Note that with Solaris 8, free memory will mean free memory again as well.

Solaris Internals
Home page for the Solaris Internals book. Several downloads are available here. Click on "tools" then "MemTool" to start the process of obtaining MemTool.

Also I have found that on 2.6, "/usr/proc/bin/pmap -x" works even though it's undocumented. Starting with Solaris 7, it's documented.

Previous Thread | Next Thread
Test Your Knowledge in Computers #995
Difficulty: Medium
IBM announced its System/370 series in 1970 with full virtualization features.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

What are the differences between 'bash' and 'sh'

Hopefully this doesn't come off as too much of a "newbie" question or a flamebait. But I have recently begun working with a Sun Solaris box after having spent the past five years working with RedHat. From what i can tell, thing look fairly similar and the 'man' command is some help. But I've... (7 Replies)
Discussion started by: deckard
7 Replies

2. UNIX for Advanced & Expert Users

How to remove a file with a leading dash '-' in it's name?

Somehow someone created a file named '-ov' in the root directory. Given the name, the how was probably the result of some cpio command they bozo'ed. I've tried a number of different ways to get rid of it using * and ? wildcards, '\' escape patterns etc.. They all fail with " illegal option --... (3 Replies)
Discussion started by: GSalisbury
3 Replies

3. IP Networking

BELKIN 'F5D5020' 16bit PCMCIA - FreeBSD HOWTO

Hey all, I've bought a few bits from Belkin who seem quite happy to support FreeBSD! Last time I bought a UPS from them and it's still going well :D I saw this on their website that the 16bit PCMCIA card was supported under FreeBSD: http://www.belkin.com/network/F5D5020.html I went to my... (0 Replies)
Discussion started by: WIntellect
0 Replies

4. UNIX for Dummies Questions & Answers

quoting echo 'it's friday'

echo 'it's friday' why appear the > (3 Replies)
Discussion started by: yls177
3 Replies

5. UNIX for Dummies Questions & Answers

HELP! The '/var/adm/message' file increase every few seconds???

Hi, guys, I have a big problem. I've got a sun solaris 4.1.4 workstation, and the /var/adm/message file will add one row every few seconds. It becomes a large file in a short time. I wander if there are some mistakes configuring the workstation. the /var/adm/message is as follow: ... (3 Replies)
Discussion started by: cloudsmell
3 Replies

6. Filesystems, Disks and Memory

HELP! The '/var/adm/message' file increase every few seconds???

Hi, guys, I have a big problem. I've got a sun solaris 4.1.4 workstation, and the /var/adm/message file will add one row every few seconds. It soon becomes a large file. I wander if there are some mistakes configuring the workstation. the /var/adm/message is as follow: ... (1 Reply)
Discussion started by: cloudsmell
1 Replies

7. Shell Programming and Scripting

Clearify what it means under 'WHAT' when hit the 'w'-command

I wonder how I shall read the result below, especially 'what' shown below. The result was shown when I entered 'w'. E.g what is TOP? What is gosh ( what does selmgr mean?)? login@ idle JCPU PCPU what 6:15am 7:04 39 39 TOP 6:34am 6:45 45 45 TOP 6:41am ... (1 Reply)
Discussion started by: Aelgen
1 Replies

8. UNIX for Dummies Questions & Answers

'find' command question

my solaris text talks about the 'find' command... it further goes to talk about an "action" used with the find command. I am completely confused as to what the {} do with the find comand. the explanation is this: "A set of braces, {}, delimits where the file name is passed to the command from... (2 Replies)
Discussion started by: xyyz
2 Replies

9. Programming

i can't use 'make' in my computer?

I need to compile a file,but 'make' does not work.please tell me how to use it or need which tools? (3 Replies)
Discussion started by: dsun5
3 Replies

10. UNIX for Dummies Questions & Answers

How to subtract 2 hours from 'date' in shell ( /bin/sh ) script ?

I write a sh script that zip and copy to tape all files that older then 2 hours. 1. The way I choose is - touch a file with "now - 2 hours", then use fine with '! -newer' 2. Do you have any other idea to do it ? tnx. (1 Reply)
Discussion started by: yairon
1 Replies

Featured Tech Videos