Solaris

Yes i 'm using ZFS, And can u plz tell how did u calculate the memory i was not able to get that .. but yes the stats u gave were correct as per top.

The same thing is happening with my Ldoms too they start with 3.5 gb free mem and end up with 1.5G free though nothing is running in them.

If you are running ZFS and LDOMs, what is the problem you are trying to fix ? Having 8 GB free isn't a problem per se.

About calculating, the numbers reported by kstat are in pages, and pages are 8 KB on SPARC.

In the SunOS/Solaris environment, main memory/physical memory should really be called the page cache. Think of it as cache of storage rather than program/data memory. To Solaris, any empty page is wasted faster-than-disk storage. Over time, the amount of free memory will settle at some value bigger than lotsfree (google it). If half your memory is free, then you don't have any pressure on the page cache. So you are chasing numbers rather than fixing an issue.....

Thanx for info Jlliagre , its quite helpful just to add we can find how much memory zfs is using with following command

kstat zfs:0:arcstats:size which give above 5G in my server. But isn't there any way to tune zfs not to use that much memory ? or put its cache somewhere else

Yes but what's the point to tune for poorer performance when you have plenty of RAM available ?
This ZFS memory is actually available should demand arrive from applications.
Unlike the ZFS one, UFS file cache is reported it as free memory so it doesn't trigger this kind of questions.
Like what ?

you can cap the memory ZFS uses. also you can buy some ssd disks and use them as cache... but why would you do this if you've 8gig of free memory?

if you are using zfs that is where your memory goes. By design zfs caches memory. It is called the arc cache. It will release memory based on system needs but is going to take as much as there is avaliable for its cache. It is recommended not to modify this and let zfs manage it for you, however you can adjust it by modifying your /etc/system file to minimize the amount of memory that zfs can reserve for its arc. If you want to see how much zfs is using go to this site and get arcstat.pl and run it on your system Monitoring ZFS Statistic - Roman Ivanov

the output will look like so:

# ./arcstat.pl
Time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
14:20:55 343M 36M 10 34M 10 2M 10 1M 0 10G 10G
14:20:56 11 0 0 0 0 0 0 0 0 10G 10G
14:20:57 3 0 0 0 0 0 0 0 0 10G 10G
14:20:58 6 0 0 0 0 0 0 0 0 10G 10G
14:20:59 26 0 0 0 0 0 0 0 0 10G 10G
14:21:00 6 0 0 0 0 0 0 0 0 10G 10G


This shows me that zfs on my system is cacheing 10g of physical memory. I would be willing to bet you will find your 8g there.