performance of mmap() against NFS


 
Thread Tools Search this Thread
Top Forums Programming performance of mmap() against NFS
# 1  
Old 03-31-2009
performance of mmap() against NFS

Hi all.

I got a performance related problem that bothers me a lot.

we have two computer nodes(svr1 & svr2), and svr1 has mysql 4.1.20 installed as well as nfs, and svr2 can mount disk from svr2, so we can make it possible to share files between them...

the data files we'll deal with can often reach GB(or even TB) level, for this instance, we got a 4.17GB seismic data file , the job is to read data from it as input and output the data to the shared disk.

in order to improve the performance we use memory mapping instead of traditionally read/write sys calls.

if we run the program at svr1, the time consumed is merely around 8min, but things get different if we run the same program at svr2, the time consumed is 4hours more.

it supprised me a lot, the point is svr2 must use nfs to input/output data, I am wondering if the nfs is the bottleneck that makes so, and how to solve that.

code segment is as follows:
Code:
static CSDB_bool fillset(int fd, char* buf, MYSQL_RES* res, int num_chno, int trlen)
{
	//不加参数验证了,必须保证调用正确
	
	
	char* pos		= NULL; //当前位置	
	off_t bottom 	= 0; //映射区起始位置对应的道号,初始为第0道
	int size		= num_chno; //映射区能够容纳的道数
	off_t offset 	= 512;
	size_t bytesize = size * trlen;	//内存映射的大小(含线头)
	size_t sys_page_size = sysconf(_SC_PAGESIZE); //内存页的大小
	int comp		= offset % sys_page_size;	//余数
	offset 			= offset - comp;	//对齐
	
	//初始映射到文件的第一道
	//mmap的最后一个参数真麻烦!!!!
	char* map = (char*)mmap(NULL, bytesize, PROT_READ, MAP_PRIVATE, fd, offset);
	if(map == MAP_FAILED)
	{//映射失败
		perror("mmap()");
		return FALSE;
	}
	//int i = 0;
	//for(i = 0; i < num_chno; ++i)
	MYSQL_ROW row;
	int chno = 0;	
	while((row = mysql_fetch_row(res)))
	{		
		chno = atoi(row[0]);
		if(chno > bottom && chno < bottom + size - 1)
		{		
			pos = map + (chno - bottom) * trlen;
			if(offset == 0)
			{
				memcpy(buf, pos + 512, trlen);	
			}
			else
			{
				memcpy(buf, pos + comp, trlen);	
			}		
					
		}
		else
		{
			offset 	= 512 + (size_t)chno * trlen;
			comp	= offset % sys_page_size;	//余数
			offset 	= offset - comp;	//对齐
			//munmap(map, bytesize);
			//map = (char*)mmap(NULL, bytesize, PROT_READ, MAP_PRIVATE, fd, offset);
			map = (char*)mmap(map, bytesize, PROT_READ, MAP_PRIVATE | MAP_FIXED, fd, offset);
			if(map == MAP_FAILED)
			{
				return FALSE;
			}
			bottom = chno;			
			pos = map;
			if(offset == 0)
			{
				memcpy(buf, pos + 512, trlen);	
			}
			else
			{
				memcpy(buf, pos + comp, trlen);	
			}	
			
		}
		buf = buf + trlen;			
	}
	//卸载映射
	munmap(map, bytesize);
	return TRUE;
}

any one got clue please give me some suggestion
thanks a lot in advances

ben

Last edited by jim mcnamara; 03-31-2009 at 11:28 AM.. Reason: add code tags
# 2  
Old 03-31-2009
Consider tuning NFS - you did not mention the OS, but here is a link, you can use google for other links for NFS tuning.

http://www.redhat.com/f/pdf/rhel4/NFSv4WP.pdf

Plus your code is hard to read -- please use [ code ] [ /code ] tags (I added spaces so you could see the the tags themselves).
# 3  
Old 03-31-2009
One note: why are you calling mmap() INSIDE a loop? Usually one call suffices.
# 4  
Old 04-01-2009
Thanks so much

The OS is rhel4, so the link you offered just hits it.

Quote:
Originally Posted by jim mcnamara
One note: why are you calling mmap() INSIDE a loop? Usually one call suffices.
The files we deal with are often too large to map it into address space at one time, so I just map a part of file at a time, and check if the data I need is in already mapped, if so, memcpy it, or I should remap it.

have I made myself clear, sorry about my poor englishSmilie, do you have any better solution of this kind of problem?
# 5  
Old 04-02-2009
Try NFS tuning first. You may also want to increase virtual memory on the client.
Also, if you can, get the sysadmin to turn off atime updates on your NFS mounted filesystem.
# 6  
Old 04-02-2009
Why memcpy an mmap-ed block anywhere? This has little advantage over just read()-ing it, because memcpy forces it to page in the entire section anyway. Ideally, whatever function needs this data could use the mmap-ed blocks directly, which would have the advantage of paging in only the data it actually uses.
# 7  
Old 04-02-2009
thanks Jim, I already learnt the tuning skill, and I will give it a try once I get the chance.

and also thank Corona, yes there're many potential issues in design time, but now there seems to be no way to change that, at least this version.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

NFS Performance Issue

Hi, I have a conflict. I have 2 servers: the 1st creates an application logs and the 2nd is an empty server. I want to export (read-only) from the 1st server by NFS the directory that contains the logs to the 2nd server. Now, the question is: If in the 2nd server i'm doing a lot of... (3 Replies)
Discussion started by: moshesa
3 Replies

2. Emergency UNIX and Linux Support

mmap

I want to know whether this is possile or ever been tried out. I want to obtain a chuck of memory using mmap() I do it so : n = mmap(0, 8000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); And hold on to that memory, when a process requests for memory, some memory is... (2 Replies)
Discussion started by: xerox
2 Replies

3. Programming

mmap

hai, How do we map 'n' number of files into memory by using mmap system call?? Thanks in advance...... (5 Replies)
Discussion started by: andrew.paul
5 Replies

4. Homework & Coursework Questions

mmap

Descriptions: Develop a program that uses mmap() to map a file to memory space. Prepare such a file by yourself and do the follows. <LI class=MsoNormal>Display the content of the file after mapping; <LI class=MsoNormal>Output how many digits included in the file; <LI class=MsoNormal>Replace... (1 Reply)
Discussion started by: gokult
1 Replies

5. Programming

mmap()

how to use mmap() to map a file to memory space. Do you have any simple program???? Because I have to implement lot of concepts into it. (5 Replies)
Discussion started by: gokult
5 Replies

6. UNIX for Dummies Questions & Answers

mmap()

how to use mmap() to map a file to memory space. Do you have any simple program???? Because I have to implement lot of concepts into it. (3 Replies)
Discussion started by: gokult
3 Replies

7. Solaris

mmap() on 64 bit m/c

Dear Experts, i have a problem related to mmap(), when i run my program on sun for 64 bit which is throwing SIGBUS when it encounters mmap() function, what is the reason how to resolve this one, because it is working for 32 bit. with regards, vidya. (2 Replies)
Discussion started by: vin_pll
2 Replies

8. Filesystems, Disks and Memory

NFS performance

Hi, In our environment (HP-UX B.11.11) we mainly use NFS v3 resources as the storage solution for our application. It worked fine, however we have recently noticed big I/O performance degradation that affected the application. Is there any command or utility that can help us measure I/O... (2 Replies)
Discussion started by: piooooter
2 Replies

9. Red Hat

NFS performance stats on Linux

Hi everyone, The last two days I'm researching results of NFS operations on Linux, and I noticed some time difference when read and write. cat /proc/version : Linux version 2.6.9-42.ELsmp (bhcompile@hs20-bc1-1.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-2)) #1 SMP Wed Jul 12... (1 Reply)
Discussion started by: sysgate
1 Replies

10. Filesystems, Disks and Memory

mmap

Hello. I'm writing some random access i/o software on Solaris 8 using mmap64 to memory map large files (my test file is ~25 GB). The abbreviated code fragment is: fd = open(cbuf,O_RDONLY|O_LARGEFILE); struct stat statbuf; fstat(fd,&statbuf); off_t len =... (0 Replies)
Discussion started by: gusm
0 Replies
Login or Register to Ask a Question