First, a process's memory is divided into pieces called segments. There is a text segment and a code segment which are required to exist by standards and there may be others. malloc() allocates free space at the end of the data seqment. If there isn't enough free space, malloc will use the sbrk() system call to add space. Thus malloc can and usually does make a process grow. free() on the other hand will never shrink a process. It just adds the area being freed to the space available for subsequent malloc() calls.

Next, plock(PROCLOCK) attempts to lock the entire text and data seqments into core. Looking at your post and your code, I gather that you think the plock(PROCLOCK) somehow will only lock whatever the last malloc() call allocated. plock() doesn't know or care about malloc(), it operates on entire segments.

So your program first malloc'ed 10MB which increased the size of your process by about 10MB. At this point, plock(PROCLOCK) worked. Then you freed the area and malloc'ed 40MB. Well you get use the spare 10MB that you freed, but malloc() must increase your process by another 30MB to satisfy the request. At this point, your process is now too big to be locked into memory. It stays too big when you do the free(). It won't need to grow as you malloc the final 10MB, it just allocates 10MB from the 40MB that you freed. But it still can't be locked.