Sorry, a word was missing in my last post that was making it confusing. I was already understanding the mask concept the way you describe it.
The bottom line is, regardless of the "umask vs mask" and "wrx vs rwx" typos in the initial command posted, the behavior the OP is observing is correct and there is no way to directly achieve what he wants.
ZFS NFSv4 ACLs wouldn't help either in that case as existing files permissions aren't affected by the inheritance flags.
In any case, assuming the directory ACL are properly set, it would be possible for the user to overcome the limitation by first renaming the file to edit, then copying that file to the original name and finally removing the copy. eg:
Code:
$ id
uid=xxxxx(jlliagre) gid=xxxxx(jlliagre)
$ su
...
# mkdir d
# pe setfacl -r -m user:jlliagre:rwx d
# date > d/f
# getfacl d
# file: d
# owner: root
# group: root
user::rwx
user:jlliagre:rwx #effective:rwx
group::r-x #effective:r-x
mask:rwx
other:r-x
# getfacl d/f
# file: d/f
# owner: root
# group: root
user::rw-
group::r-- #effective:r--
mask:r--
other:r--
# ^D
$ cd d
/tmp/mnt/d
$ ls -l
total 2
-rw-r--r-- 1 root root 30 Jul 4 11:32 f
$ date >> f
/bin/ksh: f: cannot create
$ mv f f1
$ cp f1 f
$ rm -f f1
$ date >> f
$ ls -l
total 2
-rw-r--r-- 1 jlliagre jlliagre 60 Jul 4 11:32 f