I'm not too clear on how tcpd would tell the difference between users either, since it would seem to happen before it hands over the connection! Some of its
documentation hints about IDENT protocol or RFC931, which could mean having to do custom configuration on the client not just the server. And that probably assumes the username on the client machine matches what they want to login as in the first place.
All in all it would be much better to do it cleanly inside vsftp, but that doesn't look possible either! It
does have per-client configuration settings, cheerfully ignored whenever they specify limits on things already happened -- like specifying an IP to connect from. Its user allow/deny list cannot specify IP addresses either, it's just a dumb text file of one user per line.
If you absolutely have to use vsftpd, you might need to set up a separate daemon for that one user, restricted to some internal subnet. Or, if the user's not internal, over some VPN.