When I want to use the editor ,whether vi or textedit,it prompts for entering a key. How can I disable this function?many thanks for offering solutions!

How are you entering the command? Vi has a -x option for encryption. Check that you don't have vi as an alias for vi -x (use the alias command.

From the vi man page:

-x Encryption option; when used, vi simulates the X com-
mand of ex and prompts the user for a key. This key is
used to encrypt and decrypt text using the algorithm
of the crypt command. The X command makes an educated
guess to determine whether text read in is encrypted
or not. The temporary buffer file is encrypted also,
using a transformed version of the key typed in for
the -x option. If an empty encryption key is entered
(that is, if the return key is pressed right after the
prompt), the file will not be encrypted. This is a
good way to decrypt a file erroneously encrypted with
a mistyped encryption key, such as a backspace or undo
key.

If the file is encrypted, then you will not see the file correctly if you just do a regular vi on it (the text will still be encrypted).