Code:

awk '{printf("%050s\t%s\n", toupper($0), $0)}' hex-data | LC_COLLATE=C sort -k1,1 | cut -f2

If you take a look at the C/POSIX collation sequence, you'll see that the sequences A-F and a-f both sort after 0-9 in the POSIX/C locale. So long as the hex numbers do not mix case and are of the same length, the string collation sequence is congruent to a hex digits sequence.

LC_COLLATE Category in the POSIX Locale

If you have a list of hex numbers, you can decorate them with a column of zero-padded, case-folded numbers, sort on that column, then output the original numbers.

Code:

$ cat hex-data
3e80
3EF8
460
$ awk '{printf("%050s\t%s\n", toupper($0), $0)}' hex-data | LC_COLLATE=C sort -k1,1 | cut -f2
460
3e80
3EF8

Regards,
Alister