Shell Programming and Scripting

I want to filter out the special character whose ascii value doesn't fall within the range "[\x80-\xFF]" .

Example:� or Ć. So in that case is there any defined range which will filter out this characters.

I can filter those which falls withing "[\x80-\xFF]" . Need to filter those special chracter which doesn't fall within the "[\x80-\xFF]" range

You might want to use the :print: character class the POSIX BRE regexp provide and negate that: [^:print:] and see how far that gets you.

Basically there is no pattern for what constitutes a printable or non-printable character: character "\9", which is a TAB just is that by convention, not because it is - in principle - any different from "\10" or "\8".

You might also want to identify your locale, which may establish so-called collating sequences. See more about this at this page.

I hope this helps.

bakunin

I don't know of an RE context where \9 would represent a tab (although \0x9 and \011 would when using an ASCII based character set).

The print character class is identified by [:print:]. A BRE matching a character in the print class is [[:print:]] and a BRE matching a character that is not in the print class is [^[:print:]]. The BRE [^:print:] would match any character other than :, i, n, p, r, and t.

Note also that the print class does not include control characters. To select a UTF-8 character that is not a character in the 7-bit ASCII character set (actually select each byte of one of those characters), you could use the BRE [^[:ctrl:][:print:]] while in the C or POSIX locale.

But, when working with ASCII, UTF-8, and 8859-* character sets, just filtering out bytes with the high order bit set should be sufficient.

If you're trying to identify characters greater than 0xff, those aren't characters, they will be encoded in some way and ordinary regular expressions won't match them. Some GNU tools may have extended features to support them.

I know this is not about python per se, but there are REGEX tools for extended character sets, unicode being one of those sets:

regex - matching unicode characters in python regular expressions - Stack Overflow

UNIX in general is not unicode centric so Corona's answer pretty much stands for most regex engines.

The PCRE supports a lot of encoded charsets. You can download it here:
PCRE - Browse /pcre/8.30 at SourceForge.net

I have a file which consists all the special character which lies within "[\x80-\xFF]" and speical character beyond this range.

So in order to identify all of them by their byte values what can be done.

Moderator's Comments:

edit by bakunin: please leave out the HTML-tags. You don't need them

To repeat:

How is your file encoded?