Shell Programming and Scripting

Hi,

Is there a way to identify the lines in a file having extended ascii characters and display the same?

For instance I have a file abc.txt having below data

Code:

aaa|bbb|111|This is first line
aaa|bbb|222|This is secõnd line
aaa|bbb|333|This is third line
aaa|bbb|444|This is foùrth line

Since the 2nd and 4th line contains the extended ascii characters (õ and ù), I would like to print those lines. Any ideas?

Code:

perl -ne 'print "$_" if /[\x80-\xFF]/' abc.txt

Thanks for the solution. It works perfectly fine. There is one more issue though. I'm trying to identify the number of bytes, lines and max line length using below command.

Code:

wc -l -c -L

But it ends up in error whenever it comes across any line having extended ascii characters in it and ends up with below error.

Code:

Invalid or imcomplete multibyte or wide character

Any solution to ensure it doesn't fail? I dont want to remove the extd. ascii characters from the line. One way I could think is to create a temp file, replace the extd. ascii characters with a normal one (e.g. "a") and then run the command but then again not familar with how to replace the same

---------- Post updated 07-17-14 at 12:32 AM ---------- Previous update was 07-16-14 at 11:27 PM ----------

Looking around the net I found couple of approaches which I can use on the same lines perl solution was suggested.

To find out the lines having extended ascii characters

Code:

grep -P "[\x80-\xFF]" abc.txt > abc1.txt

To replace the extended ascii character with a single byte character(a).

Code:

LANG=C sed 's/[\x80-\xFF]/a/g' abc.txt > abc2.txt

and once its done the wc can be used on the resultant file.

This approach seems to be working fine for me. Please suggest if there is a better approach that can be used.

Just a longhand very quick example; OSX 10.7.5, default bash terminal:-

Code:

#!/bin/bash
# len_line.sh
text='secõnd'
printf "$text" > /tmp/data
# Character length.
printf "Character length = ${#text}\n\n"
# Real length.
printf "Real length ="
wc -c < /tmp/data
echo ""
# Hexdump as proof.
hexdump -C < /tmp/data

Results:-

Code:

Last login: Thu Jul 17 07:59:20 on ttys000
AMIGA:barrywalker~> ./len_line.sh
Character length = 6

Real length =       7

00000000  73 65 63 c3 b5 6e 64                              |sec..nd|
00000007
AMIGA:barrywalker~> _

---------- Post updated at 12:24 PM ---------- Previous update was at 08:11 AM ----------

Using CygWin, default bash terminal manually:-

Code:

AMIGA:~> text='secõnd'
AMIGA:~> printf "$text" | wc -c
7
AMIGA:~> chars=$(printf "$text" | wc -c)
AMIGA:~> echo "$chars"
7
AMIGA:~> _