Shell Programming and Scripting

In a UTF-8 locale, the two-byte sequence specified by the C string "\347\03" is not a valid character, and it looks like gawk is discarding th contents of that line (or at least the characters starting with the invalid byte sequence and following characters up to the end of the line) are being discarded without printing a diagnostic. The standards only specify the behavior of awk when the files it reads are text files, so it is allowed to do anything it wants in this case. (By definition, file1 is not a text file since it contains byte sequences that are not valid characters in the current locale.) As RudiC suggested, using the C locale (which uses a single-byte character set with all byte sequences being valid characters) instead of the en_US.UTF-8 locale (which uses a variable number of bytes to encode a character and some sequences do not form valid characters) should do what you want in this case. (Note, however, that even in the C locale, NUL bytes are not valid in a text file and, if the file is not an empty file, the last byte of the file must be a <newline> character, and no line in the file can be longer than LINE_MAX bytes long. On most systems, LINE_MAX is 2048. The standards don't allow LINE_MAX to be less than 2048. The command:

Code:

getconf LINE_MAX

will give you the limit on your system.) From what we have seen so far (i.e., the first three lines), file1 appears to be a valid text file in the C locale.

Most of it went over my head only thing I understood that awk is not recognizing as a text file and so it isn't working. Ok so I tried and set the variable . After that command was still showing value empty but when I do it was showing value C. So that should be fine.

After that I ran sed command

Code:

sed 's/^..//' file1 > file2

and it didn't work, it just printed identical file in file2 with all junk characters as it is. Resulting file2 is same size as file1. So sed command didn't work.

I tried cut command and it works for all lines except the first line. On first line it leaves character and for all other lines it removed junk characters as expected.

Code:

cut -c 2- file1 > file2

. Let me know if this gives you any clue why cut command removes all junk characters except for 1st line.

We JUST want you to set LC_ALL=C in the environment of the awk, sed, or cut command you're trying to run. We aren't trying to get you to change the environment in your shell for other commands that you might run later. We NEVER suggested that you issue the command:

Code:

LC_ALL=C

by itself.

If cut -c2- is giving you what you want with the locale still set to en_US.UTF-8, cut is acting strangely (or at least inconsistently) when compared to awk with the same input. The cut -c option works on characters (just like awk substr()) and the two byte sequence at the start of each line you have shown us in file1 is NOT a character in any locale that uses UTF-8 as it's underlying codeset.

You could use:

Code:

cut -b3- file1 > file2

to throw away the first two bytes (not characters) of every line from file1 independent of locale.

Or you could use the suggestion RudiC provided:

Code:

LC_ALL=C sed 's/^..//' file1 > file2

or, using the same syntax, with any of the awk suggestions we've provided, such as:

Code:

LC_ALL=c awk '{print subset($0, 3)}' file1 > file2

to set the locale for sed or awk only without affecting the locale that would be used by any other utility you (or anyone else) would run later in that login session or in any other login sessions currently active on your system.

All of these commands worked like charm

Code:

cut -b3- file1 > file2
LC_ALL=c awk '{print subset($0, 3)}' file1 > file2
LC_ALL=C sed 's/^..//' file1 > file2

Thank you so much "Don Cragun" and "RudiC".