UTF-8,16,32 character lengths using awk


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users UTF-8,16,32 character lengths using awk
# 1  
Old 11-30-2015
UTF-8,16,32 character lengths using awk

Hi All,

I am trying to obtain count of characters using awk, but "length" function returns a value of 1 for 2-byte or 3-byte characters as well unlike wc -c command.

I have tried to use the below commands within awk function, but it does not seem to work
Code:
{
cmd="wc -c "stringtocheck
( cmd ) | getline Count
print Count
}

Can you please give me a solution. I have tried other options such as echo -e, print, but none seems to work.

Thanks & Regards
# 2  
Old 11-30-2015
What exactly is going wrong? Do you want byte counts or char counts? As to awk, my version reports the same byte count as does wc -c:
Code:
wc -c <file
57
awk '{print length}' file
43
12

(You'll have to count the two <NL> chars in)

As to your awk function - wc needs an input file, so it won't count the stringtocheck variable...
# 3  
Old 11-30-2015
Hi,

I need the byte count in a line (ignoring the newline character). Currently wc -c is giving me the byte count and length function within awk is giving me the character count.

Thanks & Regards
# 4  
Old 12-01-2015
Hi tostay2003...

If awk is so important then why not combine the two?

Longhand using OSX 10.7.5, default bash terminal:-
Code:
Last login: Tue Dec  1 13:06:10 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> ls -l binary
-rw-r--r--  1 barrywalker  staff  8000 28 Jan  2015 binary
AMIGA:barrywalker~/Desktop/Code/Shell> hexdump -C binary
00000000  4c 19 5e a6 0a 10 2e 20  9a 3e f8 be e8 6c e4 bd  |L.^.... .>...l..|
00000010  a7 f2 0c ca 66 dd 24 23  ba 01 9d 98 24 99 69 89  |....f.$#....$.i.|
00000020  30 73 e0 06 da 35 b8 91  8b cc ea 81 6a b6 ff c3  |0s...5......j...|
00000030  5d b2 67 2a 53 dc 53 33  79 f2 15 ad de e8 e0 6a  |].g*S.S3y......j|
00000040  fc 5e 64 a2 8f 50 e8 10  b3 dd 35 23 e3 3d 5b a2  |.^d..P....5#.=[.|
.
.
.
00001f00  52 63 7a 06 34 f3 04 b4  a9 77 6f e9 6b 5b 1b 1d  |Rcz.4....wo.k[..|
00001f10  dd 08 6e 65 47 9f a0 5f  e9 af cb 37 95 d6 02 c0  |..neG.._...7....|
00001f20  c3 10 53 20 33 94 7f b5  04 47 1c ac c4 b7 b0 06  |..S 3....G......|
00001f30  97 4f 84 be 95 53 1e d8  09 15 b8 20 f2 8d 42 cf  |.O...S..... ..B.|
00001f40
AMIGA:barrywalker~/Desktop/Code/Shell> awk 'BEGIN { system ( "wc -c < binary" ) }'
    8000
AMIGA:barrywalker~/Desktop/Code/Shell> _

EDIT:-
Apologies...
Ignore the above as I see that you want binary byte count per line not per file.
My error...
Is this what you are after?
Code:
#!/bin/bash
# blc
echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text
cat /tmp/text
hexdump -C /tmp/text
linenumber=1
while read -s -r line
do
	length=$( echo "$line" | wc -c )
	echo "Line $linenumber length less newline = $(($length-1))..."
	linenumber=$((linenumber+1))
done < /tmp/text

Resilts:-
Code:
Last login: Tue Dec  1 13:07:20 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc
•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á
00000000  e2 80 a2 c2 b6 c2 a7 e2  80 a2 c2 b6 e2 80 a2 c2  |................|
00000010  b6 c2 a7 e2 80 a2 61 73  64 66 68 e2 88 86 cb 99  |......asdfh.....|
00000020  e2 88 86 cb 99 c2 a9 c2  a9 cb 99 e2 88 86 cb 99  |................|
00000030  e2 88 86 cb 99 c2 a9 e2  88 86 cb 99 c3 a6 0a e2  |................|
00000040  80 a0 c2 ae e2 80 a0 c2  a5 c2 a5 e2 80 a0 c3 b8  |................|
00000050  cb 99 cb 9a e2 88 86 cb  99 e2 88 86 cb 99 c2 a9  |................|
00000060  cb 9a e2 88 86 c3 b8 e2  80 a2 c2 b6 c2 a7 c2 b6  |................|
00000070  c2 a7 c2 a2 c2 a7 e2 88  9e 41 4a 48 4b 4b c2 a7  |.........AJHKK..|
00000080  e2 80 a2 e2 80 a2 c2 ba  c2 ba c2 a9 c6 92 c6 92  |................|
00000090  c2 a9 e2 88 9a e2 88 ab  c3 a7 e2 88 ab 0a ef a3  |................|
000000a0  bf c3 94 ef a3 bf c3 94  c3 93 ef a3 bf c3 94 ef  |................|
000000b0  a3 bf c3 82 cb 9c ef a3  bf c2 a8 c3 81 cb 86 c2  |................|
000000c0  a8 c2 b0 ef ac 82 cb 87  c3 93 cb 9d cb 9d e2 97  |................|
000000d0  8a c3 94 c3 93 ef a3 bf  c4 b1 ef a3 bf c3 94 ef  |................|
000000e0  a3 bf c3 94 c3 93 c3 94  c3 93 c3 94 cb 86 c2 a8  |................|
000000f0  31 32 33 34 35 36 37 cb  86 cb 86 c3 81 cb 86 c2  |1234567.........|
00000100  a8 c3 81 cb 86 c2 a8 cb  86 c2 a8 c3 81 0a        |..............|
0000010e
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> _


Last edited by wisecracker; 12-01-2015 at 09:54 AM..
# 5  
Old 12-01-2015
The description of your problem is extremely confusing. UTF-8, UTF-16, and UTF-32 are completely different character sets and if you have a single file that contains characters from all three, determining which bytes in that file represent a <newline> character may be impossible unless you can clearly describe byte offsets in your file where there are shifts from one codeset to another and clearly describe how any program reading this file can determine what codeset is in use for any particular byte in that file.

If you are reading a file that is entirely encoded in UTF-8 (in which characters can be encoded with one to six bytes), you could tell your script that the UTF-8 input file was instead a file encoded in ISO 8859-1 (in which all characters are one byte) and count characters in lines in awk using the length() function since the <newline> character is encoded the same way in both codesets.

But, since you haven't described what the rest of your awk program is doing, we have no way to guess at whether or not this option might work for you and no way to guess if there might be other options.
# 6  
Old 12-01-2015
Although Don has very valid points this might be a starting point...
Again OSX 10.7.5, default bash terminal.

1) Manually generate a UTF8(ish) test file named /tmp/text.
Code:
echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text

2) Create a working stand alone shell script.
Code:
#!/bin/bash
# blc.sh '/full/path/to/filename'
linenumber=1
while read -s -r line
do
	length=$( echo "$line" | wc -c )
	echo "Line $linenumber length less newline = $(($length-1))..."
	linenumber=$((linenumber+1))
done < $1

3) Fit this inside an awk script.
Code:
#!/bin/sh
# blc_awk.sh /full/path/to/filename
# Note there must be a whitespace after blc.sh.
awk -v arg="$1" 'BEGIN { system ( "~/Desktop/Code/Shell/blc.sh "arg ) }'

Run in this case as blc_awk.sh /tmp/text
(In this example my blc.sh sits in the directory shown in the script just change yours to suit.)
Hope this is useful...
Results:-
Code:
Last login: Tue Dec  1 21:08:13 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
> †®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
> ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc.sh /tmp/text
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc_awk.sh /tmp/text
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> _


Last edited by wisecracker; 12-01-2015 at 05:18 PM..
# 7  
Old 12-01-2015
I read binary with awk a lot... LANG=C (environment var) is your friend.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Paste files of varying lengths

I have three files of varying lengths and different number of columns. How can I paste all three with all columns aligned? File1 ---- 123 File2 ---- 234 345 678 File3 ---- 456 789 Output should look like: 123 234 456 345 789 (6 Replies)
Discussion started by: Un1xNewb1e
6 Replies

2. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

3. Shell Programming and Scripting

Merge two files with different lengths

Hi there, I have two very long files like: file1: two fields 1 123 1 125 1 234 2 123 2 234 2 300 2 312 3 10 3 215 4 56 ... (11 Replies)
Discussion started by: ClaraW
11 Replies

4. UNIX for Dummies Questions & Answers

Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad. Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies

5. Shell Programming and Scripting

How to modify character to UTF-8 in shell script?

I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB. (5 Replies)
Discussion started by: vel4ever
5 Replies

6. Shell Programming and Scripting

Merging data from 2 files of different lengths?

Hi all, Sorry if someone has answered something like this already, but I have a problem. I am not brilliant with "awk" but think it should be the command to use to get what I am after. I have 2 files: job-file (several hundred lines like): 1018003,LONG MU WAN,1113S 1018004,LONG MU... (4 Replies)
Discussion started by: sgb2301
4 Replies

7. Shell Programming and Scripting

Read lines with different lengths in while loop

Hi there ! I need to treat files with variable line length, and process the tab-delimited words of each line. The tools I know are some basic bash scripting and sed ... I haven't got to python or perl yet. So my file looks like this obj1 0.01953 0.34576 0.04418 0.01249 obj2 0.78140... (7 Replies)
Discussion started by: jossojjos
7 Replies

8. Solaris

limit on Solaris username lengths?

Hi this question applies to Solaris 8,9,10 and opensolaris as in my environment it applies to all of these Is there a limit on the size of the username (in /etc/passwd) or indeed does there come a point where, like the 8 character limitation of passwords, the system receives the input but... (6 Replies)
Discussion started by: hcclnoodles
6 Replies

9. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

10. UNIX for Dummies Questions & Answers

Using grep to find strings of certain lengths?

I am trying to use grep to find strings of certain lengths that all start with the same letter. Is this possible?:confused: (4 Replies)
Discussion started by: crabtruck
4 Replies
Login or Register to Ask a Question