Sponsored Content
Top Forums UNIX for Advanced & Expert Users UTF-8,16,32 character lengths using awk Post 302961562 by wisecracker on Tuesday 1st of December 2015 08:14:30 AM
Old 12-01-2015
Hi tostay2003...

If awk is so important then why not combine the two?

Longhand using OSX 10.7.5, default bash terminal:-
Code:
Last login: Tue Dec  1 13:06:10 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> ls -l binary
-rw-r--r--  1 barrywalker  staff  8000 28 Jan  2015 binary
AMIGA:barrywalker~/Desktop/Code/Shell> hexdump -C binary
00000000  4c 19 5e a6 0a 10 2e 20  9a 3e f8 be e8 6c e4 bd  |L.^.... .>...l..|
00000010  a7 f2 0c ca 66 dd 24 23  ba 01 9d 98 24 99 69 89  |....f.$#....$.i.|
00000020  30 73 e0 06 da 35 b8 91  8b cc ea 81 6a b6 ff c3  |0s...5......j...|
00000030  5d b2 67 2a 53 dc 53 33  79 f2 15 ad de e8 e0 6a  |].g*S.S3y......j|
00000040  fc 5e 64 a2 8f 50 e8 10  b3 dd 35 23 e3 3d 5b a2  |.^d..P....5#.=[.|
.
.
.
00001f00  52 63 7a 06 34 f3 04 b4  a9 77 6f e9 6b 5b 1b 1d  |Rcz.4....wo.k[..|
00001f10  dd 08 6e 65 47 9f a0 5f  e9 af cb 37 95 d6 02 c0  |..neG.._...7....|
00001f20  c3 10 53 20 33 94 7f b5  04 47 1c ac c4 b7 b0 06  |..S 3....G......|
00001f30  97 4f 84 be 95 53 1e d8  09 15 b8 20 f2 8d 42 cf  |.O...S..... ..B.|
00001f40
AMIGA:barrywalker~/Desktop/Code/Shell> awk 'BEGIN { system ( "wc -c < binary" ) }'
    8000
AMIGA:barrywalker~/Desktop/Code/Shell> _

EDIT:-
Apologies...
Ignore the above as I see that you want binary byte count per line not per file.
My error...
Is this what you are after?
Code:
#!/bin/bash
# blc
echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text
cat /tmp/text
hexdump -C /tmp/text
linenumber=1
while read -s -r line
do
	length=$( echo "$line" | wc -c )
	echo "Line $linenumber length less newline = $(($length-1))..."
	linenumber=$((linenumber+1))
done < /tmp/text

Resilts:-
Code:
Last login: Tue Dec  1 13:07:20 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc
•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á
00000000  e2 80 a2 c2 b6 c2 a7 e2  80 a2 c2 b6 e2 80 a2 c2  |................|
00000010  b6 c2 a7 e2 80 a2 61 73  64 66 68 e2 88 86 cb 99  |......asdfh.....|
00000020  e2 88 86 cb 99 c2 a9 c2  a9 cb 99 e2 88 86 cb 99  |................|
00000030  e2 88 86 cb 99 c2 a9 e2  88 86 cb 99 c3 a6 0a e2  |................|
00000040  80 a0 c2 ae e2 80 a0 c2  a5 c2 a5 e2 80 a0 c3 b8  |................|
00000050  cb 99 cb 9a e2 88 86 cb  99 e2 88 86 cb 99 c2 a9  |................|
00000060  cb 9a e2 88 86 c3 b8 e2  80 a2 c2 b6 c2 a7 c2 b6  |................|
00000070  c2 a7 c2 a2 c2 a7 e2 88  9e 41 4a 48 4b 4b c2 a7  |.........AJHKK..|
00000080  e2 80 a2 e2 80 a2 c2 ba  c2 ba c2 a9 c6 92 c6 92  |................|
00000090  c2 a9 e2 88 9a e2 88 ab  c3 a7 e2 88 ab 0a ef a3  |................|
000000a0  bf c3 94 ef a3 bf c3 94  c3 93 ef a3 bf c3 94 ef  |................|
000000b0  a3 bf c3 82 cb 9c ef a3  bf c2 a8 c3 81 cb 86 c2  |................|
000000c0  a8 c2 b0 ef ac 82 cb 87  c3 93 cb 9d cb 9d e2 97  |................|
000000d0  8a c3 94 c3 93 ef a3 bf  c4 b1 ef a3 bf c3 94 ef  |................|
000000e0  a3 bf c3 94 c3 93 c3 94  c3 93 c3 94 cb 86 c2 a8  |................|
000000f0  31 32 33 34 35 36 37 cb  86 cb 86 c3 81 cb 86 c2  |1234567.........|
00000100  a8 c3 81 cb 86 c2 a8 cb  86 c2 a8 c3 81 0a        |..............|
0000010e
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> _


Last edited by wisecracker; 12-01-2015 at 09:54 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Using grep to find strings of certain lengths?

I am trying to use grep to find strings of certain lengths that all start with the same letter. Is this possible?:confused: (4 Replies)
Discussion started by: crabtruck
4 Replies

2. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

3. Solaris

limit on Solaris username lengths?

Hi this question applies to Solaris 8,9,10 and opensolaris as in my environment it applies to all of these Is there a limit on the size of the username (in /etc/passwd) or indeed does there come a point where, like the 8 character limitation of passwords, the system receives the input but... (6 Replies)
Discussion started by: hcclnoodles
6 Replies

4. Shell Programming and Scripting

Read lines with different lengths in while loop

Hi there ! I need to treat files with variable line length, and process the tab-delimited words of each line. The tools I know are some basic bash scripting and sed ... I haven't got to python or perl yet. So my file looks like this obj1 0.01953 0.34576 0.04418 0.01249 obj2 0.78140... (7 Replies)
Discussion started by: jossojjos
7 Replies

5. Shell Programming and Scripting

Merging data from 2 files of different lengths?

Hi all, Sorry if someone has answered something like this already, but I have a problem. I am not brilliant with "awk" but think it should be the command to use to get what I am after. I have 2 files: job-file (several hundred lines like): 1018003,LONG MU WAN,1113S 1018004,LONG MU... (4 Replies)
Discussion started by: sgb2301
4 Replies

6. Shell Programming and Scripting

How to modify character to UTF-8 in shell script?

I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB. (5 Replies)
Discussion started by: vel4ever
5 Replies

7. UNIX for Dummies Questions & Answers

Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad. Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies

8. Shell Programming and Scripting

Merge two files with different lengths

Hi there, I have two very long files like: file1: two fields 1 123 1 125 1 234 2 123 2 234 2 300 2 312 3 10 3 215 4 56 ... (11 Replies)
Discussion started by: ClaraW
11 Replies

9. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

10. Shell Programming and Scripting

Paste files of varying lengths

I have three files of varying lengths and different number of columns. How can I paste all three with all columns aligned? File1 ---- 123 File2 ---- 234 345 678 File3 ---- 456 789 Output should look like: 123 234 456 345 789 (6 Replies)
Discussion started by: Un1xNewb1e
6 Replies
All times are GMT -4. The time now is 01:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy