Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: UTF-8,16,32 character lengths using awk
Top Forums UNIX for Advanced & Expert Users UTF-8,16,32 character lengths using awk Post 302961612 by wisecracker on Tuesday 1st of December 2015 04:11:34 PM
Old 12-01-2015
Although Don has very valid points this might be a starting point...
Again OSX 10.7.5, default bash terminal.

1) Manually generate a UTF8(ish) test file named /tmp/text.
Code:
echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
†®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text

2) Create a working stand alone shell script.
Code:
#!/bin/bash
# blc.sh '/full/path/to/filename'
linenumber=1
while read -s -r line
do
	length=$( echo "$line" | wc -c )
	echo "Line $linenumber length less newline = $(($length-1))..."
	linenumber=$((linenumber+1))
done < $1

3) Fit this inside an awk script.
Code:
#!/bin/sh
# blc_awk.sh /full/path/to/filename
# Note there must be a whitespace after blc.sh.
awk -v arg="$1" 'BEGIN { system ( "~/Desktop/Code/Shell/blc.sh "arg ) }'

Run in this case as blc_awk.sh /tmp/text
(In this example my blc.sh sits in the directory shown in the script just change yours to suit.)
Hope this is useful...
Results:-
Code:
Last login: Tue Dec  1 21:08:13 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> echo "•¶§•¶•¶§•asdfh∆˙∆˙©©˙∆˙∆˙©∆˙æ
> †®†¥¥†ø˙˚∆˙∆˙©˚∆ø•¶§¶§¢§∞AJHKK§••ºº©ƒƒ©√∫ç∫
> ÔÔÓÔ˜¨Áˆ¨°flˇÓ˝˝◊ÔÓıÔÔÓÔÓÔˆ¨1234567ˆˆÁˆ¨Áˆ¨ˆ¨Á" > /tmp/text
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc.sh /tmp/text
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> ./blc_awk.sh /tmp/text
Line 1 length less newline = 62...
Line 2 length less newline = 94...
Line 3 length less newline = 111...
AMIGA:barrywalker~/Desktop/Code/Shell> _

Last edited by wisecracker; 12-01-2015 at 05:18 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Using grep to find strings of certain lengths?

I am trying to use grep to find strings of certain lengths that all start with the same letter. Is this possible?:confused: (4 Replies)
Discussion started by: crabtruck
4 Replies

2. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

3. Solaris

limit on Solaris username lengths?

Hi this question applies to Solaris 8,9,10 and opensolaris as in my environment it applies to all of these Is there a limit on the size of the username (in /etc/passwd) or indeed does there come a point where, like the 8 character limitation of passwords, the system receives the input but... (6 Replies)
Discussion started by: hcclnoodles
6 Replies

4. Shell Programming and Scripting

Read lines with different lengths in while loop

Hi there ! I need to treat files with variable line length, and process the tab-delimited words of each line. The tools I know are some basic bash scripting and sed ... I haven't got to python or perl yet. So my file looks like this obj1 0.01953 0.34576 0.04418 0.01249 obj2 0.78140... (7 Replies)
Discussion started by: jossojjos
7 Replies

5. Shell Programming and Scripting

Merging data from 2 files of different lengths?

Hi all, Sorry if someone has answered something like this already, but I have a problem. I am not brilliant with "awk" but think it should be the command to use to get what I am after. I have 2 files: job-file (several hundred lines like): 1018003,LONG MU WAN,1113S 1018004,LONG MU... (4 Replies)
Discussion started by: sgb2301
4 Replies

6. Shell Programming and Scripting

How to modify character to UTF-8 in shell script?

I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB. (5 Replies)
Discussion started by: vel4ever
5 Replies

7. UNIX for Dummies Questions & Answers

Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad. Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies

8. Shell Programming and Scripting

Merge two files with different lengths

Hi there, I have two very long files like: file1: two fields 1 123 1 125 1 234 2 123 2 234 2 300 2 312 3 10 3 215 4 56 ... (11 Replies)
Discussion started by: ClaraW
11 Replies

9. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

10. Shell Programming and Scripting

Paste files of varying lengths

I have three files of varying lengths and different number of columns. How can I paste all three with all columns aligned? File1 ---- 123 File2 ---- 234 345 678 File3 ---- 456 789 Output should look like: 123 234 456 345 789 (6 Replies)
Discussion started by: Un1xNewb1e
6 Replies

LEARN ABOUT DEBIAN

geometry::primitive::line

Geometry::Primitive::Line(3pm)				User Contributed Perl Documentation			    Geometry::Primitive::Line(3pm)

NAME

       Geometry::Primitive::Line - A Line

DESCRIPTION

       Geometry::Primitive::Line represents a straight curve defined by two points.

SYNOPSIS

	 use Geometry::Primitive::Line;

	 my $line = Geometry::Primitive::Line->new();
	 $line->start($point1);
	 $line->end($point2);

ATTRIBUTES

   end
       Set/Get the end point of the line.

   start
       Set/Get the start point of the line.

METHODS

   new
       Creates a new Geometry::Primitive::Line

   contains_point
       Returns true if the supplied point is 'on' the line.  Accepts either a point object or an x y pair.

   grow
       Does nothing, as I'm not sure how.  Patches or hints welcome.

   is_parallel ($other_line)
       Returns true if the supplied line is parallel to this one.

   is_perpendicular ($other_line)
       Returns true if the supplied line is perpendicular to this one.

   length
       Get the length of the line.

   point_end
       Get the end point.  Provided for Shape role.

   point_start
       Get the start point.  Provided for Shape role.

   scale
       Does nothing at the moment.

   slope
       Get the slope of the line.

   to_string
       Guess!

   y_intercept
       Returns the Y intercept of this line.

AUTHOR

       Cory Watson <gphat@cpan.org>

COPYRIGHT &; LICENSE
       You can redistribute and/or modify this code under the same terms as Perl itself.

perl v5.10.1							    2010-01-10					    Geometry::Primitive::Line(3pm)
All times are GMT -4. The time now is 06:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy