Sponsored Content
Top Forums UNIX for Dummies Questions & Answers grep and UNICODE (utf-16) file Post 302101553 by matrixmadhan on Wednesday 3rd of January 2007 02:35:43 AM
Old 01-03-2007
i think utf (8/16) format files doenst have the default end-of-line identifier in such a case, the usual tools applied to other text files cannot be used with.

Try running a wc -l on the file and post the output of number of lines, for pure utf formatted files it would return a zero; for such situations customized codes need to be written.

Smilie
 

9 More Discussions You Might Find Interesting

1. Programming

How to display unicode characters / unicode string

I have a stream of characters like "\u8BBE\u5907\u7BA1" and i want to display it. I tried following things already without any luck. 1) printf("%s",L("\u8BBE\u5907\u7BA1")); 2) printf("%lc",0x8BBE); 3) setlocale followed by fwide followed by wprintf 4) also changed the local manually... (3 Replies)
Discussion started by: jackdorso
3 Replies

2. Shell Programming and Scripting

Help with Converting UTF-8 data to Unicode

How can I get an error when converting 3rd line, since it has invalid characters abcde a®cdée a�cd� Unicode for ® = ® é = é I used "iconv -f UTF-8 -t ISO-8859-15 in.txt > out.txt" (2 Replies)
Discussion started by: arunbs
2 Replies

3. Shell Programming and Scripting

Unicode file validation

I don't want HTML_CONTENT,RICH_CONTENT,TEXT_CONTENT columns data in the file and reset of data we need to extract. Find the attached file. Need to extract date in between DI_UX_ROW_END tag. Can help me using unix command using AWK. Thanks, (2 Replies)
Discussion started by: bmk
2 Replies

4. UNIX for Dummies Questions & Answers

Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad. Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies

5. UNIX for Advanced & Expert Users

[ask]unicode utf-8 for arabic font

hlow all, i want to read arabic font in cli (cat, vi ,etc) in windows i can see the for why in linux i can't see that. this for the example وَمَنْ يَشْكُرْ فَإِنَّمَا يَشْكُرُ لِنَفْسِهِ what should i do ? i need your advice for read that font in cli...:confused: thx before (0 Replies)
Discussion started by: zvtral
0 Replies

6. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

7. Shell Programming and Scripting

Copying a file with UTF char on UNIX server

Hi, I need to run a SQL which check for special UTF char in DB. When I try to copy that in UNIX file it changes it to some wierd chat. How can in retain the UTF chars in my script? e.g. ο|π|ρ|σ|τ|υ|φ|χ|ψ Any help will be appriciated. Thanks, (14 Replies)
Discussion started by: varun22486
14 Replies

8. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies

9. Shell Programming and Scripting

Create .nfo file in ISO-8859-1 or UTF-8

Hey guys, I have a little problem, Let's say I create this script : #!/bin/sh nfo_file="/home/admin/info.nfo" echo "▒▒█ Hello █▒▒" > $nfo_fileIt seems to be okay : cat /home/admin/info.nfo ▒▒█ Hello █▒▒file -bi /home/admin/info.nfo text/plain; charset=utf-8But when I open it in a... (7 Replies)
Discussion started by: antoinelomb
7 Replies
Template::Provider::Encoding(3pm)			User Contributed Perl Documentation			 Template::Provider::Encoding(3pm)

NAME
Template::Provider::Encoding - Explicitly declare encodings of your templates SYNOPSIS
use Template::Provider::Encoding; use Template::Stash::ForceUTF8; use Template; my $tt = Template->new( LOAD_TEMPLATES => [ Template::Provider::Encoding->new ], STASH => Template::Stash::ForceUTF8->new, ); # Everything should be Unicode # (but you can pass UTF-8 bytes as well, thanks to Template::Stash::ForceUTF8) my $author = "x{5bae}x{5ddd}"; # this will emit Unicode flagged string to STDOUT. You might # probably want to binmode(STDOUT, ":encoding($enccoding)") # before process() call $tt->process($template, { author => $author }); # in your templates [% USE encoding 'utf-8' -%] My name is [% author %]. { ... whatever UTF-8 bytes } DESCRIPTION
Template::Provider::Encoding is a Template Provider subclass to decode template using its declaration. You have to declare encoding of the template in the head (1st line) of template using (fake) encoding TT plugin. Otherwise the template is handled as utf-8. [% USE encoding 'utf-8' %] Here comes utf-8 strings with [% variable %]. DIFFERNCE WITH OTHER WAYS
UNICODE option and BOM Recent TT allows "UNICODE" option to Template::Provider and by adding it Provider scans BOM (byte-order mark) to detect UTF-8/UTF-16 encoded template files. This module does basically the same thing in a different way, but IMHO adding BOM to template files is a little painful especially for non-programmers. Template::Provider::Encode Template::Provider::Encode provides a very similar way to detect Template file encodings and output the template into various encodings. This module doesn't touch output encoding of the template and instead it emits valid Unicode flagged string. I think the output encoding conversion should be done by other piece of code, especially in the framework. This module doesn't require you to specify encoding in the code, nor doesn't guess encodings. Instead it forces you to put "[% USE encoding 'foo-bar' %]" in the top of template files, which is explicit and, I think, is a good convention. AUTHOR
Tatsuhiko Miyagawa <miyagawa@bulknews.net> This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Template::Stash::ForceUTF8, Template::Provider::Encode perl v5.12.3 2007-08-01 Template::Provider::Encoding(3pm)
All times are GMT -4. The time now is 03:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy