01-03-2007
i think utf (8/16) format files doenst have the default end-of-line identifier in such a case, the usual tools applied to other text files cannot be used with.
Try running a wc -l on the file and post the output of number of lines, for pure utf formatted files it would return a zero; for such situations customized codes need to be written.
9 More Discussions You Might Find Interesting
1. Programming
I have a stream of characters like "\u8BBE\u5907\u7BA1"
and i want to display it.
I tried following things already without any luck.
1) printf("%s",L("\u8BBE\u5907\u7BA1"));
2) printf("%lc",0x8BBE);
3) setlocale followed by fwide followed by wprintf
4) also changed the local manually... (3 Replies)
Discussion started by: jackdorso
3 Replies
2. Shell Programming and Scripting
How can I get an error when converting 3rd line, since it has invalid characters
abcde
a®cdée
a�cd�
Unicode for
® = ®
é = é
I used "iconv -f UTF-8 -t ISO-8859-15 in.txt > out.txt" (2 Replies)
Discussion started by: arunbs
2 Replies
3. Shell Programming and Scripting
I don't want HTML_CONTENT,RICH_CONTENT,TEXT_CONTENT columns data in the file and reset of data we need to extract.
Find the attached file.
Need to extract date in between DI_UX_ROW_END tag.
Can help me using unix command using AWK.
Thanks, (2 Replies)
Discussion started by: bmk
2 Replies
4. UNIX for Dummies Questions & Answers
Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad.
Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies
5. UNIX for Advanced & Expert Users
hlow all,
i want to read arabic font in cli (cat, vi ,etc) in windows i can see the for why in linux i can't see that. this for the example وَمَنْ يَشْكُرْ فَإِنَّمَا يَشْكُرُ لِنَفْسِهِ
what should i do ? i need your advice for read that font in cli...:confused:
thx before (0 Replies)
Discussion started by: zvtral
0 Replies
6. Linux
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies
7. Shell Programming and Scripting
Hi,
I need to run a SQL which check for special UTF char in DB. When I try to copy that in UNIX file it changes it to some wierd chat. How can in retain the UTF chars in my script?
e.g. ο|π|ρ|σ|τ|υ|φ|χ|ψ
Any help will be appriciated.
Thanks, (14 Replies)
Discussion started by: varun22486
14 Replies
8. Shell Programming and Scripting
I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies
9. Shell Programming and Scripting
Hey guys,
I have a little problem,
Let's say I create this script :
#!/bin/sh
nfo_file="/home/admin/info.nfo"
echo "▒▒█ Hello █▒▒" > $nfo_fileIt seems to be okay :
cat /home/admin/info.nfo
▒▒█ Hello █▒▒file -bi /home/admin/info.nfo
text/plain; charset=utf-8But when I open it in a... (7 Replies)
Discussion started by: antoinelomb
7 Replies
LEARN ABOUT DEBIAN
template::provider::encoding
Template::Provider::Encoding(3pm) User Contributed Perl Documentation Template::Provider::Encoding(3pm)
NAME
Template::Provider::Encoding - Explicitly declare encodings of your templates
SYNOPSIS
use Template::Provider::Encoding;
use Template::Stash::ForceUTF8;
use Template;
my $tt = Template->new(
LOAD_TEMPLATES => [ Template::Provider::Encoding->new ],
STASH => Template::Stash::ForceUTF8->new,
);
# Everything should be Unicode
# (but you can pass UTF-8 bytes as well, thanks to Template::Stash::ForceUTF8)
my $author = "x{5bae}x{5ddd}";
# this will emit Unicode flagged string to STDOUT. You might
# probably want to binmode(STDOUT, ":encoding($enccoding)")
# before process() call
$tt->process($template, { author => $author });
# in your templates
[% USE encoding 'utf-8' -%]
My name is [% author %]. { ... whatever UTF-8 bytes }
DESCRIPTION
Template::Provider::Encoding is a Template Provider subclass to decode template using its declaration. You have to declare encoding of the
template in the head (1st line) of template using (fake) encoding TT plugin. Otherwise the template is handled as utf-8.
[% USE encoding 'utf-8' %]
Here comes utf-8 strings with [% variable %].
DIFFERNCE WITH OTHER WAYS
UNICODE option and BOM
Recent TT allows "UNICODE" option to Template::Provider and by adding it Provider scans BOM (byte-order mark) to detect UTF-8/UTF-16
encoded template files. This module does basically the same thing in a different way, but IMHO adding BOM to template files is a little
painful especially for non-programmers.
Template::Provider::Encode
Template::Provider::Encode provides a very similar way to detect Template file encodings and output the template into various encodings.
This module doesn't touch output encoding of the template and instead it emits valid Unicode flagged string. I think the output encoding
conversion should be done by other piece of code, especially in the framework.
This module doesn't require you to specify encoding in the code, nor doesn't guess encodings. Instead it forces you to put "[% USE encoding
'foo-bar' %]" in the top of template files, which is explicit and, I think, is a good convention.
AUTHOR
Tatsuhiko Miyagawa <miyagawa@bulknews.net>
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
SEE ALSO
Template::Stash::ForceUTF8, Template::Provider::Encode
perl v5.12.3 2007-08-01 Template::Provider::Encoding(3pm)