02-22-2013
Changing txt files to pure UTF-8
I have two Macs running 10.7.5.
We download .txt files from remote site to these local Macs using 'rsync -e ssh -avz...'.
The files on Mac1 are in the required format of pure UTF-8. The files on Mac2 are in UTF-8 (no BOM) which is wrong format for us; these formats are indicated using BBEdit.
Is there a Unix instruction to correct this or an instruction we can pass to rysnc?
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi, there:
I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task.
I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
Discussion started by: kotoponus
0 Replies
2. UNIX for Dummies Questions & Answers
Hi Guru's,
I need to create 3 files with the contents "ABC" using single command.
Iam using:
echo "ABC" > file1.txt file2.txt file3.txt
the above command is not working. pls help me...
With Regards / Ganapati (4 Replies)
Discussion started by: ganapati
4 Replies
3. Solaris
Dear experts,
In a directory i have both *.TXT and *.txt files. I have a script-
for file in `ls *.txt`; do
mv $file /tmp/$file
How to list both *.txt and*.TXT file in one command so that script will move both .txt or .TXT whatever it find.
br//purple (4 Replies)
Discussion started by: thepurple
4 Replies
4. Shell Programming and Scripting
Hi all,
I currently use this code to delete text from file names when they are dropped in a folder.
find . -type f | while read i;do && mv "$i" "${i//2009abc/}" ;done
I have four versions of the code that delete the unwanted text if it has 2006, 2007, 2008 and 2009 with other standard... (4 Replies)
Discussion started by: Monkey Dean
4 Replies
5. Shell Programming and Scripting
HI All,
I am coding a shell script which will pick all the .csv files in a particular directoryand write it in to a .txt file, this .txt file i will use as a source in datastage for processing.
now after the processing is done I have to move and archive all the files in the .txt file to a... (5 Replies)
Discussion started by: subhasri_2020
5 Replies
6. Shell Programming and Scripting
So I have a python program that I run, which runs accordingly to options I have listed in a text file (ie user_prefs). Now there are many options listed in this user_prefs.txt, but the one of most interest to me is that of the file path of the time series.
I have over a hundred of these time... (8 Replies)
Discussion started by: Jimmyd24
8 Replies
7. Shell Programming and Scripting
Hi,
Below are the sample data for txt files.
txt file 1
Partnumber|catgroup_id
10001082|46016
10001093|4680
10001093|386003
10001093|463004
10003251|683
10003251|63005
10003252|463005
10003252|4683
10003260|463005
10003260|4683
10003264|4683
10003264|463005
13420000|67... (5 Replies)
Discussion started by: Ankita Talukdar
5 Replies
8. Linux
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies
9. UNIX for Dummies Questions & Answers
just discover this thread. It sounds perfect for me because I am following an online course on command line tools and some instruction ... seems not be replicable on my Terminal
Here is an example on a txt file with n rows each containing 3 word separated by a space.
a b c
d e f
....
... (5 Replies)
Discussion started by: mrMe
5 Replies
10. UNIX for Beginners Questions & Answers
Dears,
I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text... (4 Replies)
Discussion started by: JeanM-1
4 Replies
LEARN ABOUT DEBIAN
template::provider::encoding
Template::Provider::Encoding(3pm) User Contributed Perl Documentation Template::Provider::Encoding(3pm)
NAME
Template::Provider::Encoding - Explicitly declare encodings of your templates
SYNOPSIS
use Template::Provider::Encoding;
use Template::Stash::ForceUTF8;
use Template;
my $tt = Template->new(
LOAD_TEMPLATES => [ Template::Provider::Encoding->new ],
STASH => Template::Stash::ForceUTF8->new,
);
# Everything should be Unicode
# (but you can pass UTF-8 bytes as well, thanks to Template::Stash::ForceUTF8)
my $author = "x{5bae}x{5ddd}";
# this will emit Unicode flagged string to STDOUT. You might
# probably want to binmode(STDOUT, ":encoding($enccoding)")
# before process() call
$tt->process($template, { author => $author });
# in your templates
[% USE encoding 'utf-8' -%]
My name is [% author %]. { ... whatever UTF-8 bytes }
DESCRIPTION
Template::Provider::Encoding is a Template Provider subclass to decode template using its declaration. You have to declare encoding of the
template in the head (1st line) of template using (fake) encoding TT plugin. Otherwise the template is handled as utf-8.
[% USE encoding 'utf-8' %]
Here comes utf-8 strings with [% variable %].
DIFFERNCE WITH OTHER WAYS
UNICODE option and BOM
Recent TT allows "UNICODE" option to Template::Provider and by adding it Provider scans BOM (byte-order mark) to detect UTF-8/UTF-16
encoded template files. This module does basically the same thing in a different way, but IMHO adding BOM to template files is a little
painful especially for non-programmers.
Template::Provider::Encode
Template::Provider::Encode provides a very similar way to detect Template file encodings and output the template into various encodings.
This module doesn't touch output encoding of the template and instead it emits valid Unicode flagged string. I think the output encoding
conversion should be done by other piece of code, especially in the framework.
This module doesn't require you to specify encoding in the code, nor doesn't guess encodings. Instead it forces you to put "[% USE encoding
'foo-bar' %]" in the top of template files, which is explicit and, I think, is a good convention.
AUTHOR
Tatsuhiko Miyagawa <miyagawa@bulknews.net>
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
SEE ALSO
Template::Stash::ForceUTF8, Template::Provider::Encode
perl v5.12.3 2007-08-01 Template::Provider::Encoding(3pm)