Sponsored Content
Top Forums Shell Programming and Scripting Unable to identify the special characters beyond the range of "[\x80-\xFF]" Post 302959476 by Abhijit Sen on Tuesday 3rd of November 2015 09:42:57 AM
Old 11-03-2015
Hi All,

I am able to convert the UTF-16 characters to UTF-8 characters.

But when it is getting converted(using iconv) as UTF-16 is 2 byte value , hence the file alignment is getting changed. Is there any way to fix this to resolve the alignment problem.

My file contains only UTF-8 value which takes 1 byte. And each line it can have only limited number of UTF-8 chracters. But during conversion few of the UTF-8 characters are getting pushed to next line which is altering the file alignment.

I have tried to use recode command but that is not working. Any help would be greatly appreciated.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to split special characters "|" using awk

Hi friends I need to splict special character "|" here. Here is my script which giving error LINE=INVTRAN|cd /home/msgGoogle TraxFolderType=`awk -F"|" '{print $1}' $LINE` filePath=`awk -F"|" '{print $2}' $LINE` echo "TraxFolderType: "$TraxFolderType echo "filePath :"$filePath ... (3 Replies)
Discussion started by: krishna9
3 Replies

2. Shell Programming and Scripting

How to remove "New line characters" and "spaces" at a time

Dear friends, following is the output of a script from which I want to remove spaces and new-line characters. Example:- Line1 abcdefghijklmnopqrstuvwxyz Line2 mnopqrstuvwxyzabcdefghijkl Line3 opqrstuvwxyzabcdefdefg Here in above example, at every starting line there is a “tab” &... (4 Replies)
Discussion started by: anushree.a
4 Replies

3. SuSE

VMDB Failure" followed by "Unable to open snapshot file"

keep getting an error when I try to revert to a snapshot: "VMDB Failure" followed by "Unable to open snapshot file" Im using vmware server 1.0.4, host OS is windows xp and guest OS is SLES. Is there anything I can do to recover the snapshot or am I in trouble!?!?! (0 Replies)
Discussion started by: s_linux
0 Replies

4. Shell Programming and Scripting

Question about special variables: "-" and "$_"

both ksh/bash support this 2 special variables, Is there any document for reference? 1) "-" is $OLDPWD 2) "$_" is last argument of previous command. (4 Replies)
Discussion started by: honglus
4 Replies

5. Shell Programming and Scripting

How to print range of lines using sed when pattern has special character "["

Hi, My input has much more lines, but few of them are below pin(IDF) { direction : input; drc_pinsigtype : signal; pin(SELDIV6) { direction : input; drc_pinsigtype : ... (3 Replies)
Discussion started by: nehashine
3 Replies

6. Shell Programming and Scripting

if [ "variable" = "numerical-range" ]; then

been a while so i'm a bit rusty and need a little help. writing a script that needs to compare $EXECHOST(a number) against a numerical range and then set a value. below isn't working but should give you folks an idea of my goal: if ; then echo "This is a 32B machine, exiting..." if ;... (4 Replies)
Discussion started by: crimso
4 Replies

7. Shell Programming and Scripting

Need HELP with AWK split. Need to check for "special characters" in string before splitting the file

Hi Experts. I'm stuck with the below AWK code where i'm trying to move the records containing any special characters in the last field to a bad file. awk -F, '{if ($NF ~ /^|^/) print >"goodfile";else print >"badfile"}' filename sample data 1,abc,def,1234,A * 2,bed,dec,342,* A ... (6 Replies)
Discussion started by: shell_boy23
6 Replies

8. Shell Programming and Scripting

finding the strings beween 2 characters "/" & "/" in .txt file

Hi all. I have a .txt file that I need to sort it My file is like: 1- 88 chain0 MASTER (FF-TE) FFFF 1962510 /TCK T FD2TQHVTT1 /jtagc/jtag_instreg/updateinstr_reg_1 dff1 (TI,SO) 2- ... (10 Replies)
Discussion started by: Behrouzx77
10 Replies

9. UNIX for Dummies Questions & Answers

PuTTY displaying "special" characters

I'm not really sure which forum this question should go into, so I'm posting it here. I work with AIX and RHEL systems using PuTTY (Release 0.60_q1.129) from a Windows 7 workstation. Some of the files we get from z/OS use "special" characters as delimiters. These characters include Hex 18... (7 Replies)
Discussion started by: derndingle
7 Replies
PPI::Token::BOM(3)					User Contributed Perl Documentation					PPI::Token::BOM(3)

NAME
PPI::Token::BOM - Tokens representing Unicode byte order marks INHERITANCE
PPI::Token::BOM isa PPI::Token isa PPI::Element DESCRIPTION
This is a special token in that it can only occur at the beginning of documents. If a BOM byte mark occurs elsewhere in a file, it should be treated as PPI::Token::Whitespace. We recognize the byte order marks identified at this URL: <http://www.unicode.org/faq/utf_bom.html#BOM> UTF-32, big-endian 00 00 FE FF UTF-32, little-endian FF FE 00 00 UTF-16, big-endian FE FF UTF-16, little-endian FF FE UTF-8 EF BB BF Note that as of this writing, PPI only has support for UTF-8 (namely, in POD and strings) and no support for UTF-16 or UTF-32. We support the BOMs of the latter two for completeness only. The BOM is considered non-significant, like white space. METHODS
There are no additional methods beyond those provided by the parent PPI::Token and PPI::Element classes. SUPPORT
See the support section in the main module AUTHOR
Chris Dolan <cdolan@cpan.org> COPYRIGHT
Copyright 2001 - 2011 Adam Kennedy. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.18.2 2011-02-25 PPI::Token::BOM(3)
All times are GMT -4. The time now is 01:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy