Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Explain iconv command

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Explain iconv command
# 1  
Old 03-18-2019
Unicode Characters in Fixed Length file
I have a requirement to remove all non-ascii characters from a fixed length file. I used the below command which is removing special characters but somehow the total record length is being truncated to one space less. If it is a multi-byte string then many characters at the end are being truncated.

Code:
 cat non-ascii.txt | iconv -f utf8 -t ascii//TRANSLIT//IGNORE > ascii.txt

Last edited by eskay; 03-18-2019 at 04:22 PM..
# 2  
Old 03-18-2019
Does not surprise me...
iconv purpose is not to remove char when not in a supposed charset, but to convert one charset to another...
How can you assume non ascii char in UTF8 like you are doing?
And if UTF8, yes converted to ASCII record length will will not be the same ASCII is sure shorter...
# 3  
Old 03-18-2019
What is the output of the file command, i.e.

Code:
file non-ascii.txt

# 4  
Old 03-18-2019
UTF-8 Unicode text, with very long lines
# 5  
Old 03-18-2019
changing utf-8 to non-utf-8 may change the length as utf-8 has variable-length characters.

For that matter, not sure how much sense a 'fixed length' UTF-8 record makes. It can be N bytes and fewer characters.
This User Gave Thanks to Corona688 For This Post:
jim mcnamara
# 6  
Old 03-18-2019
But with iconv function Ñ is converted to N.. but I need to add a space at the end as it is a multi-byte character to not change the length. Is it possible?
# 7  
Old 03-18-2019
How are these records terminated?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Please help to explain the command

su - keibatch -c ""date ; /usr/local/kei/batch/apb/bin/JKEIKYK4140.sh -run "&$C$6&" WSUKE100201"" Not clear about : date ; /usr/local/kei/batch/apb/bin/JKEIKYK4140.sh -run "&$C$6&" WSUKE100201 Please help (2 Replies)
Discussion started by: honda_city
2 Replies

2. Shell Programming and Scripting

Can any one explain this sqlplus command?

Hi , i am new to unix i need a small clarification regarding this sqlplus -s $USER_NAME/$PASSWD@$ORA_SID<< EOF >> SQL_CONN_LOG.log In the above command what is the meaning of <<EOF>> Thanks, krishna. (2 Replies)
Discussion started by: rams_krishna
2 Replies

3. UNIX for Dummies Questions & Answers

Please explain this command?

Hi, I saw this. But I don't know why we need this? ls mydir > foo.txt ## I know what this will do, it will take the results and write to the file called foo.txt ls mydir > foo.txt 2>&1 ## Don't know why we need 2>&1 Thanks. (2 Replies)
Discussion started by: samnyc
2 Replies

4. Shell Programming and Scripting

Help with command iconv

I need to convert a utf16 file to utf8. When i use the iconv command to do so it gives an error saying invalid function. When I ran the iconv -l function it did not list the utf16 and utf8 as part of its internal table. Is there anyway I can add these encodings in the library? Is there any other... (3 Replies)
Discussion started by: gaun
3 Replies

5. Shell Programming and Scripting

Characterset conversion problem using iconv command

Hi Friends, I am not able to conver character set from UTF-8 to IBM-284 throwing an error "cannot open convertor" . Could you please help me how to get out of this error. Below command is working fine iconv -f ISO8859-15 -t UTF-8 fromfile.txt > tofile.txt But the below command is... (2 Replies)
Discussion started by: sivakumarl
2 Replies

6. UNIX for Dummies Questions & Answers

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (2 Replies)
Discussion started by: Shruthi8818
2 Replies

7. Shell Programming and Scripting

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (0 Replies)
Discussion started by: Shruthi8818
0 Replies

8. Shell Programming and Scripting

Please Explain me this command

find . -type f -ctime +3 -exec mv {} /somedirectory/ \; in particular "-ctime v/s -mtime" and "difference between +3 and -3" (5 Replies)
Discussion started by: Rambo
5 Replies

9. UNIX for Dummies Questions & Answers

Can anyone explain what this command is doing?

Specifically what is the purpose of sed? What is f? Why is the 'cp f $phonefile' line needed when the script ‘goes live'? Why might that two commands following sed be commented out at the present time ( i.e., during development)? Thanks in... (2 Replies)
Discussion started by: knp808
2 Replies

10. Shell Programming and Scripting

please explain the command

Hi all , please explain the following command : perl -e 'select(undef,undef,undef,.15)' Thanks and Regards Navatha (2 Replies)
Discussion started by: Navatha
2 Replies
Login or Register to Ask a Question