Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to modify character to UTF-8 in shell script?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to modify character to UTF-8 in shell script?
# 1  
Old 01-11-2012
How to modify character to UTF-8 in shell script?
I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB.
# 2  
Old 01-11-2012
Try iconv command:
iconv(1) - Linux man page

Albert.
# 3  
Old 01-11-2012
Below is the command i am using for conversion to UTF-8

Code:
iconv -f ISO-8859-1 -t UTF-8 /loc/test.txt /loc/test1.txt

But its not sending the output to the test1.txt file.

---------- Post updated at 09:57 AM ---------- Previous update was at 08:18 AM ----------

iconv is working fine after modified my code just like the below one,

Code:
iconv -f ISO-8859-1 -t UTF-8 /loc/test.txt > /loc/test1.txt

But how to have the input file as the output file?

Code:
iconv -f ISO-8859-1 -t UTF-8 /loc/test.txt > /loc/test.txt

For the above code test.txt file becomes empty

---------- Post updated at 09:58 AM ---------- Previous update was at 09:57 AM ----------
# 4  
Old 01-11-2012
You do it by moving the new file over the old file when you're done, just like you'd do with awk, sed, and a million other shell commands. But check the output before you do so. Moving bad output overtop of good input is a big problem.
# 5  
Old 01-11-2012
Yes. You are correct but bad output of the new file will cause a problem here.
# 6  
Old 01-11-2012
Quote:
Originally Posted by vel4ever
But how to have the input file as the output file?
Code:
iconv -f ISO-8859-1 -t UTF-8 /loc/test.txt > /loc/test.txt

For the above code test.txt file becomes empty
Yes, first convert file, then rename it.
Code:
iconv -f ISO-8859-1 -t UTF-8 /loc/test.txt > /loc/test.tmp; mv loc/test.tmp /loc/test.txt

You must be careful, because you'll overwrite original file. First, make a backup. If conversion doesn't work properly, you could loose original file.
So, unless original file is kept safe, I think it's better split above expression in two: convert and rename.

Albert.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

UTF-8,16,32 character lengths using awk

Hi All, I am trying to obtain count of characters using awk, but "length" function returns a value of 1 for 2-byte or 3-byte characters as well unlike wc -c command. I have tried to use the below commands within awk function, but it does not seem to work { cmd="wc -c "stringtocheck ( cmd )... (6 Replies)
Discussion started by: tostay2003
6 Replies

2. UNIX for Dummies Questions & Answers

Modify Column Data using Shell Script

HI Guys, Input :- P081 wr1 12p0d5: 22.8 P081 wr1 12p2d18: 23.1 P149 wr1 1pxcud6/port_0_dev_7: 20.4 P149 wr1 1pxcud4/port_1_dev_10: 22.4 OutputP081 wr1 120 22.8 P081 wr1 122 23.1 P149 wr1 10 20.4 P149 wr1 11 22.4 In in First two line delete p and after d untill : In Last two line... (4 Replies)
Discussion started by: pareshkp
4 Replies

3. Shell Programming and Scripting

Modify XML tag using shell script

Hi All Need some help with a unix shell script. I have a XML file as shown below: <Root> <Service> <endPoint type="SOAP" protocol="http"> <provider>ABCD</provider> <urlRewrite>/service/xyz/getAccountDetails</urlRewrite> <timeout>30</timeout> </endPoint> </Service> <Service> <endPoint... (3 Replies)
Discussion started by: abhilwa
3 Replies

4. Shell Programming and Scripting

Modify cal command in shell script

Plz help me a) To display on the screen the sorted output of "who" along with the total no. of users. b) the same output (except the no. of users) should be in file FILE1. (2 Replies)
Discussion started by: shivasaini
2 Replies

5. UNIX for Dummies Questions & Answers

Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad. Problem is: When saving... (4 Replies)
Discussion started by: jawsnnn
4 Replies

6. Shell Programming and Scripting

Shell script to modify file in several directories

Hi, I want a script shell to automate modifying httpd.conf file for several instances of apache, save httpd.file before changing it, after modifying it and then restart apache. - Replace ServerRoot "xxxx" by ServerRoot "yyyy" of all directories : "... (4 Replies)
Discussion started by: bras39
4 Replies

7. Shell Programming and Scripting

Modify text file using shell script

Hi, I have a text file which is following format - COL VAL ABC 1 ABC 2 ABC 3 ABC 4 ABC 5 My requirement is to search for a particular value (provided by user) in the file and comment the previous entries including that as well. E.g. If I search for number 3, then the output... (6 Replies)
Discussion started by: bhupinder08
6 Replies

8. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

9. Shell Programming and Scripting

how to modify a file using shell script

Hi, i am using SuonOS and ksh. i need to add data into a file(s.txt) using a shell script. i have to pass 3 parameters and these 3 paramaters should add into the file at end of the file. File s.txt is look like, --------------------------------- column1|column2|column3 ... (1 Reply)
Discussion started by: syamkp
1 Replies

10. Shell Programming and Scripting

Is it possible in a shell script to modify itself ??

We are running a quiz and the data collected from the quiz is submitted to the database. My requirement is to write a shell script to get these submitted records. I should be able to run this shell script at any time and the records it returns should be the ones submitted after the script was... (5 Replies)
Discussion started by: sehgalniraj
5 Replies
Login or Register to Ask a Question