Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad.
Problem is: When saving the files to our windows machine in UTF-8 format, notepad inserts BOM characters
Code:

at the beginning of the text. ETL tools such as Informatica have no problem reading the files with this character, but unfortunately we validate the data before loading it, and this validation is performed by a shell script. Since the first field of the first line is no longer a valid field, the shell script fails.
One solution we tried was removing the BOM characters from the text file in Unix before processing it. This worked fine as far as the shell script was concerned, but then the ETL tool failed to read the UTF characters in the file.
My questions:
1. Is there a way to remove this issue at root, i.e. can I find a way to remove the BOM character in notepad, while saving it to UTF-8 format.
2. Can some other tool help me out to make this change instead of notepad... DOS maybe?
3. What are my options in Unix? Is there a way to remove the BOM characters without "breaking" the file in Unix? There must be, because I have seen a lot of UTF files without BOM being processed just fine earlier. I just don't know how to do it.
Unfortunately downloading+installing tools is not an option (Controlled environment at work means I would have to cut through at least half a dozen people to get something as basic as puTTY installed on my system).
Question regarding your first point: Wouldn't transferring the file in ASCII mode incorrectly transmit the UTF(japanese/spanish) characters? Also, are you suggesting skipping the "copy data from excel - paste to notepad - save to UTF8 format" step? That might again not be possible in my current situation, unless I find a way to convert the data to a proper UTF8 text file without BOM characters using a pre-installed application.
Hi All,
I am trying to obtain count of characters using awk, but "length" function returns a value of 1 for 2-byte or 3-byte characters as well unlike wc -c command.
I have tried to use the below commands within awk function, but it does not seem to work
{
cmd="wc -c "stringtocheck
( cmd )... (6 Replies)
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
Gurus,
I have a text file having only one row having following data.
640.0800 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 640.2324 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 640.3848 ... (5 Replies)
I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB. (5 Replies)
####################################################################
#NAME SL.NO TITLE SAL
####################################################################
|RAGAV S S | 12358 | SALES EXECUTIVE| | 25000
|RAJU R B | 64253 | SALES EXECUTIVE| ... (5 Replies)
hello all
i request you to give the solution for the following problem..
I want read the text file.and print the contents character by character..like if the text file contains google means..i want to print
g
go
goo
goog
googl
google
like this Using unix Shell scripting...
without using... (1 Reply)
In my command prompt I did:
sed 's/\://' mytextfile > newtextfile
But it only deleted the first instance of : in each line when some lines have multiple : appearing in each one. How can I delete all the : from the entire file? (1 Reply)
Hi,
I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Hi,
I need a script to read the n° character from a text file.
eg: if the text file contains the line "123456" ,I nedd a command to display the number 4, as an example.
I tried with awk and printf but it seems only works with words separated with spaces, but in this case I have only one word... (15 Replies)
Hi, there:
I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task.
I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
