Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Issue with UTF-8 BOM character in text file Post 302657825 by jawsnnn on Monday 18th of June 2012 11:32:50 AM
Old 06-18-2012
Issue with UTF-8 BOM character in text file

Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad.

Problem is: When saving the files to our windows machine in UTF-8 format, notepad inserts BOM characters
Code:
 

at the beginning of the text. ETL tools such as Informatica have no problem reading the files with this character, but unfortunately we validate the data before loading it, and this validation is performed by a shell script. Since the first field of the first line is no longer a valid field, the shell script fails.

One solution we tried was removing the BOM characters from the text file in Unix before processing it. This worked fine as far as the shell script was concerned, but then the ETL tool failed to read the UTF characters in the file.

My questions:
1. Is there a way to remove this issue at root, i.e. can I find a way to remove the BOM character in notepad, while saving it to UTF-8 format.
2. Can some other tool help me out to make this change instead of notepad... DOS maybe?
3. What are my options in Unix? Is there a way to remove the BOM characters without "breaking" the file in Unix? There must be, because I have seen a lot of UTF files without BOM being processed just fine earlier. I just don't know how to do it.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Finding files with UTF-8 BOM

Hi, there: I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task. I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
Discussion started by: kotoponus
0 Replies

2. UNIX for Dummies Questions & Answers

need to read 3° character from a text file

Hi, I need a script to read the n° character from a text file. eg: if the text file contains the line "123456" ,I nedd a command to display the number 4, as an example. I tried with awk and printf but it seems only works with words separated with spaces, but in this case I have only one word... (15 Replies)
Discussion started by: piltrafa
15 Replies

3. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

4. UNIX for Dummies Questions & Answers

Deleting all instances of a certain character from a text file

In my command prompt I did: sed 's/\://' mytextfile > newtextfile But it only deleted the first instance of : in each line when some lines have multiple : appearing in each one. How can I delete all the : from the entire file? (1 Reply)
Discussion started by: guitarscn
1 Replies

5. Shell Programming and Scripting

read the text file and print the content character by character..

hello all i request you to give the solution for the following problem.. I want read the text file.and print the contents character by character..like if the text file contains google means..i want to print g go goo goog googl google like this Using unix Shell scripting... without using... (1 Reply)
Discussion started by: samupnl
1 Replies

6. Shell Programming and Scripting

post-Adding character for a text file

#################################################################### #NAME SL.NO TITLE SAL #################################################################### |RAGAV S S | 12358 | SALES EXECUTIVE| | 25000 |RAJU R B | 64253 | SALES EXECUTIVE| ... (5 Replies)
Discussion started by: manas_ranjan
5 Replies

7. Shell Programming and Scripting

How to modify character to UTF-8 in shell script?

I have a shell script running to load some data from a text file to database. Text file contains some non-ASCII characters like ü. How can i convert these characters to UTF-8 codes before loading to DB. (5 Replies)
Discussion started by: vel4ever
5 Replies

8. Shell Programming and Scripting

new line after n'th character in text file

Gurus, I have a text file having only one row having following data. 640.0800 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 640.2324 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 -999.25 640.3848 ... (5 Replies)
Discussion started by: Amit.saini333
5 Replies

9. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

10. UNIX for Advanced & Expert Users

UTF-8,16,32 character lengths using awk

Hi All, I am trying to obtain count of characters using awk, but "length" function returns a value of 1 for 2-byte or 3-byte characters as well unlike wc -c command. I have tried to use the below commands within awk function, but it does not seem to work { cmd="wc -c "stringtocheck ( cmd )... (6 Replies)
Discussion started by: tostay2003
6 Replies
Net::SFTP::Foreign::Constants(3pm)			User Contributed Perl Documentation			Net::SFTP::Foreign::Constants(3pm)

NAME
Net::SFTP::Foreign::Constants - Constant definitions for Net::SFTP::Foreign SYNOPSIS
use Net::SFTP::Foreign::Constants qw(:tag SSH2_FILEXFER_VERSION); print "Protocol version is ", SSH2_FILEXFER_VERSION; DESCRIPTION
Net::SFTP::Foreign::Constants provides a list of exportable SFTP constants: for SFTP messages and commands, for file-open flags, for status messages, etc. Constants can be exported individually, or in sets identified by tag names. Net::SFTP::Foreign::Constants provides values for all of the constants listed in the SFTP protocol version 3 draft; the only thing to note is that the constants are listed with the prefix "SSH2_" instead of "SSH_". So, for example, to import the constant for the file-open command, you would write: use Net::SFTP::Foreign::Constants qw( SSH2_FXP_OPEN ); TAGS
As mentioned above, constants can either be imported individually or in sets grouped by tag names. The tag names are: :fxp Imports all of the "SSH2_FXP_*" constants: these are the constants used in the messaging protocol. :flags Imports all of the "SSH2_FXF_*" constants: these are constants used as flags sent to the server when opening files. :att Imports all of the "SSH2_FILEXFER_ATTR_*" constants: these are the constants used to construct the flag in the serialized attributes. The flag describes what types of file attributes are listed in the buffer. :status Imports all of the "SSH2_FX_*" constants: these are constants returned from a server "SSH2_FXP_STATUS" message and indicate the status of a particular operation. :error Imports all the "SFTP_ERR_*" constants used to represent high level errors: "SFTP_ERR_LOCAL_ALREADY_EXISTS", "SFTP_ERR_LOCAL_CHMOD_FAILED", "SFTP_ERR_LOCAL_OPEN_FAILED", "SFTP_ERR_LOCAL_READ_ERROR", "SFTP_ERR_LOCAL_STAT_FAILED", "SFTP_ERR_LOCAL_UTIME_FAILED", "SFTP_ERR_LOCAL_WRITE_FAILED", "SFTP_ERR_REMOTE_BAD_PACKET_SEQUENCE", "SFTP_ERR_REMOTE_BAD_PERMISSIONS", "SFTP_ERR_REMOTE_BAD_TIME", "SFTP_ERR_REMOTE_BLOCK_TOO_SMALL", "SFTP_ERR_REMOTE_CLOSE_FAILED", "SFTP_ERR_REMOTE_FSETSTAT_FAILED", "SFTP_ERR_REMOTE_FSTAT_FAILED", "SFTP_ERR_REMOTE_LSTAT_FAILED", "SFTP_ERR_REMOTE_MKDIR_FAILED", "SFTP_ERR_REMOTE_OPENDIR_FAILED", "SFTP_ERR_REMOTE_OPEN_FAILED", "SFTP_ERR_REMOTE_READDIR_FAILED", "SFTP_ERR_REMOTE_READ_FAILED", "SFTP_ERR_REMOTE_REALPATH_FAILED", "SFTP_ERR_REMOTE_REMOVE_FAILED", "SFTP_ERR_REMOTE_RENAME_FAILED", "SFTP_ERR_REMOTE_RMDIR_FAILED", "SFTP_ERR_REMOTE_READLINK_FAILED", "SFTP_ERR_REMOTE_SYMLINK_FAILED", "SFTP_ERR_REMOTE_SETSTAT_FAILED", "SFTP_ERR_REMOTE_STAT_FAILED", "SFTP_ERR_REMOTE_WRITE_FAILED" and "SFTP_ERR_REMOTE_HARDLINK_FAILED". Note: these constants are not defined on the SFTP draft. :ext Import all the "SSH2_FXE_*" constants: there are the constants defined for usage with SFTP extensions. Currently, these are supported: "SSH2_FXE_STATVFS_ST_RDONLY", "SSH_FXE_STATVFS_ST_NOSUID". There is one constant that does not fit into any of the tag sets: "SSH2_FILEXFER_VERSION", which holds the value of the SFTP protocol implemented by Net::SFTP::Foreign. AUTHOR &; COPYRIGHTS Please see the Net::SFTP::Foreign manpage for author, copyright, and license information. perl v5.14.2 2012-05-11 Net::SFTP::Foreign::Constants(3pm)
All times are GMT -4. The time now is 08:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy