Usually all the file transferred should be binary format so that nothing can be changed so that we don’t get any unknown characters
Let us consider the file name as Orgdata_UTF8.txt then output file as Orgdata.txt
As some systems add the BOM by default and some systems doesn’t add the BOM based on the operating systems and it is also the similar case with UTF-16LE format as it is sometimes recognised as UTF-16 and some as UTF-16LE based on versions and use them as needed.
Adding BOM manually
Create a new file as below Orgdata.txt and check the file type using file command to confirm that it is UTF-16LE format and then convert it as below
Use the hex coder to check if you have got the desired result of ff fe or not. This result varies depending on the type of hexdump used.
In reality both of them are same as one of them shows the reversing output.
While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format.
I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)
I'm using shell scripting in Applescript. When searching a file with the ANSEL character set (for GEDCOM files) using (grep '1 CHAR ANSEL' filepath) gives the expected result. When searching a UNICODE formatted file (utf-16), searching for text known to exist in the file using (grep '1 CHAR... (4 Replies)
Hi,
I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Hi all,
At present a file from AS400 system is being FTPed to an AIX system.
Now, a similar file needs to be sent from our Unix box (Solaris)
Is there any tool available which does the conversion in Unix from UTF-8 to EBCDIC?
Any suggestions/ pointers are really appreciated.
Thanks,... (4 Replies)
Hello,
I am trying to convert a 7bit ASCII file to UTF-8.
I have used iconv before though it can't recognize it for some reason and says unknown file encoding.
When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file
It still... (2 Replies)
Sometimes we recieve some excel files containing French/Japanese characters over the mail, and these files are manually transferred to the server by using SFTP (security is not a huge concern here). The data is changed to text format before transferring it using Notepad.
Problem is: When saving... (4 Replies)
Hello all
i have utf-8 file that i try to convert to WINDOWS-1251 on linux
without any success
the file name is utf-8 when i try to do :
file -bi test.txt
it gives me :
text/plain; charset=utf-8
when i try to convert the file i do :
/usr/bin/iconv -f UTF-8 -t WINDOWS-1251 test.txt >... (1 Reply)
Hi,
I need to run a SQL which check for special UTF char in DB. When I try to copy that in UNIX file it changes it to some wierd chat. How can in retain the UTF chars in my script?
e.g. ο|π|ρ|σ|τ|υ|φ|χ|ψ
Any help will be appriciated.
Thanks, (14 Replies)
I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Dears,
I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text... (4 Replies)
Discussion started by: JeanM-1
4 Replies
LEARN ABOUT REDHAT
mac2unix
dos2unix(1) General Commands Manual dos2unix(1)NAME
dos2unix - DOS/MAC to UNIX text file format converter
SYNOPSYS
dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]
Options:
[-hkqV] [--help] [--keepdate] [--quiet] [--version]
DESCRIPTION
This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.
OPTIONS
The following options are available:
-h --help
Print online help.
-k --keepdate
Keep the date stamp of output file same as input file.
-q --quiet
Quiet mode. Suppress all warning and messages.
-V --version
Prints version information.
-c --convmode convmode
Sets conversion mode. Simulates dos2unix under SunOS.
-o --oldfile file ...
Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.
-n --newfile infile outfile ...
New file mode. Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
used or you WILL lost your files.
EXAMPLES
Get input from stdin and write output to stdout.
dos2unix
Convert and replace a.txt. Convert and replace b.txt.
dos2unix a.txt b.txt
dos2unix -o a.txt b.txt
Convert and replace a.txt in ASCII conversion mode. Convert and replace b.txt in ISO conversion mode. Convert c.txt from Mac to Unix
ascii format.
dos2unix a.txt -c iso b.txt
dos2unix -c ascii a.txt -c iso b.txt
dos2unix -c mac a.txt b.txt
Convert and replace a.txt while keeping original date stamp.
dos2unix -k a.txt
dos2unix -k -o a.txt
Convert a.txt and write to e.txt.
dos2unix -n a.txt e.txt
Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.
dos2unix -k -n a.txt e.txt
Convert and replace a.txt. Convert b.txt and write to e.txt.
dos2unix a.txt -n b.txt e.txt
dos2unix -o a.txt -n b.txt e.txt
Convert c.txt and write to e.txt. Convert and replace a.txt. Convert and replace b.txt. Convert d.txt and write to f.txt.
dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt
DIAGNOSTICS BUGS
The program does not work properly under MSDOS in stdio processing mode. If you know why is that so, please tell me.
AUTHORS
Benjamin Lin - <blin@socs.uts.edu.au>
Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>
MISCELLANY
Tested environment:
Linux 1.2.0 with GNU C 2.5.8
SunOS 4.1.3 with GNU C 2.6.3
MS-DOS 6.20 with Borland C++ 4.02
Suggestions and bug reports are welcome.
SEE ALSO unix2dos(1)mac2unix(1)1995.03.31 dos2unix v3.0 dos2unix(1)