Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Determing the encoding of a file Post 302751859 by MIA651 on Friday 4th of January 2013 03:45:10 PM
Old 01-04-2013
Quote:
Originally Posted by DGPickett
Well, utf-8 and unicode have a pattern in their encoding. The dd command has an ebcdic decoder I have used. Might it be from big blue land?

Googling around the subject, one suggests file -i, another mentions enca enca(1): detect/convert encoding of text files - Linux man page and for solaris, auto_ef. There is a 'chardet' python based tool.
Yes tried file -i and it tells me it is a regular file. By big blue land, I assume you mean IBM? If that's the case yes I am using an AIX machine therefore auto_ef and enca are unrecognized commands. Yet to try chardet...I'll have to dig deeper. Thanks though!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

File encoding in Unix

1. I have a shell script which creates a file using cat command. How can i find what encoding the file follows (e.g. UTF8, ANSI)? 2. I want to convert that file to PC-ANSI format. How can i achieve that? I am using HP-Unix. (6 Replies)
Discussion started by: ssmallya
6 Replies

2. Shell Programming and Scripting

get the file encoding

Hello! The system is AIX 5.3 Give please command or script to get the file encoding Thanks (2 Replies)
Discussion started by: vinment
2 Replies

3. AIX

get the file encoding

Hello! The system is AIX 5.3 Give please command or script to get the file encoding (1 Reply)
Discussion started by: vinment
1 Replies

4. Shell Programming and Scripting

Dymically determing the number of check list in Zenity, How?

hi, In my project i cannot determine the number of check list initially... I will know dynamically during execution... so How to specify the number of check list dynamically in zenity Waiting for your precious Answer..... (1 Reply)
Discussion started by: shivarajM
1 Replies

5. Shell Programming and Scripting

Cygwin vi XML file encoding problem

Hi, I have got a zip (binary) file transferred from MacOS (thus it has additional __MACOSX directory packed inside). On extracting this zip, there are few *.xml files available. When I opened this *.xml file in vim editor using Cygwin (on windows) the editor displayed in the bottom. I tried... (4 Replies)
Discussion started by: royalibrahim
4 Replies

6. HP-UX

how to find the character encoding of a file in hp_ux

how to find the character encoding of a file in hp_ux (1 Reply)
Discussion started by: alokjyotibal
1 Replies

7. Shell Programming and Scripting

How to find the file encoding and updating the file encoding?

Hi, I am beginner to Unix. My requirement is to validate the encoding used in the incoming file(csv,txt).If it is encoded with UTF-8 format,then the file should remain as such otherwise i need to chnage the encoding to UTF-8. Please advice me how to proceed on this. (7 Replies)
Discussion started by: cnraja
7 Replies

8. HP-UX

Determing size of swap space

Hi Experts, Need your advise in determining the size of swap space in of the new HP-Ux server. Server is having 32G of physical memory. Ideally what amout of physical memory should be allocated as a swap space? Following document from HP suggests to have minimum swap space... (2 Replies)
Discussion started by: sai_2507
2 Replies

9. Solaris

View file encoding then change encoding.

Hi all!! I´m using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . I´m expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies

10. Shell Programming and Scripting

How to know file encoding?

how can i know what format a file is * example: UTF-8 ANSI UCS2 i am in a... (8 Replies)
Discussion started by: tricampeon81
8 Replies
auto_ef(3EXT)						    Extended Library Functions						     auto_ef(3EXT)

NAME
auto_ef, auto_ef_file, auto_ef_str, auto_ef_free, auto_ef_get_encoding, auto_ef_get_score - auto encoding finder functions SYNOPSIS
cc [ flag ... ] file... -lauto_ef [ library... ] #include <auto_ef.h> size_t auto_ef_file(auto_ef_t **info, const char *filename, int flags); size_t auto_ef_str(auto_ef_t **info, const char *buffer, size_t bufsize, int flags); void auto_ef_free(auto_ef_t *info); char *auto_ef_get_encoding(auto_ef_t info); double auto_ef_get_score(auto_ef_t info); DESCRIPTION
Auto encoding finder provides functions that find the encoding of given file or string. The auto_ef_file() function examines text in the file specified with filename and returns information on possible encodings. The info argument is a pointer to a pointer to an auto_ef_t, the location at which the pointer to the auto_ef_t array is stored upon return. The flags argument specifies the level of examination. Currently only one set of flags, exclusive each other, is available: AE_LEVEL_0, AE_LEVEL_1, AE_LEVEL_2, and AE_LEVEL_3. The AE_LEVEL_0 level is fastest but the result can be less accurate. The AE_LEVEL_3 level produces best result but can be slow. If the flags argument is unspecified, the default is AE_LEVEL_0. When another flag or set of flags are defined in the future, use the inclusive-bitwise OR operation to specify multiple flags. Information about encodings are stored in data typeauto_ef_t in the order of possibility with the most possible encoding stored first. To examine the information, use the auto_ef_get_encoding() and auto_ef_get_score() access functions. For a list of encodings with which auto_ef_file() can examine text, see auto_ef(1). If auto_ef_file() cannot determine the encoding of text, it returns 0 and stores NULL at the location pointed by info. The auto_ef_get_encoding() function returns the name of the encoding. The returned string is vaild until until the location pointed to by info is freed with auto_ef_free(). Applications should not use free(3C) to free the pointer returned by auto_ef_get_encoding(). The auto_ef_get_score() function returns the score of this encoding in the range between 0.0 and 1.0. The auto_ef_str() function is identical to auto_ef_file(), except that it examines text in the buffer specified by buffer with a maximum size of bufsize bytes, instead of text in a file. The auto_ef_free() function frees the area allocated by auto_ef_file() or by auto_ef_str(), taking as its argument the pointer stored at the location pointed to by info. RETURN VALUES
Upon successful completion, the auto_ef_file() and auto_ef_str() functions return the number of possible encodings for which information is stored. Otherwise, -1 is returned. The auto_ef_get_encoding() function returns the string that stores the encoding name. the auto_ef_get_score() function returns the score value for encoding the name with the examined text data. ERRORS
The auto_ef_file() and auto_ef_str() will fail if: EACCES Search permission is denied on a component of the path prefix, the file exists and the permissions specified by mode are denied, the file does not exist and write permission is denied for the parent directory of the file to be created, or libauto_ef cannot find the internal hashtable. EINTR A signal was caught during the execution. ENOMEM Failed to allocate area to store the result. EMFILE Too many files descriptors are currently open in the calling process. ENFILE Too many files are currently open in the system. EXAMPLES
Example 1: Specify the array index to examine stored information. Since auto_ef_file() stores the array whose elements hold information on each possible encoding, the following example specifies the array index to examine the stored information. #include <auto_ef.h> auto_ef_t *array_info; size_t number; char *encoding; number = auto_ef_file(&array_info, filename, flags); encoding = auto_ef_get_encoding(array_info[0]); auto_ef_free(array_info); ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Interface Stability |Stable | +-----------------------------+-----------------------------+ |MT-Level |MT-Safe | +-----------------------------+-----------------------------+ SEE ALSO
auto_ef(1), libauto_ef(3LIB), attributes(5) SunOS 5.10 22 Sep 2003 auto_ef(3EXT)
All times are GMT -4. The time now is 12:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy