Sponsored Content
Full Discussion: Iconv on large files
Top Forums UNIX for Advanced & Expert Users Iconv on large files Post 302980683 by tostay2003 on Wednesday 31st of August 2016 06:53:49 PM
Old 08-31-2016
Iconv on large files

Hi All,

I am using iconv to convert huge files. the process is getting killed. I tried the option in below link

Iconv for large files
i.e

HTML Code:
iconv -f UCS-2 -t UTF-8 < inputfile.txt > outputfile.txt
However, the process still gets killed. We do not have iconv-chunk.

Can you please suggest anyway to process without explicitly splitting the inputfile by myself in the script.
 

10 More Discussions You Might Find Interesting

1. Programming

about iconv

I want to use iconv.h to convert some text to another charset. The code is below: #include <stdio.h> #include <stdlib.h> #include <iconv.h> int main() { iconv_t cd; char instr="汉字"; char *inbuf; char *outbuf; unsigned int insize=7; ... (4 Replies)
Discussion started by: yong
4 Replies

2. UNIX for Dummies Questions & Answers

large files?

How do we check 'large files' is enabled on a Unix box -- HP-UX B11.11 (2 Replies)
Discussion started by: ranj@chn
2 Replies

3. UNIX for Advanced & Expert Users

iconv and xmllint

Here is my question, volume of records processed : 5M ( approx ) Its basically very simple operation that am trying to do and I had achieved the output that am interested. What am looking for really is to improve the performance, an optimized way to do that. with respect to iconv, am... (3 Replies)
Discussion started by: matrixmadhan
3 Replies

4. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234 (7 Replies)
Discussion started by: ad23
7 Replies

5. Solaris

How to safely copy full filesystems with large files (10Gb files)

Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS. I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies

6. Shell Programming and Scripting

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (0 Replies)
Discussion started by: Shruthi8818
0 Replies

7. UNIX for Dummies Questions & Answers

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (2 Replies)
Discussion started by: Shruthi8818
2 Replies

8. AIX

Installing Converter files for iconv conversion

Hi Friends, I need to install two converter packages UTF-8_IBM-284 and UTF-8_IBM-500 in the location /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* . Could you please let me know how to get this converter files. Thanks in Advance, Siva. (1 Reply)
Discussion started by: sivakumarl
1 Replies

9. Shell Programming and Scripting

Help with command iconv

I need to convert a utf16 file to utf8. When i use the iconv command to do so it gives an error saying invalid function. When I ran the iconv -l function it did not list the utf16 and utf8 as part of its internal table. Is there anyway I can add these encodings in the library? Is there any other... (3 Replies)
Discussion started by: gaun
3 Replies

10. Shell Programming and Scripting

Iconv for large files

Hi I am using iconv command to convert the encoding of files. Below is the command used: iconv -f UCS-2 -t UTF-8 inputfile.txt> outputfile.txt The command is working fine for files less than 2GB. When I try converting the files of more than 2GB size I get an error as 'Can't open file'. I... (2 Replies)
Discussion started by: vrcr
2 Replies
Iconv(3)						User Contributed Perl Documentation						  Iconv(3)

NAME
Text::Iconv - Perl interface to iconv() codeset conversion function SYNOPSIS
use Text::Iconv; $converter = Text::Iconv->new("fromcode", "tocode"); $converted = $converter->convert("Text to convert"); DESCRIPTION
The Text::Iconv module provides a Perl interface to the iconv() function as defined by the Single UNIX Specification. The convert() method converts the encoding of characters in the input string from the fromcode codeset to the tocode codeset, and returns the result. Settings of fromcode and tocode and their permitted combinations are implementation-dependent. Valid values are specified in the system documentation; the iconv(1) utility should also provide a -l option that lists all supported codesets. Utility methods Text::Iconv objects also provide the following methods: retval() returns the return value of the underlying iconv() function for the last conversion; according to the Single UNIX Specification, this value indicates "the number of non-identical conversions performed." Note, however, that iconv implementations vary widely in the interpretation of this specification. This method can be called after calling convert(), e.g.: $result = $converter->convert("lorem ipsum dolor sit amet"); $retval = $converter->retval; When called before the first call to convert(), or if an error occured during the conversion, retval() returns undef. get_attr(): This method is only available with GNU libiconv, otherwise it throws an exception. The get_attr() method allows you to query various attributes which influence the behavior of convert(). The currently supported attributes are trivialp, transliterate, and discard_ilseq, e.g.: $state = $converter->get_attr("transliterate"); See iconvctl(3) for details. To ensure portability to other iconv implementations you should first check for the availability of this method using eval {}, e.g.: eval { $conv->get_attr("trivialp") }; if ($@) { # get_attr() is not available } else { # get_attr() is available } This method should be considered experimental. set_attr(): This method is only available with GNU libiconv, otherwise it throws an exception. The set_attr() method allows you to set various attributes which influence the behavior of convert(). The currently supported attributes are transliterate and discard_ilseq, e.g.: $state = $converter->set_attr("transliterate"); See iconvctl(3) for details. To ensure portability to other iconv implementations you should first check for the availability of this method using eval {}, cf. the description of set_attr() above. This method should be considered experimental. ERRORS
If the conversion can't be initialized an exception is raised (using croak()). Handling of conversion errors Text::Iconv provides a class attribute raise_error and a corresponding class method for setting and getting its value. The handling of errors during conversion depends on the setting of this attribute. If raise_error is set to a true value, an exception is raised; otherwise, the convert() method only returns undef. By default raise_error is false. Example usage: Text::Iconv->raise_error(1); # Conversion errors raise exceptions Text::Iconv->raise_error(0); # Conversion errors return undef $a = Text::Iconv->raise_error(); # Get current setting Per-object handling of conversion errors As an experimental feature, Text::Iconv also provides an instance attribute raise_error and a corresponding method for setting and getting its value. If raise_error is undef, the class-wide settings apply. If raise_error is 1 or 0 (true or false), the object settings override the class-wide settings. Consult iconv(3) for details on errors that might occur. Conversion of undef Converting undef, e.g., $converted = $converter->convert(undef); always returns undef. This is not considered an error. NOTES
The supported codesets, their names, the supported conversions, and the quality of the conversions are all system-dependent. AUTHOR
Michael Piotrowski <mxp@dynalabs.de> SEE ALSO
iconv(1), iconv(3) perl v5.12.1 2007-10-17 Iconv(3)
All times are GMT -4. The time now is 10:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy