Sponsored Content
Full Discussion: Iconv for large files
Top Forums Shell Programming and Scripting Iconv for large files Post 302947563 by jim mcnamara on Thursday 18th of June 2015 10:38:47 PM
Old 06-18-2015
iconv on your system appears not to support largefiles.

There is an approach called chunking - try this:
https://github.com/mla/iconv-chunks

It is a perl that allows iconv to process 100MB chunks of gigantic files and output to stdout - you redirect to the new file.

Last edited by jim mcnamara; 06-18-2015 at 11:43 PM..
This User Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. Programming

about iconv

I want to use iconv.h to convert some text to another charset. The code is below: #include <stdio.h> #include <stdlib.h> #include <iconv.h> int main() { iconv_t cd; char instr="汉字"; char *inbuf; char *outbuf; unsigned int insize=7; ... (4 Replies)
Discussion started by: yong
4 Replies

2. UNIX for Dummies Questions & Answers

large files?

How do we check 'large files' is enabled on a Unix box -- HP-UX B11.11 (2 Replies)
Discussion started by: ranj@chn
2 Replies

3. UNIX for Advanced & Expert Users

iconv and xmllint

Here is my question, volume of records processed : 5M ( approx ) Its basically very simple operation that am trying to do and I had achieved the output that am interested. What am looking for really is to improve the performance, an optimized way to do that. with respect to iconv, am... (3 Replies)
Discussion started by: matrixmadhan
3 Replies

4. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234 (7 Replies)
Discussion started by: ad23
7 Replies

5. Solaris

How to safely copy full filesystems with large files (10Gb files)

Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS. I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies

6. Shell Programming and Scripting

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (0 Replies)
Discussion started by: Shruthi8818
0 Replies

7. UNIX for Dummies Questions & Answers

Help with iconv command

Hi , I am using iconv command to convert a file in UTF-16 format to UTF-8 format. This command will work for few files but for some showing an error as bad input character. But if i copy the contents of the file for which it is showing "bad input character" to a new file and perform the... (2 Replies)
Discussion started by: Shruthi8818
2 Replies

8. AIX

Installing Converter files for iconv conversion

Hi Friends, I need to install two converter packages UTF-8_IBM-284 and UTF-8_IBM-500 in the location /usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* . Could you please let me know how to get this converter files. Thanks in Advance, Siva. (1 Reply)
Discussion started by: sivakumarl
1 Replies

9. Shell Programming and Scripting

Help with command iconv

I need to convert a utf16 file to utf8. When i use the iconv command to do so it gives an error saying invalid function. When I ran the iconv -l function it did not list the utf16 and utf8 as part of its internal table. Is there anyway I can add these encodings in the library? Is there any other... (3 Replies)
Discussion started by: gaun
3 Replies

10. UNIX for Advanced & Expert Users

Iconv on large files

Hi All, I am using iconv to convert huge files. the process is getting killed. I tried the option in below link https://www.unix.com/shell-programming-and-scripting/258825-iconv-large-files.html i.e iconv -f UCS-2 -t UTF-8 < inputfile.txt > outputfile.txt However, the process still gets... (4 Replies)
Discussion started by: tostay2003
4 Replies
YAZ-ICONV(1)							     Commands							      YAZ-ICONV(1)

NAME
yaz-iconv - YAZ Character set conversion utility SYNOPSIS
yaz-iconv [-f from] [-t to] [-v] [file...] DESCRIPTION
yaz-iconv converts data in file in character set specified by from to output in character set as specified by to. This yaz-iconv utility similar to the iconv found on many POSIX systems (Glibc, Solaris, etc). If no file is specified, yaz-iconv reads from standard input. OPTIONS
-ffrom] Specify the character set from of the input file. Should be used in conjunction with option -t. -tto] Specify the character set of of the output. Should be used in conjunction with option -f. -v Print more information about the conversion process. ENCODINGS
The yaz-iconv command and the API as defined in yaz/yaz-iconv.h is a wrapper for the library system call iconv. But YAZ' iconv utility also implements conversions on its own. The table below lists characters sets (or encodings). that are supported by YAZ. Each character set is marked with either encode or decode. If an encoding is encode-enabled YAZ may convert to to the designated encoding. If an encoding is decode-enabled, YAZ may convert from the designated encoding. marc8 (encode, decode) The MARC8[1] encoding as defined by the Library of Congress. Most MARC21/USMARC records usees this encoding. marc8s (encode, decode) Like MARC8 but with conversion prefers non-combined characters in the Latin-1 plane over combined characters. marc8lossy (encode) Lossy encoding of MARC-8. marc8lossless (encode) Lossless encoding of MARC8. utf8 (encode, decode) The most commonly used UNICODE encoding on the Internet. iso8859-1 (encode, decode) ISO-8859-1, AKA Latin-1. iso5426 (decode) ISO 5426. Some MARC records (UNIMARC) uses this encoding. iso5428:1984 (encode, decode) ISO 5428:1984. advancegreek (encode, decode) An encoding for Greek used by some vendors (Advance). danmarc (decode) Danmarc (in danish)[2] is an encoding based on UNICODE which is used for DanMARC2 records. EXAMPLES
The following command converts from ISO-8859-1 (Latin-1) to UTF-8. yaz-iconv -f ISO-8859-1 -t UTF-8 -X <input.lst >output.lst FILES
prefix/bin/yaz-iconv prefix/include/yaz/yaz-iconv.h SEE ALSO
yaz(7) iconv(1) NOTES
1. MARC8 http://www.loc.gov/marc/specifications/speccharmarc8.html 2. Danmarc (in danish) http://www.kat-format.dk/danMARC2/Danmarc2.4.htm#felt+Indl.+4 YAZ 4.2.30 04/16/2012 YAZ-ICONV(1)
All times are GMT -4. The time now is 02:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy