Sponsored Content
Top Forums Shell Programming and Scripting Need help splitting huge single record file Post 302578858 by Corona688 on Friday 2nd of December 2011 03:07:00 PM
Old 12-02-2011
Those commands are trying to load the entire line into memory at once, which won't work on a 64 32-bit system since there's a 4GB limit on per-process memory and sufficient clutter in the way that you probably can't get an entire 2.5G in one chunk.

How big are the individual records? You can tell awk to use things other than \n as a record separator, changing its definition of lines:

Code:
$  awk -v RS='\\*end\\*' '1' < data
RANDOMDATA
RANDOMDATA
RANDOMDATA
RANDOMDATA
$

Use nawk or gawk if you have it.

Last edited by Corona688; 12-02-2011 at 04:39 PM..
This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

splitting a record and adding a record to a file

Hi, I am new to UNIX scripting and woiuld appreicate your help... Input file contains only one (but long) record: aaaaabbbbbcccccddddd..... Desired file: NEW RECORD #new record (hardcoded) added as first record - its length is irrelevant# aaaaa bbbbb ccccc ddddd ... ... ... (1 Reply)
Discussion started by: rsolap
1 Replies

2. UNIX for Dummies Questions & Answers

Splitting a file based on record sin another file

All, We receive a file with a large no of records (records can vary) and we have to split it into two files based on another file. e.g. File1: UHDR 2008112 "25187","00000022","00",21-APR-1991,"" ,"D",-000000519,+0000000000,"C", ,+000000000,+000000000,000000000,"2","" ,21-APR-1991... (7 Replies)
Discussion started by: er_ashu
7 Replies

3. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

4. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

5. UNIX for Advanced & Expert Users

Splitting the single csv file

Hi, I have a requiement where in i will get a single file but there will be mutiple headers. Suppose say for eg: Header1 Data... Data... Header2 Data.. Data.. Header3 Data.. Data.. I want to split each with the corresponding data into a single file. Please let me know how... (1 Reply)
Discussion started by: weknowd
1 Replies

6. Shell Programming and Scripting

Splitting & reformating a single file

I have a bif text file with the following format: d1_03 fr:23 d1_03 fr:56 d1_03 fr:67 d1_03 fr:78 d1_01 fr:35 d1_01 fr:29 d1_01 fr:45 d2_09 fr:34 d2_09 fr:78 d3_98 fr:90 d3_98 fr:104 d3_98 fr:360 I have like thousands of such lines I want to reformat this file based on column 1... (3 Replies)
Discussion started by: Lucky Ali
3 Replies

7. Shell Programming and Scripting

splitting a huge line of file into multiple lines with fixed number of columns

Hi, I have a huge file with a single line. But I want to break that line into lines of with each line having five columns. My file is like this: code: "hi","there","how","are","you?","It","was","great","working","with","you.","hope","to","work","you." I want it like this: code:... (1 Reply)
Discussion started by: rajsharma
1 Replies

8. Shell Programming and Scripting

Splitting single file into n files

Hi all, I am new to scripting and I have a requirement we have source file as HEADER 01.10.2010 14:32:37 NAYA TA0022 TA0000 20000001;20060612;99991231;K4;02;3 20000008;20080624;99991231;K4;02;3 20000026;19840724;99991231;KK;01;3 20000027;19840724;99991231;KK;01;3... (6 Replies)
Discussion started by: srk409
6 Replies

9. Shell Programming and Scripting

Fetching record based on Uniq Key from huge file.

Hi i want to fetch 100k record from a file which is looking like as below. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ... (17 Replies)
Discussion started by: lathigara
17 Replies

10. Shell Programming and Scripting

Splitting a single file to multiple files

Hi Friends , Please guide me with the code to extract multiple files from one file . The File Looks like ( Suppose a file has 2 tables list ,column length may vary ) H..- > File Header.... H....- >Table 1 Header.... D....- > Table 1 Data.... T....- >Table 1 Trailer.... H..-> Table 2... (1 Reply)
Discussion started by: AspiringD
1 Replies
SPLIT(1)							   User Commands							  SPLIT(1)

NAME
split - split a file into pieces SYNOPSIS
split [OPTION]... [INPUT [PREFIX]] DESCRIPTION
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default size is 1000 lines, and default PREFIX is `x'. With no INPUT, or when INPUT is -, read standard input. Mandatory arguments to long options are mandatory for short options too. -a, --suffix-length=N use suffixes of length N (default 2) -b, --bytes=SIZE put SIZE bytes per output file -C, --line-bytes=SIZE put at most SIZE bytes of lines per output file -d, --numeric-suffixes use numeric suffixes instead of alphabetic -e, --elide-empty-files do not generate empty output files with `-n' --filter=COMMAND write to shell COMMAND; file name is $FILE -l, --lines=NUMBER put NUMBER lines per output file -n, --number=CHUNKS generate CHUNKS output files. See below -u, --unbuffered immediately copy input to output with `-n r/...' --verbose print a diagnostic just before each output file is opened --help display this help and exit --version output version information and exit SIZE may be (or may be an integer optionally followed by) one of following: KB 1000, K 1024, MB 1000*1000, M 1024*1024, and so on for G, T, P, E, Z, Y. CHUNKS may be: N split into N files based on size of input K/N output Kth of N to stdout l/N split into N files without splitting lines l/K/N output Kth of N to stdout without splitting lines r/N like `l' but use round robin distribution r/K/N like- wise but only output Kth of N to stdout AUTHOR
Written by Torbjorn Granlund and Richard M. Stallman. REPORTING BUGS
Report split bugs to bug-coreutils@gnu.org GNU coreutils home page: <http://www.gnu.org/software/coreutils/> General help using GNU software: <http://www.gnu.org/gethelp/> Report split translation bugs to <http://translationproject.org/team/> COPYRIGHT
Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
The full documentation for split is maintained as a Texinfo manual. If the info and split programs are properly installed at your site, the command info coreutils 'split invocation' should give you access to the complete manual. GNU coreutils 8.12.197-032bb September 2011 SPLIT(1)
All times are GMT -4. The time now is 11:32 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy