Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Split a large file
Top Forums Shell Programming and Scripting Split a large file Post 302444426 by alister on Wednesday 11th of August 2010 11:17:34 PM
Old 08-12-2010
commasplit.awk:
Code:
BEGIN {
    RS = ","
    max_size = 100*2^20
}

function open_file() {
    len = 0
    fn = "file" ++i
    printf("v=[") > fn
}

function close_file() {
    printf("];") > fn
    close(fn)
}

NR == 1 {
    open_file()
}

len >= max_size {
    close_file()
    open_file()
}

{
    s = (len?",":"") $0
    printf("%s", s) > fn
    len += length(s)
}

END {
   close_file()
}

Invocation:
Code:
awk -f commasplit.awk  datafile


max_size is not a hard limit. The file size may be a bit larger, as much as max_size + length of header (3) and footer (2) + one field length - 1. If the field's value were "125000" (6), we'd be talking about a file size of 100 MiB + 10.

Regards,
Alister
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split A Large File

Hi, I have a large file(csv format) that I need to split into 2 files. The file looks something like Original_file.txt first name, family name, address a, b, c, d, e, f, and so on for over 100,00 lines I need to create two files from this one file. The condition is i need to ensure... (4 Replies)
Discussion started by: nbvcxzdz
4 Replies

2. Shell Programming and Scripting

Split a large file with patterns and size

Hi, I have a large file with a repeating pattern in it. Now i want the file split into the block of patterns with a specified no. of lines in each file. i.e. The file is like 1... 2... 2... 3... 1... 2... 3... 1... 2... 2... 2... 2... 2... 3... where 1 is the start of the block... (5 Replies)
Discussion started by: sudhamacs
5 Replies

3. Shell Programming and Scripting

Split Large File

HI, i've to split a large file which inputs seems like : Input file name_file.txt 00001|AAAA|MAIL|DATEOFBIRTHT|....... 00001|AAAA|MAIL|DATEOFBIRTHT|....... 00002|BBBB|MAIL|DATEOFBIRTHT|....... 00002|BBBB|MAIL|DATEOFBIRTHT|....... 00003|CCCC|MAIL|DATEOFBIRTHT|.......... (1 Reply)
Discussion started by: AMARA
1 Replies

4. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies

5. Shell Programming and Scripting

Splitting a large file, split command will not do.

Hello Everyone, I have a large file that needs to be split into many seperate files, however the text in between the blank lines need to be intact. The file looks like SomeText SomeText SomeText SomeOtherText SomeOtherText .... Since the number of lines of text are different for... (3 Replies)
Discussion started by: jwillis0720
3 Replies

6. Shell Programming and Scripting

Split large file based on last digit from a column

Hello, What's the best way to split a large into multiple files based on the last digit in the first column. input file: f 2738483300000x0y03772748378831x1y13478378358383x2y23743878383802x3y33787828282820x4y43748838383881x5y5 Desired Output: f0 3738483300000x0y03787828282820x4y4 f1... (9 Replies)
Discussion started by: alain.kazan
9 Replies

7. UNIX for Dummies Questions & Answers

Split large file to smaller fastly

hi , I have a requirement input file: 1 1111111111111 108 1 1111111111111 109 1 1111111111111 109 1 1111111111111 110 1 1111111111111 111 1 1111111111111 111 1 1111111111111 111 1 1111111111111 112 1 1111111111111 112 1 1111111111111 112 The output should be, (19 Replies)
Discussion started by: mechvijays
19 Replies

8. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

9. UNIX for Advanced & Expert Users

How to split large file with different record delimiter?

Hi, I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues. If the record delimiter is unix new line, I could use split command either with option l or b. The problem is that the line terminator is |##| How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies

10. UNIX for Beginners Questions & Answers

Trying To Split a Large File

Trying to split a 35gb file into 1000mb parts. My research shows I should you this. split -b 1000m file.txt and my return is "split: cannot open 'crunch1.txt' for reading: No such file or directory" so I tried split -b 1000m Documents/Wordlists/file.txt and I get nothing other than the curser just... (3 Replies)
Discussion started by: sub terra
3 Replies

LEARN ABOUT SUNOS

adjmsg

adjmsg(9F)						   Kernel Functions for Drivers 						adjmsg(9F)

NAME

       adjmsg - trim bytes from a message

SYNOPSIS

       #include <sys/stream.h>

       int adjmsg(mblk_t *mp, ssize_t len);

INTERFACE LEVEL

       Architecture independent level 1 (DDI/DKI).

PARAMETERS

       mp	Pointer to the message to be trimmed.

       len	The number of bytes to be removed.

DESCRIPTION

       The  adjmsg()  function	removes  bytes from a message. |len| (the absolute value of len) specifies the number of bytes to be removed.  The
       adjmsg() function only trims bytes across message blocks of the same type.

       The adjmsg() function finds the maximal leading sequence of message blocks of the same type as that of mp and starts removing bytes  either
       from  the  head	of that sequence or from the tail of that sequence. If len is greater than 0, adjmsg() removes bytes from the start of the
       first message block in that sequence. If len is less than 0, it removes bytes from the end of the last message block in that sequence.

       The adjmsg() function fails if |len| is greater than the number of bytes in the maximal leading sequence it finds.

       The adjmsg() function may remove any except the first zero-length message block created during  adjusting.  It may also	remove	any  zero-
       length message blocks that occur within the scope of |len|.

RETURN VALUES

       The adjmsg() function returns:

       1	Successful completion.

       0	An error occurred.

CONTEXT

       The adjmsg() function can be called from user or interrupt context.

SEE ALSO

       STREAMS Programming Guide

SunOS 5.10							    20 Nov 1996 							adjmsg(9F)
All times are GMT -4. The time now is 12:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy