Sponsored Content
Top Forums Shell Programming and Scripting Split files based on row delimiter count Post 302991209 by Scrutinizer on Tuesday 7th of February 2017 11:33:48 PM
Old 02-08-2017
If you have GNU awk (gawk) or mawk you could try something like this, which should split the file in chunks (new files ending with "-chunknr") of 20,000,000 rows where the last file contains the remainder of rows:

Code:
awk -v n=20000000 'BEGIN{ORS=RS="<EORD>"} !(NR%n-1){close(f); f=FILENAME "-" ++c}{print>f}' file

This User Gave Thanks to Scrutinizer For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

renaming files using split with a delimiter

I have a directory of files that I need to rename by splitting the first and second halves of the filenames using the delimiter "-O" and then renaming with the second half first, followed by two underscores and then the first half. For example, natfinal1995annvol1_14.pdf -O filenum-20639 will be... (2 Replies)
Discussion started by: swimulator
2 Replies

2. Shell Programming and Scripting

split file based on group count

Hi, can some one please help me to split the file based on groups. like in the below scenario x indicates the begining of the group and the file should be split each with 2 groups below there are 10 groups it should create 5 files. could you please help? (4 Replies)
Discussion started by: hitmansilentass
4 Replies

3. Shell Programming and Scripting

Split into columns based on the parameter and use & as delimiter

Here is my source, i have million lines like this on a file. disp0201.php?poc=4060&roc=1&ps=R&ooc=13&mjv=6&mov=5&rel=5&bod=155&oxi=2&omj=5&ozn=1&dav=20&cd=&daz=& drc=&mo=&sid=&lang=EN&loc=JPN I want to split this into columns in order to load in database, anything starts with"&mjv=6" as first... (13 Replies)
Discussion started by: elamurugu
13 Replies

4. Shell Programming and Scripting

split record based on delimiter

Hi, My inputfile contains field separaer is ^. 12^inms^ 13^fakdks^ssk^s3 23^avsd^ 13^fakdks^ssk^a4 I wanted to print only 2 delimiter occurence i.e 12^inms^ 23^avsd^ (4 Replies)
Discussion started by: Jairaj
4 Replies

5. Shell Programming and Scripting

KSH: Split String into smaller substrings based on count

KSH HP-SOL-Lin Cannot use xAWK I have several strings that are quite long and i want to break them down into smaller substrings. What I have String = "word1 word2 word3 word4 .....wordx" What I want String1="word1 word2" String2="word 3 word4" String3="word4 word5" Stringx="wordx... (5 Replies)
Discussion started by: nitrobass24
5 Replies

6. Shell Programming and Scripting

Split Large Files Based On Row Pattern..

Hi all. I've tried searching the web but could not find similar problem to mine. I have one large file to be splitted into several files based on the matching pattern found in each row. For example, let's say the file content: ... (13 Replies)
Discussion started by: aimy
13 Replies

7. Shell Programming and Scripting

Send email based on row count

i have below code to count number of rows in file1.txt, if the row count is more than one then i have sending an email along with file1.txt attached and fail the process(do nothing if count is <=1), if I test individually count part works good but when i include the email part its not working,... (1 Reply)
Discussion started by: srini_106
1 Replies

8. Shell Programming and Scripting

Count delimiter(~|*) each row in a file and return 1 or 0

Hi I want to check delimiter in file. Delimiter in my file is ~|* sample of file : ABC~|*edgf~|*T1J333~|*20121130 ABC~|*sdaf~|*T1J333~|*20121130 ABC~|*fsdg~|*T1J333~|*20121130 ABC~|*dfsg~|*T1J333~|*20121130 in this i want to count number delimiter occur is 4 in each row if count is... (21 Replies)
Discussion started by: MOHANP12
21 Replies

9. Shell Programming and Scripting

awk - split data based on the count

Greetings Experts, I am generating a validation query through awk and facing an issue, which I need to overcome by splitting the data based on the pattern matching count in the value of an array. File1 -- Table11@column1@date@Table21@column1@varchar(10)@d;... (4 Replies)
Discussion started by: chill3chee
4 Replies

10. Shell Programming and Scripting

Linux shell script to insert new lines based on delimiter count

The input file is a .dat file which is delimited by null (^@ in Linux). On a windows PC it looks something like this (numbers are masked with 1). https://i.imgur.com/nta2Gqp.jpg The entire file is in one row but it has multiple records - each record contains 80 fields i.e. there are 81 counts... (9 Replies)
Discussion started by: digitalnirvana
9 Replies
fmt(1)								   User Commands							    fmt(1)

NAME
fmt - simple text formatters SYNOPSIS
fmt [-cs] [-w width | -width] [inputfile...] DESCRIPTION
fmt is a simple text formatter that fills and joins lines to produce output lines of (up to) the number of characters specified in the -w width option. The default width is 72. fmt concatenates the inputfiles listed as arguments. If none are given, fmt formats text from the standard input. Blank lines are preserved in the output, as is the spacing between words. fmt does not fill nor split lines beginning with a `.' (dot), for compatibility with nroff(1). Nor does it fill or split a set of contiguous non-blank lines which is determined to be a mail header, the first line of which must begin with "From". Indentation is preserved in the output, and input lines with differing indentation are not joined (unless -c is used). fmt can also be used as an in-line text filter for vi(1). The vi command: !}fmt reformats the text between the cursor location and the end of the paragraph. OPTIONS
-c Crown margin mode. Preserve the indentation of the first two lines within a paragraph, and align the left margin of each subsequent line with that of the second line. This is useful for tagged paragraphs. -s Split lines only. Do not join short lines to form longer ones. This prevents sample lines of code, and other such formatted text, from being unduly combined. -w width | -width Fill output lines to up to width columns. OPERANDS
inputfile Input file. ENVIRONMENT VARIABLES
See environ(5) for a description of the LC_CTYPE environment variable that affects the execution of fmt. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
nroff(1), vi(1), attributes(5), environ(5) NOTES
The -width option is acceptable for BSD compatibility, but it may go away in future releases. SunOS 5.10 9 May 1997 fmt(1)
All times are GMT -4. The time now is 09:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy