Sponsored Content
Top Forums Shell Programming and Scripting Split files with formatted numbers Post 302907206 by Don Cragun on Thursday 26th of June 2014 01:01:48 PM
Old 06-26-2014
Quote:
Originally Posted by bobbygsk
I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.

Besides, how do I get header (n records) and trailer with file number or some content in it.
Simple, you slightly modify the code I gave you to put 100000 lines per output file instead of 2 lines per output file. The code I gave you already closes files when it is done with them so it only keeps one output file open at a time.

You're going to have to give us a lot more than "get header (n records) and trailer with file number or some content in it" to guess at what you want to put as headers and trailers in your files. Show us sample input and show us sample output! How is your script supposed to identify which lines are headers, which lines are trailers, and what data you want added to or removed from those headers as you copy parts of the input file to your hundreds of output files?
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to remove improperly formatted fortran output line from files, tried sed

I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in... (2 Replies)
Discussion started by: gillesc_mac
2 Replies

2. Shell Programming and Scripting

Generating formatted reports from log files

Given that I have a log file of the format: DATE ID LOG_LEVEL | EVENT 2009-07-23T14:05:11Z T-4030097550 D | MessX 2009-07-23T14:10:44Z T-4030097550 D | MessY 2009-07-23T14:34:08Z T-7298651656 D | MessX 2009-07-23T14:41:00Z T-7298651656 D | MessY 2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies

3. UNIX for Dummies Questions & Answers

Split Function Prefix Numbers

Hello, Hello, I use the following command to split a file: split -Number_of_Lines Input_File MyPrefix_ output is MyPrefix_a MyPrefix_b MyPrefix_c ...... Instead, how can I get numerical values like: MyPrefix_1 MyPrefix_2 MyPrefix_3 ...... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

4. Shell Programming and Scripting

Extracting formatted text and numbers

Hello, I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh. For example, given the following data I want to extract: and... (3 Replies)
Discussion started by: DFr0st
3 Replies

5. UNIX for Dummies Questions & Answers

Breaking a fasta formatted file into multiple files containing each gene separately

Hey, I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time: for i in *.rtf.out do awk '/^>/{f=++d".fasta"} {print > $i.out}' $i done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies

6. Shell Programming and Scripting

Split a file into multiple files based on line numbers and first column value

Hi All I have one query,say i have a requirement like the below code should be move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines. This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first... (2 Replies)
Discussion started by: sarav.shan
2 Replies

7. Shell Programming and Scripting

awk split numbers

I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into -dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25... (3 Replies)
Discussion started by: sdf
3 Replies

8. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies

9. Shell Programming and Scripting

Sum up formatted numbers with comma separation

I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values. 1,775,947,633 4,738 7,300 16,610 15,279 0 0 I tried the following code in a shell script. awk '{sum+=$5} END{print... (4 Replies)
Discussion started by: krishmaths
4 Replies
fwtmp(1M)																 fwtmp(1M)

NAME
fwtmp, wtmpfix - manipulate connect accounting records SYNOPSIS
[files] DESCRIPTION
fwtmp reads from the standard input and writes to the standard output, converting binary records of the type found in to formatted ASCII records. The ASCII version is useful to enable editing, via ed(1), bad records or for general purpose maintenance of the file. The argument is used to denote that input is in ASCII form, and output is to be written in binary form. The arguments and are independent, respectively specifying ASCII input and binary output. Therefor, is an ASCII to ASCII copy and is a binary to binary copy. should be used for reading If is not used, structure is read. wtmpfix examines the standard input or named files in format, corrects the time/date stamps to make the entries consistent, and writes to the stan- dard output. A can be used in place of files to indicate the standard input. If time/date corrections are not performed, will fault when it encounters certain date-change records. Each time the date is set, a pair of date change records is written to The first record is the old date denoted by the string old time placed in the line field and the flag placed in the type field of the structure. The second record specifies the new date, and is denoted by the string placed in the line field and the flag placed in the type field. uses these records to synchronize all time stamps in the file. nullifies date change records when writing to the standard output by setting the time field of the structure in the old date change record equal to the time field in the new date change record. This prevents and from factoring in a date change record pair more than once. In addition to correcting time/date stamps, wtmpfix checks the validity of the name field to ensure that it consists solely of alphanumeric characters or spaces. If it encounters a name that is considered invalid, it changes the login name to and writes a diagnostic to the standard error. This minimizes the risk that will fail when processing connect accounting records. DIAGNOSTICS
wtmpfix generates the following diagnostics messages: WARNINGS
generates no errors, even on garbage input. FILES
SEE ALSO
ed(1), acct(1M), acctcms(1M), acctcom(1M), acctcon(1M), acctmerg(1M), acctprc(1M), acctsh(1M), runacct(1M), acct(2), acct(4), utmp(4), wtmps(4). STANDARDS CONFORMANCE
fwtmp(1M)
All times are GMT -4. The time now is 12:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy