06-26-2014
Quote:
Originally Posted by
bobbygsk
I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.
Besides, how do I get header (n records) and trailer with file number or some content in it.
Simple, you slightly modify the code I gave you to put 100000 lines per output file instead of 2 lines per output file. The code I gave you already closes files when it is done with them so it only keeps one output file open at a time.
You're going to have to give us a lot more than "
get header (n records) and trailer with file number or some content in it" to guess at what you want to put as headers and trailers in your files. Show us sample input and show us sample output! How is your script supposed to identify which lines are headers, which lines are trailers, and what data you want added to or removed from those headers as you copy parts of the input file to your hundreds of output files?
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in... (2 Replies)
Discussion started by: gillesc_mac
2 Replies
2. Shell Programming and Scripting
Given that I have a log file of the format:
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies
3. UNIX for Dummies Questions & Answers
Hello,
Hello,
I use the following command to split a file:
split -Number_of_Lines Input_File MyPrefix_
output is
MyPrefix_a
MyPrefix_b
MyPrefix_c
......
Instead, how can I get numerical values like:
MyPrefix_1
MyPrefix_2
MyPrefix_3
...... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies
4. Shell Programming and Scripting
Hello,
I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh.
For example, given the following data
I want to extract:
and... (3 Replies)
Discussion started by: DFr0st
3 Replies
5. UNIX for Dummies Questions & Answers
Hey,
I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time:
for i in *.rtf.out
do
awk '/^>/{f=++d".fasta"} {print > $i.out}' $i
done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies
6. Shell Programming and Scripting
Hi All
I have one query,say i have a requirement like the below code should be
move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines.
This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first... (2 Replies)
Discussion started by: sarav.shan
2 Replies
7. Shell Programming and Scripting
I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into
-dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25... (3 Replies)
Discussion started by: sdf
3 Replies
8. UNIX for Beginners Questions & Answers
Hello,
I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number.
What I have tried is the below command with 2 digit numeric value
split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02)
How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies
9. Shell Programming and Scripting
I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values.
1,775,947,633
4,738
7,300
16,610
15,279
0
0
I tried the following code in a shell script.
awk '{sum+=$5} END{print... (4 Replies)
Discussion started by: krishmaths
4 Replies
fwtmp(1M) fwtmp(1M)
NAME
fwtmp, wtmpfix - manipulate connect accounting records
SYNOPSIS
[files]
DESCRIPTION
fwtmp
reads from the standard input and writes to the standard output, converting binary records of the type found in to formatted ASCII records.
The ASCII version is useful to enable editing, via ed(1), bad records or for general purpose maintenance of the file.
The argument is used to denote that input is in ASCII form, and output is to be written in binary form. The arguments and are independent,
respectively specifying ASCII input and binary output. Therefor, is an ASCII to ASCII copy and is a binary to binary copy. should be used
for reading If is not used, structure is read.
wtmpfix
examines the standard input or named files in format, corrects the time/date stamps to make the entries consistent, and writes to the stan-
dard output. A can be used in place of files to indicate the standard input. If time/date corrections are not performed, will fault when
it encounters certain date-change records.
Each time the date is set, a pair of date change records is written to The first record is the old date denoted by the string old time
placed in the line field and the flag placed in the type field of the structure. The second record specifies the new date, and is denoted
by the string placed in the line field and the flag placed in the type field. uses these records to synchronize all time stamps in the
file. nullifies date change records when writing to the standard output by setting the time field of the structure in the old date change
record equal to the time field in the new date change record. This prevents and from factoring in a date change record pair more than
once.
In addition to correcting time/date stamps, wtmpfix checks the validity of the name field to ensure that it consists solely of alphanumeric
characters or spaces. If it encounters a name that is considered invalid, it changes the login name to and writes a diagnostic to the
standard error. This minimizes the risk that will fail when processing connect accounting records.
DIAGNOSTICS
wtmpfix generates the following diagnostics messages:
WARNINGS
generates no errors, even on garbage input.
FILES
SEE ALSO
ed(1), acct(1M), acctcms(1M), acctcom(1M), acctcon(1M), acctmerg(1M), acctprc(1M), acctsh(1M), runacct(1M), acct(2), acct(4), utmp(4),
wtmps(4).
STANDARDS CONFORMANCE
fwtmp(1M)