06-26-2014
Quote:
Originally Posted by
bobbygsk
I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.
Besides, how do I get header (n records) and trailer with file number or some content in it.
Simple, you slightly modify the code I gave you to put 100000 lines per output file instead of 2 lines per output file. The code I gave you already closes files when it is done with them so it only keeps one output file open at a time.
You're going to have to give us a lot more than "
get header (n records) and trailer with file number or some content in it" to guess at what you want to put as headers and trailers in your files. Show us sample input and show us sample output! How is your script supposed to identify which lines are headers, which lines are trailers, and what data you want added to or removed from those headers as you copy parts of the input file to your hundreds of output files?
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in... (2 Replies)
Discussion started by: gillesc_mac
2 Replies
2. Shell Programming and Scripting
Given that I have a log file of the format:
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies
3. UNIX for Dummies Questions & Answers
Hello,
Hello,
I use the following command to split a file:
split -Number_of_Lines Input_File MyPrefix_
output is
MyPrefix_a
MyPrefix_b
MyPrefix_c
......
Instead, how can I get numerical values like:
MyPrefix_1
MyPrefix_2
MyPrefix_3
...... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies
4. Shell Programming and Scripting
Hello,
I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh.
For example, given the following data
I want to extract:
and... (3 Replies)
Discussion started by: DFr0st
3 Replies
5. UNIX for Dummies Questions & Answers
Hey,
I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time:
for i in *.rtf.out
do
awk '/^>/{f=++d".fasta"} {print > $i.out}' $i
done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies
6. Shell Programming and Scripting
Hi All
I have one query,say i have a requirement like the below code should be
move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines.
This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first... (2 Replies)
Discussion started by: sarav.shan
2 Replies
7. Shell Programming and Scripting
I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into
-dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25... (3 Replies)
Discussion started by: sdf
3 Replies
8. UNIX for Beginners Questions & Answers
Hello,
I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number.
What I have tried is the below command with 2 digit numeric value
split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02)
How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies
9. Shell Programming and Scripting
I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values.
1,775,947,633
4,738
7,300
16,610
15,279
0
0
I tried the following code in a shell script.
awk '{sum+=$5} END{print... (4 Replies)
Discussion started by: krishmaths
4 Replies
diskusg(1M) diskusg(1M)
NAME
diskusg - generate disk accounting data by user ID
SYNOPSIS
[options] [files]
DESCRIPTION
generates intermediate disk accounting information from data in files, or the standard input if omitted. outputs lines on the standard
output, one per user, in the following format:
uid login #blocks
where:
uid User's numerical user ID,
login User's login name, and
#blocks Total number of disk blocks allocated to this user.
normally reads only the inodes of file systems for disk accounting. In this case, files are the special filenames of these devices.
Options
recognizes the following options:
Input data is already in
output format. combines all lines for a single user into a single line.
verbose. Print a list on standard error of all files that are charged to no one.
Ignore the data on those file systems whose file system name is in
fnmlist. fnmlist is a list of file system names, separated by commas or enclosed within quotes. compares each name
in this list with the file system name stored in the volume ID if it exists.
Use file as the name of the password file to generate login names. is used by default.
Write records to
file of files that are charged to no one. Records consist of the special file name, the inode number, and the user
ID.
The output of is normally the input to (see acct(1M)) which generates total accounting records that can be merged with other accounting
records. is normally run in (see acctsh(1M)).
EXAMPLES
The following generates daily disk accounting information:
FILES
used for user- ID-to-login-name conversions
SEE ALSO
acct(1M), acctsh(1M), volcopy(1M), acct(4), vxdiskusg(1M).
STANDARDS CONFORMANCE
diskusg(1M)