06-26-2014
I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.
Besides, how do I get header (n records) and trailer with file number or some content in it.
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in... (2 Replies)
Discussion started by: gillesc_mac
2 Replies
2. Shell Programming and Scripting
Given that I have a log file of the format:
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies
3. UNIX for Dummies Questions & Answers
Hello,
Hello,
I use the following command to split a file:
split -Number_of_Lines Input_File MyPrefix_
output is
MyPrefix_a
MyPrefix_b
MyPrefix_c
......
Instead, how can I get numerical values like:
MyPrefix_1
MyPrefix_2
MyPrefix_3
...... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies
4. Shell Programming and Scripting
Hello,
I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh.
For example, given the following data
I want to extract:
and... (3 Replies)
Discussion started by: DFr0st
3 Replies
5. UNIX for Dummies Questions & Answers
Hey,
I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time:
for i in *.rtf.out
do
awk '/^>/{f=++d".fasta"} {print > $i.out}' $i
done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies
6. Shell Programming and Scripting
Hi All
I have one query,say i have a requirement like the below code should be
move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines.
This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first... (2 Replies)
Discussion started by: sarav.shan
2 Replies
7. Shell Programming and Scripting
I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into
-dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25... (3 Replies)
Discussion started by: sdf
3 Replies
8. UNIX for Beginners Questions & Answers
Hello,
I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number.
What I have tried is the below command with 2 digit numeric value
split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02)
How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies
9. Shell Programming and Scripting
I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values.
1,775,947,633
4,738
7,300
16,610
15,279
0
0
I tried the following code in a shell script.
awk '{sum+=$5} END{print... (4 Replies)
Discussion started by: krishmaths
4 Replies
split(n) Tcl Built-In Commands split(n)
__________________________________________________________________________________________________________________________________________________
NAME
split - Split a string into a proper Tcl list
SYNOPSIS
split string ?splitChars?
_________________________________________________________________
DESCRIPTION
Returns a list created by splitting string at each character that is in the splitChars argument. Each element of the result list will con-
sist of the characters from string that lie between instances of the characters in splitChars. Empty list elements will be generated if
string contains adjacent characters in splitChars, or if the first or last character of string is in splitChars. If splitChars is an empty
string then each character of string becomes a separate element of the result list. SplitChars defaults to the standard white-space char-
acters.
EXAMPLES
Divide up a USENET group name into its hierarchical components:
split "comp.lang.tcl.announce" .
-> comp lang tcl announce
See how the split command splits on every character in splitChars, which can result in information loss if you are not careful:
split "alpha beta gamma" "temp"
-> al {ha b} {} {a ga} {} a
Extract the list words from a string that is not a well-formed list:
split "Example with {unbalanced brace character"
-> Example with {unbalanced brace character
Split a string into its constituent characters
split "Hello world" {}
-> H e l l o { } w o r l d
PARSING RECORD-ORIENTED FILES
Parse a Unix /etc/passwd file, which consists of one entry per line, with each line consisting of a colon-separated list of fields:
## Read the file
set fid [open /etc/passwd]
set content [read $fid]
close $fid
## Split into records on newlines
set records [split $content "
"]
## Iterate over the records
foreach rec $records {
## Split into fields on colons
set fields [split $rec ":"]
## Assign fields to variables and print some out...
lassign $fields
userName password uid grp longName homeDir shell
puts "$longName uses [file tail $shell] for a login shell"
}
SEE ALSO
join(n), list(n), string(n)
KEYWORDS
list, split, string
Tcl split(n)