09-02-2009
Quote:
Originally Posted by
methyl
What Operating System and version do you use?
Does your version of the "split" command have the "-a" switch? See "man split".
Splitting he files twice is indeed twice the work. Using shell script for serious data processing has its merits for quick development of one-off jobs and prototypes. For a regular job on this scale personally I would choose a high level language which has no issues in closing file descriptors mid-process.
I am intrigued why you would want to break 10,000,000 records into 5,000 record chunks? To my mind it just creates complications in volume processing.
Other integration applications are not supported for the large size files. For that purpose we will split the large file into small once's and pass it.
We are using the SunSolaris5.10
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I want to split a file containing millions of records.
I am issuing the command
split -l 20000 filename which will split the file in 20K records each.
It works fine except in some files, data after one particular field is lost( the field with space).
Say the record is ... (4 Replies)
Discussion started by: superprogrammer
4 Replies
2. UNIX for Dummies Questions & Answers
I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this.
For example:
split -l 3000000 filename.txt
This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies
3. Shell Programming and Scripting
Hi all,
i have a .ksh script which is, among other stuff, splitting a file and saveing the filenames into variables for further processing:
# file split before ftp and put result filenames into variables
if ]; then
awk '{close(f);f=$1}{sub("^","");print > f".TXT"}' $_ftpfile
set B*.TXT... (0 Replies)
Discussion started by: spidermike
0 Replies
4. Shell Programming and Scripting
Hi,
I have a data file xyz.dat similar to the one given below,
2345|98|809||x|969|0
2345|98|809||y|0|537
2345|97|809||x|544|0
2345|97|809||y|0|651
9685|98|809||x|321|0
9685|98|809||y|0|357
9685|98|709||x|687|0
9685|98|709||y|0|234
2315|98|809||x|564|0
2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies
5. Shell Programming and Scripting
Hi all,
I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example:
... (2 Replies)
Discussion started by: cul8er
2 Replies
6. Shell Programming and Scripting
Hi ,
I do have a fixedwidth flatfile that has data for 10 different datasets each identified by the first two digits in the flatfile.
01 in the first two digit position refers to Set A
02 in the first two digit position refers to Set B and so on
I want to genrate 10 different files from my... (6 Replies)
Discussion started by: okkadu
6 Replies
7. Shell Programming and Scripting
Hello;
I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies
8. Shell Programming and Scripting
I need to split a file if it is over 2GB in size (or any size), preferably split on the lines. I have figured out how to get the file size using awk, and I can split the file based on the number of lines (which I got with wc -l) but I can't figure out how to connect them together in the script.
... (6 Replies)
Discussion started by: ssemple2000
6 Replies
9. Shell Programming and Scripting
Good day all
I need some helps,
say that I have data like below, each field separated by a tab
DATE NAME ADDRESS
15/7/2012 LX a.b.c
15/7/2012 LX1 a.b.c
16/7/2012 AB a.b.c
16/7/2012 AB2 a.b.c
15/7/2012 LX2 a.b.c... (2 Replies)
Discussion started by: alexyyw
2 Replies
10. UNIX for Beginners Questions & Answers
Dear Users,
Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file
input file.txt
scaffold1 928 929 C/T +
scaffold1 942 943 G/C +
scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
LEARN ABOUT CENTOS
funtbl
funtbl(1) SAORD Documentation funtbl(1)
NAME
funtbl - extract a table from Funtools ASCII output
SYNOPSIS
funtable [-c cols] [-h] [-n table] [-p prog] [-s sep] <iname>
DESCRIPTION
[NB: This program has been deprecated in favor of the ASCII text processing support in funtools. You can now perform fundisp on funtools
ASCII output files (specifying the table using bracket notation) to extract tables and columns.]
The funtbl script extracts a specified table (without the header and comments) from a funtools ASCII output file and writes the result to
the standard output. The first non-switch argument is the ASCII input file name (i.e. the saved output from funcnts, fundisp, funhist,
etc.). If no filename is specified, stdin is read. The -n switch specifies which table (starting from 1) to extract. The default is to
extract the first table. The -c switch is a space-delimited list of column numbers to output, e.g. -c "1 3 5" will extract the first
three odd-numbered columns. The default is to extract all columns. The -s switch specifies the separator string to put between columns.
The default is a single space. The -h switch specifies that column names should be added in a header line before the data is output. With-
out the switch, no header is prepended. The -p program switch allows you to specify an awk-like program to run instead of the default
(which is host-specific and is determined at build time). The -T switch will output the data in rdb format (i.e., with a 2-row header of
column names and dashes, and with data columns separated by tabs). The -help switch will print out a message describing program usage.
For example, consider the output from the following funcnts command:
[sh] funcnts -sr snr.ev "ann 512 512 0 9 n=3"
# source
# data file: /proj/rd/data/snr.ev
# arcsec/pixel: 8
# background
# constant value: 0.000000
# column units
# area: arcsec**2
# surf_bri: cnts/arcsec**2
# surf_err: cnts/arcsec**2
# summed background-subtracted results
upto net_counts error background berror area surf_bri surf_err
---- ------------ --------- ------------ --------- --------- --------- ---------
1 147.000 12.124 0.000 0.000 1600.00 0.092 0.008
2 625.000 25.000 0.000 0.000 6976.00 0.090 0.004
3 1442.000 37.974 0.000 0.000 15936.00 0.090 0.002
# background-subtracted results
reg net_counts error background berror area surf_bri surf_err
---- ------------ --------- ------------ --------- --------- --------- ---------
1 147.000 12.124 0.000 0.000 1600.00 0.092 0.008
2 478.000 21.863 0.000 0.000 5376.00 0.089 0.004
3 817.000 28.583 0.000 0.000 8960.00 0.091 0.003
# the following source and background components were used:
source_region(s)
----------------
ann 512 512 0 9 n=3
reg counts pixels sumcnts sumpix
---- ------------ --------- ------------ ---------
1 147.000 25 147.000 25
2 478.000 84 625.000 109
3 817.000 140 1442.000 249
There are four tables in this output. To extract the last one, you can execute:
[sh] funcnts -s snr.ev "ann 512 512 0 9 n=3" | funtbl -n 4
1 147.000 25 147.000 25
2 478.000 84 625.000 109
3 817.000 140 1442.000 249
Note that the output has been re-formatted so that only a single space separates each column, with no extraneous header or comment informa-
tion.
To extract only columns 1,2, and 4 from the last example (but with a header prepended and tabs between columns), you can execute:
[sh] funcnts -s snr.ev "ann 512 512 0 9 n=3" | funtbl -c "1 2 4" -h -n 4 -s " "
#reg counts sumcnts
1 147.000 147.000
2 478.000 625.000
3 817.000 1442.000
Of course, if the output has previously been saved in a file named foo.out, the same result can be obtained by executing:
[sh] funtbl -c "1 2 4" -h -n 4 -s " " foo.out
#reg counts sumcnts
1 147.000 147.000
2 478.000 625.000
3 817.000 1442.000
SEE ALSO
See funtools(7) for a list of Funtools help pages
version 1.4.2 January 2, 2008 funtbl(1)