Sponsored Content
Full Discussion: Split file based on size
Top Forums Shell Programming and Scripting Split file based on size Post 302604848 by Ygor on Tuesday 6th of March 2012 06:31:11 AM
Old 03-06-2012
Try something like...
Code:
awk '(t=substr($0,1,4))!=p&&c>50000000{n++;c=0}{print $0 > sprintf("outfile%02d",n); p=t; c+=length+1}' infile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a large file with patterns and size

Hi, I have a large file with a repeating pattern in it. Now i want the file split into the block of patterns with a specified no. of lines in each file. i.e. The file is like 1... 2... 2... 3... 1... 2... 3... 1... 2... 2... 2... 2... 2... 3... where 1 is the start of the block... (5 Replies)
Discussion started by: sudhamacs
5 Replies

2. Shell Programming and Scripting

Split a file based on a pattern

Dear all, I have a large file which is composed of 8000 frames, what i would like to do is split the file into 8000 single files names file.pdb.1, file.pdb.2 etc etc each frame in the large file is seperated by a "ENDMDL" flag so my thinking is to use this flag a a point to split the files... (4 Replies)
Discussion started by: Mish_99
4 Replies

3. UNIX for Dummies Questions & Answers

split files based on size

I have a few txt files in some directory and I need to check their sizes one by one. If any of them are greater than 5mb then I need to split the file in two. Can someone help? Thanks. (6 Replies)
Discussion started by: khanvader
6 Replies

4. Shell Programming and Scripting

Split a file based on pattern and size

Hello, I have a large file (2GB) that I would like to split based on pattern and size. I've used the following command to split the file (token is "HELLO") awk '/HELLO/{i++}{print > "file"i}' input.txt and the output is similar to the following (i included filesize in KB): 10 ... (2 Replies)
Discussion started by: jl487
2 Replies

5. Shell Programming and Scripting

Split file based on file size in Korn script

I need to split a file if it is over 2GB in size (or any size), preferably split on the lines. I have figured out how to get the file size using awk, and I can split the file based on the number of lines (which I got with wc -l) but I can't figure out how to connect them together in the script. ... (6 Replies)
Discussion started by: ssemple2000
6 Replies

6. Shell Programming and Scripting

Split the file based on pattern

Hi , I have huge files around 400 mb, which has clob data and have diffeent scenarios: I am trying to pass scenario number as parameter and and get required modified file based on the scenario number and criteria. Scenario 1: file name : scenario_1.txt ... (2 Replies)
Discussion started by: sol_nov
2 Replies

7. UNIX for Dummies Questions & Answers

Split file based on column

i have file1.txt asdas|csada|130310|0423|A1|canberra sdasd|sfdsf|130426|2328|A1|sydney Expected output : on eaceh third and fourth colum, split into each two characters asdas|csada|13|03|10|04|23|A1|canberra sdasd|sfdsf|13|04|26|23|28|A1|sydney (10 Replies)
Discussion started by: radius
10 Replies

8. Shell Programming and Scripting

Split file based on records

I have to split a file based on number of lines and the below command works fine: split -l 2 Inputfile -d OutputfileMy input file contains header, detail and trailor info as below: H D D D D TMy split files for the above command contains: First File: H DSecond File: ... (11 Replies)
Discussion started by: Ajay Venkatesan
11 Replies

9. Shell Programming and Scripting

Split the File based on Size

I have a file that is about 7 GB in size. The requirement is I should split the file equally in such a way that the size of the split files is less than 2Gb. If the file is less than 2gb, than nothing needs to be done. ( need to done using shell script) Thanks, (4 Replies)
Discussion started by: rudoraj
4 Replies

10. UNIX for Beginners Questions & Answers

File Size Split up based on Month

Hi, I have a directory in Unix and there are folders available in the directory. Files are created on different month and now i have a requirement to calculate size of the folder on month basis. Is there any Unix command to check this please?? Thanks (6 Replies)
Discussion started by: Nivas
6 Replies
funhist(1)							SAORD Documentation							funhist(1)

NAME
funhist - create a 1D histogram of a column (from a FITS binary table or raw event file) or an image SYNOPSIS
funhist [-n|-w|-T] <iname> [column] [[lo:hi:]bins] OPTIONS
-n # normalize bin value by the width of each bin -w # specify bin width instead of number of bins in arg3 -T # output in rdb/starbase format (tab separators) DESCRIPTION
funhist creates a one-dimensional histogram from the specified columns of a FITS Extension binary table of a FITS file (or from a non-FITS raw event file), or from a FITS image or array, and writes that histogram as an ASCII table. Alternatively, the program can perform a 1D projection of one of the image axes. The first argument to the program is required, and specifies the Funtools file: FITS table or image, raw event file, or array. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, and filters. For a table, the second argument also is required. It specifies the column to use in generating the histogram. If the data file is of type image (or array), the column is optional: if "x" (or "X"), "y" (or "Y") is specified, then a projection is performed over the x (dim1) or y (dim2) axes, respectively. (That is, this projection will give the same results as a histogram performed on a table containing the equiva- lent x,y event rows.) If no column name is specified or "xy" (or "XY") is specified for the image, then a histogram is performed on the values contained in the image pixels. The argument that follows is optional and specifies the number of bins to use in creating the histogram and, if desired, the range of bin values. For image and table histograms, the range should specify the min and max data values. For image histograms on the x and y axes, the range should specify the min and max image bin values. If this argument is omitted, the number of output bins for a table is calcu- lated either from the TLMIN/TLMAX headers values (if these exist in the table FITS header for the specified column) or by going through the data to calculate the min and max value. For an image, the number of output bins is calculated either from the DATAMIN/DATAMAX header val- ues, or by going through the data to calculate min and max value. (Note that this latter calculation might fail if the image cannot be fit in memory.) If the data are floating point (table or image) and the number of bins is not specified, an arbitrary default of 128 is used. For binary table processing, the -w (bin width) switch can be used to specify the width of each bin rather than the number of bins. Thus: funhist test.ev pha 1:100:5 means that 5 bins of width 20 are used in the histogram, while: funhist -w test.ev pha 1:100:5 means that 20 bins of width 5 are used in the histogram. The data are divvied up into the specified number of bins and the resulting 1D histogram (or projection) is output in ASCII table format. For a table, the output displays the low_edge (inclusive) and hi_edge (exclusive) values for the data. For example, a 15-row table con- taining a "pha" column whose values range from -7.5 to 7.5 can be processed thus: [sh] funhist test.ev pha # data file: /home/eric/data/test.ev # column: pha # min,max,bins: -7.5 7.5 15 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 22 -7.50000000 -6.50000000 2 21 -6.50000000 -5.50000000 3 20 -5.50000000 -4.50000000 4 19 -4.50000000 -3.50000000 5 18 -3.50000000 -2.50000000 6 17 -2.50000000 -1.50000000 7 16 -1.50000000 -0.50000000 8 30 -0.50000000 0.50000000 9 16 0.50000000 1.50000000 10 17 1.50000000 2.50000000 11 18 2.50000000 3.50000000 12 19 3.50000000 4.50000000 13 20 4.50000000 5.50000000 14 21 5.50000000 6.50000000 15 22 6.50000000 7.50000000 [sh] funhist test.ev pha 1:6 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 6 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 16 0.50000000 1.50000000 2 17 1.50000000 2.50000000 3 18 2.50000000 3.50000000 4 19 3.50000000 4.50000000 5 20 4.50000000 5.50000000 6 21 5.50000000 6.50000000 [sh] funhist test.ev pha 1:6:3 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 3 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 33 0.50000000 2.50000000 2 37 2.50000000 4.50000000 3 41 4.50000000 6.50000000 For a table histogram, the -n(normalize) switch can be used to normalize the bin value by the width of the bin (i.e., hi_edge-lo_edge): [sh] funhist -n test.ev pha 1:6:3 # data file: test.ev # column: pha # min,max,bins: 0.5 6.5 3 # width normalization (val/(hi_edge-lo_edge)) is applied bin value lo_edge hi_edge ------ --------------------- --------------------- --------------------- 1 16.50000000 0.50000000 2.50000000 2 6.16666667 2.50000000 4.50000000 3 4.10000000 4.50000000 6.50000000 This could used, for example, to produce a light curve with values having units of counts/second instead of counts. For an image histogram, the output displays the low and high image values (both inclusive) used to generate the histogram. For example, in the following example, 184 pixels had a value of 1, 31 had a value of 2, while only 2 had a value of 3,4,5,6, or 7: [sh] funhist test.fits # data file: /home/eric/data/test.fits # min,max,bins: 1 7 7 bin value lo_val hi_val ------ --------------------- --------------------- --------------------- 1 184.00000000 1.00000000 1.00000000 2 31.00000000 2.00000000 2.00000000 3 2.00000000 3.00000000 3.00000000 4 2.00000000 4.00000000 4.00000000 5 2.00000000 5.00000000 5.00000000 6 2.00000000 6.00000000 6.00000000 7 2.00000000 7.00000000 7.00000000 For the axis projection of an image, the output displays the low and high image bins (both inclusive) used to generate the projection. For example, in the following example, 21 counts had their X bin value of 2, etc.: [sh] funhist test.fits x 2:7 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 6 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 21.00000000 2.00000000 2.00000000 2 20.00000000 3.00000000 3.00000000 3 19.00000000 4.00000000 4.00000000 4 18.00000000 5.00000000 5.00000000 5 17.00000000 6.00000000 6.00000000 6 16.00000000 7.00000000 7.00000000 [sh] funhist test.fits x 2:7:2 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 2 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 60.00000000 2.00000000 4.00000000 2 51.00000000 5.00000000 7.00000000 You can use gnuplot or other plotting programs to graph the results, using a script such as: #!/bin/sh sed -e '1,/---- .*/d /^$/,$d' | awk ' BEGIN{print "set nokey; set title "funhist"; set xlabel "bin"; set ylabel "counts"; plot "-" with boxes"} {print $3, $2, $4-$3}' | gnuplot -persist - 1>/dev/null 2>&1 Similar plot commands are supplied in the script funhist.plot: funhist test.ev pha ... | funhist.plot gnuplot SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funhist(1)
All times are GMT -4. The time now is 09:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy