Sponsored Content
Top Forums UNIX for Beginners Questions & Answers sed awk: split a large file to unique file names Post 302980154 by MadeInGermany on Wednesday 24th of August 2016 01:42:55 PM
Old 08-24-2016
I like the shell solution. It can become a little more I/O efficient (matters when the output files are written to a network file system):
Code:
p_col1=""
while read col1 rest
do
  if [ "$col1" != "$p_col1" ]
  then
    p_col1=$col1
    exec 3>"$col1".txt
  fi
  echo "$col1 $rest" >&3
done < Input_file
exec 3>&-

The input file must be sorted on col1 (otherwise: remove previous output files and append with exec 3>>"$col1".txt)

Last edited by MadeInGermany; 08-24-2016 at 04:42 PM.. Reason: Removed comment about awk - close() releases the file descriptors!
This User Gave Thanks to MadeInGermany For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

2. UNIX for Dummies Questions & Answers

split a file with unique sets

This may sound like a trivial problem, but I still need some help: I have a file with ids and I want to split it 'n' ways (could be any number) into files: 1 1 1 2 2 3 3 4 5 5 Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may... (8 Replies)
Discussion started by: ChicagoBlues
8 Replies

3. Shell Programming and Scripting

extract unique pattern from large text file

Hi All, I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like 12345- , i tried with egrep ,eg : egrep "+" text.txt but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
Discussion started by: shijujoe
19 Replies

4. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

5. UNIX for Dummies Questions & Answers

Get List of Unique File Names

I have a large directory of web pages. I am doing a search through the web pages using grep and would like to get a list of unique file names of search results. The following command works fine to give me a list of file names where term appears: grep -l term *.html However, since these are... (3 Replies)
Discussion started by: rjulich
3 Replies

6. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

7. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

8. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

9. Shell Programming and Scripting

sed and awk not working on a large record file

Hi All, I have a very large single record file. abc;date||bcd;efg|......... pqr;stu||record_count;date when i do wc -l on this file it gives me "0" records, coz of missing line feed. my problem is there is an extra pipe that is coming at the end of this record like... (6 Replies)
Discussion started by: Gurkamal83
6 Replies

10. Linux

Split a large textfile (one file) into multiple file to base on ^L

Hi, Anyone can help, I have a large textfile (one file), and I need to split into multiple file to break each file into ^L. My textfile ========== abc company abc address abc contact ^L my company my address my contact my skills ^L your company your address ========== (3 Replies)
Discussion started by: fspalero
3 Replies
funtable(1)							SAORD Documentation						       funtable(1)

NAME
funtable - copy selected rows from a Funtools file to a FITS binary table SYNOPSIS
funtable [-a] [-i|-z] [-m] [-s cols] <iname> <oname> [columns] OPTIONS
-a # append to existing output file as a table extension -i # for image data, only generate X and Y columns -m # for tables, write a separate file for each region -s "col1 ..." # columns on which to sort -z # for image data, output zero-valued pixels DESCRIPTION
funtable selects rows from the specified FITS Extension (binary table only) of a FITS file, or from a non-FITS raw event file, and writes those rows to a FITS binary table file. It also will create a FITS binary table from an image or a raw array file. The first argument to the program specifies the FITS file, raw event file, or raw array file. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, and filters. The second argument is the output FITS file. If "stdout" is specified, the FITS binary table is written to the standard output. By default, all columns of the input file are copied to the output file. Selected columns can be output using an optional third argument in the form: "column1 column1 ... columnN" The funtable program generally is used to select rows from a FITS binary table using Table Filters and/or Spatial Region Filters. For example, you can copy only selected rows (and output only selected columns) by executing in a command such as: [sh] funtable "test.ev[pha==1&&pi==10]" stdout "x y pi pha" | fundisp stdin X Y PHA PI ------- ------- ------- --------- 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 1 10 The special column $REGION can be specified to write the region id of each row: [sh $] funtable "test.ev[time-(int)time>=.99&&annulus(0 0 0 10 n=3)]" stdout 'x y time $REGION' | fundisp stdin X Y TIME REGION -------- -------- --------------------- ---------- 5 -6 40.99000000 3 4 -5 59.99000000 2 -1 0 154.99000000 1 -2 1 168.99000000 1 -3 2 183.99000000 2 -4 3 199.99000000 2 -5 4 216.99000000 2 -6 5 234.99000000 3 -7 6 253.99000000 3 Here only rows with the proper fractional time and whose position also is within one of the three annuli are written. Columns can be excluded from display using a minus sign before the column: [sh $] funtable "test.ev[time-(int)time>=.99]" stdout "-time" | fundisp stdin X Y PHA PI DX DY -------- -------- -------- ---------- ----------- ----------- 5 -6 5 -6 5.50 -6.50 4 -5 4 -5 4.50 -5.50 -1 0 -1 0 -1.50 0.50 -2 1 -2 1 -2.50 1.50 -3 2 -3 2 -3.50 2.50 -4 3 -4 3 -4.50 3.50 -5 4 -5 4 -5.50 4.50 -6 5 -6 5 -6.50 5.50 -7 6 -7 6 -7.50 6.50 All columns except the time column are written. In general, the rules for activating and de-activating columns are: o If only exclude columns are specified, then all columns but the exclude columns will be activated. o If only include columns are specified, then only the specified columns are activated. o If a mixture of include and exclude columns are specified, then all but the exclude columns will be active; this last case is ambiguous and the rule is arbitrary. In addition to specifying columns names explicitly, the special symbols + and - can be used to activate and de-activate all columns. This is useful if you want to activate the $REGION column along with all other columns. According to the rules, the syntax "$REGION" only acti- vates the region column and de-activates the rest. Use "+ $REGION" to activate all columns as well as the region column. Ordinarily, only the selected table is copied to the output file. In a FITS binary table, it sometimes is desirable to copy all of the other FITS extensions to the output file as well. This can be done by appending a '+' sign to the name of the extension in the input file name. For example, the first command below copies only the EVENT table, while the second command copies other extensions as well: [sh] funtable "/proj/rd/data/snr.ev[EVENTS]" events.ev [sh] funtable "/proj/rd/data/snr.ev[EVENTS+]" eventsandmore.ev If the input file is an image or a raw array file, then funtable will generate a FITS binary table from the pixel values in the image. Note that it is not possible to specify the columns to output (using command-line argument 3). Instead, there are two ways to create such a binary table from an image. By default, a 3-column table is generated, where the columns are "X", "Y", and "VALUE". For each pixel in the image, a single row (event) is generated with the "X" and "Y" columns assigned the dim1 and dim2 values of the image pixel, respectively and the "VALUE" column assigned the value of the pixel. With sort of table, running funhist on the "VALUE" column will give the same results as running funhist on the original image. If the -i ("individual" rows) switch is specified, then only the "X" and "Y" columns are generated. In this case, each positive pixel value in the image generates n rows (events), where n is equal to the integerized value of that pixel (plus 0.5, for floating point data). In effect, -i approximately recreates the rows of a table that would have been binned into the input image. (Of course, this is only approximately correct, since the resulting x,y positions are integerized.) If the -s [col1 col2 ... coln] ("sort") switch is specified, the output rows of a binary table will be sorted using the specified columns as sort keys. The sort keys must be scalar columns and also must be part of the output file (i.e. you cannot sort on a column but not include it in the output). This facility uses the _sort program (included with funtools), which must be accessible via your path. For binary tables, the -m ("multiple files") switch will generate a separate file for each region in the filter specification i.e. each file contains only the rows from that region. Rows which pass the filter but are not in any region also are put in a separate file. The separate output file names generated by the -m switch are produced automatically from the root output file to contain the region id of the associated region. (Note that region ids start at 1, so that the file name associated with id 0 contains rows that pass the filter but are not in any given region.) Output file names are generated as follows: o A $n specification can be used anywhere in the root file name (suitably quoted to protect it from the shell) and will be expanded to be the id number of the associated region. For example: funtable -m input.fits'[cir(512,512,1);cir(520,520,1)...]' 'foo.goo_$n.fits' will generate files named foo.goo_0.fits (for rows not in any region but still passing the filter), foo.goo_1.fits (rows in region id #1, the first region), foo.goo_2.fits (rows in region id #2), etc. Note that single quotes in the output root are required to protect the '$' from the shell. o If $n is not specified, then the region id will be placed before the first dot (.) in the filename. Thus: funtable -m input.fits'[cir(512,512,1);cir(520,520,1)...]' foo.evt.fits will generate files named foo0.evt.fits (for rows not in any region but still passing the filter), foo1.evt.fits (rows in region id #1), foo2.evt.fits (rows in region id #2), etc. o If no dot is specified in the root output file name, then the region id will be appended to the filename. Thus: funtable -m input.fits'[cir(512,512,1);cir(520,520,1)...]' 'foo_evt' will generate files named foo_evt0 (for rows not in any region but still passing the filter), foo_evt1 (rows in region id #1), foo_evt2 (rows in region id #2), etc. The multiple file mechanism provide a simple way to generate individual source data files with a single pass through the data. By default, a new FITS file is created and the binary table is written to the first extension. If the -a (append) switch is specified, the table is appended to an existing FITS file as a BINTABLE extension. Note that the output FITS file must already exist. If the -z ("zero" pixel values) switch is specified and -i is not specified, then pixels having a zero value will be output with their "VALUE" column set to zero. Obviously, this switch does not make sense when individual events are output. SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funtable(1)
All times are GMT -4. The time now is 10:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy