Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Help with extracting data and plotting Post 302303897 by rnturner on Friday 3rd of April 2009 09:22:26 PM
Old 04-03-2009
I'll assume you're on a UNIX system and Perl and gnuplot are available.

The following (cheap and pretty dirty) Perl script will extract the two columns you're interested in:


Code:
#!/usr/bin/perl
#
#  Name: extract
#  Use: extract data.file
#
if ( $#ARGV == 0 )
  {              # First comand line arg is data file name
    $input_data = $ARGV[0];
    $output_data = $input_data . ".extracted_data";
  }
else
  {
    printf STDERR "Usage: extract data-file-name\n";
    exit 1;
  }

$extract_data = 0;   # Initially "false"

open(OUTPUT, '>', $output_data);

open (DATA, '<', $input_data);

RECORD:
while (<DATA>)
  {
    chomp;
    s/^\s+//;                # Remove leading whitespace
    if ( length($_) <= 0 )
      {       # Skipping empty lines
	next RECORD;
      }
    if ( /E\/N/ )
      {
	if ( / Ko / )
	  {             # Found end of the header records. Data follows...
	    $extract_data = -1;
	    next RECORD;
	  }
      }
    if ( $extract_data )
      {
	@values = split(/\s+/, $_);
	printf OUTPUT "%s %s\n", $values[0], $values[1];
      }
  }

close(DATA);

close(OUTPUT);

exit;

(Sorry... the forum software does awful things to the indentation of the script and I'm not sure how to prevent that.)

This would be run using:

Code:
   chmod u+x extract
   ./extract data.file

and produce

Code:
   data.file.extracted_data

A simple gnuplot command file to plot two curves might be:

set terminal postscript landscape;
plot "data.file.extracted_data" with lines, "data2.file.extracted_data" with lines;

If these were saved in a file named "gnuplot.cmds", you would generate the plot using:
Code:
gnuplot < gnuplot.cmds > plotted_data.ps

If you need to plot more than two curves, add another datafile name containing extracted data file to the end of the "plot" command and rerun the above gnuplot command. You could also add extra commands to control the X and Y axes, etc. (above the line containing the "plot" command).

Hope this helps.

--
Rick

Last edited by Yogesh Sawant; 04-04-2009 at 07:11 AM.. Reason: added code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting certain data from a sentence

How do I delete text in each line from the first character up to a certain pattern, ie. 'qmuser.' and then delete from the next occurence of a dot to the end of the sentence For example: - LTSB Renewal Notice Travel Pack --- d \qmaster\spool1\qmuser.8664_LM245_20031216094308.ps.0 From this... (7 Replies)
Discussion started by: dbrundrett
7 Replies

2. UNIX for Advanced & Expert Users

Gnuplot question: plotting 3D data in map view

I have a simple gnuplot question. I have a set of points (list of x,y,z values; irregularly spaced, i.e. no grid) that I want to plot. I want the plot to look like this: - map view (no 3D view) - color of each point should depend on z-value. - I want to define my own color scale - plot should... (1 Reply)
Discussion started by: karman
1 Replies

3. Shell Programming and Scripting

Extracting ICMP Output for Plotting

Dear, I want to perform a plotting using xgraph, and the plotting data (ping.txt) is as below. For the graph I just want to plot the time for x-axis (line count) and the RTT for y-axis (time in ms). Below are script i write for that purpose but it seen did not work. Any guide for me because i... (1 Reply)
Discussion started by: Paris Heng
1 Replies

4. UNIX for Dummies Questions & Answers

Plotting Data within UNIX

I have a set of data that looks similar to the following in UNIX: 0.12_0.008_fall_ff.out:bisect return: 0.08056640625 0.12_0.04_fall_ff.out:bisect return: 0.07470703125 0.12_0.12_fall_ff.out:bisect return: 0.06298828125 0.12_0.24_fall_ff.out:bisect return: 0.05126953125 Previously I have... (4 Replies)
Discussion started by: EDALBNUG
4 Replies

5. UNIX for Dummies Questions & Answers

Extracting Data Using SED

Given the following text in a file named extract.txt: listenPort:=25 smtpDestination:=2 enableSSL:= I am trying to extract only the value 2 following smtpDestination:= Someone had suggested I use: sed -e "s/^smtpDestination:=\(.*\)$/\1/" extract.txt but this returns: listenPort:=25 2 ... (2 Replies)
Discussion started by: cleanden
2 Replies

6. Shell Programming and Scripting

extracting data from a string

Hi there, I have a bunch of vlan tagged network interfaces that are named as follows e1000g111000 e1000g99001 e1000g3456000 nge2002 where the 'e1000g' and 'nge' parts of the name are the driver, the red and blue bits above define the VLAN and the last digit on the end defines the... (3 Replies)
Discussion started by: rethink
3 Replies

7. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

8. Shell Programming and Scripting

extracting data

I have a txt file of the following format >ab_ qwerty >rt_ hfjkil >Ty2 hglashglkasghkf; >P2 aklhfklflkkgfgkfl >ui_ vnllkdskkkffkfkkf >we32 vksksjksj;lslsf'sk's's .... ..... I want to split this big file based on the header (>) (5 Replies)
Discussion started by: Lucky Ali
5 Replies

9. Shell Programming and Scripting

Help with extracting data within parentheses

This is my input file: a|b|c(ef)|g|h(km)|p My output file should look like: a|b|ef|g|km|p That is, pipe is the delimiter. The data within pipe must be displayed as it is but if it encounters any data within parentheses, then only the data within parentheses has to be displayed ( the data... (2 Replies)
Discussion started by: ksatish89
2 Replies

10. Shell Programming and Scripting

Fun with terminal plotting JSON data at the command line

One of the great thing about unix is the ability to pipe multiple programs together to manipulate data. Plain, unstructured text is the most common type of data that is passed between programs, but these days JSON is becoming more popular. I thought it would be fun to pipe together some command... (1 Reply)
Discussion started by: kbrazil
1 Replies
funhist(1)							SAORD Documentation							funhist(1)

NAME
funhist - create a 1D histogram of a column (from a FITS binary table or raw event file) or an image SYNOPSIS
funhist [-n|-w|-T] <iname> [column] [[lo:hi:]bins] OPTIONS
-n # normalize bin value by the width of each bin -w # specify bin width instead of number of bins in arg3 -T # output in rdb/starbase format (tab separators) DESCRIPTION
funhist creates a one-dimensional histogram from the specified columns of a FITS Extension binary table of a FITS file (or from a non-FITS raw event file), or from a FITS image or array, and writes that histogram as an ASCII table. Alternatively, the program can perform a 1D projection of one of the image axes. The first argument to the program is required, and specifies the Funtools file: FITS table or image, raw event file, or array. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions, and filters. For a table, the second argument also is required. It specifies the column to use in generating the histogram. If the data file is of type image (or array), the column is optional: if "x" (or "X"), "y" (or "Y") is specified, then a projection is performed over the x (dim1) or y (dim2) axes, respectively. (That is, this projection will give the same results as a histogram performed on a table containing the equiva- lent x,y event rows.) If no column name is specified or "xy" (or "XY") is specified for the image, then a histogram is performed on the values contained in the image pixels. The argument that follows is optional and specifies the number of bins to use in creating the histogram and, if desired, the range of bin values. For image and table histograms, the range should specify the min and max data values. For image histograms on the x and y axes, the range should specify the min and max image bin values. If this argument is omitted, the number of output bins for a table is calcu- lated either from the TLMIN/TLMAX headers values (if these exist in the table FITS header for the specified column) or by going through the data to calculate the min and max value. For an image, the number of output bins is calculated either from the DATAMIN/DATAMAX header val- ues, or by going through the data to calculate min and max value. (Note that this latter calculation might fail if the image cannot be fit in memory.) If the data are floating point (table or image) and the number of bins is not specified, an arbitrary default of 128 is used. For binary table processing, the -w (bin width) switch can be used to specify the width of each bin rather than the number of bins. Thus: funhist test.ev pha 1:100:5 means that 5 bins of width 20 are used in the histogram, while: funhist -w test.ev pha 1:100:5 means that 20 bins of width 5 are used in the histogram. The data are divvied up into the specified number of bins and the resulting 1D histogram (or projection) is output in ASCII table format. For a table, the output displays the low_edge (inclusive) and hi_edge (exclusive) values for the data. For example, a 15-row table con- taining a "pha" column whose values range from -7.5 to 7.5 can be processed thus: [sh] funhist test.ev pha # data file: /home/eric/data/test.ev # column: pha # min,max,bins: -7.5 7.5 15 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 22 -7.50000000 -6.50000000 2 21 -6.50000000 -5.50000000 3 20 -5.50000000 -4.50000000 4 19 -4.50000000 -3.50000000 5 18 -3.50000000 -2.50000000 6 17 -2.50000000 -1.50000000 7 16 -1.50000000 -0.50000000 8 30 -0.50000000 0.50000000 9 16 0.50000000 1.50000000 10 17 1.50000000 2.50000000 11 18 2.50000000 3.50000000 12 19 3.50000000 4.50000000 13 20 4.50000000 5.50000000 14 21 5.50000000 6.50000000 15 22 6.50000000 7.50000000 [sh] funhist test.ev pha 1:6 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 6 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 16 0.50000000 1.50000000 2 17 1.50000000 2.50000000 3 18 2.50000000 3.50000000 4 19 3.50000000 4.50000000 5 20 4.50000000 5.50000000 6 21 5.50000000 6.50000000 [sh] funhist test.ev pha 1:6:3 # data file: /home/eric/data/test.ev # column: pha # min,max,bins: 0.5 6.5 3 bin value lo_edge hi_edge ------ --------- --------------------- --------------------- 1 33 0.50000000 2.50000000 2 37 2.50000000 4.50000000 3 41 4.50000000 6.50000000 For a table histogram, the -n(normalize) switch can be used to normalize the bin value by the width of the bin (i.e., hi_edge-lo_edge): [sh] funhist -n test.ev pha 1:6:3 # data file: test.ev # column: pha # min,max,bins: 0.5 6.5 3 # width normalization (val/(hi_edge-lo_edge)) is applied bin value lo_edge hi_edge ------ --------------------- --------------------- --------------------- 1 16.50000000 0.50000000 2.50000000 2 6.16666667 2.50000000 4.50000000 3 4.10000000 4.50000000 6.50000000 This could used, for example, to produce a light curve with values having units of counts/second instead of counts. For an image histogram, the output displays the low and high image values (both inclusive) used to generate the histogram. For example, in the following example, 184 pixels had a value of 1, 31 had a value of 2, while only 2 had a value of 3,4,5,6, or 7: [sh] funhist test.fits # data file: /home/eric/data/test.fits # min,max,bins: 1 7 7 bin value lo_val hi_val ------ --------------------- --------------------- --------------------- 1 184.00000000 1.00000000 1.00000000 2 31.00000000 2.00000000 2.00000000 3 2.00000000 3.00000000 3.00000000 4 2.00000000 4.00000000 4.00000000 5 2.00000000 5.00000000 5.00000000 6 2.00000000 6.00000000 6.00000000 7 2.00000000 7.00000000 7.00000000 For the axis projection of an image, the output displays the low and high image bins (both inclusive) used to generate the projection. For example, in the following example, 21 counts had their X bin value of 2, etc.: [sh] funhist test.fits x 2:7 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 6 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 21.00000000 2.00000000 2.00000000 2 20.00000000 3.00000000 3.00000000 3 19.00000000 4.00000000 4.00000000 4 18.00000000 5.00000000 5.00000000 5 17.00000000 6.00000000 6.00000000 6 16.00000000 7.00000000 7.00000000 [sh] funhist test.fits x 2:7:2 # data file: /home/eric/data/test.fits # column: X # min,max,bins: 2 7 2 bin value lo_bin hi_bin ------ --------------------- --------------------- --------------------- 1 60.00000000 2.00000000 4.00000000 2 51.00000000 5.00000000 7.00000000 You can use gnuplot or other plotting programs to graph the results, using a script such as: #!/bin/sh sed -e '1,/---- .*/d /^$/,$d' | awk ' BEGIN{print "set nokey; set title "funhist"; set xlabel "bin"; set ylabel "counts"; plot "-" with boxes"} {print $3, $2, $4-$3}' | gnuplot -persist - 1>/dev/null 2>&1 Similar plot commands are supplied in the script funhist.plot: funhist test.ev pha ... | funhist.plot gnuplot SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funhist(1)
All times are GMT -4. The time now is 07:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy