Sponsored Content
Top Forums Shell Programming and Scripting How to make AWK process an input file many many times? Post 302421321 by radoulov on Friday 14th of May 2010 05:34:02 AM
Old 05-14-2010
If you need to process the same data multiple times and the data is not that large (GBs), you'd better read it into an array (or multiple arrays) and process it in memory after reading the input.

If the data size does not fit into memory, you can manipulate the ARGV array (just like I showed you in one of your previous posts).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make my command work at all times

hi all, This is a very basic question. I want to make the command work at all times. i'm working on Suse-Linux and "clear" command is used to clear the contents of screen. I want to use only "cls" instead of "clear" command. i tried alias cls=clear , but its working only for a temporary... (3 Replies)
Discussion started by: wxwidgets
3 Replies

2. Shell Programming and Scripting

how to start a process and make it sleep for 5 mins and then kill that process

how to start a process and make it sleep for 5 mins and then kill that process (6 Replies)
Discussion started by: shrao
6 Replies

3. Shell Programming and Scripting

Can process substitution be used as an input file to another program?

Hey, I was trying to figure out how to launch a program from the command line, and it works if you pass it a config file. I was thinking about writing a script to dynamically create the config file and pass it to the command using something like command substitution (so I don't actually have to... (3 Replies)
Discussion started by: bj0
3 Replies

4. UNIX for Dummies Questions & Answers

make 1 new file using records of 5 input files

can you make me an awk script that will: read the 1st record from each input file, file1 file2 file3 file4 and file5 and write them to file6. then..... read the 2nd record from each input file, file1 file2 file3 file4 and file5 and append them to file6. then..... read the 3rd record... (1 Reply)
Discussion started by: kenneth.mcbride
1 Replies

5. Shell Programming and Scripting

How to make the parent process to wait for the child process

Hi All, I have two ksh script. 1st script calls the 2nd script and the second script calls an 'C' program. I want 1st script to wait until the 'C' program completes. I cant able to get the process id for the 'C' program (child process) to make the 1st script to wait for the second... (7 Replies)
Discussion started by: sennidurai
7 Replies

6. Shell Programming and Scripting

How to read the input from a file in background process?

Hi I have a script to run some other scripts automatically. But while running the script it should take the input value from a text file instead of taking from the keyboard. Please find last two lines of the script below. Here ans should be taken from a text file ineerly without displaying this... (1 Reply)
Discussion started by: shirdi
1 Replies

7. Shell Programming and Scripting

Reading a file several times with awk

Hi everyone, I was wondering if it's possible to read a file ("file2" in my example) more than once. In this example I want to print file2 entirely for each lines of file1: awk -F$'\t' '{ print $0 while ((getline < "file2") > 0) { print "\t"$0 } }' file1 It... (4 Replies)
Discussion started by: anthony.cros
4 Replies

8. Shell Programming and Scripting

The scripts not able to make the file to size 0, every times it go back to its original size

#!/bin/sh ########################################################################################################## #This script is being used for AOK application for cleaning up the .out files and zip it under logs directory. # IBM # Created #For pdocap201/pdoca202 .out files for AOK #1.... (0 Replies)
Discussion started by: mridul10_crj
0 Replies

9. Shell Programming and Scripting

make loop from input file

Hi Guys, I have file A.txt PP97 PP66 PP87 PP66 PP47 PP57 PP44 PP20 PP66 PP16 PP13 PP51 PP68 PP70 PP75 PP30 (2 Replies)
Discussion started by: asavaliya
2 Replies

10. Shell Programming and Scripting

How to check same process running 10 times?

At run time Without knowing job name how to check the job running in specific user "ABCD" ,If the running job duplicate more then 10 then script it self send alert message to the users with the process ID name so that will kill the processed to avoid hung issue ,tried below script please check and... (15 Replies)
Discussion started by: Kalia
15 Replies
Linfit(3)						User Contributed Perl Documentation						 Linfit(3)

NAME
PDL::Fit::Linfit - routines for fitting data with linear combinations of functions. DESCRIPTION
This module contains routines to perform general curve-fits to a set (linear combination) of specified functions. Given a set of Data: (y0, y1, y2, y3, y4, y5, ...ynoPoints-1) The fit routine tries to model y as: y' = beta0*x0 + beta1*x1 + ... beta_noCoefs*x_noCoefs Where x0, x1, ... x_noCoefs, is a set of functions (curves) that the are combined linearly using the beta coefs to yield an approximation of the input data. The Sum-Sq error is reduced to a minimum in this curve fit. Inputs: $data This is your data you are trying to fit. Size=n $functions 2D array. size (n, noCoefs). Row 0 is the evaluation of function x0 at all the points in y. Row 1 is the evaluation of of function x1 at all the points in y, ... etc. Example of $functions array Structure: $data is a set of 10 points that we are trying to model using the linear combination of 3 functions. $functions = ( [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ], # Constant Term [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ], # Linear Slope Term [ 0, 2, 4, 9, 16, 25, 36, 49, 64, 81] # quadradic term ) SYNOPSIS
$yfit = linfit1d $data, $funcs FUNCTIONS
linfit1d 1D Fit linear combination of supplied functions to data using min chi^2 (least squares). Usage: ($yfit, [$coeffs]) = linfit1d [$xdata], $data, $fitFuncs, [Options...] Signature: (xdata(n); ydata(n); $fitFuncs(n,order); [o]yfit(n); [o]coeffs(order)) Uses a standard matrix inversion method to do a least squares/min chi^2 fit to data. Returns the fitted data and optionally the coefficients. One can thread over extra dimensions to do multiple fits (except the order can not be threaded over - i.e. it must be one fixed set of fit functions "fitFuncs". The data is normalised internally to avoid overflows (using the mean of the abs value) which are common in large polynomial series but the returned fit, coeffs are in unnormalised units. # Generate data from a set of functions $xvalues = sequence(100); $data = 3*$xvalues + 2*cos($xvalues) + 3*sin($xvalues*2); # Make the fit Functions $fitFuncs = cat $xvalues, cos($xvalues), sin($xvalues*2); # Now fit the data, Coefs should be the coefs in the linear combination # above: 3,2,3 ($yfit, $coeffs) = linfit1d $data,$fitFuncs; Options: Weights Weights to use in fit, e.g. 1/$sigma**2 (default=1) perl v5.12.1 2009-12-29 Linfit(3)
All times are GMT -4. The time now is 12:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy