Sponsored Content
Full Discussion: Fill in missing Data
Top Forums Shell Programming and Scripting Fill in missing Data Post 302609933 by Corona688 on Tuesday 20th of March 2012 03:28:28 PM
Old 03-20-2012
Why not fill them in with zeroes?

Code:
$ awk -F"," -v OFS="," '{ for (N=2; N<=NF; N++) if (!$N || ($N==" ")) $N=0 } 1' data
Wed Feb 01 09:00:02 EST 2012,0,0,0.4,0.3,0,0.3,0,0.3,0,0.5,0,0.3,0,0,0.4,0.3,0
Wed Feb 01 09:00:11 EST 2012,0,0,0,0,0,0,0.2,0,0,0,0,0,0,0,0,0,0
Wed Feb 01 09:00:22 EST 2012,0.7,0.4,0.5,0.4,0.3,0.4,0,0.4,0.5,0.6,0.3,0.4,0,0.1,0.5,0.4,0
Wed Feb 01 09:00:42 EST 2012,0.9,1.0,0.6,0.5,0,0,0,0,0,0,0,0.5,0,0,0,0,0
Wed Feb 01 09:01:02 EST 2012,1.0,0.8,0,0,0,0.6,0,0,0,0,0,0.6,0,0,0,0,0
Wed Feb 01 09:01:22 EST 2012,0,0.7,0.7,0.6,0.5,0,0,0.5,0,0,0,0,0,0,0,0,0
Wed Feb 01 09:01:42 EST 2012,0.9,0.6,0.6,0.5,0,0,0,0.4,0.7,0.7,0.4,0.5,0,0,0.6,0.5,0
Wed Feb 01 09:02:02 EST 2012,0.8,0,0.5,0.4,0,0,0,1.1,0,0,0.3,0.4,0,0,0,0,0
Wed Feb 01 09:02:22 EST 2012,0.9,0,0,0,0,0,0,0.7,0,0.6,0,0,0,0,0,0,0

$

Or do you want the missing value to be filled in with data from last time? That'd make more sense than interpolating with data from other servers...

Code:
awk -F"," -v OFS="," '{ for (N=2; N<=NF; N++) if (!$N || ($N==" ")) { $N=D[N] } else { D[N]=$N } } 1' data
Wed Feb 01 09:00:02 EST 2012,,,0.4,0.3,,0.3,,0.3,,0.5,,0.3,,,0.4,0.3,
Wed Feb 01 09:00:11 EST 2012,,,0.4,0.3,,0.3,0.2,0.3,,0.5,,0.3,,,0.4,0.3,
Wed Feb 01 09:00:22 EST 2012,0.7,0.4,0.5,0.4,0.3,0.4,0.2,0.4,0.5,0.6,0.3,0.4,,0.1,0.5,0.4,
Wed Feb 01 09:00:42 EST 2012,0.9,1.0,0.6,0.5,0.3,0.4,0.2,0.4,0.5,0.6,0.3,0.5,,0.1,0.5,0.4,
Wed Feb 01 09:01:02 EST 2012,1.0,0.8,0.6,0.5,0.3,0.6,0.2,0.4,0.5,0.6,0.3,0.6,,0.1,0.5,0.4,
Wed Feb 01 09:01:22 EST 2012,1.0,0.7,0.7,0.6,0.5,0.6,0.2,0.5,0.5,0.6,0.3,0.6,,0.1,0.5,0.4,
Wed Feb 01 09:01:42 EST 2012,0.9,0.6,0.6,0.5,0.5,0.6,0.2,0.4,0.7,0.7,0.4,0.5,,0.1,0.6,0.5,
Wed Feb 01 09:02:02 EST 2012,0.8,0.6,0.5,0.4,0.5,0.6,0.2,1.1,0.7,0.7,0.3,0.4,,0.1,0.6,0.5,
Wed Feb 01 09:02:22 EST 2012,0.9,0.6,0.5,0.4,0.5,0.6,0.2,0.7,0.7,0.6,0.3,0.4,,0.1,0.6,0.5,

$


Last edited by Corona688; 03-20-2012 at 04:34 PM..
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Fill in missing numbers in range

I need to edit a list of numbers on the following form: 1 1.0 2 1.4 5 2.1 7 1.9 I want: 1 1.0 2 1.4 3 0.0 4 0.0 5 2.1 6 0.0 7 1.9 (i want to add the missing number in column 1 together with 0.0 in column 2). I guess it is rather trivial but i didn't even manage to read column... (5 Replies)
Discussion started by: bistru
5 Replies

2. Shell Programming and Scripting

fill in missing columns

It can't be that hard, but I just can't figure it out: I have file like: File Sub-brick M_1 S_1 M_2 S_2 M_4 S_4 ... xxx 1 214 731 228 621 132 578 ... and would like to get 0 0 where M_3 S_3 is missing xxx 1 214 731 228 621 0 0 132 578 ... I wrote following script, but can't figure out... (3 Replies)
Discussion started by: avvk
3 Replies

3. Shell Programming and Scripting

Fill missing numbers in second column with zeros

Hi All, I have 100 files with names like this: 1.dat, 2.dat, 3.dat until 100.dat. My dat files look like this: 42323 0 438939 1 434 0 0.9383 3434 120.23 3 234 As you can see in the second column, some numbers are missing. I want to fill those missing places with 0's in all... (3 Replies)
Discussion started by: shoaibjameel123
3 Replies

4. Shell Programming and Scripting

Fill missing values with 2

Hi All, I have 100 .txt files which look like this: 3 4 5 6 7 Now, some files have some numbers missing in them and they look like this: 4 5 6 (6 Replies)
Discussion started by: shoaibjameel123
6 Replies

5. Shell Programming and Scripting

fill in last column of data

Hello, I am fairly new to awk, and I have the following problem. My file has missing data in the last column, and the program I am pre-processing this file for cannot interpret correctly shortened rows (it just wraps the data around). Is there a way to force awk to create the same... (6 Replies)
Discussion started by: timert34
6 Replies

6. Shell Programming and Scripting

Fill data if number range is given

Hi I want to get all numbers if number range is given as input. Eg: INPUT FILE 100-105 107 108-112 OUTPUT REQUIRED: 100 101 102 103 104 105 107 108 109 110 111 112 How can I do it using shell? :confused: Thanks in advance. (11 Replies)
Discussion started by: dashing201
11 Replies

7. Shell Programming and Scripting

Fill in missing rows with zero to have uniform table

Hello, I have two files of same structure except some rows are missing randomly in each file. How do I fill the missing rows to have the exact ID column (S01 ~ S96) and rest columns filled with "0" with awk? The purpose of this step is to join the two files side by side. The closest thread is... (17 Replies)
Discussion started by: yifangt
17 Replies

8. UNIX for Beginners Questions & Answers

Fill in missing hours and interpolate values using awk.

I have a time series data like this 40754,35.6931,51.3092,201610160700,21.0 40754,35.6931,51.3092,201610160800,23.0 40754,35.6931,51.3092,201610160900,24.0 40754,35.6931,51.3092,201610161000,24.0 40754,35.6931,51.3092,201610161300,25.0 40754,35.6931,51.3092,201610161400,23.0... (6 Replies)
Discussion started by: emirzaei
6 Replies

9. Shell Programming and Scripting

Fill in missing values

Hi, I have a data sample as shown below. I want to fill in the left column so that the line will be continuous. For example, between 1 and 5 should be 2,3,4. And corresponding values in the right column will be 0. Thus the expected data should look like that: 1 1 1 10 1 2 1 3 1 5 1 6 2 0... (6 Replies)
Discussion started by: theanh0508
6 Replies
Basic(3)						User Contributed Perl Documentation						  Basic(3)

NAME
PDL::Basic -- Basic utility functions for PDL DESCRIPTION
This module contains basic utility functions for creating and manipulating piddles. Most of these functions are simplified interfaces to the more flexible functions in the modules PDL::Primitive and PDL::Slices. SYNOPSIS
use PDL::Basic; FUNCTIONS
xvals Fills a piddle with X index values $x = xvals($somearray); $x = xvals([OPTIONAL TYPE],$nx,$ny,$nz...); etc. see zeroes. perldl> print xvals zeroes(5,10) [ [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] [0 1 2 3 4] ] yvals Fills a piddle with Y index values $x = yvals($somearray); yvals(inplace($somearray)); $x = yvals([OPTIONAL TYPE],$nx,$ny,$nz...); etc. see zeroes. perldl> print yvals zeroes(5,10) [ [0 0 0 0 0] [1 1 1 1 1] [2 2 2 2 2] [3 3 3 3 3] [4 4 4 4 4] [5 5 5 5 5] [6 6 6 6 6] [7 7 7 7 7] [8 8 8 8 8] [9 9 9 9 9] ] zvals Fills a piddle with Z index values $x = zvals($somearray); zvals(inplace($somearray)); $x = zvals([OPTIONAL TYPE],$nx,$ny,$nz...); etc. see zeroes. perldl> print zvals zeroes(3,4,2) [ [ [0 0 0] [0 0 0] [0 0 0] [0 0 0] ] [ [1 1 1] [1 1 1] [1 1 1] [1 1 1] ] ] xlinvals X axis values between endpoints (see xvals). $a = zeroes(100,100); $x = $a->xlinvals(0.5,1.5); $y = $a->ylinvals(-2,-1); # calculate Z for X between 0.5 and 1.5 and # Y between -2 and -1. $z = f($x,$y); "xlinvals", "ylinvals" and "zlinvals" return a piddle with the same shape as their first argument and linearly scaled values between the two other arguments along the given axis. ylinvals Y axis values between endpoints (see yvals). See xlinvals for more information. zlinvals Z axis values between endpoints (see zvals). See xlinvals for more information. hist Create histogram of a piddle $hist = hist($data,[$min,$max,$step]); ($xvals,$hist) = hist($data,[$min,$max,$step]); If requested, $xvals gives the computed bin centres A nice idiom (with PDL::Graphics::PGPLOT) is bin hist $data; # Plot histogram perldl> p $y [13 10 13 10 9 13 9 12 11 10 10 13 7 6 8 10 11 7 12 9 11 11 12 6 12 7] perldl> $h = hist $y,0,20,1; # hist with step 1, min 0 and 20 bins perldl> p $h [0 0 0 0 0 0 2 3 1 3 5 4 4 4 0 0 0 0 0 0] whist Create a weighted histogram of a piddle $hist = whist($data, $wt, [$min,$max,$step]); ($xvals,$hist) = whist($data, $wt, [$min,$max,$step]); If requested, $xvals gives the computed bin centres. $data and $wt should have the same dimensionality and extents. A nice idiom (with PDL::Graphics::PGPLOT) is bin whist $data, $wt; # Plot histogram perldl> p $y [13 10 13 10 9 13 9 12 11 10 10 13 7 6 8 10 11 7 12 9 11 11 12 6 12 7] perldl> $wt = grandom($y->nelem) perldl> $h = whist $y, $wt, 0, 20, 1 # hist with step 1, min 0 and 20 bins perldl> p $h [0 0 0 0 0 0 -0.49552342 1.7987439 0.39450696 4.0073722 -2.6255299 -2.5084501 2.6458365 4.1671676 0 0 0 0 0 0] sequence Create array filled with a sequence of values $a = sequence($b); $a = sequence [OPTIONAL TYPE], @dims; etc. see zeroes. perldl> p sequence(10) [0 1 2 3 4 5 6 7 8 9] perldl> p sequence(3,4) [ [ 0 1 2] [ 3 4 5] [ 6 7 8] [ 9 10 11] ] rvals Fills a piddle with radial distance values from some centre. $r = rvals $piddle,{OPTIONS}; $r = rvals [OPTIONAL TYPE],$nx,$ny,...{OPTIONS}; Options: Centre => [$x,$y,$z...] # Specify centre Center => [$x,$y.$z...] # synonym. Squared => 1 # return distance squared (i.e., don't take the square root) perldl> print rvals long,7,7,{Centre=>[2,2]} [ [2 2 2 2 2 3 4] [2 1 1 1 2 3 4] [2 1 0 1 2 3 4] [2 1 1 1 2 3 4] [2 2 2 2 2 3 4] [3 3 3 3 3 4 5] [4 4 4 4 4 5 5] ] For a more general metric, one can define, e.g., sub distance { my ($a,$centre,$f) = @_; my ($r) = $a->allaxisvals-$centre; $f->($r); } sub l1 { sumover(abs($_[0])); } sub euclid { use PDL::Math 'pow'; pow(sumover(pow($_[0],2)),0.5); } sub linfty { maximum(abs($_[0])); } so now distance($a, $centre, &euclid); will emulate rvals, while "&l1" and "&linfty" will generate other well-known norms. axisvals Fills a piddle with index values on Nth dimension $z = axisvals ($piddle, $nth); This is the routine, for which xvals, yvals etc are mere shorthands. "axisvals" can be used to fill along any dimension. Note the 'from specification' style (see zeroes) is not available here, for obvious reasons. allaxisvals Generates a piddle with index values $z = allaxisvals ($piddle); "allaxisvals" produces an array with axis values along each dimension, adding an extra dimension at the start. "allaxisvals($piddle)->slice("($nth)")" will produce the same result as "axisvals($piddle,$nth)" (although with extra work and not inplace). It's useful when all the values will be required, as in the example given of a generalized rvals. transpose transpose rows and columns. $b = transpose($a); $b = ~$a; Also bound to the "~" unary operator in PDL::Matrix. perldl> $a = sequence(3,2) perldl> p $a [ [0 1 2] [3 4 5] ] perldl> p transpose( $a ) [ [0 3] [1 4] [2 5] ] perl v5.8.0 2001-10-24 Basic(3)
All times are GMT -4. The time now is 12:28 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy