Sponsored Content
Top Forums Shell Programming and Scripting Removing repeating lines from a data frame (AWK) Post 302539976 by gd9629 on Tuesday 19th of July 2011 09:13:31 AM
Old 07-19-2011
The "'DATE TIME" header is in the .csv file, I think it appears once I combined the two columns using the GAWK code I posted above.

The result from
Code:
head -1 /u/Picarro/DataLog/2011/june/june.dat | cat -Te

The result
Code:
DATE                      TIME                      FRAC_DAYS_SINCE_JAN1      FRAC_HRS_SINCE_JAN1       EPOCH_TIME                ALARM_STATUS              species                   solenoid_valves           MPVPosition               OutletValve               CavityPressure            CavityTemp                WarmBoxTemp               EtalonTemp                DasTemp                   CO2_sync                  CO2_dry_sync              CH4_sync                  CH4_dry_sync              H2O_sync                  ^M$

To clarify,

The headers shown above are in the .dat files (both single files and big combined file)

The "// DATE TIME" (replaces the separate "DATE" and "TIME" header into one column) header arises in the .csv file, after the .dat file has been 'GAWKed'.

Does that help?
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Omit repeating lines

Can someone help me with the following 2 objectives? 1) The following command is just an example. It gets a list of all print jobs. From there I am trying to extract the printer name. It works with the following command: lpstat -W "completed" -o | awk -F- '{ print $1}' Problem is, I want... (6 Replies)
Discussion started by: TheCrunge
6 Replies

2. Shell Programming and Scripting

frame multiple lines into one

Hi, i have a file with contents like below ( any number of entries can be there) 111 222 333 444 555 i need to make another file with single line like below: 111,222,333,444,555 (without ending , ) TIA Prvn (8 Replies)
Discussion started by: prvnrk
8 Replies

3. UNIX for Advanced & Expert Users

removing frame charecters

Hi I have a requirement as follows. My Input file is as follows. COL1,COL2,COL3,COL4,COL5 987,2,3~7~5,400~468~598,0005~4687~5980 1111,2,2~7,400~468,0005~897 Expected OUTPUT ============ COL1,COL2,COL3,COL4,COL5 987,2,3,400,0005 987,2,7,468,4687 987,2,5,598,5980 1111,2,2,400,0005... (6 Replies)
Discussion started by: tkbharani
6 Replies

4. Shell Programming and Scripting

Merging non-repeating columns of lines

Hello, I have file to work with. It has 5 columns. The first three, altogether, constitutes the position. The 4th column contains some values for downstream analysis and the fifth column contains some values that I want to add to 4th column (only if they happen to be in the same position). My... (5 Replies)
Discussion started by: menenuh
5 Replies

5. UNIX for Dummies Questions & Answers

Extract repeating data from file

I want to extract the last rows of a data file, similar to that one below: C1 xxx C2 rrr C3 ttt .... Cn-1 hhh Cn bbb C1 yyy C2 sss C3 uuu ... Cn-1 iii Cn ccc ... I just want to extract the final rows between C1 and Cn at each data file. n is not a constant,... (2 Replies)
Discussion started by: natasha
2 Replies

6. Shell Programming and Scripting

awk removing data before or after a pattern

I have the following data: 01:00:00 29 10 20 41 01:20:00 18 6 34 42 01:40:00 28 5 24 43 02:00:01 11 7 8 74 02:20:01 19 15 12 54 02:40:01 1 4 0 95 03:00:01 1... (6 Replies)
Discussion started by: BeefStu
6 Replies

7. UNIX for Dummies Questions & Answers

Remove groups of repeating lines

I know uniq exists, but am not sure how to remove repeating lines when they are groups of two different lines repeating themselves, without using sort. I need them to be sorted in the original order, just to remove repeats. cd /media/AUDIO/WAVE/9780743518673/mp3 ~/Desktop/mp3-to-m4b... (1 Reply)
Discussion started by: glev2005
1 Replies

8. Shell Programming and Scripting

perform actions at specific locations in data frame

Hi everyone, I got a data frame like the one below and and would like to do the following: Ignore the first 3 rows and check in all following rows the second position. If the value is >500, subtract 100. Example DF: ABC 22 DE 12 BCD 223 GH 12 EFG 2104 DH ... (4 Replies)
Discussion started by: TuAd
4 Replies

9. Shell Programming and Scripting

awk : collecting all data between two time frame

Hi Experts , I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00" ... (8 Replies)
Discussion started by: zenkarthi
8 Replies

10. Shell Programming and Scripting

Removing certain lines from results - awk

im using the code below to monitor a file: gawk '{ a += gsub("(^| )accepted( |$)", "&") a += gsub("(^| )open database( |$)", "&") } END { for (i in a) printf("%s=%s\n", i, a) }' /var/log/syslog the code is searching the syslog file for the string "accepted" and "open... (2 Replies)
Discussion started by: SkySmart
2 Replies
TIME2POSIX(3)						   BSD Library Functions Manual 					     TIME2POSIX(3)

NAME
time2posix, time2posix_z, posix2time, posix2time_z, -- convert seconds since the Epoch LIBRARY
Standard C Library (libc, -lc) SYNOPSIS
#include <time.h> time_t time2posix(time_t t); time_t time2posix_z(const timezone_t tz, time_t t); time_t posix2time(time_t t); time_t posix2time_z(const timezone_t tz, time_t t); DESCRIPTION
IEEE Std 1003.1 (``POSIX.1'') legislates that a time_t value of 536457599 shall correspond to Wed Dec 31 23:59:59 UTC 1986. This effectively implies that POSIX time_t's cannot include leap seconds and, therefore, that the system time must be adjusted as each leap occurs. If the time package is configured with leap-second support enabled, however, no such adjustment is needed and time_t values continue to increase over leap events (as a true `seconds since...' value). This means that these values will differ from those required by POSIX by the net number of leap seconds inserted since the Epoch. Typically this is not a problem as the type time_t is intended to be (mostly) opaque -- time_t values should only be obtained-from and passed-to functions such as time(3), localtime(3), localtime_r(3), localtime_rz(3), mktime(3), mktime_z(3), and difftime(3). However, POSIX gives an arithmetic expression for directly computing a time_t value from a given date/time, and the same relationship is assumed by some (usually older) applications. Any programs creating/dissecting time_t's using such a relationship will typically not handle intervals over leap seconds correctly. The time2posix(), time2posix_z(), posix2time(), and posix2time_z() functions are provided to address this time_t mismatch by converting between local time_t values and their POSIX equivalents. This is done by accounting for the number of time-base changes that would have taken place on a POSIX system as leap seconds were inserted or deleted. These converted values can then be used in lieu of correcting the older applications, or when communicating with POSIX-compliant systems. time2posix() and time2posix_z() are single-valued. That is, every local time_t corresponds to a single POSIX time_t. posix2time() and posix2time() are less well-behaved: for a positive leap second hit the result is not unique, and for a negative leap second hit the corre- sponding POSIX time_t doesn't exist so an adjacent value is returned. Both of these are good indicators of the inferiority of the POSIX rep- resentation. The ``z'' variants of the two functions behave exactly like their counterparts, but they operate in the given tz argument which was previ- ously allocated using tzalloc(3) and are re-entrant. The following table summarizes the relationship between a time_t and its conversion to, and back from, the POSIX representation over the leap second inserted at the end of June, 1993. DATE TIME T X=time2posix(T) posix2time(X) 93/06/30 23:59:59 A+0 B+0 A+0 93/06/30 23:59:60 A+1 B+1 A+1 or A+2 93/07/01 00:00:00 A+2 B+1 A+1 or A+2 93/07/01 00:00:01 A+3 B+2 A+3 A leap second deletion would look like... DATE TIME T X=time2posix(T) posix2time(X) ??/06/30 23:59:58 A+0 B+0 A+0 ??/07/01 00:00:00 A+1 B+2 A+1 ??/07/01 00:00:01 A+2 B+3 A+2 [Note: posix2time(B+1) => A+0 or A+1] If leap-second support is not enabled, local time_t's and POSIX time_t's are equivalent, and both time2posix() and posix2time() degenerate to the identity function. SEE ALSO
difftime(3), localtime(3), localtime_r(3), localtime_rz(3), mktime(3), mktime_z(3), time(3), tzalloc(3) BSD
December 4, 2010 BSD
All times are GMT -4. The time now is 08:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy