07-19-2011
Removing repeating lines from a data frame (AWK)
Hey Guys!
I have written a code which combines lots of files into one big file(.csv).
However, each of the original files had headers on the first line, and now that I've combined the files the headers are interspersed throughout the new combined data frame. For example, throughout the data I will have a line;
//DATE TIME
FRAC_DAYS_SINCE_JAN1
FRAC_HRS_SINCE_JAN1 EPOCH_TIME
ALARM_STATUS
with each of these headers in a different column.
Is there a way of removing these lines when these headers arise?
Thanks!
Last edited by gd9629; 07-19-2011 at 06:50 AM..
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Can someone help me with the following 2 objectives?
1) The following command is just an example. It gets a list of all print jobs. From there I am trying to extract the printer name. It works with the following command:
lpstat -W "completed" -o | awk -F- '{ print $1}'
Problem is, I want... (6 Replies)
Discussion started by: TheCrunge
6 Replies
2. Shell Programming and Scripting
Hi,
i have a file with contents like below ( any number of entries can be there)
111
222
333
444
555
i need to make another file with single line like below:
111,222,333,444,555 (without ending , )
TIA
Prvn (8 Replies)
Discussion started by: prvnrk
8 Replies
3. UNIX for Advanced & Expert Users
Hi
I have a requirement as follows. My Input file is as follows.
COL1,COL2,COL3,COL4,COL5
987,2,3~7~5,400~468~598,0005~4687~5980
1111,2,2~7,400~468,0005~897
Expected OUTPUT
============
COL1,COL2,COL3,COL4,COL5
987,2,3,400,0005
987,2,7,468,4687
987,2,5,598,5980
1111,2,2,400,0005... (6 Replies)
Discussion started by: tkbharani
6 Replies
4. Shell Programming and Scripting
Hello,
I have file to work with. It has 5 columns. The first three, altogether, constitutes the position. The 4th column contains some values for downstream analysis and the fifth column contains some values that I want to add to 4th column (only if they happen to be in the same position).
My... (5 Replies)
Discussion started by: menenuh
5 Replies
5. UNIX for Dummies Questions & Answers
I want to extract the last rows of a data file, similar to that one below:
C1 xxx
C2 rrr
C3 ttt
....
Cn-1 hhh
Cn bbb
C1 yyy
C2 sss
C3 uuu
...
Cn-1 iii
Cn ccc
...
I just want to extract the final rows between C1 and Cn at each data file. n is not a constant,... (2 Replies)
Discussion started by: natasha
2 Replies
6. Shell Programming and Scripting
I have the following data:
01:00:00 29 10 20 41
01:20:00 18 6 34 42
01:40:00 28 5 24 43
02:00:01 11 7 8 74
02:20:01 19 15 12 54
02:40:01 1 4 0 95
03:00:01 1... (6 Replies)
Discussion started by: BeefStu
6 Replies
7. UNIX for Dummies Questions & Answers
I know uniq exists, but am not sure how to remove repeating lines when they are groups of two different lines repeating themselves, without using sort. I need them to be sorted in the original order, just to remove repeats.
cd /media/AUDIO/WAVE/9780743518673/mp3
~/Desktop/mp3-to-m4b... (1 Reply)
Discussion started by: glev2005
1 Replies
8. Shell Programming and Scripting
Hi everyone,
I got a data frame like the one below and and would like to do the following:
Ignore the first 3 rows and check in all following rows the second position. If the value is >500, subtract 100.
Example DF:
ABC 22 DE 12
BCD 223 GH 12
EFG 2104 DH ... (4 Replies)
Discussion started by: TuAd
4 Replies
9. Shell Programming and Scripting
Hi Experts ,
I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp
for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00"
... (8 Replies)
Discussion started by: zenkarthi
8 Replies
10. Shell Programming and Scripting
im using the code below to monitor a file:
gawk '{
a += gsub("(^| )accepted( |$)", "&")
a += gsub("(^| )open database( |$)", "&")
} END {
for (i in a)
printf("%s=%s\n", i, a)
}' /var/log/syslog
the code is searching the syslog file for the string "accepted" and "open... (2 Replies)
Discussion started by: SkySmart
2 Replies
comb(1) General Commands Manual comb(1)
Name
comb - combine delta versions of SCCS file
Syntax
comb [-o] [-s] [-psid] [-clist] files
Description
The command generates a shell procedure which, when run, will reconstruct the given SCCS files. For further information, see The recon-
structed files are generally smaller than the original files. The arguments may be specified in any order, but all keyletter arguments
apply to all named SCCS files. If a directory is named, behaves as though each file in the directory were specified as a named file,
except non-SCCS files (last component of the path name does not begin with s.) and unreadable files, which are silently ignored. If a
name of - is given, the standard input is read; each line of the standard input is taken to be the name of an SCCS file to be processed.
The generated shell procedure is written on the standard output.
Each keyletter argument is explained as though only one named file is to be processed, but the effects of any keyletter argument apply
independently to each named file.
Options
-clist Preserves specified deltas. See for the syntax of a list. All other deltas are discarded.
-o Causes the reconstructed file to be accessed at the release of the delta to be created. Otherwise the reconstructed file would
be accessed at the most recent ancestor. Use of the -o keyletter may decrease the size of the reconstructed SCCS file. It may
also alter the shape of the delta tree of the original file.
-pSID Indicates oldest delta to be preserved. All older deltas are discarded in the reconstructed file.
-s Generates a shell procedure which produces a report. This report gives the file name, size (in blocks) after combining, original
size (also in blocks), and percentage change computed by:
100 * (original - combined) / original
It is recommended that before any SCCS files are actually combined, one should use this option to determine exactly how much
space is saved by the combining process.
If no keyletter arguments are specified, will preserve only leaf deltas and the minimal number of ancestors needed to preserve the tree.
Restrictions
The command may rearrange the shape of the tree of deltas. It may not save any space; in fact, it is possible for the reconstructed file
to actually be larger than the original.
Diagnostics
See for explanations.
Files
s.COMB The name of the reconstructed SCCS file.
comb????? Temporary.
See Also
admin(1), delta(1), get(1), help(1), prs(1), sccs(1), sccsfile(5)
Guide to the Source Code Control System
comb(1)