Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Sort and uniq lines of a file while keeping a header line Post 302314909 by Digby on Monday 11th of May 2009 04:28:14 AM
Old 05-11-2009
Sort and uniq lines of a file while keeping a header line

So, I have a file that has some duplicate lines. The file has a header line that I would like to keep at the top.

I could do this by extracting the header from the file, 'sort -u' the remaining lines, and recombine them. But they are quite big, so if there is a way to do it with a single command, that would be great.

Anyone any ideas? Thanks in advance.

Here's an example file

Code:
MAKEBITS 1.0 574331 /home/woodd/workspace2/ibis_3d/scripts/pdblist2fps.py
a139_371_frag1 1 2 3 4 5 6 10 11
a139_371_frag2 2 4 5 158 159 160 161 162
a139_371_frag3 2 6 159 160 161 258 259 260
a139_371_frag4 1 2 3 4 5 6 10 11
a139_371_frag5 1 2 3 4 5 6 10 11
a139_371_frag6 1 2 3 4 5 6 10 11
a139_IMD_frag1 1 3 4 6 57 58 59 89
a139_371_frag1 1 2 3 4 5 6 10 11
a139_371_frag2 2 4 5 158 159 160 161 162
a139_371_frag3 2 6 159 160 161 258 259 260
a139_371_frag4 1 2 3 4 5 6 10 11
a139_371_frag5 1 2 3 4 5 6 10 11
a139_371_frag6 1 2 3 4 5 6 10 11
a139_IMD_frag1 1 3 4 6 57 58 59 89

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to sort standard input without first line < Header >

Do somebody have idea How to sort standard input without first line which in my case it's header Example: Cnt|VT |STAT|Date |Time |From |Alert Message |Instance | 125| | | 260308 |160026 |ZAMUAT2|ifpollq... (8 Replies)
Discussion started by: pp56825
8 Replies

2. UNIX for Dummies Questions & Answers

duplicated lines not recognized by sort and uniq

Hello all, I've got a strange behaviour of sort and uniq commands: they do not recognise apparently duplicated lines in a file (already sorted). The lines are identical by eye, but they must differ in smth, because when they are put in two files, those have slightly different size. What can make... (8 Replies)
Discussion started by: roussine
8 Replies

3. UNIX for Dummies Questions & Answers

find uniq lines in file, using the first field of line

Hello all, new to unix and have just found the forum. I think I will be here quite often, and hope that in time i will be able to provide soem help, role on not being a newbie anymore :) I have a question which iI am hoping someone could help me with. If i have a file with lines in in thus... (8 Replies)
Discussion started by: grom
8 Replies

4. Shell Programming and Scripting

unix sort according to a header line

Hi, I have a file with a header line, followed by some contents. How can I sort the file according to header lines? eg. /* abcd_005*/ a bc /* abcd_001*/ d e /* abcd_002*/ x y desired output: /*abcd_001*/ (0 Replies)
Discussion started by: neil.0412
0 Replies

5. UNIX for Advanced & Expert Users

$cat file|sort|uniq

I have a file with list of redundant server names and want only unique ones of those. I have used following command but still redudant ones are listing $cat file|sort|uniq where could be the problem. Thanks, Srinivas (3 Replies)
Discussion started by: srinivas Juluri
3 Replies

6. UNIX for Dummies Questions & Answers

Sort a tab file with header.

How to sort a tab delimited file first on col1 and then on col2. Also I need to keep the header intact. file.txt val1 val2 val3 val4 a b c d m n o p e f g h i j k l ... (3 Replies)
Discussion started by: mary271
3 Replies

7. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

8. Shell Programming and Scripting

How do a distinct from a file using sort uniq in bash?

I have an output file .dat. From this file i have to do a distinct of the ID using the sort uniq command in bash script. How can i do it? i found : sort -u ${FILEOUT_DAT} but i don't think is my solution because the id isn't specified.. is there other solution? (7 Replies)
Discussion started by: punticci
7 Replies

9. Shell Programming and Scripting

Need to sort text keeping first line always first

I have a file is created from standard output. I have put a leading space to force the first line to collate low vis a vis the rest of the lines. If I pass the entire file to the Linux sort, it ignores the leading space and the first line appears in somewhere in the list. If I add lots of... (15 Replies)
Discussion started by: lsatenstein
15 Replies

10. Shell Programming and Scripting

Find header in a text file and prepend it to all lines until another header is found

I've been struggling with this one for quite a while and cannot seem to find a solution for this find/replace scenario. Perhaps I'm getting rusty. I have a file that contains a number of metrics (exactly 3 fields per line) from a few appliances that are collected in parallel. To identify the... (3 Replies)
Discussion started by: verdepollo
3 Replies
SNMPDELTA(1)							     Net-SNMP							      SNMPDELTA(1)

NAME
snmpdelta - Monitor delta differences in SNMP Counter values SYNOPSIS
snmpdelta [ common options ] [-Cf] [ -Ct ] [ -Cs ] [ -CS ] [ -Cm ] [ -CF configfile ] [ -Cl ] [ -Cp period ] [ -CP Peaks ] [ -Ck ] [ -CT ] AGENT OID [ OID ... ] DESCRIPTION
snmpdelta will monitor the specified integer valued OIDs, and report changes over time. AGENT identifies a target SNMP agent, which is instrumented to monitor the given objects. At its simplest, the AGENT specification will consist of a hostname or an IPv4 address. In this situation, the command will attempt communication with the agent, using UDP/IPv4 to port 161 of the given target host. See snmpcmd(1) for a full list of the possible formats for AGENT. OID is an object identifier which uniquely identifies the object type within a MIB. Multiple OIDs can be specified on a single snmpdelta command. OPTIONS
COMMON OPTIONS Please see snmpcmd(1) for a list of possible values for COMMON OPTIONS as well as their descriptions. -Cf Don't fix errors and retry the request. Without this option, if multiple oids have been specified for a single request and if the request for one or more of the oids fails, snmpdelta will retry the request so that data for oids apart from the ones that failed will still be returned. Specifying -Cf tells snmpdelta not to retry a request, even if there are multiple oids specified. -Ct Flag will determine time interval from the monitored entity. -Cs Flag will display a timestamp. -CS Generates a "sum count" in addition to the individual instance counts. The "sum count" is the total of all the individual deltas for each time period. -Cm Prints the max value ever attained. -CF configfile Tells snmpdelta to read it's configuration from the specified file. This options allows the input to be set up in advance rather than having to be specified on the command line. -Cl Tells snmpdelta to write it's configuration to files whose names correspond to the MIB instances monitored. For example, snmpdelta -Cl localhost ifInOctets.1 will create a file "localhost-ifInOctets.1". -Cp Specifies the number of seconds between polling periods. Polling constitutes sending a request to the agent. The default polling period is one second. -CP peaks Specifies the reporting period in number of polling periods. If this option is specified, snmpdelta polls the agent peaks number of times before reporting the results. The result reported includes the average value over the reporting period. In addition, the highest polled value within the reporting period is shown. -Ck When the polling period (-Cp) is an increment of 60 seconds and the timestamp is displayed in the output (-Cs), then the default display shows the timestamp in the format hh:mm mm/dd. This option causes the timestamp format to be hh:mm:ss mm/dd. -CT Makes snmpdelta print its output in tabular form. -Cv vars/pkt Specifies the maximum number of oids allowed to be packaged in a single PDU. Multiple PDUs can be created in a single request. The default value of variables per packet is 60. This option is useful if a request response results in an error becaues the packet is too big. EXAMPLES
$ snmpdelta -c public -v 1 -Cs localhost IF-MIB::ifInUcastPkts.3 IF-MIB::ifOutUcastPkts.3 [20:15:43 6/14] ifInUcastPkts.3 /1 sec: 158 [20:15:43 6/14] ifOutUcastPkts.3 /1 sec: 158 [20:15:44 6/14] ifInUcastPkts.3 /1 sec: 184 [20:15:44 6/14] ifOutUcastPkts.3 /1 sec: 184 [20:15:45 6/14] ifInUcastPkts.3 /1 sec: 184 [20:15:45 6/14] ifOutUcastPkts.3 /1 sec: 184 [20:15:46 6/14] ifInUcastPkts.3 /1 sec: 158 [20:15:46 6/14] ifOutUcastPkts.3 /1 sec: 158 [20:15:47 6/14] ifInUcastPkts.3 /1 sec: 184 [20:15:47 6/14] ifOutUcastPkts.3 /1 sec: 184 [20:15:48 6/14] ifInUcastPkts.3 /1 sec: 184 [20:15:48 6/14] ifOutUcastPkts.3 /1 sec: 184 [20:15:49 6/14] ifInUcastPkts.3 /1 sec: 158 [20:15:49 6/14] ifOutUcastPkts.3 /1 sec: 158 ^C $ snmpdelta -c public -v 1 -Cs -CT localhost IF-MIB:ifInUcastPkts.3 IF-MIB:ifOutcastPkts.3 localhost ifInUcastPkts.3 ifOutUcastPkts.3 [20:15:59 6/14] 184.00 184.00 [20:16:00 6/14] 158.00 158.00 [20:16:01 6/14] 184.00 184.00 [20:16:02 6/14] 184.00 184.00 [20:16:03 6/14] 158.00 158.00 [20:16:04 6/14] 184.00 184.00 [20:16:05 6/14] 184.00 184.00 [20:16:06 6/14] 158.00 158.00 ^C The following example uses a number of options. Since the Cl option is specified, the output is sent to a file and not to the screen. $ snmpdelta -c public -v 1 -Ct -Cs -CS -Cm -Cl -Cp 60 -CP 60 interlink.sw.net.cmu.edu .1.3.6.1.2.1.2.2.1.16.3 .1.3.6.1.2.1.2.2.1.16.4 fi SEE ALSO
snmpcmd(1), variables(5). 4th Berkeley Distribution 25 Jul 2003 SNMPDELTA(1)
All times are GMT -4. The time now is 04:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy