Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Removing the lines which are same as 1st line Post 302582093 by ryran on Thursday 15th of December 2011 01:02:37 AM
Old 12-15-2011
Quote:
Originally Posted by gowtham.varma
cat <inputfile> | sort -n | uniq
1) You don't need to use cat there.
2) Why are you using numeric sorting? I'm gonna guess the -n option doesn't do what you think it does.
3) "sort | uniq" == "sort -u"
4) This might be a bad solution. The OP only wants to get rid of duplicate headers. We can't assume there are no other duplicate lines in his input.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing end of line to merge multiple lines

I'm sure this will be an easy question for you experts out there, but I have been searching the forum and working on this for a couple hours now and can't get it right. I have a very messy data file that I am trying to tidy up - one of the issues is some records are split into multiple lines: ... (4 Replies)
Discussion started by: tink
4 Replies

2. Shell Programming and Scripting

Removing empty lines(space) between two lines containing strings

Hi, Please provide shell script to Remove empty lines(space) between two lines containing strings in a file. Input File : A1/EXT "BAP_BSC6/07B/00" 844 090602 1605 RXOCF-465 PDTR11 1 SITE ON BATTERY A2/EXT... (3 Replies)
Discussion started by: sudhakaryadav
3 Replies

3. Shell Programming and Scripting

replace only 1st word of a line if it comes in the subsequent lines at same postion.

I have a file like this.. Maharastra Mumbai worli Maharastra Mumbai navy maharatra Pune Maharastra Nagpur Karnataka Bangalore Karnataka Mysore Karnataka Mangalore Punjab Amritsar punjab Jalandar my expected outcome should be like this Maharastra Mumbai worli ---------- ... (9 Replies)
Discussion started by: geeko
9 Replies

4. UNIX for Dummies Questions & Answers

Get only the 1st and 8th line from every 10 lines

Hi, I have 1000 line text file. I need only the 1st and 8th line from every set of 10 lines, that is, 1,8,11,18,21,21,28,31,38,... lines etc into a text file. Please let me know how I can achieve the same. Regards, Don (2 Replies)
Discussion started by: donisback
2 Replies

5. UNIX for Dummies Questions & Answers

append following lines to 1st line, every 3 lines

I have output like this: USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 ... where USER_ID is a unique user login followed by their login timestamp and... (6 Replies)
Discussion started by: MaindotC
6 Replies

6. Shell Programming and Scripting

Removing carriage return/line feeds on multiple lines

I would like to remove carriage returns/line feeds in a text file, but in a specific cadence: Read first line (Header Line 1), remove cr/lf at the end (replace it with a space ideally); Read the next line (Line of Text 2), leave the cr/lf intact; Read the next line, remove the cr/lf; Read... (14 Replies)
Discussion started by: tomr2012
14 Replies

7. Shell Programming and Scripting

File w/ many line pairs--how do I order based on 1st lines only?

I have a file in which the data is stored in pairs of lines. The first line (beginining with ">") is a header, the second line is a sequence. I would like to sort the file by species name. Desired output for the example file: I can use sort -t'_' -k2 to alphabetize headers in the... (1 Reply)
Discussion started by: pathunkathunk
1 Replies

8. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

9. Shell Programming and Scripting

Joining broken lines and removing empty lines

Hi - I have req to join broken lines and remove empty lines but should NOT be in one line. It has to be as is line by line. The challenge here is there is no end of line/start of line char. thanks in advance Source:- 2003-04-34024|04-10-2003|Claims|Claim|01-13-2003|Air Bag:Driver;... (7 Replies)
Discussion started by: Jackceasar123
7 Replies

10. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies
UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME
uniq -- report or filter out repeated lines in a file SYNOPSIS
uniq [-c | -d | -u] [-i] [-f num] [-s chars] [input_file [output_file]] DESCRIPTION
The uniq utility reads the specified input_file comparing adjacent lines, and writes a copy of each unique input line to the output_file. If input_file is a single dash ('-') or absent, the standard input is read. If output_file is absent, standard output is used for output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first. The following options are available: -c Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. -d Only output lines that are repeated in the input. -f num Ignore the first num fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adjacent fields by blanks. Field numbers are one based, i.e. the first field is field one. -s chars Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first num fields will be ignored. Character numbers are one based, i.e. the first character is character one. -u Only output lines that are not repeated in the input. -i Case insensitive comparison of lines. DIAGNOSTICS
The uniq utility exits 0 on success, and >0 if an error occurs. COMPATIBILITY
The historic +number and -number options have been deprecated but are still supported in this implementation. SEE ALSO
sort(1) STANDARDS
The uniq utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. HISTORY
A uniq command appeared in Version 3 AT&T UNIX. BSD
June 6, 1993 BSD
All times are GMT -4. The time now is 04:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy