Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Removing the lines which are same as 1st line Post 302582093 by ryran on Thursday 15th of December 2011 01:02:37 AM
Old 12-15-2011
Quote:
Originally Posted by gowtham.varma
cat <inputfile> | sort -n | uniq
1) You don't need to use cat there.
2) Why are you using numeric sorting? I'm gonna guess the -n option doesn't do what you think it does.
3) "sort | uniq" == "sort -u"
4) This might be a bad solution. The OP only wants to get rid of duplicate headers. We can't assume there are no other duplicate lines in his input.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing end of line to merge multiple lines

I'm sure this will be an easy question for you experts out there, but I have been searching the forum and working on this for a couple hours now and can't get it right. I have a very messy data file that I am trying to tidy up - one of the issues is some records are split into multiple lines: ... (4 Replies)
Discussion started by: tink
4 Replies

2. Shell Programming and Scripting

Removing empty lines(space) between two lines containing strings

Hi, Please provide shell script to Remove empty lines(space) between two lines containing strings in a file. Input File : A1/EXT "BAP_BSC6/07B/00" 844 090602 1605 RXOCF-465 PDTR11 1 SITE ON BATTERY A2/EXT... (3 Replies)
Discussion started by: sudhakaryadav
3 Replies

3. Shell Programming and Scripting

replace only 1st word of a line if it comes in the subsequent lines at same postion.

I have a file like this.. Maharastra Mumbai worli Maharastra Mumbai navy maharatra Pune Maharastra Nagpur Karnataka Bangalore Karnataka Mysore Karnataka Mangalore Punjab Amritsar punjab Jalandar my expected outcome should be like this Maharastra Mumbai worli ---------- ... (9 Replies)
Discussion started by: geeko
9 Replies

4. UNIX for Dummies Questions & Answers

Get only the 1st and 8th line from every 10 lines

Hi, I have 1000 line text file. I need only the 1st and 8th line from every set of 10 lines, that is, 1,8,11,18,21,21,28,31,38,... lines etc into a text file. Please let me know how I can achieve the same. Regards, Don (2 Replies)
Discussion started by: donisback
2 Replies

5. UNIX for Dummies Questions & Answers

append following lines to 1st line, every 3 lines

I have output like this: USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 USER_ID 12/31/69 19:00:00 12/31/69 19:00:00 ... where USER_ID is a unique user login followed by their login timestamp and... (6 Replies)
Discussion started by: MaindotC
6 Replies

6. Shell Programming and Scripting

Removing carriage return/line feeds on multiple lines

I would like to remove carriage returns/line feeds in a text file, but in a specific cadence: Read first line (Header Line 1), remove cr/lf at the end (replace it with a space ideally); Read the next line (Line of Text 2), leave the cr/lf intact; Read the next line, remove the cr/lf; Read... (14 Replies)
Discussion started by: tomr2012
14 Replies

7. Shell Programming and Scripting

File w/ many line pairs--how do I order based on 1st lines only?

I have a file in which the data is stored in pairs of lines. The first line (beginining with ">") is a header, the second line is a sequence. I would like to sort the file by species name. Desired output for the example file: I can use sort -t'_' -k2 to alphabetize headers in the... (1 Reply)
Discussion started by: pathunkathunk
1 Replies

8. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

9. Shell Programming and Scripting

Joining broken lines and removing empty lines

Hi - I have req to join broken lines and remove empty lines but should NOT be in one line. It has to be as is line by line. The challenge here is there is no end of line/start of line char. thanks in advance Source:- 2003-04-34024|04-10-2003|Claims|Claim|01-13-2003|Air Bag:Driver;... (7 Replies)
Discussion started by: Jackceasar123
7 Replies

10. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies
SORT(1) 						      General Commands Manual							   SORT(1)

NAME
sort - sort a file of ASCII lines SYNOPSIS
sort [-bcdfimnru] [-tc] [-o name] [+pos1] [-pos2] file ... OPTIONS
-b Skip leading blanks when making comparisons -c Check to see if a file is sorted -d Dictionary order: ignore punctuation -f Fold upper case onto lower case -i Ignore nonASCII characters -m Merge presorted files -n Numeric sort order -o Next argument is output file -r Reverse the sort order -t Following character is field separator -u Unique mode (delete duplicate lines) EXAMPLES
sort -nr file # Sort keys numerically, reversed sort +2 -4 file # Sort using fields 2 and 3 as key sort +2 -t: -o out # Field separator is : sort +.3 -.6 # Characters 3 through 5 form the key DESCRIPTION
Sort sorts one or more files. If no files are specified, stdin is sorted. Output is written on standard output, unless -o is specified. The options +pos1 -pos2 use only fields pos1 up to but not including pos2 as the sort key, where a field is a string of characters delim- ited by spaces and tabs, unless a different field delimiter is specified with -t. Both pos1 and pos2 have the form m.n where m tells the number of fields and n tells the number of characters. Either m or n may be omitted. SEE ALSO
comm(1), grep(1), uniq(1). SORT(1)
All times are GMT -4. The time now is 10:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy