How to delete lines in a file that have duplicates or derive the lines that aper once Post: 302335007

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

delete blank lines or lines with spaces only

hi there i'm trying to delete blank lines and or lines with spaces only from a series of files in an directory. to do so, i'm using this: for files in `ls /users/myname/pesop* 2>/dev/null` do grep -v ^$ $files > newfile mv newfile $files done now, this works great for blank lines but...

2. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section...

3. Shell Programming and Scripting

sed problem - delete all lines until a match on 2 lines

First of all, I know this can be more eassily done with perl or other scripting languages but, that's not the issue. I need this in sed. (or wander if it's possible ) I got a file (trace file to recreate the control file from oracle for the dba boys) which contains some lines another line...

4. UNIX for Dummies Questions & Answers

How get only required lines & delete the rest of the lines in file

Hiiii I have a file which contains huge data as a.dat: PDE 1990 1 9 18 51 28.90 24.7500 95.2800 118.0 6.1 0.0 BURMA event name: 010990D time shift: 7.3000 half duration: 5.0000 latitude: 24.4200 longitude: 94.9500 depth: 129.6000 Mrr: ...

5. Shell Programming and Scripting

need to delete all lines from a group of files except the 1st 2 lines

Hello, I have a group of text files with many lines in each file. I need to delete all the lines in each and only leave 2 lines in each file.

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to...

7. Shell Programming and Scripting

search and replace, when found, delete multiple lines, add new set of lines?

hey guys, I tried searching but most 'search and replace' questions are related to one liners. Say I have a file to be replaced that has the following: $ cat testing.txt TESTING AAA BBB CCC DDD EEE FFF GGG HHH ENDTESTING This is the input file:

8. Shell Programming and Scripting

Sed/awk to delete single lines that aren't touching other lines

Hello, I'm trying to figure out how to use sed or awk to delete single lines in a file. By single, I mean lines that are not touching any other lines (just one line with white space above and below). Example: one two three four five six seven eight I want it to look like:

9. Shell Programming and Scripting

Delete 40 lines after every 24 lines from a file

Hello, I have file of more than 10000 lines. I want to delete 40 lines after every 20 lines. e.g from a huge file, i want to delete line no from 34 - 74, then 94 - 134 etc and so on. Please let me know how i can do it. Best regards,

10. UNIX for Beginners Questions & Answers

Delete multiple lines between blank lines containing two patterns

Hi all, I'm looking for a way (sed or awk) to delete multiple lines between blank lines containing two patterns ex: user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 16...

LEARN ABOUT V7

uniq

UNIQ(1) 						      General Commands Manual							   UNIQ(1)

NAME

       uniq - report repeated lines in a file

SYNOPSIS

       uniq [ -udc [ +n ] [ -n ] ] [ input [ output ] ]

DESCRIPTION

       Uniq  reads  the  input file comparing adjacent lines.  In the normal case, the second and succeeding copies of repeated lines are removed;
       the remainder is written on the output file.  Note that repeated lines must be adjacent in order to be found; see sort(1).  If the -u  flag
       is  used, just the lines that are not repeated in the original file are output.	The -d option specifies that one copy of just the repeated
       lines is to be written.	The normal mode output is the union of the -u and -d mode outputs.

       The -c option supersedes -u and -d and generates an output report in default style but with each line preceded by a count of the number	of
       times it occurred.

       The n arguments specify skipping an initial portion of each line in the comparison:

       -n      The  first n fields together with any blanks before each are ignored.  A field is defined as a string of non-space, non-tab charac-
	       ters separated by tabs and spaces from its neighbors.

       +n      The first n characters are ignored.  Fields are skipped before characters.

SEE ALSO

       sort(1), comm(1)

																	   UNIQ(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

delete blank lines or lines with spaces only

Discussion started by: vascobrito

2. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Discussion started by: RichElks

3. Shell Programming and Scripting

sed problem - delete all lines until a match on 2 lines

Discussion started by: plelie2

4. UNIX for Dummies Questions & Answers

How get only required lines & delete the rest of the lines in file

Discussion started by: reva