Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicates lines in a files
Top Forums UNIX for Dummies Questions & Answers Remove duplicates lines in a files Post 302737995 by jyoung on Friday 30th of November 2012 07:37:36 AM
Old 11-30-2012
sort and uniq
I have an input file like so:
Code:
apples
peaches
pears
apples
peaches
pears
apples
peaches
pears

I sort it like so:
Code:
sort oldfruit > myfruit

I then use the uniq command like so:
Code:
uniq myfruit newfruit

and the result is:
Code:
apples
peaches
pears

This was done in AIX 5.2 from the command line, but should work the same in ksh.
Last edited by zaxxon; 11-30-2012 at 10:30 AM.. Reason: misunderstood intentions completely. reversed everything, sorry.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies

2. Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

I have two files and need to compare the two files and to remove the matching lines from both the files (4 Replies)
Discussion started by: shellscripter
4 Replies

3. Shell Programming and Scripting

How to delete lines in a file that have duplicates or derive the lines that aper once

Input: a b b c d d I need: a c I know how to get this (the lines that have duplicates) : b d sort file | uniq -d But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies

4. Shell Programming and Scripting

compare files and then remove some lines

Hi everyone I have a dilemma and I'm hoping someone has an answer for me. I have two files: # cat masterfile line3 line4 line5 line6 line7 # cat tempfile line1 line2 line3 line4 I want to compare tempfile with masterfile. (3 Replies)
Discussion started by: soliberus
3 Replies

5. Shell Programming and Scripting

Remove the files that have less than certain lines

Hi all, I'm a newbie and I'm sorry if my question is too simple. I'm having problem to delete the files that have less than certain lines, say 16. #!/bin/tcsh set filen = `sh -c 'ls *csv 2> /dev/null'` foreach fil (${filen}) if ]; then rm -f ${filen} fi end exit ... (2 Replies)
Discussion started by: GoldenFire
2 Replies

6. Shell Programming and Scripting

Compare one files with strings from another + remove lines

Have two files and want to compare the content of file1 with file2. When matched remove the line. awk 'NR==FNR {b; next} !(b in $0)' file1 file2file1 1. if match 2. removefile2 1. this line has to be removed if match 2. this line has a match, remove 3. this line has no match, no removingThe... (3 Replies)
Discussion started by: sdf
3 Replies

7. Shell Programming and Scripting

How to remove the last 3 lines from many files?

Hello, I need to run a command or shell script that will remove the last 3 lines from every .js file that is under the directory /var/ww/vhost/ Can you please help ? thank you. (22 Replies)
Discussion started by: MaRiOsGR
22 Replies

8. Shell Programming and Scripting

Remove duplicates from two files using n characters for comparison

Hi All, I have two files file1 123456CRTGHG 125437CRNDGF 126537CRDDGF file2 123456CRTZHC 124567CJHGHG 125987CJHGDF I need to compare the two files and any records in file 1 and 2 based on initial n characters (6 in example) need to be ignored. string separated by unprintable... (2 Replies)
Discussion started by: Bruble
2 Replies

9. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

10. Shell Programming and Scripting

Remove lines from output in files using awk

I have two large files (~250GB) that I am trying to remove the where GT: 0/0 or 1/1 or 2/2 for both files. I was going to use a bash with the below awk, which I think will find each line but how do I remove that line is that condition is found? Thank you :). Input 20 60055 . A ... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT OSX

locale::codes::langfam

Locale::Codes::LangFam(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangFam(3pm)

NAME

       Locale::Codes::LangFam - standard codes for language extension identification

SYNOPSIS

	  use Locale::Codes::LangFam;

	  $lext = code2langfam('apa');		       # $lext gets 'Apache languages'
	  $code = langfam2code('Apache languages');    # $code gets 'apa'

	  @codes   = all_langfam_codes();
	  @names   = all_langfam_names();

DESCRIPTION

       The "Locale::Codes::LangFam" module provides access to standard codes used for identifying language families, such as those as defined in
       ISO 639-5.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 639-5
       language family codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language families. A code set may be specified using either a name, or a
       constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lext = code2langfam('apa','alpha');
	  $lext = code2langfam('apa',LOCALE_LANGFAM_ALPHA);

       The codesets currently supported are:

       alpha
	   This is the set of three-letter (lowercase) codes from ISO 639-5 such as 'apa' for Apache languages.

	   This is the default code set.

ROUTINES

       code2langfam ( CODE [,CODESET] )
       langfam2code ( NAME [,CODESET] )
       langfam_code2code ( CODE ,CODESET ,CODESET2 )
       all_langfam_codes ( [CODESET] )
       all_langfam_names ( [CODESET] )
       Locale::Codes::LangFam::rename_langfam  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangFam::add_langfam  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangFam::delete_langfam  ( CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangFam::delete_langfam_alias  ( NAME )
       Locale::Codes::LangFam::rename_langfam_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::delete_langfam_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.loc.gov/standards/iso639-5/id.php
	   ISO 639-5 .

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2012 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.16.2							    2012-10-11					       Locale::Codes::LangFam(3pm)
All times are GMT -4. The time now is 11:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy