Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk script to remove duplicate rows in line
Top Forums Shell Programming and Scripting awk script to remove duplicate rows in line Post 302318412 by devtakh on Thursday 21st of May 2009 12:53:19 PM
Old 05-21-2009
Code:
 tr ';' '\n' < file| awk -F, '{if(!a[$1]) print $0;a[$1]=$0}'


-Devaraj Takhellambam
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies

2. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

3. Shell Programming and Scripting

To remove date and duplicate rows from a log file using unix commands

Hi, I have a log file having size of 48mb. For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages. The following things to be done : 1) To remove all the date and time from the log file 2) To remove all the... (1 Reply)
Discussion started by: Pank10
1 Replies

4. Ubuntu

delete duplicate rows with awk files

Hi every body I have some text file with a lots of duplicate rows like this: 165.179.568.197 154.893.836.174 242.473.396.153 165.179.568.197 165.179.568.197 165.179.568.197 154.893.836.174 how can I delete the repeated rows? Thanks Saeideh (2 Replies)
Discussion started by: sashtari
2 Replies

5. Shell Programming and Scripting

remove consecutive duplicate rows

I have some data that looks like, 1 3300665.mol 3300665 5177008 102.093 2 3300665.mol 3300665 5177008 102.093 3 3294015.mol 3294015 5131552 102.114 4 3294015.mol 3294015 5131552 102.114 5 3293734.mol 3293734 5129625 104.152 6 3293734.mol ... (13 Replies)
Discussion started by: LMHmedchem
13 Replies

6. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

7. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

8. Shell Programming and Scripting

How to duplicate rows using awk or any other method?

I want to duplicate each row in my file Egfile.txt Name State Age Jack NJ 34 John MA 23 Jessica FL 45 I want the code to produce this output Name State Age Jack NJ 34 Jack NJ 34 John MA 23 John MA 23 Jessica FL 45 Jessica FL 45 (6 Replies)
Discussion started by: sidnow
6 Replies

9. UNIX for Dummies Questions & Answers

Using awk to remove duplicate line if field is empty

Hi all, I've got a file that has 12 fields. I've merged 2 files and there will be some duplicates in the following: FILE: 1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100 2. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, (EMPTY) 3. CDC, 54321, TEST3,... (4 Replies)
Discussion started by: tugar
4 Replies

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ... (12 Replies)
Discussion started by: Tim2424
12 Replies

LEARN ABOUT MOJAVE

locale::codes::langvar

Locale::Codes::LangVar(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangVar(3pm)

NAME

       Locale::Codes::LangVar - standard codes for language variation identification

SYNOPSIS

	  use Locale::Codes::LangVar;

	  $lvar = code2langvar('acm');		       # $lvar gets 'Mesopotamian Arabic'
	  $code = langvar2code('Mesopotamian Arabic'); # $code gets 'acm'

	  @codes   = all_langvar_codes();
	  @names   = all_langvar_names();

DESCRIPTION

       The "Locale::Codes::LangVar" module provides access to standard codes used for identifying language variations, such as those as defined in
       the IANA language registry.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default IANA language
       registry codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language variations. A code set may be specified using either a name, or
       a constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lvar = code2langvar('en','alpha-2');
	  $lvar = code2langvar('en',LOCALE_CODE_ALPHA_2);

       The codesets currently supported are:

       alpha
	   This is the set of alphanumeric codes from the IANA language registry, such as 'arevela' for Eastern Armenian.

	   This code set is identified with the symbol "LOCALE_LANGVAR_ALPHA".

	   This is the default code set.

ROUTINES

       code2langvar ( CODE [,CODESET] )
       langvar2code ( NAME [,CODESET] )
       langvar_code2code ( CODE ,CODESET ,CODESET2 )
       all_langvar_codes ( [CODESET] )
       all_langvar_names ( [CODESET] )
       Locale::Codes::LangVar::rename_langvar  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangVar::add_langvar  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangVar::delete_langvar  ( CODE [,CODESET] )
       Locale::Codes::LangVar::add_langvar_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangVar::delete_langvar_alias  ( NAME )
       Locale::Codes::LangVar::rename_langvar_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangVar::add_langvar_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangVar::delete_langvar_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.iana.org/assignments/language-subtag-registry
	   The IANA language subtag registry.

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2013 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.18.2							    2014-01-06					       Locale::Codes::LangVar(3pm)
All times are GMT -4. The time now is 06:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy