Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicate lines from a file
Top Forums Shell Programming and Scripting Remove duplicate lines from a file Post 302882349 by bartus11 on Monday 6th of January 2014 09:53:23 AM
Old 01-06-2014
Try:
Code:
nawk 'NR==1{x=$0;print}$0!=x' file

Last edited by bartus11; 01-06-2014 at 11:17 AM.. Reason: changed from awk to nawk
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

4. UNIX for Dummies Questions & Answers

How to delete or remove duplicate lines in a file

Hi please help me how to remove duplicate lines in any file. I have a file having huge number of lines. i want to remove selected lines in it. And also if there exists duplicate lines, I want to delete the rest & just keep one of them. Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies

5. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies

6. Shell Programming and Scripting

Remove duplicate lines from first file comparing second file

Hi, I have two files with below data:: file1:- 123|aaa|ppp 445|fff|yyy 999|ttt|jjj 555|hhh|hhh file2:- 445|fff|yyy 555|hhh|hhh The records present in file1, not present in file 2 should be writtent to the out put file. output:- 123|aaa|ppp 999|ttt|jjj Is there any one line... (3 Replies)
Discussion started by: gani_85
3 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies

LEARN ABOUT DEBIAN

sdl::gfx::imagefilter

pods::SDL::GFX::ImageFilter(3pm)			User Contributed Perl Documentation			  pods::SDL::GFX::ImageFilter(3pm)

NAME

       SDL::GFX::ImageFilter - image filtering functions

CATEGORY

       TODO, GFX

METHODS

   MMX_detect
       int gfx_image_MMX_detect()      CODE:	       SDL_imageFilterMMXdetect();

   MMX_off
       void gfx_image_MMX_off()      CODE:	     SDL_imageFilterMMXoff();

   MMX_on
       void gfx_image_MMX_on()	    CODE:	    SDL_imageFilterMMXon();

   add
       int gfx_image_add(Src1, Src2, Dest, length)	unsigned char *Src1	 unsigned char *Src2	  unsigned char *Dest	   int length
	    CODE:	    RETVAL = SDL_imageFilterAdd(Src1, Src2, Dest, length);	OUTPUT: 	  RETVAL

   mean
       int gfx_image_mean(Src1, Src2, Dest, length)	 unsigned char *Src1	  unsigned char *Src2	   unsigned char *Dest	    int length
	    CODE:	    RETVAL = SDL_imageFilterMean(Src1, Src2, Dest, length);	 OUTPUT:	   RETVAL

   sub
       int gfx_image_sub(Src1, Src2, Dest, length)	unsigned char *Src1	 unsigned char *Src2	  unsigned char *Dest	   int length
	    CODE:	    RETVAL = SDL_imageFilterSub(Src1, Src2, Dest, length);	OUTPUT: 	  RETVAL

   abs_diff
       int gfx_image_abs_diff(Src1, Src2, Dest, length)      unsigned char *Src1      unsigned char *Src2      unsigned char *Dest	int length
	    CODE:	    RETVAL = SDL_imageFilterAbsDiff(Src1, Src2, Dest, length);	    OUTPUT:	      RETVAL

   mult
       int gfx_image_mult(Src1, Src2, Dest, length)	 unsigned char *Src1	  unsigned char *Src2	   unsigned char *Dest	    int length
	    CODE:	    RETVAL = SDL_imageFilterMult(Src1, Src2, Dest, length);	 OUTPUT:	   RETVAL

   mult_nor
       int gfx_image_mult_nor(Src1, Src2, Dest, length)      unsigned char *Src1      unsigned char *Src2      unsigned char *Dest	int length
	    CODE:	    RETVAL = SDL_imageFilterMultNor(Src1, Src2, Dest, length);	    OUTPUT:	      RETVAL

   mult_div_by_2
       int gfx_image_mult_div_by_2(Src1, Src2, Dest, length)	  unsigned char *Src1	   unsigned char *Src2	    unsigned char *Dest      int
       length	   CODE:	   RETVAL = SDL_imageFilterMultDivby2(Src1, Src2, Dest, length);      OUTPUT:		RETVAL

   mult_div_by_4
       int gfx_image_mult_div_by_4(Src1, Src2, Dest, length)	  unsigned char *Src1	   unsigned char *Src2	    unsigned char *Dest      int
       length	   CODE:	   RETVAL = SDL_imageFilterMultDivby4(Src1, Src2, Dest, length);      OUTPUT:		RETVAL

   bit_and
       int gfx_image_bit_and(Src1, Src2, Dest, length)	    unsigned char *Src1      unsigned char *Src2      unsigned char *Dest      int length
	    CODE:	    RETVAL = SDL_imageFilterBitAnd(Src1, Src2, Dest, length);	   OUTPUT:	     RETVAL

   bit_or
       int gfx_image_bit_or(Src1, Src2, Dest, length)	   unsigned char *Src1	    unsigned char *Src2      unsigned char *Dest      int length
	    CODE:	    RETVAL = SDL_imageFilterBitOr(Src1, Src2, Dest, length);	  OUTPUT:	    RETVAL

   div
       int gfx_image_div(Src1, Src2, Dest, length)	unsigned char *Src1	 unsigned char *Src2	  unsigned char *Dest	   int length
	    CODE:	    RETVAL = SDL_imageFilterDiv(Src1, Src2, Dest, length);	OUTPUT: 	  RETVAL

   bit_negation
       int gfx_image_bit_negation(Src1, Dest, length)	   unsigned char *Src1	    unsigned char *Dest      int length      CODE:
		 RETVAL = SDL_imageFilterBitNegation(Src1, Dest, length);      OUTPUT:		 RETVAL

   add_byte
       int gfx_image_add_byte(Src1, Dest, length, C)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned char C
	    CODE:	    RETVAL = SDL_imageFilterAddByte(Src1, Dest, length, C);	 OUTPUT:	   RETVAL

   add_uint
       int gfx_image_add_uint(Src1, Dest, length, C)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned int C
	    CODE:	    RETVAL = SDL_imageFilterAddUint(Src1, Dest, length, C);	 OUTPUT:	   RETVAL

   add_byte_to_half
       int gfx_image_add_byte_to_half(Src1, Dest, length, C)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned char
       C      CODE:	      RETVAL = SDL_imageFilterAddByteToHalf(Src1, Dest, length, C);	 OUTPUT:	   RETVAL

   sub_byte
       int gfx_image_sub_byte(Src1, Dest, length, C)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned char C
	    CODE:	    RETVAL = SDL_imageFilterSubByte(Src1, Dest, length, C);	 OUTPUT:	   RETVAL

   sub_uint
       int gfx_image_sub_uint(Src1, Dest, length, C)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned int C
	    CODE:	    RETVAL = SDL_imageFilterSubUint(Src1, Dest, length, C);	 OUTPUT:	   RETVAL

   shift_right
       int gfx_image_shift_right(Src1, Dest, length, N)      unsigned char *Src1      unsigned char *Dest      int length      unsigned char N
	    CODE:	    RETVAL = SDL_imageFilterShiftRight(Src1, Dest, length, N);	    OUTPUT:	      RETVAL

   shift_right_uint
       int gfx_image_shift_right_uint(Src1, Dest, length, N)	  unsigned char *Src1	   unsigned char *Dest	    int length	    unsigned char
       N      CODE:	      RETVAL = SDL_imageFilterShiftRightUint(Src1, Dest, length, N);	  OUTPUT:	    RETVAL

   mult_by_byte
       int gfx_image_mult_by_byte(Src1, Dest, length, C)      unsigned char *Src1      unsigned char *Dest	int length	unsigned char C
	    CODE:	    RETVAL = SDL_imageFilterMultByByte(Src1, Dest, length, C);	    OUTPUT:	      RETVAL

   shift_right_and_mult_by_byte
       int gfx_image_shift_right_and_mult_by_byte(Src1, Dest, length, N, C)	 unsigned char *Src1	  unsigned char *Dest	   int length
	    unsigned char N	 unsigned char C      CODE:	      RETVAL = SDL_imageFilterShiftRightAndMultByByte(Src1, Dest, length, N, C);
	    OUTPUT:	      RETVAL

   shift_left_byte
       int gfx_image_shift_left_byte(Src1, Dest, length, N)	 unsigned char *Src1	  unsigned char *Dest	   int length	   unsigned char N
	    CODE:	    RETVAL = SDL_imageFilterShiftLeftByte(Src1, Dest, length, N);      OUTPUT:		 RETVAL

   shift_left_uint
       int gfx_image_shift_left_uint(Src1, Dest, length, N)	 unsigned char *Src1	  unsigned char *Dest	   int length	   unsigned char N
	    CODE:	    RETVAL = SDL_imageFilterShiftLeftUint(Src1, Dest, length, N);      OUTPUT:		 RETVAL

   shift_left
       int gfx_image_shift_left(Src1, Dest, length, N)	    unsigned char *Src1      unsigned char *Dest      int length      unsigned char N
	    CODE:	    RETVAL = SDL_imageFilterShiftLeft(Src1, Dest, length, N);	   OUTPUT:	     RETVAL

   binarize_using_threshold
       int gfx_image_binarize_using_threshold(Src1, Dest, length, T)	  unsigned char *Src1	   unsigned char *Dest	    int length
	    unsigned char T	 CODE:		 RETVAL = SDL_imageFilterBinarizeUsingThreshold(Src1, Dest, length, T);      OUTPUT:
		 RETVAL

   clip_to_range
       int gfx_image_clip_to_range(Src1, Dest, length, Tmin, Tmax)	unsigned char *Src1	 unsigned char *Dest	  int length	  unsigned
       char Tmin      unsigned char Tmax      CODE:	      RETVAL = SDL_imageFilterClipToRange(Src1, Dest, length, Tmin, Tmax);	OUTPUT:
		 RETVAL

   normalize_linear
       int gfx_image_normalize_linear(Src1, Dest, length, Cmin, Cmax, Nmin, Nmax)      unsigned char *Src1	unsigned char *Dest	 int
       length	   int Cmin	 int Cmax      int Nmin      int Nmax	   CODE:	   RETVAL = SDL_imageFilterNormalizeLinear(Src1, Dest,
       length, Cmin, Cmax, Nmin, Nmax);      OUTPUT:	       RETVAL

   convolve_kernel_3x3_divide
       int gfx_image_convolve_kernel_3x3_divide(Src, Dest, rows, columns, Kernel, Divisor)	unsigned char *Src	unsigned char *Dest
	    int rows	  int columns	   Sint16 *Kernel      unsigned char Divisor	  CODE: 	  RETVAL =
       SDL_imageFilterConvolveKernel3x3Divide(Src, Dest, rows, columns, Kernel, Divisor);      OUTPUT:		 RETVAL

   convolve_kernel_5x5_divide
       int gfx_image_convolve_kernel_5x5_divide(Src, Dest, rows, columns, Kernel, Divisor)	unsigned char *Src	unsigned char *Dest
	    int rows	  int columns	   Sint16 *Kernel      unsigned char Divisor	  CODE: 	  RETVAL =
       SDL_imageFilterConvolveKernel5x5Divide(Src, Dest, rows, columns, Kernel, Divisor);      OUTPUT:		 RETVAL

   convolve_kernel_7x7_divide
       int gfx_image_convolve_kernel_7x7_divide(Src, Dest, rows, columns, Kernel, Divisor)	unsigned char *Src	unsigned char *Dest
	    int rows	  int columns	   Sint16 *Kernel      unsigned char Divisor	  CODE: 	  RETVAL =
       SDL_imageFilterConvolveKernel7x7Divide(Src, Dest, rows, columns, Kernel, Divisor);      OUTPUT:		 RETVAL

   convolve_kernel_9x9_divide
       int gfx_image_convolve_kernel_9x9_divide(Src, Dest, rows, columns, Kernel, Divisor)	unsigned char *Src	unsigned char *Dest
	    int rows	  int columns	   Sint16 *Kernel      unsigned char Divisor	  CODE: 	  RETVAL =
       SDL_imageFilterConvolveKernel9x9Divide(Src, Dest, rows, columns, Kernel, Divisor);      OUTPUT:		 RETVAL

   convolve_kernel_3x3_shift_right
       int gfx_image_convolve_kernel_3x3_shift_right(Src, Dest, rows, columns, Kernel, NRightShift)	 unsigned char *Src	 unsigned char
       *Dest	  int rows	int columns	 Sint16 *Kernel      unsigned char NRightShift	    CODE:	    RETVAL =
       SDL_imageFilterConvolveKernel3x3ShiftRight(Src, Dest, rows, columns, Kernel, NRightShift);      OUTPUT:		 RETVAL

   convolve_kernel_5x5_shift_right
       int gfx_image_convolve_kernel_5x5_shift_right(Src, Dest, rows, columns, Kernel, NRightShift)	 unsigned char *Src	 unsigned char
       *Dest	  int rows	int columns	 Sint16 *Kernel      unsigned char NRightShift	    CODE:	    RETVAL =
       SDL_imageFilterConvolveKernel5x5ShiftRight(Src, Dest, rows, columns, Kernel, NRightShift);      OUTPUT:		 RETVAL

   convolve_kernel_7x7_shift_right
       int gfx_image_convolve_kernel_7x7_shift_right(Src, Dest, rows, columns, Kernel, NRightShift)	 unsigned char *Src	 unsigned char
       *Dest	  int rows	int columns	 Sint16 *Kernel      unsigned char NRightShift	    CODE:	    RETVAL =
       SDL_imageFilterConvolveKernel7x7ShiftRight(Src, Dest, rows, columns, Kernel, NRightShift);      OUTPUT:		 RETVAL

   convolve_kernel_9x9_shift_right
       int gfx_image_convolve_kernel_9x9_shift_right(Src, Dest, rows, columns, Kernel, NRightShift)	 unsigned char *Src	 unsigned char
       *Dest	  int rows	int columns	 Sint16 *Kernel      unsigned char NRightShift	    CODE:	    RETVAL =
       SDL_imageFilterConvolveKernel9x9ShiftRight(Src, Dest, rows, columns, Kernel, NRightShift);      OUTPUT:		 RETVAL

   sobel_x
       int gfx_image_sobel_x(Src, Dest, rows, columns)	    unsigned char *Src	    unsigned char *Dest      int rows	   int columns	    CODE:
		 RETVAL = SDL_imageFilterSobelX(Src, Dest, rows, columns);	OUTPUT: 	  RETVAL

   sobel_x_shift_right
       int gfx_image_sobel_x_shift_right(Src, Dest, rows, columns, NRightShift)      unsigned char *Src      unsigned char *Dest      int rows
	    int columns      unsigned char NRightShift	    CODE:	    RETVAL = SDL_imageFilterSobelXShiftRight(Src, Dest, rows, columns,
       NRightShift);	  OUTPUT:	    RETVAL

   align_stack
       void gfx_image_align_stack()	 CODE:		 SDL_imageFilterAlignStack();

   restore_stack
       void gfx_image_restore_stack()	   CODE:	   SDL_imageFilterRestoreStack();

AUTHORS

       See "AUTHORS" in SDL.

perl v5.14.2							    2012-05-28					  pods::SDL::GFX::ImageFilter(3pm)
All times are GMT -4. The time now is 08:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy