How to remove duplicate lines of a record without changing the order

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ...

2. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR

3. Shell Programming and Scripting

Find and remove duplicate record and print list

Gents, I needs to delete duplicate values and only get uniq values based in columns 2-27 Always we should keep the last record found... I need to store one clean file and other with the duplicate values removed. Input : S3033.0 7305.01 0 420123.8...

4. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do...

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later!

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few...

7. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused:

8. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but...

9. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but...

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file...

LEARN ABOUT DEBIAN

pdl::char

Char(3pm)						User Contributed Perl Documentation						 Char(3pm)

NAME

       PDL::Char -- PDL subclass which allows reading and writing of fixed-length character strings as byte PDLs

SYNOPSIS

	use PDL;
	use PDL::Char;

	my $pchar = PDL::Char->new( [['abc', 'def', 'ghi'],['jkl', 'mno', 'pqr']] );

	$pchar->setstr(1,0,'foo');

	print $pchar; # 'string' bound to "", perl stringify function
	# Prints:
	# [
	#  ['abc' 'foo' 'ghi']
	#  ['jkl' 'mno' 'pqr']
	# ]

	print $pchar->atstr(2,0);
	# Prints:
	# ghi

DESCRIPTION

       This subclass of PDL allows one to manipulate PDLs of 'byte' type as if they were made of fixed length strings, not just numbers.

       This type of behavior is useful when you want to work with charactar grids.  The indexing is done on a string level and not a character
       level for the 'setstr' and 'atstr' commands.

       This module is in particular useful for writing NetCDF files that include character data using the PDL::NetCDF module.

FUNCTIONS

   new
       Function to create a byte PDL from a string, list of strings, list of list of strings, etc.

	# create a new PDL::Char from a perl array of strings
	$strpdl = PDL::Char->new( ['abc', 'def', 'ghij'] );

	# Convert a PDL of type 'byte' to a PDL::Char
	$strpdl1 = PDL::Char->new (sequence (byte, 4, 5)+99);

	$pdlchar3d = PDL::Char->new([['abc','def','ghi'],['jkl', 'mno', 'pqr']]);

   string
       Function to print a character PDL (created by 'char') in a pretty format.

	$char = PDL::Char->new( [['abc', 'def', 'ghi'], ['jkl', 'mno', 'pqr']] );
	print $char; # 'string' bound to "", perl stringify function
	# Prints:
	# [
	#  ['abc' 'def' 'ghi']
	#  ['jkl' 'mno' 'pqr']
	# ]

	# 'string' is overloaded to the "" operator, so:
	# print $char;
	# should have the same effect.

   setstr
       Function to set one string value in a character PDL.  The input position is the position of the string, not a character in the string.  The
       first dimension is assumed to be the length of the string.

       The input string will be null-padded if the string is shorter than the first dimension of the PDL.  It will be truncated if it is longer.

	$char = PDL::Char->new( [['abc', 'def', 'ghi'], ['jkl', 'mno', 'pqr']] );
	$char->setstr(0,1, 'foobar');
	print $char; # 'string' bound to "", perl stringify function
	# Prints:
	# [
	#  ['abc' 'def' 'ghi']
	#  ['foo' 'mno' 'pqr']
	# ]
	$char->setstr(2,1, 'f');
	print $char; # 'string' bound to "", perl stringify function
	# Prints:
	# [
	#  ['abc' 'def' 'ghi']
	#  ['foo' 'mno' 'f']	  -> note that this 'f' is stored "f"
	# ]

   atstr
       Function to fetch one string value from a PDL::Char type PDL, given a position within the PDL.  The input position of the string, not a
       character in the string.  The length of the input string is the implied first dimension.

	$char = PDL::Char->new( [['abc', 'def', 'ghi'], ['jkl', 'mno', 'pqr']] );
	print $char->atstr(0,1);
	# Prints:
	# jkl

perl v5.14.2							    2011-03-30								 Char(3pm)

UNIX for Advanced & Expert Users