Remove duplicate rows of a file based on a value of a column

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code...

2. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ...

3. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323...

4. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file...

5. UNIX for Dummies Questions & Answers

merging rows into new file based on rows and first column

I have 2 files, file01= 7 columns, row unknown (but few) file02= 7 columns, row unknown (but many) now I want to create an output with the first field that is shared in both of them and then subtract the results from the rest of the fields and print there e.g. file 01 James|0|50|25|10|50|30...

6. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX...

7. UNIX for Dummies Questions & Answers

How to get remove duplicate of a file based on many conditions

Hii Friends.. I have a huge set of data stored in a file.Which is as shown below a.dat: RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ...

8. Shell Programming and Scripting

Remove duplicate line detail based on column one data

My input file: AVI.out <detail>named as the RRM .</detail> AVI.out <detail>Contains 1 RRM .</detail> AR0.out <detail>named as the tellurite-resistance.</detail> AWG.out <detail>Contains 2 HTH .</detail> ADV.out <detail>named as the DENR family.</detail> ADV.out ...

9. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is...

10. Shell Programming and Scripting

To remove date and duplicate rows from a log file using unix commands

Hi, I have a log file having size of 48mb. For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages. The following things to be done : 1) To remove all the date and time from the log file 2) To remove all the...

LEARN ABOUT PHP

array_column

ARRAY_COLUMN(3) 							 1							   ARRAY_COLUMN(3)

array_column - Return the values from a single column in the input array

SYNOPSIS

       array array_column (array  $array, mixed  $column_key, [mixed  $index_key = null])

DESCRIPTION

       array_column(3)	returns  the  values  from  a  single  column of the $array, identified by the $column_key. Optionally, you may provide an
       $index_key to index the values in the returned array by the values from the $index_key column in the input array.

PARAMETERS

	      o $array
		- A multi-dimensional array (record set) from which to pull a column of values.

	      o $column_key
		- The column of values to return. This value may be the integer key of the column you wish to retrieve, or it may  be  the  string
		key  name  for an associative array. It may also be NULL to return complete arrays (useful together with $index_key to reindex the
		array).

	      o $index_key
		- The column to use as the index/keys for the returned array. This value may be the integer key of the column, or it  may  be  the
		string key name.

RETURN VALUES

	Returns an array of values representing a single column from the input array.

EXAMPLES

       Example #1

	      Get column of first names from recordset

	      <?php
	      // Array representing a possible record set returned from a database
	      $records = array(
		  array(
		      'id' => 2135,
		      'first_name' => 'John',
		      'last_name' => 'Doe',
		  ),
		  array(
		      'id' => 3245,
		      'first_name' => 'Sally',
		      'last_name' => 'Smith',
		  ),
		  array(
		      'id' => 5342,
		      'first_name' => 'Jane',
		      'last_name' => 'Jones',
		  ),
		  array(
		      'id' => 5623,
		      'first_name' => 'Peter',
		      'last_name' => 'Doe',
		  )
	      );

	      $first_names = array_column($records, 'first_name');
	      print_r($first_names);
	      ?>

	      The above example will output:

	      Array
	      (
		  [0] => John
		  [1] => Sally
		  [2] => Jane
		  [3] => Peter
	      )

       Example #2

	       Get column of last names from recordset, indexed by the "id" column

	      <?php
	      // Using the $records array from Example #1
	      $last_names = array_column($records, 'last_name', 'id');
	      print_r($last_names);
	      ?>

	      The above example will output:

	      Array
	      (
		  [2135] => Doe
		  [3245] => Smith
		  [5342] => Jones
		  [5623] => Doe
	      )

PHP Documentation Group 													   ARRAY_COLUMN(3)

UNIX for Dummies Questions & Answers