Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns
Top Forums UNIX for Beginners Questions & Answers Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns Post 303032089 by pedro88 on Monday 11th of March 2019 07:08:33 AM
Old 03-11-2019
Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns
Hello all,
I need to filter a dataframe composed of several columns of data to remove the duplicates according to one of the columns. I did it with pandas. In the main time, I need that the last column that contains all different data ( not redundant) is conserved in the output like this:
Code:
A         B           C             D
a1        b1           c1            d1
a2       b2          c2           d2

output:
Code:
A         B           C             D
ad        bd       cd            d1,d2

where ad bd and cd are the dereplicated output rows and in D we have that for each of the unique rows we have all the data separated by a comma in one single cell for each unique row.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting table cells in a script

I'd like to use sed or awk to do this but I'm weak on both along with RE. Looking for a way with sed or awk to count for the 7th table data within a table row and if the condition is met to delete "<td>and everything in between </td>". Since the table header start on a specific line each time, that... (15 Replies)
Discussion started by: phpfreak
15 Replies

2. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

3. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

4. UNIX for Dummies Questions & Answers

Two files; if cells match then copy over other columns

My current issue is dealing with two space delimited files. The first file has column 1 as the sample ID's, then columns 2 - n as the observations. The second file has column 1 as the sample ID's, column 2 as the mother ID's, column 3 as the father ID's, column 4 as the gender, and column 5... (3 Replies)
Discussion started by: Renyulb28
3 Replies

5. UNIX Desktop Questions & Answers

Using grep to remove cells instead of lines

I would like to use grep to remove certain strings from a text file but I can't use the grep -v option because it removes the whole line that includes the string whereas I just want to remove the string. How do I go about doing that? My input file: Magmas CEU rs12542019 CPNE1 RBM12 CEU... (1 Reply)
Discussion started by: evelibertine
1 Replies

6. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

7. Shell Programming and Scripting

Remove Duplicates on multiple Key Columns and get the Latest Record from Date/Time Column

Hi Experts , we have a CDC file where we need to get the latest record of the Key columns Key Columns will be CDC_FLAG and SRC_PMTN_I and fetch the latest record from the CDC_PRCS_TS Can we do it with a single awk command. Please help.... (3 Replies)
Discussion started by: vijaykodukula
3 Replies

8. Shell Programming and Scripting

Remove duplicates by keeping the order intact

Hello friends, I have a file with duplicate lines. I could eliminate duplicate lines by running sort <file> |uniq >uniq_file and it works fine BUT it changes the order of the entries as it we did "sort". I need to remove duplicates and also need to keep the order/sequence of entries. I... (1 Reply)
Discussion started by: magnus29
1 Replies

9. UNIX for Beginners Questions & Answers

Merge cells in all rows of a HTML table dynamically.

Hello All, I have visited many pages in Unix.com and could find out one solution for merging the HTML cells in the 1st row. (Unable to post the complete URL as I should not as per website rules). But, however I try, I couldn't achieve this merging to happen for all other rows of HTML... (17 Replies)
Discussion started by: Mounika
17 Replies

10. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

LEARN ABOUT PHP

db2_special_columns

DB2_SPECIAL_COLUMNS(3)							 1						    DB2_SPECIAL_COLUMNS(3)

db2_special_columns - Returns a result set listing the unique row identifier columns for a table

SYNOPSIS

       resource db2_special_columns (resource  $connection, string  $qualifier, string	$schema, string  $table_name, int  $scope)

DESCRIPTION

	Returns a result set listing the unique row identifier columns for a table.

PARAMETERS

	      o $connection
		- A valid connection to an IBM DB2, Cloudscape, or Apache Derby database.

	      o $qualifier
		- A qualifier for DB2 databases running on OS/390 or z/OS servers. For other databases, pass NULL or an empty string.

	      o $schema
		- The schema which contains the tables.

	      o $table_name
		- The name of the table.

	      o $scope
		-  Integer  value representing the minimum duration for which the unique row identifier is valid. This can be one of the following
		values:

	      +--------------+--------------------------------------+---+
	      |Integer value |					    |	|
	      | 	     |					    |	|
	      | 	     |		   SQL constant 	    |	|
	      | 	     |					    |	|
	      | 	     |		   Description		    |	|
	      | 	     |					    |	|
	      +--------------+--------------------------------------+---+
	      |      0	     |					    |	|
	      | 	     |					    |	|
	      | 	     |		 SQL_SCOPE_CURROW	    |	|
	      | 	     |					    |	|
	      | 	     | Row identifier is valid	only  while |	|
	      | 	     | the cursor is positioned on the row. |	|
	      | 	     |					    |	|
	      |      1	     |					    |	|
	      | 	     |					    |	|
	      | 	     |	      SQL_SCOPE_TRANSACTION	    |	|
	      | 	     |					    |	|
	      | 	     | Row  identifier	is  valid  for	the |	|
	      | 	     | duration of the transaction.	    |	|
	      | 	     |					    |	|
	      |      2	     |					    |	|
	      | 	     |					    |	|
	      | 	     |		SQL_SCOPE_SESSION	    |	|
	      | 	     |					    |	|
	      | 	     | Row  identifier	is  valid  for	the |	|
	      | 	     | duration of the connection.	    |	|
	      | 	     |					    |	|
	      +--------------+--------------------------------------+---+
RETURN VALUES

	Returns  a  statement resource with a result set containing rows with unique row identifier information for a table. The rows are composed
       of the following columns:

       +------------+---------------------------------------------------+
       |Column name |							|
       |	    |							|
       |	    |			 Description			|
       |	    |							|
       +------------+---------------------------------------------------+
       |   SCOPE    |							|
       |	    |							|
       |	    |							|
       |	    |							|
       |	    |	     box, tab (|); c | c | c |	.   T{	Integer |
       |	    |	     value					|
       |	    |							|
       |	    |			SQL constant			|
       |	    |							|
       |	    |			 Description			|
       |	    |							|
       +------------+---------------------------------------------------+
       |     0	    |							|
       |	    |							|
       |	    |		      SQL_SCOPE_CURROW			|
       |	    |							|
       |	    | Row  identifier is valid only while the cursor is |
       |	    | positioned on the row.				|
       |	    |							|
       |     1	    |							|
       |	    |							|
       |	    |		    SQL_SCOPE_TRANSACTION		|
       |	    |							|
       |	    | Row identifier is valid for the duration	of  the |
       |	    | transaction.					|
       |	    |							|
       |     2	    |							|
       |	    |							|
       |	    |		      SQL_SCOPE_SESSION 		|
       |	    |							|
       |	    | Row  identifier  is valid for the duration of the |
       |	    | connection.					|
       |	    |							|
       +------------+---------------------------------------------------+
       T} T{ COLUMN_NAME

       T} |T{ Name of the unique column.

       T} T{ DATA_TYPE

       T} |T{ SQL data type for the column.

       T} T{ TYPE_NAME

       T} |T{ Character string representation of the SQL data type for the column.

       T} T{ COLUMN_SIZE

       T} |T{ An integer value representing the size of the column.

       T} T{ BUFFER_LENGTH

       T} |T{
	Maximum number of bytes necessary to store data from this column.

       T} T{ DECIMAL_DIGITS

       T} |T{
	The scale of the column, or NULL where scale is not applicable.

       T} T{ NUM_PREC_RADIX

       T} |T{
	An integer value of either 10 (representing an exact numeric data type), 2 (representing an approximate numeric data type), or NULL  (rep-
       resenting a data type for which radix is not applicable).

       T} T{ PSEUDO_COLUMN

       T} |T{ Always returns 1.

       T}

SEE ALSO

       db2_column_privileges(3),  db2_columns(3),  db2_foreign_keys(3), db2_primary_keys(3), db2_procedure_columns(3), db2_procedures(3), db2_sta-
       tistics(3), db2_table_privileges(3), db2_tables(3).

PHP Documentation Group 												    DB2_SPECIAL_COLUMNS(3)
All times are GMT -4. The time now is 05:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy