Sponsored Content
Top Forums Shell Programming and Scripting CSV File:Filter duplicate records from column1 & another column having unique record Post 303010465 by as7951 on Tuesday 2nd of January 2018 05:26:54 AM
Old 01-02-2018
Hi Experts,

Apologies in case i am disturbing you with my posts.
I am not much good with awk scripting but I do shell scripting and try to learn more with the issues i come across
But sincerely i need to know work around for this query.

I tried the below code, but it is not working as per my expectation.
It is working when column 2 contains unique value in every row, but if row 2 and row 5 contains same value, it prints "error".
Code:
awk -F"|" '
{++CNT[$1]
}
{++ABC[$2]
}

(CNT[$1] && ABC[$2] > 1) { print "error"
        }
'

Request if you can help to improve.

I need to have file suppose that contains duplicate values in column 1 then against those duplicate value in column 2 there should be unique values
In above sample file.
There wont' be 2 line per document number, there can be any number of duplicate values, it can be more than 5 or even 50
Yes, there should be non-identical number in column2(Line number) per Document number(column1) and there is no limit on number, they just has to be non duplicate.
if column 1 contain duplicate values in row then corresponding to those duplicate values in row column 2 should contain non duplicate values


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 01-02-2018 at 06:32 AM.. Reason: Added CODE tags.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

return a list of unique values of a column from csv format file

Hi all, I have a huge csv file with the following format of data, Num SNPs, 549997 Total SNPs,555352 Num Samples, 157 SNP, SampleID, Allele1, Allele2 A001,AB1,A,A A002,AB1,A,A A003,AB1,A,A ... ... ... I would like to write out a list of unique SNP (column 1). Could you... (3 Replies)
Discussion started by: phoeberunner
3 Replies

2. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

3. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

4. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

5. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

6. Linux

Filter a .CSV file based on the 5th column values

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013",""... (2 Replies)
Discussion started by: dhruuv369
2 Replies

7. Shell Programming and Scripting

Output first unique record in csv file

Hi, I have to output a new csv file from an input csv file with first unique value in the first column. input csv file color product id status green 102 pass yellow 201 hold yellow 202 keep green 101 ok green 103 hold yellow 203 ... (5 Replies)
Discussion started by: Chris LAU
5 Replies

8. Linux

To get all the columns in a CSV file based on unique values of particular column

cat sample.csv ID,Name,no 1,AAA,1 2,BBB,1 3,AAA,1 4,BBB,1 cut -d',' -f2 sample.csv | sort | uniq this gives only the 2nd column values Name AAA BBB How to I get all the columns of CSV along with this? (1 Reply)
Discussion started by: sanvel
1 Replies

9. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

10. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies
cdk_position(3) 					     Library Functions Manual						   cdk_position(3)

NAME
cdk_position - curses widget positioning key-bindings SYNOPSIS
Position methods for the Cdk widgets allow the user to move the widget around the screen via the cursor/keypad keys. DESCRIPTION
The following key bindings can be used to move the widget around the screen. +---------------------------------------------------------------+ |Key Action | +---------------------------------------------------------------+ |Up Arrow Moves the widget up one row. | |Down Arrow Moves the widget down one row. | |Left Arrow Moves the widget left one column | |Right Arrow Moves the widget right one column | |1 Moves the widget down one row and left one | | column. | |2 Moves the widget down one row. | |3 Moves the widget down one row and right one | | column. | |4 Moves the widget left one column | |5 Centers the widget both vertically and hori- | | zontally. | |6 Moves the widget right one column | |7 Moves the widget up one row and left one | | column. | |8 Moves the widget up one row. | |9 Moves the widget up one row and right one | | column. | |t Moves the widget to the top of the screen. | |b Moves the widget to the bottom of the | | screen. | |l Moves the widget to the left of the screen. | |r Moves the widget to the right of the screen. | |c Centers the widget between the left and | | right of the window. | |C Centers the widget between the top and bot- | | tom of the window. | |Escape Returns the widget to its original position. | |Return Exits the function and leaves the wid- | | getwhere it was. | +---------------------------------------------------------------+ +---------------------------------------------------------------+ The numbers 0-9 are chosen to correspond to the numeric keypad on your terminal. Cdk does not handle the application mode keypad control sequences, since those are not defined via terminfo. To use the numeric keypad you should ensure that it is in normal (or NumLock) mode. SEE ALSO
cdk(3), cdk_binding(3), cdk_display(3), cdk_screen(3) cdk_position(3)
All times are GMT -4. The time now is 03:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy