Sponsored Content
Top Forums Shell Programming and Scripting Filter duplicate records from csv file with condition on one column Post 303010202 by jim mcnamara on Thursday 28th of December 2017 11:28:18 AM
Old 12-28-2017
This is the kind of question that needs to have:
Sample good input that will not be "filtered"
Sample bad input -> expected output

Without this start we cannot help.

What code have you tried? Please show us where you are in your attempt.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

2. Shell Programming and Scripting

Apply condition on fixed width file and filter records

Dear members.. I have a fixed width file. Requirement is as below:- 1. Scan each record from this fixed width file 2. Check for value under field no "6" equals to "ABC". If yes, then filter this record into the output file Please suggest a unix command to achieve this, my guess awk might... (6 Replies)
Discussion started by: sureshg_sampat
6 Replies

3. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

4. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

5. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

6. Linux

Filter a .CSV file based on the 5th column values

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013",""... (2 Replies)
Discussion started by: dhruuv369
2 Replies

7. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

8. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies

10. UNIX for Beginners Questions & Answers

Filtering records of a csv file based on a value of a column

Hi, I tried filtering the records in a csv file using "awk" command listed below. awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile The output always has all the entries. The same command worked for different users from one of the forum links. content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies
TRIG(3pm)						User Contributed Perl Documentation						 TRIG(3pm)

NAME
PDL::GSLSF::TRIG - PDL interface to GSL Special Functions DESCRIPTION
This is an interface to the Special Function package present in the GNU Scientific Library. SYNOPSIS
FUNCTIONS
gsl_sf_sin Signature: (double x(); double [o]y(); double [o]e()) Sin(x) with GSL semantics. gsl_sf_sin does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_cos Signature: (double x(); double [o]y(); double [o]e()) Cos(x) with GSL semantics. gsl_sf_cos does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_hypot Signature: (double x(); double xx(); double [o]y(); double [o]e()) Hypot(x,xx) with GSL semantics. gsl_sf_hypot does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_complex_sin Signature: (double zr(); double zi(); double [o]x(); double [o]y(); double [o]xe(); double [o]ye()) Sin(z) for complex z gsl_sf_complex_sin does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_complex_cos Signature: (double zr(); double zi(); double [o]x(); double [o]y(); double [o]xe(); double [o]ye()) Cos(z) for complex z gsl_sf_complex_cos does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_complex_logsin Signature: (double zr(); double zi(); double [o]x(); double [o]y(); double [o]xe(); double [o]ye()) Log(Sin(z)) for complex z gsl_sf_complex_logsin does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_lnsinh Signature: (double x(); double [o]y(); double [o]e()) Log(Sinh(x)) with GSL semantics. gsl_sf_lnsinh does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_lncosh Signature: (double x(); double [o]y(); double [o]e()) Log(Cos(x)) with GSL semantics. gsl_sf_lncosh does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_polar_to_rect Signature: (double r(); double t(); double [o]x(); double [o]y(); double [o]xe(); double [o]ye()) Convert polar to rectlinear coordinates. gsl_sf_polar_to_rect does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_rect_to_polar Signature: (double x(); double y(); double [o]r(); double [o]t(); double [o]re(); double [o]te()) Convert rectlinear to polar coordinates. return argument in range [-pi, pi]. gsl_sf_rect_to_polar does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_angle_restrict_symm Signature: (double [o]y()) Force an angle to lie in the range (-pi,pi]. gsl_sf_angle_restrict_symm does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_angle_restrict_pos Signature: (double [o]y()) Force an angle to lie in the range [0,2 pi). gsl_sf_angle_restrict_pos does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_sin_err Signature: (double x(); double dx(); double [o]y(); double [o]e()) Sin(x) for quantity with an associated error. gsl_sf_sin_err does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. gsl_sf_cos_err Signature: (double x(); double dx(); double [o]y(); double [o]e()) Cos(x) for quantity with an associated error. gsl_sf_cos_err does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles. AUTHOR
This file copyright (C) 1999 Christian Pellegrin <chri@infis.univ.trieste.it> All rights reserved. There is no warranty. You are allowed to redistribute this software / documentation under certain conditions. For details, see the file COPYING in the PDL distribution. If this file is separated from the PDL distribution, the copyright notice should be included in the file. The GSL SF modules were written by G. Jungman. perl v5.14.2 2012-05-30 TRIG(3pm)
All times are GMT -4. The time now is 11:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy