Sponsored Content
Top Forums Shell Programming and Scripting Filter file to remove duplicate values in first column Post 302977903 by drl on Saturday 23rd of July 2016 09:03:34 AM
Old 07-23-2016
Hi.

Seems like sort with unique option works for me:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate remove all identical lines, sort.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C sort pass-fail

FILE=${1-data1}

pl " Input data file $FILE:"
head $FILE

pl " Expected output:"
head expected-output.txt

pl " Results:"
sort -u -k1,1 $FILE |
tee f1

pass-fail f1 expected-output.txt

exit 0

producing
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.4 (jessie) 
bash GNU bash 4.3.30
sort (GNU coreutils) 8.23
pass-fail - ( local: RepRev 1.6, ~/bin/pass-fail, 2016-07-23 )

-----
 Input data file data1:
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
0       compound_00     -3.5054     -1.1207     -2.4372
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114

-----
 Expected output:
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016

-----
 Results:
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016

-----
 Comparison of 7 created lines with 7 lines of desired results:
 Succeeded -- files (computed) f1 and (standard) expected-output.txt have same content.

The pass-fail code is basically just a wrapper around cmp for some extra checking and reporting.

Best wishes ... cheers, drl
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

2. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

3. Shell Programming and Scripting

Check to identify duplicate values at first column in csv file

Hello experts, I have a requirement where I have to implement two checks on a csv file: 1. Check to see if the value in first column is duplicate, if any value is duplicate script should exit. 2. Check to verify if the value at second column is between "yes" or "no", if it is anything else... (4 Replies)
Discussion started by: avikaljain
4 Replies

4. Linux

Filter a .CSV file based on the 5th column values

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013",""... (2 Replies)
Discussion started by: dhruuv369
2 Replies

5. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

6. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

7. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

8. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
tabs(1) 							   User Commands							   tabs(1)

NAME
tabs - set tabs on a terminal SYNOPSIS
tabs [ -n | --file [ [-code] | -a | -a2 | -c | -c2 | -c3 | -f | -p | -s | -u] ] [ +m [n]] [-T type] tabs [-T type] [ + m [n]] n1 [ , n2 ,...] DESCRIPTION
The tabs utility sets the tab stops on the user's terminal according to a tab specification, after clearing any previous settings. The user's terminal must have remotely settable hardware tabs. OPTIONS
The following options are supported. If a given flag occurs more than once, the last value given takes effect: -T type tabs needs to know the type of terminal in order to set tabs and margins. type is a name listed in term(5). If no -T flag is supplied, tabs uses the value of the environment variable TERM. If the value of TERM is NULL or TERM is not defined in the environment (see environ(5)), tabs uses ansi+tabs as the terminal type to provide a sequence that will work for many terminals. +m[n] The margin argument may be used for some terminals. It causes all tabs to be moved over n columns by making column n+1 the left margin. If +m is given without a value of n, the value assumed is 10. For a TermiNet, the first value in the tab list should be 1, or the margin will move even further to the right. The normal (leftmost) margin on most terminals is obtained by +m0. The margin for most terminals is reset only when the +m flag is given explicitly. Tab Specification Four types of tab specification are accepted. They are described below: canned, repetitive (-n), arbitrary (n1,n2,...), and file (-file). If no tab specification is given, the default value is -8, that is, UNIX system ``standard'' tabs. The lowest column number is 1. Note: For tabs, column 1 always refers to the leftmost column on a terminal, even one whose column markers begin at 0, for example, the DASI 300, DASI 300s, and DASI 450. Canned -code Use one of the codes listed below to select a canned set of tabs. If more than one code is specified, the last code option will be used. The legal codes and their meanings are as follows: -a 1,10,16,36,72 Assembler, IBM S/370, first format -a2 1,10,16,40,72 Assembler, IBM S/370, second format -c 1,8,12,16,20,55 COBOL, normal format -c2 1,6,10,14,49 COBOL compact format (columns 1-6 omitted). Using this code, the first typed character corresponds to card column 7, one space gets you to column 8, and a tab reaches column 12. Files using this tab setup should include a format specification as follows (see fspec(4)): <:t-c2 m6 s66 d:> -c3 1,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,67 COBOL compact format (columns 1-6 omitted), with more tabs than -c2. This is the recommended format for COBOL. The appropriate format specification is (see fspec(4)): <:t-c3 m6 s66 d:> -f 1,7,11,15,19,23 FORTRAN -p 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61 PL/I -s 1,10,55 SNOBOL -u 1,12,20,44 UNIVAC 1100 Assembler Repetitive -n A repetitive specification requests tabs at columns 1+n, 1+2*n, etc., where n is a single-digit decimal number. Of particular importance is the value 8: this represents the UNIX system ``standard'' tab setting, and is the most likely tab setting to be found at a terminal. When -0 is used, the tab stops are cleared and no new ones are set. Arbitrary See OPERANDS. File -file If the name of a file is given, tabs reads the first line of the file, searching for a format specification (see fspec(4)). If it finds one there, it sets the tab stops according to it, otherwise it sets them as -8. This type of specification may be used to make sure that a tabbed file is printed with correct tab settings, and would be used with the pr command: example% tabs - file; pr file Tab and margin setting is performed via the standard output. OPERANDS
The following operand is supported: n1[,n2,...] The arbitrary format consists of tab-stop values separated by commas or spaces. The tab-stop values must be positive deci- mal integers in ascending order. Up to 40 numbers are allowed. If any number (except the first one) is preceded by a plus sign, it is taken as an increment to be added to the previous value. Thus, the formats 1,10,20,30, and 1,10,+10,+10 are considered identical. EXAMPLES
Example 1: Using the tabs command The following command is an example using -code ( canned specification) to set tabs to the settings required by the IBM assembler: columns 1, 10, 16, 36, 72: example% tabs -a The next command is an example of using -n (repetitive specification), where n is 8, causes tabs to be set every eighth position: 1+(1*8), 1+(2*8), ... which evaluate to columns 9, 17, ...: example% tabs -8 This command uses n1,n2,... (arbitrary specification) to set tabs at columns 1, 8, and 36: example% tabs 1,8,36 The last command is an example of using -file (file specification) to indicate that tabs should be set according to the first line of $HOME/fspec.list/att4425 (see fspec(4)). example% tabs -$HOME/fspec.list/att4425 ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of tabs: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. TERM Determine the terminal type. If this variable is unset or null, and if the -T option is not specified, terminal type ansi+tabs will be used. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
expand(1), newform(1), pr(1), stty(1), tput(1), fspec(4), terminfo(4), attributes(5), environ(5), term(5), standards(5) NOTES
There is no consistency among different terminals regarding ways of clearing tabs and setting the left margin. tabs clears only 20 tabs (on terminals requiring a long sequence), but is willing to set 64. The tabspec used with the tabs command is different from the one used with the newform command. For example, tabs -8 sets every eighth position; whereas newform -i-8 indicates that tabs are set every eighth position. SunOS 5.10 1 Feb 1995 tabs(1)
All times are GMT -4. The time now is 11:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy