Duplicate check by passing external parameter


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicate check by passing external parameter
# 1  
Old 01-03-2017
Duplicate check by passing external parameter

I have a code which is using to find duplicates in a files based on column.Below is the same code which is used to find duplicates in my file based on column 1

Code:
awk -F'|' '{if (x[$1]) { x_count[$1]++; print $0; if (x_count[$1] == 1) { print x[$1] } } x[$1] = $0}' FileName >Dup_File.txt

But my requirement here is that, I want to make this command global , where I will be able to pass the column position as an external parameter.

if we pass the external parameter to the above code to check for column two and three then the code should look like below,

Code:
awk -F'|' '{if (x[$1]) { x_count[$2,$3]++; print $0; if (x_count[$2,$3] == 1) { print x[$2,$3] } } x[$2,$3] = $0}' FileName >Dup_File.txt

# 2  
Old 01-03-2017
A few questions:
- WHAT is your request? A malfunction? An error? A non-satisfactory result?
- Do you know how to pass variables to awk?
- Why do you consistently use $2,$3 as the array index except for the first x[$1]?
- Why do you print $0 from its second occurrence and again for the exact second time?
Some sample data might help...
# 3  
Old 01-03-2017
Ok You can leave ignore my previous post.So let me put my requirement here.

I have a pipe delimited file. I am trying to create a UNIX script which can be used across different files to print the duplicates line from the file.

For ex. I have two files A.txt and B.txt

A.txt
Code:
123|345|asd
122|ASD|DEF
123|ASW|231

For this file A.txt I need to print duplicate records based on first column.So my expected out put is

Code:
123|345|asd
123|ASW|231

B.txt
Code:
34|aw|asd
33|aq|qw
54|aq|qw

For this file B.txt I need to print duplicate records based on 2nd and 3rd column.My expected output is

Code:
33|aq|qw
54|aq|qw

So I need to write one script, where I can pass the column positions based on which the duplicate need to be checked can be passed as external parameter.

For the first file, sh test_script.sh 1
For the second file , sh test_script.sh "2,3" something like this
# 4  
Old 01-03-2017
Try
Code:
awk -F'|' '
NR == 1 {n = split (FLDS, ARRX, ",")
        }

        {IX = ""
         for (i=1; i<=n; i++) IX = IX FS $ARRX[i]
        }

!x[IX]  {x[IX] = $0
         L[IX] = 1
         next
        }
L[IX]   {print x[IX]
         L[IX] = 0
        }
1
' FLDS="2,3" file
33|aq|qw
54|aq|qw

These 2 Users Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Passing parameter more than 9

Hi, I've written a script where eleven parameter to be passed from command line which is inserting into an oracle table, it is working but the tenth and 11th parameter are not accepting as given it is referring to 1st parameter. HERE IS THE SCRIPT #!/bin/ksh #set -o echo $*... (4 Replies)
Discussion started by: sankar
4 Replies

2. Shell Programming and Scripting

Passing external variable to awk

Hi, I am trying to write a bash script in which I need to pass a external variable to the awk program. I tired using -v but it not accepting the value. Here is my sample code. #!/usr/bin/bash ###################################################################################### ####... (5 Replies)
Discussion started by: jpkumar10
5 Replies

3. Shell Programming and Scripting

Passing parameter to script, and split the parameter

i am passing input parameter 'one_two' to the script , the script output should display the result as below one_1two one_2two one_3two if then echo " Usage : <$0> <DATABASE> " exit 0 else for DB in 1 2 3 do DBname=`$DATABASE | awk -F "_" '{print $1_${DB}_$2}` done fi (5 Replies)
Discussion started by: only4satish
5 Replies

4. Shell Programming and Scripting

passing file extension using external variable

Hi, How can I modify the FILETYPE command ? I want to provide the file extension, like txt, root .? Thanks, #!/bin/bash FROM=$1 TO=$2 FILETYPE=$3 ... (4 Replies)
Discussion started by: nrjrasaxena
4 Replies

5. Shell Programming and Scripting

using an awk internal variable as parameter for an external array

Hello, I am running a bash script under linux which first defines an CA-array like j=0 num1=120.00 num2=10.00 until do CA='echo $num1 + $j*$num2' j=$ done within the later awk section of this same script I want to read data from a file. If the value of the second column is... (3 Replies)
Discussion started by: MotAah
3 Replies

6. Shell Programming and Scripting

passing arguments to external script

Hi! I have a python script that requires arguments and these arguments are file paths. This script works fine when executed like this: /my_python_script "file_path1" "file_path2" (i added quotes as some file names may have weird characters) the issue happens when i launch my python script... (14 Replies)
Discussion started by: gigagigosu
14 Replies

7. Shell Programming and Scripting

Passing answers to external program from KSH

I have asked this before but I haven't had any luck so far getting this to work. I use RCS(revision control system). When it runs if I pass the value 'unlock' to $3 its reassigned to $unlock. When I run the command (rcs -u'version number' 'filename') ti will ask me 1-(Do you want to break the lock... (5 Replies)
Discussion started by: pjones006
5 Replies

8. Shell Programming and Scripting

wrong parameter passing!

Hi all I have a script which will take input as filename and passes it to a java program. It is as follows -------------------------------- FILENAME=$1 echo $FILENAME ${JAVA_HOME}/bin/java -cp DateProvider $FILENAME ------------------------------------------------- when I execute the same... (2 Replies)
Discussion started by: malle
2 Replies

9. Shell Programming and Scripting

parameter passing

Hallo everyone, This is my problem below: /home/cerebrus/pax=>vat class2.sh ksh: vat: not found /home/cerebrus/pax=>cat class2.sh #!/bin/ksh set -x bdf|grep appsdev|awk '{ print $5 }'> class3 dd={cat class3} echo $dd /home/cerebrus/pax=> /home/cerebrus/pax=>./class2.sh + bdf +... (8 Replies)
Discussion started by: kekanap
8 Replies

10. Shell Programming and Scripting

Passing value to an external program problem...

This code is in my 'case' statement and it all else works fine. The problem I have is that the value in 'procno' is not passed on to the external program (fireit). It is passing all zeros instead of the actual process number. By the time I get to this case statement, I know the "Number" and... (1 Reply)
Discussion started by: giannicello
1 Replies
Login or Register to Ask a Question