Duplicate value with different index


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicate value with different index
# 1  
Old 11-19-2014
Duplicate value with different index

Hello Gents,

Please give a help with this case

Input

Code:
10001010G1
10001010G1
10001010G1
10001010G2
10001010G3
10001012G1
10001012G1
10001012G1
10001012G1
10001014G1
10001014G1
10001014G2

The case is to check that all duplicate values have the same index
values are in columns 1-8, index are in column 9-10.
in the first case the first value duplicate (10001010) have index G1, G2 and G3, and also the 3rd value duplicate (10001014) has G1 and G2.

Then I would like to get a output
like this

Code:
Value 10001010 has different code G1 G2 G3
Value 10001014 has different code G1 G2

if there is not errors found can i get a msg ..

Code:
Not errors found

Thanks for your help
# 2  
Old 11-19-2014
This bash-script should do the trick:
Code:
#!/bin/bash

declare -A array

while read line
do
    case "${array[${line:0:8}]}" in
        *${line:8:2}*)
            ;;
        *)
            array[${line:0:8}]="${array[${line:0:8}]} ${line:8:2}";;
    esac
done

for val in "${!array[@]}"
do
    if [[ ${array[$val]} == \ *\ * ]]
    then
        echo Value $val has different code ${array[$val]}
        found=1
    fi
done

[[ -z $found ]] && echo Not errors found

Execute it with
Code:
script.sh <inputfile

This User Gave Thanks to hergp For This Post:
# 3  
Old 11-19-2014
Thanks for the answer. The script works fine Smilie
but takes a long time to process.
Is there a fast way to do it with awk?
# 4  
Old 11-19-2014
Try this
Code:
{
    val = substr ($1, 1, 8);
    cod = substr ($1, 9, 2);

    if (index (array[val], cod) == 0) array[val] = array[val] " " cod;
}

END {
    for (val in array)
        if (match (array[val], " .* .*"))
        {
            print "Value " val " has different code" array[val];
            found=1
        }

    if (found == 0) print "Not errors found"
}

Execute with
Code:
awk -f script.awk <inputfile

This User Gave Thanks to hergp For This Post:
# 5  
Old 11-19-2014
Dear hergp

Thaks a lot for your help, I have many files to merge before I get the complete file to read whith your scritp

how I can at the file to be read inside of the script

example i will merge many files with extention .xls (example), then

Code:
cat *.xls* > database

how to add this in your script and read the file created

So I will tape the name of the script and by default he will know that the name of the file to be read will be
Code:
database

regards
# 6  
Old 11-19-2014
awk can read multiple files in one run, just use all filenames or a wildcard expression on the command line
Code:
awk -f script.awk *.xls

You do not need cat for this. Or if you need the intermediate file database, then try
Code:
cat *.xls | tee database | awk -f script.awk

Hope this helps.
This User Gave Thanks to hergp For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Adding sequential index to duplicate strings

I have a text file in the following format >Homo sapiens KQKCLYNLPFKRNLEGCRERCSLVIQIPRCCKGYFGRDCQACPGGPDAPCNNRGVCLDQY SATGECKCNTGFNGTACEMCWPGRFGPDCLPCGCSDHGQCDDGITGSGQCLCETGWTGPS CDTQAVLPAVCTPPCSAHATCKENNTCECNLDYEGDGITCTVVDFCKQDNGGCAKVARCS... (2 Replies)
Discussion started by: jerrild
2 Replies

2. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

3. Shell Programming and Scripting

Need Help in Index

I am having data in XML format and trying to extract codes form two fields called <String>, below is the data. <Node>tollfree<Condition>BooleanOperator<Operation>AND</Operation><Condition>BooleanOperator<Operation>NOT</Operation><Condition>FieldSelection<Field Context="ALL fields"... (7 Replies)
Discussion started by: rramkrishnas
7 Replies

4. UNIX and Linux Applications

Index server

Hi guys, I have postgresql server with huge amount of data, nearly 2 billion records. each record is at most 50 bytes(4 integer fields). I need to build index on all column to do fast reporting. but indexes becomes bloat after some time. almost 80% of database size is because of its huge... (0 Replies)
Discussion started by: majid.merkava
0 Replies

5. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

6. Shell Programming and Scripting

Sort from start index and end index in line

Hi All, I have a file (FileNames.txt) which contains the following data in it. $ cat FileNames.txt MYFILE17XXX208Sep191307.csv MYFILE19XXX208Sep192124.csv MYFILE20XXX208Sep192418.csv MYFILE22XXX208Sep193234.csv MYFILE21XXX208Sep193018.csv MYFILE24XXX208Sep194053.csv... (5 Replies)
Discussion started by: krish_indus
5 Replies

7. Shell Programming and Scripting

duplicate index names renamed

Hello everyone ! Please have a minute and see if you know how to script this I have a file like this: "create table .... ... create index n112 on ... ... create table ... .... create index n113 on... ... create table ... create index n112 on ...! duplicate ... (1 Reply)
Discussion started by: sotoc79
1 Replies

8. UNIX for Dummies Questions & Answers

wh inode index starts from 1 unlike array index (0)

brothers why inode index starts from 1 unlike array inex which starts from 0 its a question from the design of unix operating system of maurice j.bach i need to know the answer urgently...someone help please (1 Reply)
Discussion started by: sairamdevotee
1 Replies

9. Filesystems, Disks and Memory

why the inode index of file system starts from 1 unlike array index(0)

why do inode indices starts from 1 unlike array indexes which starts from 0 its a question from "the design of unix operating system" of maurice j bach id be glad if i get to know the answer quickly :) (0 Replies)
Discussion started by: sairamdevotee
0 Replies

10. Shell Programming and Scripting

What is index?

hi, :) In a shell script i came accross the following lines 1.for i in ` find /home/oracle -name ch' 2.do 3.echo $i 4.idx=`expr index $i .` 5.done Here iam not able to understand the porpose of the word "index" in line 4. any help ? cheers RRK (3 Replies)
Discussion started by: ravi raj kumar
3 Replies
Login or Register to Ask a Question