Highlighting duplicate string on a line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Highlighting duplicate string on a line
# 1  
Old 09-11-2014
Highlighting duplicate string on a line

Hi all

I have a grep written to pull out values; below (in the code snip-it) is an example of the output.
What I'm struggling to do, and looking for assistance on, is identifying the lines that have duplicate strings.
For example 74859915K74859915K in the below is 74859915K repeated twice but 32575310100014 is not a whole repeating value so I don't want to see it.

In my head (and what I'm unable to do) I want to do something like count it's length, split it in half and confirm the first half matches the second half... I'm open to suggestions as there may be a better way to do it.

Background - these values are in multiple files within an xml tag <foo></foo>. My grep is extracting them and removing the xml tags with sed leaving just the below output... it's the next step where I want to only have the true dupes.

Many thanks in advance.

Code:
74859915K74859915K
0B153858340B15385834
MUNS0-0000000001MUNS0-0000000001
10594556C10594556C
0B982730630B98273063
Q1818002FQ1818002F
78883385D78883385D
44871376D44871376D
B14513386B14513386
016797265C016797265C
0A120861950A12086195
025691290Z025691290Z
31262294G31262294G
B57312068B57312068
16803742B16803742B
723029268723029268
A50470772A50470772
B64841927B64841927
32575310100014
50836566B50836566B
499984

# 2  
Old 09-11-2014
A clunky way in a shell script would be to:-
  • Read the file and loop for each line
    • Calculate half the line length with ((half=${#line}/2))
    • Build up a string of question marks for the given length (each on represents a single character)
    • Use variable substitution to split the line
    • Compare the original with the half you have (twice)
    • If there is a match, take action one way, if not, the other way
  • Repeat for the remainder of the lines.
Does this seem a sensible logic to you? If so, we can help you code where you are stuck.

What do you think?


Robin
# 3  
Old 09-11-2014
Thanks for the reply Robin; you're on the same page as me.

Not being one to sit back and expect it to be written for me I've had a go with that pointer you gave me but I'm getting some strange results from it.
I thought I'd start simple and built up. I was half expecting (excuse the pun) that the below would output half the length of the string and store it in $half then using that combined with an awk sub string I would be able to just out put the first half of the string. The idea being I could store that in a variable do the same for the second half by getting the awk substr to start at $half for $half and I could compare the two. If they match then output if they don't then bin them.

Code:
while read line; do
((half=${#line}/2))
echo $line | awk '{print substr($0,1,$half)}'
done < $TEMP_1

That doesn't give me the output I was expecting. $TEMP_1 is a file name which contains the values one per line as per my previous post.
# 4  
Old 09-11-2014
While you are one the right track, it is best not to call an external program inside a loop because that will make it very slow. You could do it all in shell inside the loop, or use a single utility instead of a shell loop..

--
Another option would be to use a back reference in a regex :
Code:
grep '^\(.*\)\1$' file

The anchors ^ and $ make sure the two identical patterns glued together form the whole line...
This User Gave Thanks to Scrutinizer For This Post:
# 5  
Old 09-11-2014
Well good for you. We all learn better by trying, rather than being spoon-fed. With a nice pun like that, are you British?

You might need $1 in your awk rather than $0

It should still work though. This will give you the first half of each line, so you'd need to catch and compare that to the original, something like:-
Code:
while read line; do
   ((half=${#line}/2))
   halfline=`echo $line | awk '{print substr($0,1,$half)}'`
   if [ "${halfline}${halfline}" = "${line}" ]
   then
      echo "${line} is a duplicated entry"
   else
      echo "${line} is not repeated"
   fi
done < $TEMP_1 > logfile

Personally, I'd replace the awk with a substitution, so you are not calling awk over and again, something like this:-
Code:
while read line; do
   ((half=${#line}/2))
   h=1                                           # Set a counter
   mask=                                         # Null the variable
   until [ $h -gt $half ]                        # Loop until counter is right
   do
      mask="${mask}?"                            # Add a ? (single character wildcard)
      ((h=$h+1))
   done
   halfline="${line#${mask}}"                    # Split the line
   if [ "${halfline}${halfline}" = "${line}" ]   # Match twice the split line with the original
   then
      echo "${line} is a duplicated entry"
   else
      echo "${line} is not repeated"
   fi
done < $TEMP_1 > logfile


Does that suit? Does it work even.........Smilie?


Robin
This User Gave Thanks to rbatte1 For This Post:
# 6  
Old 09-11-2014
Spot on Scrutinizer that does exactily what I need it to do; both as a pipe on the end of my original grep or in the loop whilst reading each line.
Code:
grep 'somestuff' | sed 's/afew bits/g' | grep '^\(.*\)\1$' |

Code:
while read line; do
echo $line | grep '^\(.*\)\1$'
done < $TEMP_1

I'm not going to pretend I know what it's doing. Can you recommend some reading on this? is it know as back referencing within normal regex?

Robin - Thank you. Whilst Scrutinizer has answered it I'm still going to read and digest your reply so that I understand how what I was trying to achieve should work. All good learning.

Thank you both.
These 2 Users Gave Thanks to brighty For This Post:
# 7  
Old 09-11-2014
May I present my approach Smilie
Code:
#!/bin/bash

while read value; do 
 len=${#value}
 center=`expr $len / 2`
 firsthalf=${value:0:center}
 secondhalf=${value:center:len}
  if [ "$firsthalf" == "$secondhalf" ]; then
   echo "$value"
  fi
done <values >truedupes

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to get duplicate string

Hi All, I have a requirement where I have to get the duplicate string count and uniq error message. Below is my file: Rejected - Error on table TableA, column ColA. Error String 1. Rejected - Error on table TableA, column ColB. Error String 2. Rejected - Error on table TableA, column... (6 Replies)
Discussion started by: Deekhari
6 Replies

2. Shell Programming and Scripting

Highlighting duplicate string on a line

Hi all I have a grep written to pull out values; below (in the code snip-it) is an example of the output. What I'm struggling to do, and looking for assistance on, is identifying the lines that have duplicate strings. For example 74859915K74859915K in the below is 74859915K repeated twice but... (3 Replies)
Discussion started by: brighty
3 Replies

3. Red Hat

How to add a new string at the end of line by searching a string on the same line?

Hi, I have a file which is an extract of jil codes of all autosys jobs in our server. Sample jil code: ************************** permission:gx,wx date_conditions:yes days_of_week:all start_times:"05:00" condition: notrunning(appDev#box#ProductLoad)... (1 Reply)
Discussion started by: raghavendra
1 Replies

4. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

5. Shell Programming and Scripting

Remove not only the duplicate string but also the keyword of the string in Perl

Hi Perl users, I have another problem with text processing in Perl. I have a file below: Linux Unix Linux Windows SUN MACOS SUN SUN HP-AUX I want the result below: Unix Windows SUN MACOS HP-AUX so the duplicate string will be removed and also the keyword of the string on... (2 Replies)
Discussion started by: askari
2 Replies

6. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

7. Shell Programming and Scripting

Delete duplicate in certain number of string

Hi, do you have awk or sed sommand taht will delete duplicate lines like. sample: server1-log1-14 server1-log2-14 superserver-time-2 superserver-log-2 output: server-log1-14 superserver-time-2 thansk (2 Replies)
Discussion started by: kenshinhimura
2 Replies

8. Shell Programming and Scripting

filtering out duplicate substrings, regex string from a string

My input contains a single word lines. From each line data.txt prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 Desired output --> data_out.txt prjtestBlaBla prjthisBlaBla... (8 Replies)
Discussion started by: kchinnam
8 Replies

9. Shell Programming and Scripting

How to remove duplicate sentence/string in perl?

Hi, I have two strings like this in an array: For example: @a=("Brain aging is associated with a progressive imbalance between intracellular concentration of Reactive Oxygen Species","Brain aging is associated with a progressive imbalance between intracellular concentration of Reactive... (9 Replies)
Discussion started by: vanitham
9 Replies

10. UNIX for Dummies Questions & Answers

removing line and duplicate line

Hi, I have 3 lines in a text file that is similar to this (as a result of a diff between 2 files): 35,36d34 < DATA.EVENT.EVENT_ID.s = "3661208" < DATA.EVENT.EVENT_ID.s = "3661208" I am trying to get it down to just this: DATA.EVENT.EVENT_ID.s = "3661208" How can I do this?... (11 Replies)
Discussion started by: ocelot
11 Replies
Login or Register to Ask a Question