Removing duplicate files from list with different path

vino · #1 (**permalink**) 05-12-2005

I have a list which contains all the jar files shipped with the product I am involved with. Now, in this list I have some jar files which appear again and again. But these jar files are present in different folders.
My input file looks like this

Code:

/path/1/to a.jar
/path/2/to a.jar
/path/1/to b.jar
/path/1/to c.jar
/path/1/to d.jar
/path/2/to c.jar

Now I need to remove the duplicate entries i.e. remove the extra a.jar, c.jar and others like wise.

Final list would be like this:

Code:

/path/1/to a.jar
/path/1/to b.jar
/path/2/to c.jar
/path/1/to d.jar

This is the script I have so far..

Code:

#! /bin/sh
cp jar.txt jarnew.txt
for file in $(cat jar.txt)
do
        FILE1=`basename $file`
        for dup in $(cat jar.txt)
        do
        FILE2=`basename $dup`
        if [ "$file" != "$dup" -a "$FILE1" == "$FILE2" ] ; then
        sed -e '/($dup)/ d' <jarnew.txt >jarnew.txt.tmp
        echo "$FILE1 $FILE2"
        mv jarnew.txt.tmp jarnew.txt
        fi;
        done
done

But with this, the main functionality of the script is not working. i.e. the if condition is not working as required. Could be the sed problem or the logic.

I am getting the jarnew.txt as good as the jar.txt

Any pointers on how to proceed ?

Vino

muthukumar · #2 (**permalink**) 05-12-2005

Try with this script as,

#!/bin/sh
> final.jar
while read line; do

FILE=`basename $line`;
DIR=`echo $line | awk '{ print $(NF-1) }'`;
if [[ $FILE == "c.jar" && $DIR == "2" ]]
then
echo $line >> final.jar
elif [[ $DIR == "1" && $FILE != "c.jar" ]]
echo $line >> final.jar
fi

done < jar.txt

You will get result. Check it.

Just Ice · #3 (**permalink**) 05-12-2005

on the assumption that you only have filenames in the list ...

Code:

sort -t"/" -u +3 jar.txt > jarnew.txt

vino · #4 (**permalink**) 05-12-2005

JustIce,

I dont think sort is a possible soution. The path length varies i.e. the directory structure is different for files. Some of them have a depth of 3.. others a depth of more than 3.

Vino

Just Ice · #5 (**permalink**) 05-12-2005

Code:

#! /bin/ksh

jarlist=`awk -F"/" '{print $NF}' jar.txt | sort -u`
for jarfile in $jarlist
do
    grep $jarfile jar.txt | uniq
done > jarnew.txt

exit 0

vino · #6 (**permalink**) 05-12-2005

JustIce,

Just tried out your solution. It sorts the entries. And I can still see the duplicate entries.

Vino

vino · #7 (**permalink**) 05-12-2005

Muthu,

I dont intend to specify any particular jar file like in the way you have mentioned in
f [[ $FILE == "c.jar" && $DIR == "2" ]]

Rather, I would have it generalized.

How do you go about that ?

Vino

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
List files with full path	mr_bold	Shell Programming and Scripting	3	10-07-2008 12:19 PM
List Duplicate	vakharia Mahesh	Shell Programming and Scripting	22	09-24-2007 08:45 AM
list all files with full path of the file	Sowser	UNIX for Advanced & Expert Users	4	02-13-2007 05:46 PM
removing duplicate lines from a file	ocelot	UNIX for Dummies Questions & Answers	4	01-25-2007 11:02 AM
Removing duplicate lines ignore case	hellsd	UNIX for Dummies Questions & Answers	17	12-02-2004 10:47 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: