Removing duplicate files from list with different path - Shell Programming and Scripting

vino · #1 05-12-2005

I have a list which contains all the jar files shipped with the product I am involved with. Now, in this list I have some jar files which appear again and again. But these jar files are present in different folders.
My input file looks like this

Code:

/path/1/to a.jar
/path/2/to a.jar
/path/1/to b.jar
/path/1/to c.jar
/path/1/to d.jar
/path/2/to c.jar

Now I need to remove the duplicate entries i.e. remove the extra a.jar, c.jar and others like wise.

Final list would be like this:

Code:

/path/1/to a.jar
/path/1/to b.jar
/path/2/to c.jar
/path/1/to d.jar

This is the script I have so far..

Code:

#! /bin/sh
cp jar.txt jarnew.txt
for file in $(cat jar.txt)
do
        FILE1=`basename $file`
        for dup in $(cat jar.txt)
        do
        FILE2=`basename $dup`
        if [ "$file" != "$dup" -a "$FILE1" == "$FILE2" ] ; then
        sed -e '/($dup)/ d' <jarnew.txt >jarnew.txt.tmp
        echo "$FILE1 $FILE2"
        mv jarnew.txt.tmp jarnew.txt
        fi;
        done
done

But with this, the main functionality of the script is not working. i.e. the if condition is not working as required. Could be the sed problem or the logic.

I am getting the jarnew.txt as good as the jar.txt

Any pointers on how to proceed ?

Vino

muthukumar · #2 05-12-2005

Try with this script as,

#!/bin/sh
> final.jar
while read line; do

FILE=`basename $line`;
DIR=`echo $line | awk '{ print $(NF-1) }'`;
if [[ $FILE == "c.jar" && $DIR == "2" ]]
then
echo $line >> final.jar
elif [[ $DIR == "1" && $FILE != "c.jar" ]]
echo $line >> final.jar
fi

done < jar.txt

You will get result. Check it.

Just Ice · #3 05-12-2005

on the assumption that you only have filenames in the list ...

Code:

sort -t"/" -u +3 jar.txt > jarnew.txt

vino · #4 05-12-2005

Muthu,

I dont intend to specify any particular jar file like in the way you have mentioned in
f [[ $FILE == "c.jar" && $DIR == "2" ]]

Rather, I would have it generalized.

How do you go about that ?

Vino

muthukumar · #5 05-12-2005

Generally scripts are written based on pattern change. In your requirement on input, only c.jar is taken from path/2/. I have simulated your input to required output.

And, your input and output is not being generallized so that script is given with using speicific filenames.

vino · #6 05-12-2005

There are so many jar files. If I can collect these jar files manually, then I might as well do away with the script.

I need to get the jar files from the list, dynamically.

Vino

vino · #7 05-12-2005

JustIce,

I dont think sort is a possible soution. The path length varies i.e. the directory structure is different for files. Some of them have a depth of 3.. others a depth of more than 3.

Vino

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
List files with full path	mr_bold	Shell Programming and Scripting	3	10-07-2008 09:19 AM
List Duplicate	vakharia Mahesh	Shell Programming and Scripting	22	09-24-2007 05:45 AM
list all files with full path of the file	Sowser	UNIX for Advanced & Expert Users	4	02-13-2007 02:46 PM
removing duplicate lines from a file	ocelot	UNIX for Dummies Questions & Answers	4	01-25-2007 08:02 AM
Removing duplicate lines ignore case	hellsd	UNIX for Dummies Questions & Answers	17	12-02-2004 07:47 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode