Duplicate files and output list


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicate files and output list
# 1  
Old 10-23-2012
Duplicate files and output list

Gents,

I have a file like this.

Code:
1 1
1 2
2 3
2 4
2 5
3 6
3 7
4 8
5 9

I would like to get something like it
Code:
1 1 2
2 3 4 5
3 6 7

Thanks in advance for your support Smilie

Last edited by Scott; 10-23-2012 at 05:42 PM.. Reason: Code tags
# 2  
Old 10-23-2012
Code:
awk '{a[$1]=a[$1]?a[$1]" "$2:$0}END{for(i in a){print a[i]}}' file

This User Gave Thanks to pamu For This Post:
# 3  
Old 10-23-2012
also try:
Code:
awk '{a[$1]=a[$1] " " $2}END{for(i in a){print i a[i]}}' infile | sort

This User Gave Thanks to rdrtx1 For This Post:
# 4  
Old 10-23-2012
Thansk a lot its works perfect...

---------- Post updated at 01:12 PM ---------- Previous update was at 01:01 PM ----------

Please, how I can get only the duplicate files
Code:
1 1 2
2 3 4 5
3 6 7


Last edited by Scott; 10-23-2012 at 05:42 PM.. Reason: Code tags
# 5  
Old 10-23-2012
Quote:
Originally Posted by jiam912
Please, how I can get only the duplicate files
1 1 2
2 3 4 5
3 6 7
What do you mean by this...
# 6  
Old 10-23-2012
If you mean duplicate lines then use:
Code:
uniq -d

# 7  
Old 10-23-2012
I found the solution like this
Code:
awk '{a[$1]=a[$1] " " $2}END{for(i in a){print i a[i]}}' infile | sort | awk '{if(NF>2)print $0}'

Thanks

Last edited by Scott; 10-23-2012 at 05:42 PM.. Reason: Code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Using find to output list of files with specific strings

This is my problem, I am using the following code to extract the file names with specific strings 0.01: find ./ -name "*.txt" -exec grep -H '0.01' {} + It works wonders with a small sample. However, when I use it in a real scenario it produces an empty file -even though I am sure there are... (11 Replies)
Discussion started by: Xterra
11 Replies

2. Shell Programming and Scripting

How to list files names and sizes in a directory and output result to the file?

Hi , I'm trying to list the files and output is written to a file. But when I execute the command , the output file is being listed. How to exclude it ? /tmp file1.txt file2.txt ls -ltr |grep -v '-' | awk print {$9, $5} > output.txt cat output.txt file1.txt file2.txt output.txt (8 Replies)
Discussion started by: etldeveloper
8 Replies

3. Shell Programming and Scripting

List files output only for the last line

Hi, "ls -tl directory1" will list files to be sorted in mtime, but I don't want to see all the files in each directory, I want to only see output the last line (the oldest mtime) for each directory. $ ls -tl test1 -rw-r--r-- 1 hce hce 1714397 May 30 2013 b.txt -rw-r--r-- 1 hce hce 4678 May... (2 Replies)
Discussion started by: hce
2 Replies

4. Shell Programming and Scripting

List duplicate files based on Name and size

Hello, I have a huge directory (with millions of files) and need to find out duplicates based on BOTH file name and File size. I know fdupes but it calculates MD5 which is very time-consuming and especially it takes forever as I have millions of files. Can anyone please suggest a script or... (7 Replies)
Discussion started by: prvnrk
7 Replies

5. Shell Programming and Scripting

How to process select list of files and output to the same file?

Hi, I've a list of files ac_info.tps, subscription_array.tps, .......and many other files one of the file, bin_range_list.tps has the following content CREATE OR REPLACE TYPE "BIN_RANGE_LIST" AS TABLE OF BIN_RANGE_ELEM; / grant execute on... (4 Replies)
Discussion started by: jediwannabe
4 Replies

6. Shell Programming and Scripting

list files command output

Hi All, Below is the 2 different ouputs of the command "ls -lrt", my question is what exactly "total 0" & "total 8" means here ? $ ls -rtl total 0 -rw-r--r-- 1 oracle dba 0 Feb 10 20:16 c -rw-r--r-- 1 oracle dba 0 Feb 10 20:16 b -rw-r--r-- 1... (1 Reply)
Discussion started by: kannan84
1 Replies

7. Shell Programming and Scripting

I need a script to find socials in files and output a list of those files

I am trying to find socail security numbers in files in (and under) a specific directory and output a list of the files where they are found... the format would be with no dashes just 9 numeric characters in a row. I have tried this: find /DirToLookIn -exec grep '\{9\}' /dev/null {} \; >>... (1 Reply)
Discussion started by: NewSolarisAdmin
1 Replies

8. Shell Programming and Scripting

Find duplicate value comparing 2 files and create an output

I need a perl script which will create an output file after comparing two diff file in a directory path: /export/home/abc/file1 /export/home/abc/file2 File Format: <IP>TAB<DeviceName><TAB>DESCRIPTIONS file1: 10.1.2.1.3<tab>abc123def<tab>xyz.mm1.ppp.... (2 Replies)
Discussion started by: ricky007
2 Replies

9. Shell Programming and Scripting

Dynamically redirect output to duplicate files ???

Hi There are many posts in this forum regarding reditecting output, but mine is a different problem, please have a look. My shell script is redirecting output to a log file dynamically. That is it is using - exec > log1.txt 2>&1 Hence all the traces are appearing in the log1.txt. I want... (3 Replies)
Discussion started by: nsinha
3 Replies

10. Shell Programming and Scripting

Removing duplicate files from list with different path

I have a list which contains all the jar files shipped with the product I am involved with. Now, in this list I have some jar files which appear again and again. But these jar files are present in different folders. My input file looks like this /path/1/to a.jar /path/2/to a.jar /path/1/to... (10 Replies)
Discussion started by: vino
10 Replies
Login or Register to Ask a Question

Featured Tech Videos