10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
values=(1 2 3 5 4 2 3 1 6 8 3 5 )
#i need the output like this by removing the duplicates
#i dont need sorting in my program
#plz explain me as simple using for loop
#os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
2. Shell Programming and Scripting
Issue: Within awk script, remove the duplicate occurrences that are space (1 single space character) separated
Description: I am processing 2 files using awk and during processing, I am building an array and there are duplicates on this; how can I delete the duplicates... (3 Replies)
Discussion started by: chill3chee
3. Shell Programming and Scripting
Hi I have a below file structure.
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
4. Shell Programming and Scripting
I have a file with the following format:
fields seperated by "|"
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
5. Shell Programming and Scripting
I need some help to remove duplicates from a file before merging.
I have got 2 files:
file1 has data in format
the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file
file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
6. Shell Programming and Scripting
I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file.
Can both the awk... (4 Replies)
Discussion started by: siramitsharma
7. Shell Programming and Scripting
I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different.
I need to go from this:
... (2 Replies)
Discussion started by: locoroco
8. Shell Programming and Scripting
I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another.
Thanks (6 Replies)
Discussion started by: dvah
9. Shell Programming and Scripting
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
10. Shell Programming and Scripting
I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it.
Currently, I am using:
sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins.
Is there any other faster way... (15 Replies)
Discussion started by: radhika