10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi I have a below file structure.
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
2. Shell Programming and Scripting
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
3. Shell Programming and Scripting
I have a file with the following format:
fields seperated by "|"
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
4. Shell Programming and Scripting
I have searched many threads for possible close solution. But I was unable to get simlar scenario.
I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate).
12 NIL ABD LON
11 NIL ABC... (6 Replies)
Discussion started by: sybadm
5. Shell Programming and Scripting
I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different.
I need to go from this:
... (2 Replies)
Discussion started by: locoroco
6. Shell Programming and Scripting
Hi guys! I'm trying to eliminate some duplicates from a file but I'm like this :wall: !!!
My file looks like this:
What I need is to eliminate all the duplicates considering the first column (in this... (6 Replies)
Discussion started by: gabrysfe
7. Shell Programming and Scripting
I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another.
Thanks (6 Replies)
Discussion started by: dvah
8. Shell Programming and Scripting
I need a script that removes the duplicate records and write it to a new file
for example I have a file named test.txt and it looks like
I want to pick only $1 and compare with the next record and the output should be
abcd.23... (6 Replies)
Discussion started by: antointoronto
9. Shell Programming and Scripting
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
10. Shell Programming and Scripting
I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it.
Currently, I am using:
sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins.
Is there any other faster way... (15 Replies)
Discussion started by: radhika