Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Eleminating Duplicate IPs from a text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Eleminating Duplicate IPs from a text file
# 1  
Old 01-24-2008
Eleminating Duplicate IPs from a text file
Hey Guys
I need to eleminate duplicate IP's from a text file using bash.Any suggestions.Appreciate your help guys.

--CoolKid
# 2  
Old 01-24-2008
Lightbulb
Try sort -u
# 3  
Old 01-24-2008
Tried it..It is not working
# 4  
Old 01-24-2008
Quote:
Originally Posted by coolkid
Hey Guys
I need to eleminate duplicate IP's from a text file using bash.Any suggestions.Appreciate your help guys.

--CoolKid
Hey,

uniq
This filter removes duplicate lines from a sorted file. It is often seen in a pipe coupled with sort.
cat list−1 list−2 list−3 | sort | uniq > final.list
# Concatenates the list files,
# sorts them,
# removes duplicate lines,
# and finally writes the result to an output file.

Try this logic, hope you'll get the desired one.

Thanks
Varun.Smilie
# 5  
Old 01-24-2008
Thanks for the help folks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sum duplicate values in text file through awk between dates

I need to sum values in text file in case duplicate row are present with same name and different value below is example of data in file i have and format i need. Data in text file 20170308 PM,U,2 PM,U,113 PM,I,123 DA,U,135 DA,I,113 DA,I,1 20170309 PM,U,2 PM,U,1 PM,I,123 PM,I,1... (3 Replies)
Discussion started by: Adfire
3 Replies

2. Shell Programming and Scripting

Removing duplicate sequences and modifying a text file

Hi. I've tried several different programs to try and solve this problem, but none of them seem to have done exactly what I want (and I need the file in a very specific format). I have a large file of DNA sequences in a multifasta file like this, with around 15 000 genes: ... (2 Replies)
Discussion started by: 4galaxy7
2 Replies

3. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

4. Shell Programming and Scripting

Duplicate each field in a text file

Hello all, I am searching for a solution to the following problem: Given input such as this: I would like to find a way to output this: Thanks in advance! (4 Replies)
Discussion started by: hydrabane
4 Replies

5. Shell Programming and Scripting

Duplicate rows in a text file

notes: i am using cygwin and notepad++ only for checking this and my OS is XP. #!/bin/bash typeset -i totalvalue=(wc -w /cygdrive/c/cygwinfiles/database.txt) typeset -i totallines=(wc -l /cygdrive/c/cygwinfiles/database.txt) typeset -i columnlines=`expr $totalvalue / $totallines` awk -F' ' -v... (5 Replies)
Discussion started by: whitecross
5 Replies

6. Linux

Regex for plucking out IPs and CIDRs from text file?

Hello to the unix.com community. I have a mess of text. What I would like to do I pluck out IP addresses and CIDR notations only. I thought I would try something like this /usr/bin/grep -o '\{1,3\}\.\{1,3\}\.\{1,3\}\.\{1,3\}\\' /Path/To/File But there are a few problems wit this. ... (3 Replies)
Discussion started by: TroubleNow345
3 Replies

7. Shell Programming and Scripting

Shellscript to sort duplicate files listed in a text file

I have many pdf's scattered across 4 machines. There is 1 location where I have other Pdf's maintained. But the issues it the 4 machines may have duplicate pdf's among themselves, but I want just 1 copy of each so that they can be transfered to that 1 location. What I have thought is: 1) I have... (11 Replies)
Discussion started by: deaddevil
11 Replies

8. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

9. Shell Programming and Scripting

duplicate line in a text file

i would like to scan file in for duplicate lines, and print the duplicates to another file, oh and it has to be case insensitive. example line1 line2 line2 line3 line4 line4 outputfile: line2 line4 any ideas (5 Replies)
Discussion started by: nixguy
5 Replies

10. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies
Login or Register to Ask a Question