Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Finding duplicates in a file excluding specific pattern

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding duplicates in a file excluding specific pattern
# 1  
Old 06-11-2013
Finding duplicates in a file excluding specific pattern
I have unix file like below
>newuser
newuser
<hello
hello
newone

I want to find the unique values in the file(excluding <,>),so that the out put should be
>newuser
<hello
newone

can any body tell me what is command to get this new file.
# 2  
Old 06-11-2013
Code:
sed 's/^[<>]//' file | uniq

# 3  
Old 06-11-2013
This removes the <>, since I am not sure how to handle what to print
>newuser or newuser
Code:
awk '{gsub(/[<>]/,"")} !a[$0]++' file
newuser
hello
newone

# 4  
Old 06-11-2013
Thanks for the reply.
But it is priniting the output as
newuser
hello
newone
But I want the output with the < or > symbols like
>newuser
<hello
newone (while searching only we should neglect the < or >)

---------- Post updated at 02:13 AM ---------- Previous update was at 02:07 AM ----------

I want to print the output the with < or >.Any command is there to do that
# 5  
Old 06-11-2013
Code:
 awk '{if(/^[<>]/){a=substr($0,2)} else {a=$1}; if (!b[a]++){print}}' filename

This User Gave Thanks to pravin27 For This Post:
shiva2985
# 6  
Old 06-11-2013
Thanks Pravin,It worked
# 7  
Old 06-11-2013
Can be shorten some
Code:
awk '{if(/^[<>]/){a=substr($0,2)} else {a=$1}} !b[a]++' filename

even some more
Code:
awk '{a=(/^[<>]/)?substr($0,2):$1} !b[a]++' filename

Last edited by Jotne; 06-11-2013 at 05:15 AM.. Reason: fixed typo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract specific line in an html file starting and ending with specific pattern to a text file

Hi This is my first post and I'm just a beginner. So please be nice to me. I have a couple of html files where a pattern beginning with "http://www.site.com" and ending with "/resource.dat" is present on every 241st line. How do I extract this to a new text file? I have tried sed -n 241,241p... (13 Replies)
Discussion started by: dejavo
13 Replies

2. Shell Programming and Scripting

Sed: printing lines AFTER pattern matching EXCLUDING the line containing the pattern

'Hi I'm using the following code to extract the lines(and redirect them to a txt file) after the pattern match. But the output is inclusive of the line with pattern match. Which option is to be used to exclude the line containing the pattern? sed -n '/Conn.*User/,$p' > consumers.txt (11 Replies)
Discussion started by: essem
11 Replies

3. UNIX for Dummies Questions & Answers

Grep from pattern file without removing duplicates?

I have been using grep to output whole lines using a pattern file with identifiers (fileA): fig|562.2322.peg.1 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.7 From fileB with corresponding identifiers in the second column: NODE_0 fig|562.2322.peg.1 peg ... (2 Replies)
Discussion started by: Mauve
2 Replies

4. Shell Programming and Scripting

Search for duplicates and delete but remain the first one based on a specific pattern

Hi all, I have been trying to delete duplicates based on a certain pattern but failed to make it works. There are more than 1 pattern which are duplicated but i just want to remove 1 pattern only and remain the rest. I cannot use awk '!x++' inputfile.txt or sed '/pattern/d' or use uniq and sort... (7 Replies)
Discussion started by: redse171
7 Replies

5. Shell Programming and Scripting

Finding the pattern and replacing the pattern inside the file

i have little challenge, help me out.i have a file where i have a value declared and and i have to replace the value when called. for example i have the value for abc and ccc. now i have to substitute the value of value abc and ccc in the place of them. Input File: go to &abc=ddd; if... (16 Replies)
Discussion started by: saaisiva
16 Replies

6. UNIX for Dummies Questions & Answers

Finding new file, but excluding directory..

hi, I need to find files that have been created less than 3 days ago. However, I need to only search specific directories. I've searched about the net and found some useful commands such as : find . -type d -name 'dir_to_exclude' -prune -o -print -mtime -3 however I cannot get it... (2 Replies)
Discussion started by: horhif
2 Replies

7. Shell Programming and Scripting

Finding 4 current files having specific File Name pattern

Hi All, I am trying to find 4 latest files inside one folder having following File Name pattern and store them into 4 different variables and then use for processing in my shell script. File name is fixed length. 1) Each file starts with = ABCJmdmfbsjop letters + 7 Digit Number... (6 Replies)
Discussion started by: lancesunny
6 Replies

8. UNIX for Dummies Questions & Answers

removing duplicates of a pattern from a file

hey all, I need some help. I have a text file with names in it. My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns.. for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies

9. Shell Programming and Scripting

Remove duplicates from File from specific location

How can i remove the duplicate lines from a file, for example sample123456Sample testing123456testing XXXXX131323XXXXX YYYYY423432YYYYY fsdfdsf123456gsdfdsd all the duplicates from column 6-12 , must be deleted. I want to consider the first row, if same comes in the given range i want to... (1 Reply)
Discussion started by: gopikgunda
1 Replies

10. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies
Login or Register to Ask a Question