Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Deleting repeated lines by keeping only one.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting repeated lines by keeping only one.
# 1  
Old 09-25-2012
Deleting repeated lines by keeping only one.
Dear Buddies,
Need ur help once again.
I have a flat file with around 20 million lines (Huge file it is). However, many of the lines are of no use hence I want to remove it. To find and delete such lines we have certain codes written at the starting of each line. Basis that we can delete the lines. However its not as easy as it is looking.
Following is the example.

consumer_list.temp
Code:
Name: Anushree Aggarwal 
Tel1: 022-42158473
Tel2: 9965821475
Add1: Blah blah blah blah
Add2: Blah blah
Add3: Blah blah blah
Gndr: Female
Name: Rucha Chheda 
Tel1: 022-42158499
Tel2: 8325698501
Add1: Blah blah  
Add2: Blah blah blah
Add3: Blah blah blah blah
Gndr: Female
Name: Priyanka Rathi 
Tel1: 022-42158482
Tel2: 9658231492
Tel3: 021-23654125
Add1: Blah blah blah blah
Add2: Blah blah
Add3: Blah blah blah
Add4: Blah blah 
Gndr: Female

In above mentioned example multiple telephones and addresses are given. In output I want to take only one out of what ever number of telephones and addresses are provided.

Output should be as follows.

consumer_sorted_list.temp
Code:
Name: Anushree Aggarwal 
Tel1: 022-42158473
Add1: Blah blah blah blah
Gndr: Female
Name: Rucha Chheda 
Tel1: 022-42158499
Add1: Blah blah  
Gndr: Female
Name: Priyanka Rathi 
Tel1: 022-42158482
Add1: Blah blah blah blah
Gndr: Female

In short between "Name" and "Gndr" whatever information is provided it should appear only once.I am unable to think for a logic.
Need your help.
Thanks
Anu.
Last edited by Scrutinizer; 09-25-2012 at 06:59 AM.. Reason: code tags
# 2  
Old 09-25-2012
try this..

Code:
 awk '/^Name:/ || /^Tel1:/ || /^Add1:/ || /^Gndr:/' file

EDIT:

after looking at bmk's solution - same as egrep...
Code:
 grep -E "Name|Tel1|Add1|Gndr"  file

Last edited by pamu; 09-25-2012 at 07:30 AM..
# 3  
Old 09-25-2012
Try like...
Code:
grep -r "Name\|Tel1\|Add1\|Gndr" test1.txt

---------- Post updated at 05:12 AM ---------- Previous update was at 05:08 AM ----------

it's same as
Code:
egrep 'Name:|Tel1:|Add1:|Gndr:' test1.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines which has been repeated 4 times

Remove duplicate lines which has been repeated 4 times attached test.txt below command tried and not getting expect output. for i in `cat test.txt | uniq` do num=`cat test.txt | grep $i | wc -l` echo $i $num done test.txt ... (17 Replies)
Discussion started by: Kalia
17 Replies

2. UNIX for Beginners Questions & Answers

Export lines that have first entry repeated 5 times or above

Dears i want to extract lines only that have first entry repeated 3 times or above , ex data : -bash-3.00$ cat INTCONT-IS.CSV M205-00-106_AMDRN:1-0-6-22,12-662-4833,intContact,2016-11-15 02:32:16,50 M205-00-106_AMDRN:1-0-23-17,12-616-0462,intContact,2016-11-15 02:32:23,50... (5 Replies)
Discussion started by: is2_egypt
5 Replies

3. Shell Programming and Scripting

Repeated lines-case sensitive

Hi, users file contains below names i have a requirement to keep only one case sensitive user. For e.g if user name is "aaa" then only aaa should be there in the file and other matching users(AAA,aaA) should be deleted. Tried multiple options but no luck can you please help. aaa abc AAA... (2 Replies)
Discussion started by: Satyak
2 Replies

4. Shell Programming and Scripting

Compare two files with repeated lines

Hi all, I've been trying to write a script to compare two files. This is what I want: file 1: a 1 2 b 5 9 c 4 7 file 2: a a c a b Output: a 1 2 a 1 2 (2 Replies)
Discussion started by: ernesto561
2 Replies

5. Shell Programming and Scripting

Getting most repeated 3 lines

Hi all if i want to get the 3 lines of the most repeated lines here tony,1,x tony,1,x tony,2,x tony,2,x tony,3,x tony,4,x tony,5,x adam,1,y to get output tony,1,x tony,2,x tony3,x adam,1,y (3 Replies)
Discussion started by: teefa
3 Replies

6. Shell Programming and Scripting

Remove regularly repeated lines

How can i delete some regular repeated lines in a file? example: in_file EDGE 1 2 12 EDGE 2 3 23 EDGE 3 4 34 EDGE 5 6 56 EDGE 6 7 67 EDGE 7 8 78 EDGE 9 10 910 EDGE 10 11 1011 EDGE 11 12 1112 EDGE 13 14 1314 EDGE 14 15 1415 EDGE 15 16 1516 EDGE 17 18 1718 EDGE 18 19 1819 EDGE 19... (8 Replies)
Discussion started by: saeed.soltani
8 Replies

7. Shell Programming and Scripting

need to print lines between repeated pattern

Hi all, I have a file that looks like this: uid=bessemsj version: 1 dn: cn=Desk SpecialAdminDesk, ou=Desks, dc=DSS,c=nl,o=Vodafone dn: cn=DSS Advisors, ou=Groups, dc=DSS,c=nl,o=Vodafone dn: cn=DSS Dispatcher,ou=Groups,dc=DSS,c=nl,o=Vodafone dn: cn=Desk Retention Desk,ou=Desks,... (13 Replies)
Discussion started by: Eman_in_forum
13 Replies

8. Shell Programming and Scripting

Keeping only 6 logs and deleting the rest...

Hi Guys, I have folder where the logs are generated (which is quite a few). I want to keep the recent n logs and delete the rest of them.. I am able to sort them by using ls -t.. but how to take it forward.. Can you people help me.. Thanks for the help in advance, Regards,... (3 Replies)
Discussion started by: mac4rfree
3 Replies

9. Shell Programming and Scripting

Deleting repeated strings in column 2

Hi to all, I have a file where the subject could contain "Summarized Availability Report" or only "Summarized Report" If the subject is "Summarized Availability Report" I want to apply it Scrip1 and if the subject is "Summarized Report" I want to apply it Scrip2. 1-) I would like you... (5 Replies)
Discussion started by: cgkmal
5 Replies

10. UNIX for Dummies Questions & Answers

deleting a line but keeping the same file

Hi, I want to delete a line in a file that contains a string. I tried: grep -v "mystring" Myfile > Myfile But this makes the Myfile empty. I read that I need to do something like: grep -v "mystring" Myfile > Myfile.new rm Myfile mv Myfile.new Myfile Is there a way to avoid creating a... (2 Replies)
Discussion started by: laiko
2 Replies
Login or Register to Ask a Question