Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Filtering multiple entries in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Filtering multiple entries in a file
# 1  
Old 08-19-2006
Filtering multiple entries in a file
Hi,
I have a requirement to filter multiple entries in a text file.
Entries are a pair of 2 lines .Like

line1
line2
.......
........
Here line1, line2 consist of one pair. line1 and line2 contain different data.
There can be multiple entries for the same pair. And there can be 'n' such pairs having multiple entries.
Is there a method available in perl/shell scripting to filter the multiple entries.
Could any one guide me to achieve this? Any help is appreciated.

Thanks in advance.
--------------------
Suman
# 2  
Old 08-19-2006
You want to remove duplicate lines?
Code:
awk '!arr[$0]++' duplicatedfile > noduplicates_in_this_file

# 3  
Old 08-22-2006
Excellent!!!.


Could you explain me the logic in the command?

Thanks & Regards,
-------------------
Suman
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Avoid overwriting backup file when multiple entries need to replace in one file input from another

Hello, I have been working on script in which search and replace the multiple pattern. 1. update_params.sh read the multiple pattern from input file ParamMapping.txt(old_entry|New_entry) and passing this values one by one to change_text.sh 2. change_text.sh read... (0 Replies)
Discussion started by: ketanraut
0 Replies

2. Shell Programming and Scripting

Awk/sed : help on:Filtering multiple lines to one:

Experts Good day, I want to filter multiple lines of same error of same day , to only 1 error of each day, the first line from the log. Here is the file: May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out) May 26 11:29:19 cmihpx02 vmunix: NFS... (4 Replies)
Discussion started by: rveri
4 Replies

3. Shell Programming and Scripting

Filtering first file columns based on second file column

Hi friends, I have one file like below. (.csv type) SNo,data1,data2 1,1,2 2,2,3 3,3,2 and another file like below. Exclude data1 where Exclude should be treated as column name in file2. I want the output shown below. SNo,data2 1,2 2,3 3,2 Where my data1 column got removed from... (2 Replies)
Discussion started by: ks_reddy
2 Replies

4. UNIX for Dummies Questions & Answers

Filtering records from 1 file based on some manipulation doen on second file

Hi, I am looking for an awk script which should help me to meet the following requirement: File1 has records in following format INF: FAILEd RECORD AB1234 INF: FAILEd RECORD PQ1145 INF: FAILEd RECORD AB3215 INF: FAILEd RECORD AB6114 ............................ (2 Replies)
Discussion started by: mintu41
2 Replies

5. Shell Programming and Scripting

Filtering a file for unique entries

Hi All, I am trying to filter out a txt file containing 2 columns. Which has got website name and response time as below. Website_name response_time yahoo 22 google 21 yahoo 34 rediff 45 google 43 rediff 31 I want filter the above file with unique website names which has got highest... (5 Replies)
Discussion started by: anilreddy103
5 Replies

6. Shell Programming and Scripting

Filtering Multiple variables from a single column

Hi, I am currently filtering a file, "BUILD_TIMES", that has multiple column of information in it. An example of the data is as follows; Fri Nov 5 15:31:33 2010 00:28:17 R7_BCGNOFJ_70.68 Fri Nov 5 20:57:41 2010 00:07:21 R7_ADJCEL_80.6 Wed Nov 10 17:33:21 2010 00:01:13 R7_BCTTEST3_80.1X... (7 Replies)
Discussion started by: crunchie
7 Replies

7. Shell Programming and Scripting

Filtering issues with multiple columns in a single file

Hi, I am new to unix and would greatly appreciate some help. I have a file containing multiple colums containing different sets of data e.g. File 1: John Ireland 27_December_69 Mary England 13_March_55 Mike France 02_June_80 I am currently using the awk... (10 Replies)
Discussion started by: crunchie
10 Replies

8. UNIX for Dummies Questions & Answers

Filtering a file

I have a list of directories looking something like; /usr/local/1/in /usr/local/1/out /usr/local/1/archive /usr/local/2/in /usr/local/2/out /usr/local/2/archive /usr/local/3/in /usr/local/3/out /usr/local/3/archive Is there a way I can filter the out and archive directories so I... (5 Replies)
Discussion started by: JayC89
5 Replies

9. Shell Programming and Scripting

Filtering multiple files with variables

Hi, I spend few hours already searching this forum, but cannot find the solution matching exactly my case. I have multiple log files, I need to filter them so I get info about certain event. So we have files: LOGA.txt LOGB.txt LOGC.txt LOGD.txt LOGE.txt 1. I need to grep lines in... (10 Replies)
Discussion started by: Vitoriung
10 Replies

10. UNIX for Dummies Questions & Answers

problem in filtering the file

-------------------------------------------------------------------------------- Hi, Plz help me out with this. I have some requirement like this..... I have a file like this... * CS sent email (11.20) CALYPSO 1031276 9076673 CDSHY FAILED Nov 19 2007 7:28AM OASYS: Unable to find CUSTOMER... (0 Replies)
Discussion started by: adityam
0 Replies
Login or Register to Ask a Question