Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

AWK - Pattern Matching & Replacing - Performance

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AWK - Pattern Matching & Replacing - Performance
# 1  
Old 09-03-2010
AWK - Pattern Matching & Replacing - Performance
Experts,

I am a beginner to Unix Shell Scripting
We have source as a flat file which contains CTRL+F character as the delimiter. We need to count the number of records in the file (CTRL+F) to perform file validation
Following command being used:
Code:
awk '{cnt+=gsub(//,"&")}END {print cnt}' Sri.dat


This takes only fewer seconds to show the count if the file contains less than 25K records. We have files with count exceeding 1000K and this command takes more than one hour to complete

Could you please suggest any other command similar to the above requirement and provide the count within fewer seconds / minutes. The commands can be either in Unix or Perl

Please help

Thanks & regards
Sri
Last edited by radoulov; 09-03-2010 at 09:49 AM.. Reason: Code tags, please!
# 2  
Old 09-03-2010
Try this:

Code:
perl -06ne'END { print $. }' infile

This User Gave Thanks to radoulov For This Post:
srivijay81
# 3  
Old 09-03-2010
Code:
tr -dc '\006' < Sri.dat |wc -c

This User Gave Thanks to binlib For This Post:
srivijay81
# 4  
Old 09-06-2010
AWK - Pattern Matching & Replacing - Performance Reply to Thread
Thank you so much, it worked as expected
Need another detail. In our source file, we used to receive '~^' as delimiter, through the script we will replace this with 'ACK' ('\006'). Could you please provide the how the above command can be used for the delimiter '~^'. Need to know the hexadecimal value for this delimiter '~^'

Thanks & regards
Srik
# 5  
Old 09-06-2010
Quote:
Originally Posted by srivijay81
Thank you so much, it worked as expected
Need another detail. In our source file, we used to receive '~^' as delimiter, through the script we will replace this with 'ACK' ('\006'). Could you please provide the how the above command can be used for the delimiter '~^'. Need to know the hexadecimal value for this delimiter '~^'

Thanks & regards
Srik
Something like this?
Code:
awk '{gsub("~^","\006")}1' file > newfile

This User Gave Thanks to Franklin52 For This Post:
srivijay81
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Performance assessment of using single or combined pattern matching

Hi, I want to know which pattern matching technique will be giving better performance and quick result. I will be having the patterns in a file and want to read that patterns and search through a whole file of say 70 MB size. whether if i initially create a pattern matching string while... (7 Replies)
Discussion started by: ananan
7 Replies

2. Shell Programming and Scripting

How to print & and \n while replacing with sed/awk?

string="din&esh\nisgood" File.txt: the name is sed "s#\#${string}#g" File.txt Output am getting: the name is dinesh is good Expected output: the name is din&esh\nisgood The input string is dynamic it will be keep on changing am able to handle & by placing \& in the string.. (5 Replies)
Discussion started by: dineshaila
5 Replies

3. Shell Programming and Scripting

awk Array matching and replacing from master file.

I have an awk related question that I was hoping you all could help with. I am given 2 input files named OLDFILE and NEWFILE, and a Master file named MASTERFILE. They can be seen below. OLDFILE: a a a a a f g 4 5 7 8 1 2 3 (1 Reply)
Discussion started by: tiktak292
1 Replies

4. Shell Programming and Scripting

Replacing lines matching a multi-line pattern (sed/perl/awk)

Dear Unix Forums, I am hoping you can help me with a pattern matching problem. What am I trying to do? I want to replace multiple lines of a text file (that match a multi-line pattern) with a single line of text. These patterns can span several lines and do not always have the same number of... (10 Replies)
Discussion started by: thefang
10 Replies

5. Shell Programming and Scripting

Pattern Matching & replacing of content in file1 with file2

I have file 1 & file 2 with content mentioned below. I want to get the output as shown in file3. Requirement: check the content of column 1 & column 2, if value of column 1 in file1 matches with first column of file2 then remaining columns(2&3) of file2 should get replaced, also if value of... (4 Replies)
Discussion started by: siramitsharma
4 Replies

6. Shell Programming and Scripting

Pattern matching & storing result in variable

Hi! i'm trying to parse textfiles against a pattern and storing the result in a variable. The strings i want to get are embraced by and can occur several times in one line, so e.g. some text anything else endwhat i have so far: #!/bin/bash for f in $* do exec 3<&0 exec 0<$f ... (2 Replies)
Discussion started by: thoni
2 Replies

7. Shell Programming and Scripting

matching a pattern in a file & prefixing a word

Hi, In my shell script i have to match a patten in a file , if found i have to prefix the entair line by a "word" eg. pattern = "aaa" prefix= #123 file: bbbb xxx zzzz aaaa qqqq kkkk outPut file: bbbb xxx ... (5 Replies)
Discussion started by: shivarajM
5 Replies

8. UNIX for Dummies Questions & Answers

Pattern matching New to Sed & Awk

Hello, Despite reading the Pattern Matching chapter in the O'Reilly Sed & Awk book several times and looking at numerous examples, I cannot seem to get any kind of conditional script to work in my awk scripts! I am able to do the basic awk and grep script to capture the data but when I do with... (0 Replies)
Discussion started by: pg55
0 Replies

9. Shell Programming and Scripting

Replacing all except for the matching pattern

Hi all I need to replace all characters in a file except for the matching pattern eg. I need to replace all character with '*' except for the pattern "abc" Input "sdfhgsdf abc ##%$#abcsdfh sdfjkfff" Output "******abc******abc*************" Request for single liner solution (1 Reply)
Discussion started by: raghav288
1 Replies

10. Shell Programming and Scripting

help with finding & replacing pattern in a file

Hi everyone. Could u be so kind and help me with on "simple" shell script? 1. i need to search a file line by line for a pattern. example of a lines in that file 2947 domain = feD,id = 00 0A 02 48 17 1E 1D 39 DE 00 0E 00,Name Values:snNo = f10 Add AttFlag = 0 2. i need to find... (0 Replies)
Discussion started by: dusoo
0 Replies
Login or Register to Ask a Question