awk not working as expected with BIG files ...


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk not working as expected with BIG files ...
# 1  
Old 02-24-2005
Question awk not working as expected with BIG files ...

I am facing some strange problem.

I know, there is only one record in a file 'test.txt' which starts with 'X'

I ensure that with following command,
awk /^X/ test.txt | wc -l

This gives me output = '1'.

Now I take out this record out of the file, as follows :

awk /^X/ test.txt > XRecord.txt

Then I take out all records which do not start with 'X'.
Following is the command for the same,

awk '$0!~/^X/ test.txt > NotXRecord.txt

After doing this, Number of records with NotXRecord.txt is not 1 record less than number of records with 'test.txt'.

Now the same sequence above, works fine with small files, say uptil 100 thousands of records.

But it do not work for a file, with 4million rows.

Does any one have valuable hints to share?
# 2  
Old 02-24-2005
What is the problem are you getting here ??

awk may be too slow here.

Have you tried grep or sed ? for this problem ?


My understanding is , you may get good results with sed or grep
performance wise.

Have a try ....
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk matching script not working as expected

This is my ubuntu version: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial $ /bin/awk -V | head -n1 bash: /bin/awk: No such file or directory I have gotten a script that helps me to parse,... (14 Replies)
Discussion started by: delbroooks
14 Replies

2. Shell Programming and Scripting

awk gsub not working as expected

Hi Experts, Need your kind help with gsub awk. Below is my pattern:"exec=1_host_cnt=100_dup=4_NameTag=targetSrv_500.csv","'20171122112948"," 100"," 1"," 1"," 4","400","","", " aac sample exec ""hostname=XXXXX commandline='timeout 10 openssl speed -multi 2 ; exit 0'"" ","-1","-1","1","... (6 Replies)
Discussion started by: pradyumnajpn10
6 Replies

3. UNIX for Advanced & Expert Users

Manipulate files with find and fuser not working as expected on SunOs

Greetings, For housekeeping, I use the following command: find /some/path -type f -name "*log*" ! -exec fuser -s "{}" 2>/dev/null \; -exec ls -lh {} \; It finds all log files not currently in use by a process and manipulates them. This command always works on linux and redhat machines,... (2 Replies)
Discussion started by: dampio
2 Replies

4. Shell Programming and Scripting

awk command not working as expected

Following one line of awk code removes first 3 characters from each line but when I run the same code on another linux platform it doesn't work and only prints blank lines for each record. Can anyone please explain why this doesn't work? (31 Replies)
Discussion started by: later_troy
31 Replies

5. Shell Programming and Scripting

awk not working as expected in script

Dear all, I had script which used to work, but recently it is not working as expected. I have command line in my shell script to choose the following format from the output_elog and perform some task afterwards on As you see, I want all numbers in foramt following RED mark except for... (12 Replies)
Discussion started by: emily
12 Replies

6. Shell Programming and Scripting

bash variable (set via awk+sed) not working as expected

Hi! Been working on a script and I've been having a problem. I've finally narrowed it down to this variable I'm setting: servername=$(awk -v FS=\/ '{ print $7 } blah.txt | sed 's\/./-/g' | awk -v FS=\- '{print $1}')" This will essentially pare down a line like this: ... (7 Replies)
Discussion started by: creativedynamo
7 Replies

7. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

8. Shell Programming and Scripting

awk with really big files

Hi, I have a text file that is around 7Gb which is basically a matrix of numbers (FS is a space and RS is \n). I need the most efficient way of plucking out a number from a specified row and column in the file. For example, for the value at row 15983, col 26332, I'm currently I'm using: ... (1 Reply)
Discussion started by: Jonny2Vests
1 Replies

9. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

10. Shell Programming and Scripting

Var substitution in awk - not working as expected

countA=`awk '/X/''{print substr($0,38,1)}' fName | wc -l` countB=`wc -l fName | awk '{print int($1)}'` echo > temp ratio=`awk -va=$countA -vc=$countB '{printf "%.4f", a/c}' temp` After running script for above I am getting an error as : awk: 0602-533 Cannot find or open file -vc=25. The... (3 Replies)
Discussion started by: videsh77
3 Replies
Login or Register to Ask a Question