Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk not working as expected with BIG files ...
Top Forums Shell Programming and Scripting awk not working as expected with BIG files ... Post 63318 by videsh77 on Thursday 24th of February 2005 05:53:08 AM
Old 02-24-2005
Question awk not working as expected with BIG files ...
I am facing some strange problem.

I know, there is only one record in a file 'test.txt' which starts with 'X'

I ensure that with following command,
awk /^X/ test.txt | wc -l

This gives me output = '1'.

Now I take out this record out of the file, as follows :

awk /^X/ test.txt > XRecord.txt

Then I take out all records which do not start with 'X'.
Following is the command for the same,

awk '$0!~/^X/ test.txt > NotXRecord.txt

After doing this, Number of records with NotXRecord.txt is not 1 record less than number of records with 'test.txt'.

Now the same sequence above, works fine with small files, say uptil 100 thousands of records.

But it do not work for a file, with 4million rows.

Does any one have valuable hints to share?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Var substitution in awk - not working as expected

countA=`awk '/X/''{print substr($0,38,1)}' fName | wc -l` countB=`wc -l fName | awk '{print int($1)}'` echo > temp ratio=`awk -va=$countA -vc=$countB '{printf "%.4f", a/c}' temp` After running script for above I am getting an error as : awk: 0602-533 Cannot find or open file -vc=25. The... (3 Replies)
Discussion started by: videsh77
3 Replies

2. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

3. Shell Programming and Scripting

awk with really big files

Hi, I have a text file that is around 7Gb which is basically a matrix of numbers (FS is a space and RS is \n). I need the most efficient way of plucking out a number from a specified row and column in the file. For example, for the value at row 15983, col 26332, I'm currently I'm using: ... (1 Reply)
Discussion started by: Jonny2Vests
1 Replies

4. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

5. Shell Programming and Scripting

bash variable (set via awk+sed) not working as expected

Hi! Been working on a script and I've been having a problem. I've finally narrowed it down to this variable I'm setting: servername=$(awk -v FS=\/ '{ print $7 } blah.txt | sed 's\/./-/g' | awk -v FS=\- '{print $1}')" This will essentially pare down a line like this: ... (7 Replies)
Discussion started by: creativedynamo
7 Replies

6. Shell Programming and Scripting

awk not working as expected in script

Dear all, I had script which used to work, but recently it is not working as expected. I have command line in my shell script to choose the following format from the output_elog and perform some task afterwards on As you see, I want all numbers in foramt following RED mark except for... (12 Replies)
Discussion started by: emily
12 Replies

7. Shell Programming and Scripting

awk command not working as expected

Following one line of awk code removes first 3 characters from each line but when I run the same code on another linux platform it doesn't work and only prints blank lines for each record. Can anyone please explain why this doesn't work? (31 Replies)
Discussion started by: later_troy
31 Replies

8. UNIX for Advanced & Expert Users

Manipulate files with find and fuser not working as expected on SunOs

Greetings, For housekeeping, I use the following command: find /some/path -type f -name "*log*" ! -exec fuser -s "{}" 2>/dev/null \; -exec ls -lh {} \; It finds all log files not currently in use by a process and manipulates them. This command always works on linux and redhat machines,... (2 Replies)
Discussion started by: dampio
2 Replies

9. Shell Programming and Scripting

awk gsub not working as expected

Hi Experts, Need your kind help with gsub awk. Below is my pattern:"exec=1_host_cnt=100_dup=4_NameTag=targetSrv_500.csv","'20171122112948"," 100"," 1"," 1"," 4","400","","", " aac sample exec ""hostname=XXXXX commandline='timeout 10 openssl speed -multi 2 ; exit 0'"" ","-1","-1","1","... (6 Replies)
Discussion started by: pradyumnajpn10
6 Replies

10. Shell Programming and Scripting

awk matching script not working as expected

This is my ubuntu version: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial $ /bin/awk -V | head -n1 bash: /bin/awk: No such file or directory I have gotten a script that helps me to parse,... (14 Replies)
Discussion started by: delbroooks
14 Replies

LEARN ABOUT DEBIAN

ldns-gen-zone

ldns-gen-zone(1)					      General Commands Manual						  ldns-gen-zone(1)

NAME

       ldns-gen-zone - read a zonefile and print it while adding DS records and extra RR's

SYNOPSIS

       ldns-gen-zone ZONEFILE

DESCRIPTION

       ldns-gen-zone reads a DNS zone file and prints it.

       It is build for speed, not for a nice formatting. The output has one resource record per line and no pretty-printing makeup.

       DNSSEC  data  (NSEC, NSEC3, RRSIG or DNSKEY) is not stripped. You may want to use ldns-read-zone for that. Existing DS records are also not
       stripped.

       The idea is to use this tool for quickly generating a representative artificial zonefile from a real zonefile, to use it for  testing  pur-
       poses.

OPTIONS

       -a NUM Adds  NUM  extra	artificial  NS	RRSets to the output.  The RRSets owner names start with 'xn--' in an attempt to ensure uniqueness
	      (nl.-zone does not support IDN's - and this tool was written with that knowledge in mind).

	      An artificial NS RRSet has two NS records; ns1.example.com and ns2.example.com.

       -p NUM Add NUM% of DS RRSets to the NS RRSets (anywhere between 1-4 DS records per RRSet).

       -o ORIGIN
	      Sets an $ORIGIN, which can be handy if the one in the zonefile is set to '@' for example. If there is an $ORIGIN	in  the  zonefile,
	      this option will silently be ignored.

       -s     This is the recommended way of processing large zones that are already sorted and canonicalized (ie lowercase). It skips the sorting
	      and canonicalization step that is required for properly grouping RRSets together (before adding any DS  records  to  them.  Skipping
	      this step will speed things up.

	      It is not recommended to use this option if you want to add DS records to unsorted, non-canonicalized zones.

       -h     Show usage and exit.

       -v     Show version and exit.

EXAMPLES

       ldns-gen-zone -a 100000 -p 10 -s ./zonefile.txt
	      Read  a  zonefile,  add  100.000 artificial NS RRSets and 10% of DS records, print it to standard output. Don't sort (will only work
	      well if the input zonefile is already sorted and canonicalized).

       ldns-gen-zone -p 10 -s -o nl zonefile.txt | named-compilezone -s relative -i none -o zonefile_10.txt nl /dev/stdin
	      This creates a nicely formatted zone file with the help of named-compilezone.  It adds 10% DS records to the .nl zone, reformats	it
	      and saves it as zonefile_10.txt.

AUTHOR

       Initially written by Marco Davids, several modifications added by Miek Gieben, both from SIDN.

REPORTING BUGS

       Report bugs to <ldns-team@nlnetlabs.nl>.

BUGS

       Only undiscovered ones.

CAVEATS

       May require a machine with a considerable amount of memory for large zone files.

       Fake  DS records hashes are generated as digest type SHA-256 (RFC4509). Be aware not to change the DIGESTTYPE #define in the source code in
       anything else but 2 if you want to keep things realistic.

       Despite a number of efforts, this program is still not the fastest in the world.

COPYRIGHT

       Copyright (C) 2010 SIDN. This is free software. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

								   10 June 2010 						  ldns-gen-zone(1)
All times are GMT -4. The time now is 03:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy