Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: awk not working as expected with BIG files ...
Top Forums Shell Programming and Scripting awk not working as expected with BIG files ... Post 63318 by videsh77 on Thursday 24th of February 2005 05:53:08 AM
Old 02-24-2005
Question awk not working as expected with BIG files ...
I am facing some strange problem.

I know, there is only one record in a file 'test.txt' which starts with 'X'

I ensure that with following command,
awk /^X/ test.txt | wc -l

This gives me output = '1'.

Now I take out this record out of the file, as follows :

awk /^X/ test.txt > XRecord.txt

Then I take out all records which do not start with 'X'.
Following is the command for the same,

awk '$0!~/^X/ test.txt > NotXRecord.txt

After doing this, Number of records with NotXRecord.txt is not 1 record less than number of records with 'test.txt'.

Now the same sequence above, works fine with small files, say uptil 100 thousands of records.

But it do not work for a file, with 4million rows.

Does any one have valuable hints to share?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Var substitution in awk - not working as expected

countA=`awk '/X/''{print substr($0,38,1)}' fName | wc -l` countB=`wc -l fName | awk '{print int($1)}'` echo > temp ratio=`awk -va=$countA -vc=$countB '{printf "%.4f", a/c}' temp` After running script for above I am getting an error as : awk: 0602-533 Cannot find or open file -vc=25. The... (3 Replies)
Discussion started by: videsh77
3 Replies

2. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

3. Shell Programming and Scripting

awk with really big files

Hi, I have a text file that is around 7Gb which is basically a matrix of numbers (FS is a space and RS is \n). I need the most efficient way of plucking out a number from a specified row and column in the file. For example, for the value at row 15983, col 26332, I'm currently I'm using: ... (1 Reply)
Discussion started by: Jonny2Vests
1 Replies

4. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

5. Shell Programming and Scripting

bash variable (set via awk+sed) not working as expected

Hi! Been working on a script and I've been having a problem. I've finally narrowed it down to this variable I'm setting: servername=$(awk -v FS=\/ '{ print $7 } blah.txt | sed 's\/./-/g' | awk -v FS=\- '{print $1}')" This will essentially pare down a line like this: ... (7 Replies)
Discussion started by: creativedynamo
7 Replies

6. Shell Programming and Scripting

awk not working as expected in script

Dear all, I had script which used to work, but recently it is not working as expected. I have command line in my shell script to choose the following format from the output_elog and perform some task afterwards on As you see, I want all numbers in foramt following RED mark except for... (12 Replies)
Discussion started by: emily
12 Replies

7. Shell Programming and Scripting

awk command not working as expected

Following one line of awk code removes first 3 characters from each line but when I run the same code on another linux platform it doesn't work and only prints blank lines for each record. Can anyone please explain why this doesn't work? (31 Replies)
Discussion started by: later_troy
31 Replies

8. UNIX for Advanced & Expert Users

Manipulate files with find and fuser not working as expected on SunOs

Greetings, For housekeeping, I use the following command: find /some/path -type f -name "*log*" ! -exec fuser -s "{}" 2>/dev/null \; -exec ls -lh {} \; It finds all log files not currently in use by a process and manipulates them. This command always works on linux and redhat machines,... (2 Replies)
Discussion started by: dampio
2 Replies

9. Shell Programming and Scripting

awk gsub not working as expected

Hi Experts, Need your kind help with gsub awk. Below is my pattern:"exec=1_host_cnt=100_dup=4_NameTag=targetSrv_500.csv","'20171122112948"," 100"," 1"," 1"," 4","400","","", " aac sample exec ""hostname=XXXXX commandline='timeout 10 openssl speed -multi 2 ; exit 0'"" ","-1","-1","1","... (6 Replies)
Discussion started by: pradyumnajpn10
6 Replies

10. Shell Programming and Scripting

awk matching script not working as expected

This is my ubuntu version: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial $ /bin/awk -V | head -n1 bash: /bin/awk: No such file or directory I have gotten a script that helps me to parse,... (14 Replies)
Discussion started by: delbroooks
14 Replies

LEARN ABOUT DEBIAN

mongofiles

MONGOFILES(1)							  Mongo Database						     MONGOFILES(1)

NAME

       mongofiles - a simple GridFS interface

SYNOPSIS

       mongofiles [OPTIONS]

DESCRIPTION

       mongofiles is used to list, get, and insert files in the database.

EXAMPLES

       mongofiles list
	      lists files in test.fs.files

       mongofiles put README.txt
	      inserts the file README.txt into the collection test.fs.files

       mongofiles get photo.jpg
	      retrieves photo.jpg from test.fs.files and saves it locally

OPTIONS

       --help show usage information

       -h, --host HOST
	      mongo host to which to connect

       -d, --db DB
	      database to use (default DB=test)

       -c, --collection COLLECTION (default COLLECTION=fs.files)
	      collection to use

       --command [list|search|put|get]
	      execute a command

       --file FILE
	      filename for get or put

       list   list all files.  takes an optional filename.  the file has to start with the filename

       search search all files for something that contains the string

COPYRIGHT

       Copyright 2007-2009 10gen

SEE ALSO

       For more information, please refer to the MongoDB wiki, available at http://www.mongodb.org.

AUTHOR

       Kristina Chodorow

10gen								     June 2009							     MONGOFILES(1)
All times are GMT -4. The time now is 07:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy