Sponsored Content
Full Discussion: awk eating too much memory?
Top Forums Shell Programming and Scripting awk eating too much memory? Post 302560570 by anishkumarv on Friday 30th of September 2011 08:04:34 AM
Old 09-30-2011
Hi all,

Sorry for the late Reply this is my Script Using this script we are using for domain count.


Code:
#!/bin/bash

current_date=`date +%d-%m-%Y_%H.%M.%S`
today=`date +%d%m%Y`
yesterday=`date -d 'yesterday' '+%d%m%Y'`
RootPath=/var/domaincount/biz/
LOG=/var/tmp/Intelliscanlog/biz/bizcount$current_date.log

cd $RootPath
echo Intelliscan Process started for .BIZ TLD $current_date >> $LOG

#################################################################################################
## Using Wget Downloading the Zone files it will try only one time
if ! wget --tries=1 --ftp-user=USERNAME --ftp-password=PASSWORD ftp://ftp.URL/zone.gz

then
    echo Download Not Success Domain count Failed With Error >> $LOG
    exit 1
fi
###The downloaded file in Gunzip format from that we need to unzip and start the domain count process####

gunzip zone.gz
mv biz.zone $RootPath/$today.biz

###### It will start the Count #####
awk '/^[^ ]+ IN NS/ && !_[$1]++{print $1; tot++}END{print "\nTotal",tot,"Domains"}' $RootPath/$today.biz > $RootPath/zonefile/$today.biz
awk '/Total/ {print $2}' $RootPath/zonefile/$today.biz > $RootPath/$today.count

### Calculation Part
a=$(< $RootPath/$today.count)
b=$(< $RootPath/$yesterday.count)
c=$(awk 'NR==FNR{a[$0];next} $0 in a{tot++}END{print tot}' $RootPath/zonefile/$today.biz $RootPath/zonefile/$yesterday.biz)


echo "$current_date Today Count For BIZ TlD $a" >> $LOG
echo "$current_date New Registration Domain Counts $((c - a))" >> $LOG
echo "$current_date Deleted Domain Counts $((c - b))" >> $LOG
cat $LOG | mail -s "BIZ Tld Count log" 07anis@gmail.com

Using This Script we remove the duplicates from the file and count the unique domains.

Its Typo error, my overall load average become more than 8 and awk using that time maximum amount of cpu usage.

Last edited by anishkumarv; 09-30-2011 at 09:23 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Hosting Service Eating Space

Dear Group, I am not much used to UNIX. The company I am hosting wiht refuses to help me with this trouble, but as near as I can see, it is NOT my trouble. I have had this service for over a year. I just renewed for another year and all of a sudden the disk quota has been disappearing. I... (3 Replies)
Discussion started by: cindy
3 Replies

2. UNIX for Dummies Questions & Answers

Eating memory

Hello I run Gentoo Linux on my computer: Athlon XP 1700+ ~1,46 mhz 512 mb ram After a while, my computer works really slow, and when I cat /proc/meminfo, I see that I only have 8mb of 512 mb free! How is that possible? I dont run anything I can think of that eats that amount of... (4 Replies)
Discussion started by: Maestin
4 Replies

3. UNIX for Dummies Questions & Answers

/proc is eating my disk man

hi I have an sun ultra 5 running a firewall which has logging enabled (essential). The disk is sliced up with /proc on / (c0t0d0s0). / is sliced at 3 gig. My problem is this, one afternoon, a manager asked me to retrieve some firewall logs, so i went into the relevant directory (also on the /... (3 Replies)
Discussion started by: hcclnoodles
3 Replies

4. What is on Your Mind?

What are you eating ?

Hi, guys ! I was wondering... how many of you are vegetarians ? and why ? (31 Replies)
Discussion started by: Sergiu-IT
31 Replies

5. Solaris

This application is eating up the CPU

Hi, I am not very much fmiliar with Solaris OS. My main concern for posting is One application is eating 50% of CPU and I cannot run that application, If I perform any action in that application it takes real long time. I have solaris installed on my development machine.I have my application... (11 Replies)
Discussion started by: pandu345
11 Replies

6. Shell Programming and Scripting

Memory exhausted in awk

Dear All, I have executed a awk script in linux box which consists of 21 Million records.And i have two mapping files of 500 and 5200 records.To my surprise i found an error awk: cmd. line:19: (FILENAME=/home/FILE FNR=21031272) fatal: Memory exhausted. Is there any limitation for records... (3 Replies)
Discussion started by: cskumar
3 Replies

7. Solaris

Sendmail is eating high memory

Hi, I have installed sendmail on my solaris server. But sendmail its up high memory. its eat upto around 9-10 GB memory. What to do in this ? Thanks NeeleshG (6 Replies)
Discussion started by: neel.gurjar
6 Replies

8. Shell Programming and Scripting

[bash] IF is eating my loops

Hi! Could someone explain me why the below code is printing the contents of IF block 5 times instead of 0? #!/bin/bash VAR1="something" VAR2="something" for((i=0;i<10;i++)) do if(($VAR1=~$VAR2)) then echo VAR1: $VAR1 echo... (3 Replies)
Discussion started by: machinogodzilla
3 Replies

9. Shell Programming and Scripting

AWK Memory Limit ?

Is there an input file memory limit for awk? I have a 38Mb text file that I am trying to print out certatin lines and add a string to the end of that line. When I excute the script on the 38Mb file the string I am adding is put on a new line. If I do the same with a smaller file the... (3 Replies)
Discussion started by: cold_Que
3 Replies

10. Shell Programming and Scripting

how to find a job which is writing a big file and eating up space?

how to find a job which is writing a big file and eating up space? (3 Replies)
Discussion started by: rush2andy
3 Replies
Date::Manip::Migration5to6(3pm) 			User Contributed Perl Documentation			   Date::Manip::Migration5to6(3pm)

NAME
Date::Manip::Migration5to6 - how to upgrade from 5.xx to 6.00 SYNOPSIS
When upgrading from Date::Manip 5.xx to 6.00, a few changes may be necessary to your scripts. The Date::Manip::Changes5to6 document lists in more detail the ways in which Date::Manip changed, but very few of these actually entail changes to your script. It should be noted that once the changes are made to your script, it will no longer run correctly in 5.xx. NECESSARY AND SUGGESTED CHANGES
The following changes are necessary, or strongly suggested: Reading config files with Date_Init If you use Date_Init to read any config files (if you do business mode calculations, you probably do), you should remove all of the following config variables from your call to Date_Init: GlobalCnf=FILE PersonalCnf=FILE PathSep=* IgnoreGlobalCnf=* PersonalCnfPath=* and replace them with: ConfigFile=FILE where FILE is now the full path to a config file. Also, the ConfigFile argument should be the first argument in Date_Init. Date_ConvTZ The Date_ConvTZ function has changed. It should now take 3 arguments: $date = Date_ConvTZ($date,$from,$to); If $from is not given, it defaults to the local time zone. If $to is not given, it defaults to the local time zone. The date is converted from the $from time zone into the $to time zone. Both should be any time zone (or alias) supported by Date::Manip. The old $errlevel argument is no longer handled. ConvTZ and TZ config variables If you use either the ConvTZ or TZ config variables, you should replace them with either SetDate or ForceDate. See the Date::Manip::Config document for information. The TZ variable will continue to work until Dec 2013 at which point it will be removed. Other deprecated config variables The following config variables have been deprecated, but will continue to function (though they will be removed at a future date): IntCharSet (removed Oct 2012) TZ (removed Dec 2013) The following variables have been removed. If you use any of them, you may need to modify your scripts: GlobalCnf PersonalCnf PathSep IgnoreGlobalCnf PersonalCnfPath ConvTZ Internal TodayIsMidnight DeltaSigns UpdateCurrTZ ResetWorkdDay today, yesterday, tomorrow If you parse the strings "today", "yesterday", or "tomorrow" in order to get the time now, or 24 hours in the past/future, this will no longer work. These strings now refer strictly to the date (so "today" is the current day at midnight, "yesterday" is the previous day at midnight, etc.). To get the time now, 24 hours ago, or 24 hours in the future, you would need to parse the strings "now", "-24:00:00", or "+24:00:00" respectively. Do not use Memoize In 5.xx, it was documented that you could use the module Memoize to speed up Date::Manip, especially when sorting dates. This information is no longer accurate. Using Memoize in conjunction with Date::Manip should have little impact on performance, and may lead to incorrect results, especially if you change config variables. Please see the Date::Manip::Changes5to6 (GENERAL CHANGES) document for more information. If you find other instances where it is necessary to modify your script, please email me so that I can add that information to this document. BUGS AND QUESTIONS
Please refer to the Date::Manip::Problems documentation for information on submitting bug reports or questions to the author. SEE ALSO
Date::Manip - main module documentation LICENSE
This script is free software; you can redistribute it and/or modify it under the same terms as Perl itself. AUTHOR
Sullivan Beck (sbeck@cpan.org) perl v5.14.2 2012-06-02 Date::Manip::Migration5to6(3pm)
All times are GMT -4. The time now is 02:25 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy