Sponsored Content
Full Discussion: awk eating too much memory?
Top Forums Shell Programming and Scripting awk eating too much memory? Post 302560570 by anishkumarv on Friday 30th of September 2011 08:04:34 AM
Old 09-30-2011
Hi all,

Sorry for the late Reply this is my Script Using this script we are using for domain count.


Code:
#!/bin/bash

current_date=`date +%d-%m-%Y_%H.%M.%S`
today=`date +%d%m%Y`
yesterday=`date -d 'yesterday' '+%d%m%Y'`
RootPath=/var/domaincount/biz/
LOG=/var/tmp/Intelliscanlog/biz/bizcount$current_date.log

cd $RootPath
echo Intelliscan Process started for .BIZ TLD $current_date >> $LOG

#################################################################################################
## Using Wget Downloading the Zone files it will try only one time
if ! wget --tries=1 --ftp-user=USERNAME --ftp-password=PASSWORD ftp://ftp.URL/zone.gz

then
    echo Download Not Success Domain count Failed With Error >> $LOG
    exit 1
fi
###The downloaded file in Gunzip format from that we need to unzip and start the domain count process####

gunzip zone.gz
mv biz.zone $RootPath/$today.biz

###### It will start the Count #####
awk '/^[^ ]+ IN NS/ && !_[$1]++{print $1; tot++}END{print "\nTotal",tot,"Domains"}' $RootPath/$today.biz > $RootPath/zonefile/$today.biz
awk '/Total/ {print $2}' $RootPath/zonefile/$today.biz > $RootPath/$today.count

### Calculation Part
a=$(< $RootPath/$today.count)
b=$(< $RootPath/$yesterday.count)
c=$(awk 'NR==FNR{a[$0];next} $0 in a{tot++}END{print tot}' $RootPath/zonefile/$today.biz $RootPath/zonefile/$yesterday.biz)


echo "$current_date Today Count For BIZ TlD $a" >> $LOG
echo "$current_date New Registration Domain Counts $((c - a))" >> $LOG
echo "$current_date Deleted Domain Counts $((c - b))" >> $LOG
cat $LOG | mail -s "BIZ Tld Count log" 07anis@gmail.com

Using This Script we remove the duplicates from the file and count the unique domains.

Its Typo error, my overall load average become more than 8 and awk using that time maximum amount of cpu usage.

Last edited by anishkumarv; 09-30-2011 at 09:23 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Hosting Service Eating Space

Dear Group, I am not much used to UNIX. The company I am hosting wiht refuses to help me with this trouble, but as near as I can see, it is NOT my trouble. I have had this service for over a year. I just renewed for another year and all of a sudden the disk quota has been disappearing. I... (3 Replies)
Discussion started by: cindy
3 Replies

2. UNIX for Dummies Questions & Answers

Eating memory

Hello I run Gentoo Linux on my computer: Athlon XP 1700+ ~1,46 mhz 512 mb ram After a while, my computer works really slow, and when I cat /proc/meminfo, I see that I only have 8mb of 512 mb free! How is that possible? I dont run anything I can think of that eats that amount of... (4 Replies)
Discussion started by: Maestin
4 Replies

3. UNIX for Dummies Questions & Answers

/proc is eating my disk man

hi I have an sun ultra 5 running a firewall which has logging enabled (essential). The disk is sliced up with /proc on / (c0t0d0s0). / is sliced at 3 gig. My problem is this, one afternoon, a manager asked me to retrieve some firewall logs, so i went into the relevant directory (also on the /... (3 Replies)
Discussion started by: hcclnoodles
3 Replies

4. What is on Your Mind?

What are you eating ?

Hi, guys ! I was wondering... how many of you are vegetarians ? and why ? (31 Replies)
Discussion started by: Sergiu-IT
31 Replies

5. Solaris

This application is eating up the CPU

Hi, I am not very much fmiliar with Solaris OS. My main concern for posting is One application is eating 50% of CPU and I cannot run that application, If I perform any action in that application it takes real long time. I have solaris installed on my development machine.I have my application... (11 Replies)
Discussion started by: pandu345
11 Replies

6. Shell Programming and Scripting

Memory exhausted in awk

Dear All, I have executed a awk script in linux box which consists of 21 Million records.And i have two mapping files of 500 and 5200 records.To my surprise i found an error awk: cmd. line:19: (FILENAME=/home/FILE FNR=21031272) fatal: Memory exhausted. Is there any limitation for records... (3 Replies)
Discussion started by: cskumar
3 Replies

7. Solaris

Sendmail is eating high memory

Hi, I have installed sendmail on my solaris server. But sendmail its up high memory. its eat upto around 9-10 GB memory. What to do in this ? Thanks NeeleshG (6 Replies)
Discussion started by: neel.gurjar
6 Replies

8. Shell Programming and Scripting

[bash] IF is eating my loops

Hi! Could someone explain me why the below code is printing the contents of IF block 5 times instead of 0? #!/bin/bash VAR1="something" VAR2="something" for((i=0;i<10;i++)) do if(($VAR1=~$VAR2)) then echo VAR1: $VAR1 echo... (3 Replies)
Discussion started by: machinogodzilla
3 Replies

9. Shell Programming and Scripting

AWK Memory Limit ?

Is there an input file memory limit for awk? I have a 38Mb text file that I am trying to print out certatin lines and add a string to the end of that line. When I excute the script on the 38Mb file the string I am adding is put on a new line. If I do the same with a smaller file the... (3 Replies)
Discussion started by: cold_Que
3 Replies

10. Shell Programming and Scripting

how to find a job which is writing a big file and eating up space?

how to find a job which is writing a big file and eating up space? (3 Replies)
Discussion started by: rush2andy
3 Replies
YESTERDAY(1)						      General Commands Manual						      YESTERDAY(1)

NAME
yesterday - print file names from the dump SYNOPSIS
yesterday [ -c ] [ -date ] files ... DESCRIPTION
Yesterday prints the names of the files from the most recent dump. Since dumps are done early in the morning, yesterday's files are really in today's dump. For example, if today is March 17, 1992, yesterday /adm/users prints /n/dump/1992/0317/adm/users In fact, the implementation is to select the most recent dump in the current year, so the dump selected may not be from today. With option -c, yesterday copies the dump file to the current directory. The date option selects other day's dumps, with a format of 2, 4, 6, or 8 digits of the form dd, mmdd, yymmdd, or yyyymmdd. Yesterday does not guarantee that the string it prints represents an existing file. EXAMPLES
Back up to yesterday's MIPS binary of vc: cd /mips/bin yesterday -c vc Temporarily back up to March 1's MIPS C library to see if a program runs correctly when loaded with it: bind `{yesterday -0301 /mips/lib/libc.a} /mips/lib/libc.a rm v.out mk v.out FILES
/n/dump SOURCE
/rc/bin/yesterday SEE ALSO
fs(4) BUGS
It's hard to use this command without singing. YESTERDAY(1)
All times are GMT -4. The time now is 04:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy