09-02-2009
Awk Performance Issues
Hi All,
I'm facing an issue in my awk script.
The script is processing a large text file having the details of a number of persons, each person's details being written from 100 to 250 tags as given below:
100 START|
101klklk|
...
245 opr|
246 55|
250 END|
100 START|
...
245 pp|
246 65|
250 END|
In the code it writes each individual person's info into a temporary file and then process it for customization
We increased the tags from 250 to 450 leaving almost 200 tags blank.
like from 245 tag to 250 where moved to 445 to 450.(tags from 245-444 will not be coming in the file for now)
But after doing this rearrangment awk script has performance issues of almost double the time. So did changing these tags/identifiers degraded performance?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I need help in awk please help immediatly.
This below function is taking lot of time
Please help me to fine tune it so that it runs faster.
The file count is around 3million records
# Process Body
processbody() {
#set -x
while read line
do
... (18 Replies)
Discussion started by: icefish
18 Replies
2. Solaris
using the internal 2 drives mirror was created using raidctl on 100's of our servers . sometime when one drive fails we dont face any issue & we replace the drive with out any problem . but sometimes when one drive fails , system becomes unresponsive and doesnot allow us to login , the only way to... (1 Reply)
Discussion started by: skamal4u
1 Replies
3. Programming
Hi,
I have written a program in C and have to test the return value of the functions. So the normal way of doin this wud b
int rc
rc=myfunction(input);
if(rc=TRUE){
}
else{
}
But instead of doing this I have called the function in the if() condition. Does this have any... (2 Replies)
Discussion started by: sidmania
2 Replies
4. Solaris
Hi all,
I decided to replace my linux router/firewall with Solaris 11 express. This is a pppoe connection directly to my server...no router boxes. I got everything setup, but the performance is terrible on the NAT....really slow. A web page that loads on the server instantly will take... (3 Replies)
Discussion started by: vectox
3 Replies
5. AIX
Hi,
We have GPFS 3.4 Installed on two AIX 6.1 Nodes. We have 3 GPFS Mount points:
/abc01 4TB (Comprises of 14 x 300GB disks from XIV SAN)
/abc02 4TB (Comprises of 14 x 300GB disks from XIV SAN)
/abc03 1TB ((Comprises of Multiple 300GB disks from XIV SAN)
Now these 40... (1 Reply)
Discussion started by: aixromeo
1 Replies
6. Solaris
I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands
zfs list
zfs snapshot pool/u01@new
zfs send pool/u01@new | zfs -F receive newpool/u01
Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies
7. Solaris
Hello everyone,
recently we have been experiencing performance issues with chmod. We managed to narrow it down to getcwd.
The following folder exists:
/Folder1/subfol1/subfol2/subfol3
cd /Folder1/subfol1/subfol2/subfol3
truss -D pwd 2>&1 | grep getcwd
0.0001... (4 Replies)
Discussion started by: KotekBury
4 Replies
8. AIX
Good Day Everyone,
Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies
9. UNIX for Dummies Questions & Answers
hi guys
right now I have 6 Virtual Machines (VMs) running on Vmware ESXi 5.1 and attached to Storage SAN.
All these run Suse Linux 11 SP1 x64.
All of a sudden 1 of these VMs is running very slow making high CPU usage and I see al wait % kinda high 40-50%. Apparently since I don't own this... (5 Replies)
Discussion started by: karlochacon
5 Replies
10. What is on Your Mind?
For years we blocked Baiduspider due to the fact their bots do not obey the robots.txt directive and can really hurt site performance when they unleash 100 bots on the site each pulling pages many times per second.
Last year, I unblocked Baiduspider's IP addresses, and now the problem is back.
... (1 Reply)
Discussion started by: Neo
1 Replies
LEARN ABOUT CENTOS
pmdatxmon
PMDATXMON(1) General Commands Manual PMDATXMON(1)
NAME
pmdatxmon - txmon performance metrics domain agent (PMDA)
SYNOPSIS
$PCP_PMDAS_DIR/txmon/pmdatxmon [-d domain] [-l logfile] [-U username] tx_type ...
$PCP_PMDAS_DIR/txmon/txrecord [-l]
$PCP_PMDAS_DIR/txmon/txrecord tx_type servtime [tx_type servtime ... ]
$PCP_PMDAS_DIR/txmon/genload
DESCRIPTION
pmdatxmon is an example Performance Metrics Domain Agent (PMDA) which exports a small number of performance metrics from a simulated trans-
action monitor.
The txmon PMDA is shipped as both binary and source code and is designed to be an aid for PMDA developers; the txmon PMDA demonstrates how
performance data can be exported from an application (in this case txrecord) to the PCP infrastructure via a shared memory segment. As a
matter of convenience, pmdatxmon creates (and destroys on exit) the shared memory segment.
The tx_type arguments are arbitrary unique tags used to identify different transaction types.
The txrecord application simulates the processing of one or more transactions identified by tx_type and with an observed service time of
servtime .
With the -l option, txrecord displays the current summary of the transaction activity from the shared memory segment.
genload is a shell and awk(1) script that acts as a front-end to txrecord to generate a constant load of simulated transaction activity.
A brief description of the pmdatxmon command line options follows:
-d It is absolutely crucial that the performance metrics domain number specified here is unique and consistent. That is, domain should
be different for every PMDA on the one host, and the same domain number should be used for the same PMDA on all hosts.
-l Location of the log file. By default, a log file named txmon.log is written in the current directory of pmcd(1) when pmdatxmon is
started, i.e. $PCP_LOG_DIR/pmcd. If the log file cannot be created or is not writable, output is written to the standard error
instead.
-U User account under which to run the agent. The default is the unprivileged "pcp" account in current versions of PCP, but in older
versions the superuser account ("root") was used by default.
INSTALLATION
If you want access to the names, help text and values for the txmon performance metrics, do the following as root:
# cd $PCP_PMDAS_DIR/txmon
# ./Install
You will be prompted for the tx_type tags.
If you want to undo the installation, do the following as root:
# cd $PCP_PMDAS_DIR/txmon
# ./Remove
pmdatxmon is launched by pmcd(1) and should never be executed directly. The Install and Remove scripts notify pmcd(1) when the agent is
installed or removed.
FILES
$PCP_PMCDCONF_PATH
command line options used to launch pmdatxmon
$PCP_PMDAS_DIR/txmon/help
default help text file for the txmon metrics
$PCP_PMDAS_DIR/txmon/Install
installation script for the pmdatxmon agent
$PCP_PMDAS_DIR/txmon/Remove
undo installation script for the pmdatxmon agent
$PCP_LOG_DIR/pmcd/txmon.log
default log file for error messages and other information from pmdatxmon
PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the
file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configura-
tion file, as described in pcp.conf(5).
SEE ALSO
PCPIntro(1), pmcd(1), pmdasample(1), pmdatrivial(1), txmonvis(1) and PMDA(3).
Performance Co-Pilot PCP PMDATXMON(1)