Sponsored Content
Top Forums Shell Programming and Scripting Remove lines if some data is the same: Centos5 / bash Post 302676223 by oly_r on Tuesday 24th of July 2012 09:33:22 AM
Old 07-24-2012
Remove lines if some data is the same: Centos5 / bash

Ok what i have is 8 separate files based on how many IP's are associated with the domain. I want to limit duplication of ip's within the individual files (ie. i don't care if the same ip is in 1 IP file and 4 IP file). Can someone help me with how to go through the files and remove lines that have have an IP already in a previous line (within that same file). In the 2 IP file the What and Where lines should be removed since each has an ip in the Who line. In the 4 IP file when the spock line is removed since it has an ip from Kirk that file is then ok since the McCoy line is then no longer duplicating 10.100.200.200. Hopefully this is understandable (it makes sense to me Smilie ). I have been able to get rid of domains that have all the same ip's associated by using sort -k? -k? -u to ignore the first field, but can't figure out how to do single ip's from a line and test against other lines Smilie.

I'm doing this on a CentOS 5.8 box in a bash script (whole lot more processing going on all around this portion).


EXAMPLES: Colon separated lines in each file
domain.com:ip:ip:

1 IP file

Any.com:192.168.10.100:
Where.edu:192.168.10.200:


2 IP file

Who.com: 192.168.10.300:192.168.10.200:
What.gov:10.0.0.150:192.168.10.300:
Where.biz:192.168.10.200:10.10.0.10:
When.tv:192.168.10.10:192.168.10.11:

4 IP file

Kirk.ufp:10.0.100.100:10.0.200.100:10.0.200.200:10.0.100.200:
Spock.vsa:10.100.100.100:10.100.100.200:10.100.200.200:10.0.100.100
Mccoy.ama:10.100.200.200:192.168.200.200:192.168.100.200:192.168.100.201
 

10 More Discussions You Might Find Interesting

1. Programming

pthread_mutex_trylock() overwrites global variable on CentOS5

Hi all, I am new to linux and got problem with pthread_mutex_trylock(). I have used mutex in my code. When I try to call pthread_mutex_trylock() on RECURSIVE type of mutex it overwrites adjacent memory location (that is global variable of type structure say x, memory allocated using malloc()). ... (5 Replies)
Discussion started by: liveshell
5 Replies

2. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

3. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

4. Red Hat

Adding EML to CentOS5.5

Hi! I've just installed CentOS 5.5 on HP Proliant DL380G5 with two FC HBA and now I'm struggling with adding HP EML103e connected through two Brocade SAN switches. Tape library has two tape drives, each in different fabric. Robotics connected to both fabrics. Zoning is ok - OS sees drives'... (0 Replies)
Discussion started by: Sapfeer
0 Replies

5. Shell Programming and Scripting

how to write a function to get data under spesific lines ? using bash

I have a text file called ( bvhz ) contains data : Subscriber Data ID = 2 Customer = 99 Data ID = 4 Customer = cf99 Data ID = 5 Customer = c99 Data ID = 11 Customer = 9n9 Subscriber Data ID = 1 Customer = 9ds9 Data ID = 2 Customer = 9sad9 Data ID = 3 Customer = f99... (1 Reply)
Discussion started by: teefa
1 Replies

6. Shell Programming and Scripting

Converting variable space width data into CSV data in bash

Hi All, I was wondering how I can convert each line in an input file where fields are separated by variable width spaces into a CSV file. Below is the scenario what I am looking for. My Input data in inputfile.txt 19 15657 15685 Sr2dReader 107.88 105.51... (4 Replies)
Discussion started by: vharsha
4 Replies

7. Shell Programming and Scripting

Parsing XML (and insert data) then output data (bash / Solaris)

Hi folks I have a script I wrote that basically parses a bunch of config and xml files works out were to add in the new content then spits out the data into a new file. It all works - apart from the xml and config file format in the new file with XML files the original XML (that ends up in... (2 Replies)
Discussion started by: dfinch
2 Replies

8. Hardware

Bumblebee for CentOS5/RHEL5

I have a laptop running CentOS 5.9 with switchable/hybrid graphics (Intel integrated and Nvidia). For now, have managed to successfully install Nvidia driver using NVIDIA-Linux-x86_64-331.20.run file from Nvidia website, but failed to start xserver. To use both adapters I need to install... (0 Replies)
Discussion started by: cooltoad
0 Replies

9. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

10. UNIX for Beginners Questions & Answers

Bash to remove find and remove specific extension

The bash below executes and does find all the .bam files in each R_2019 folder. However set -x shows that the .bam extension only gets removed from one .bam file in each folder (appears to be the last in each). Why is it not removing the extension from each (this is $SAMPLE)? Thank you :). set... (4 Replies)
Discussion started by: cmccabe
4 Replies
PMDABASH(1)						      General Commands Manual						       PMDABASH(1)

NAME
pmdabash - Bourne-Again SHell trace performance metrics domain agent SYNOPSIS
$PCP_PMDAS_DIR/bash/pmdabash [-C] [-d domain] [-l logfile] [-I interval] [-t timeout] [-U username] configfile DESCRIPTION
pmdabash is an experimental Performance Metrics Domain Agent (PMDA) which exports "xtrace" events from a traced bash(1) process. This includes the command execution information that would usually be sent to standard error with the set -x option to the shell. Event metrics are exported showing each command executed, the function name and line number in the script, and a timestamp. Additionally, the process identifier for the shell and its parent process are exported. This requires bash version 4 or later. A brief description of the pmdabash command line options follows: -d It is absolutely crucial that the performance metrics domain number specified here is unique and consistent. That is, domain should be different for every PMDA on the one host, and the same domain number should be used for the same PMDA on all hosts. -l Location of the log file. By default, a log file named bash.log is written in the current directory of pmcd(1) when pmdabash is started, i.e. $PCP_LOG_DIR/pmcd. If the log file cannot be created or is not writable, output is written to the standard error instead. -s Amount of time (in seconds) between subsequent evaluations of the shell trace file descriptor(s). The default is 2 seconds. -m Maximum amount of memory to be allowed for each event queue (one per traced process). The default is 2 megabytes. -U User account under which to run the agent. The default is the unprivileged "pcp" account in current versions of PCP, but in older versions the superuser account ("root") was used by default. INSTALLATION
In order for a host to export the names, help text and values for the bash performance metrics, do the following as root: # cd $PCP_PMDAS_DIR/bash # ./Install As soon as an instrumented shell script (see INSTRUMENTATION selection below) is run, with tracing enabled, new metric values will appear - no further setup of the agent is required. If you want to undo the installation, do the following as root: # cd $PCP_PMDAS_DIR/bash # ./Remove pmdabash is launched by pmcd(1) and should never be executed directly. The Install and Remove scripts notify pmcd(1) when the agent is installed or removed. INSTRUMENTATION
In order to allow the flow of event data between a bash(1) script and pmdabash, the script should take the following actions: #!/bin/sh source $PCP_DIR/etc/pcp.sh pcp_trace on $@ # enable tracing echo "awoke, $count" pcp_trace off # disable tracing The tracing can be enabled and disabled any number of times by the script. On successful installation of the agent, several metrics will be available: $ pminfo bash bash.xtrace.numclients bash.xtrace.maxmem bash.xtrace.queuemem bash.xtrace.count bash.xtrace.records bash.xtrace.parameters.pid bash.xtrace.parameters.parent bash.xtrace.parameters.lineno bash.xtrace.parameters.function bash.xtrace.parameters.command When an instrumented script is running, the generation of event records can be verified using the pmevent(1) command, as follows: $ pmevent -t 1 -x '' bash.xtrace.records host: localhost samples: all bash.xtrace.records["4538 ./test-trace.sh 1 2 3"]: 5 event records 10:00:05.000 --- event record [0] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 43 bash.xtrace.parameters.command "true" 10:00:05.000 --- event record [1] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 45 bash.xtrace.parameters.command "(( count++ ))" 10:00:05.000 --- event record [2] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 46 bash.xtrace.parameters.command "echo 'awoke, 3'" 10:00:05.000 --- event record [3] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 47 bash.xtrace.parameters.command "tired 2" 10:00:05.000 --- event record [4] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 38 bash.xtrace.parameters.function "tired" bash.xtrace.parameters.command "sleep 2" FILES
$PCP_PMCDCONF_PATH command line options used to launch pmdabash $PCP_PMDAS_DIR/bash/help default help text file for the bash metrics $PCP_PMDAS_DIR/bash/Install installation script for the pmdabash agent $PCP_PMDAS_DIR/bash/Remove undo installation script for pmdabash $PCP_LOG_DIR/pmcd/bash.log default log file for error messages and other information from pmdabash PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configura- tion file, as described in pcp.conf(5). SEE ALSO
bash(1), pmevent(1) and pmcd(1). Performance Co-Pilot PCP PMDABASH(1)
All times are GMT -4. The time now is 09:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy