Sponsored Content
Full Discussion: Remove duplicate nodes
Top Forums Shell Programming and Scripting Remove duplicate nodes Post 302769389 by user8 on Tuesday 12th of February 2013 05:42:07 AM
Old 02-12-2013
This works for me (using gawk):
Code:
gawk -F',| ' '!(a[$1,$2]++ + a[$2,$1]++)'

These 5 Users Gave Thanks to user8 For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies

2. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

3. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

4. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

5. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

6. Shell Programming and Scripting

Remove subsequent duplicate only

Hi, I've been trying to dig myself out of this, but nothing has worked out yet. I have an input like this: 1-Num1 1-Num2 2-Num3 3-Num4 1-Num5 3-Num11 2-Num11 1-Num13 1-Num16 3-Num18 4-Num19 2-Num20 1-Num22 3-Num23 (11 Replies)
Discussion started by: jamie_123
11 Replies

7. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

8. Shell Programming and Scripting

How To Remove Duplicate Based on the Value?

Hi , Some time i got duplicated value in my files , bundle_identifier= B Sometext=ABC bundle_identifier= A bundle_unit=500 Sometext123=ABCD bundle_unit=400 i need to check if there is a duplicated values or not if yes , i need to check if the value is A or B when Bundle_Identified ,... (2 Replies)
Discussion started by: OTNA
2 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies
NODEATTR(1)							     NODEATTR							       NODEATTR(1)

NAME
nodeattr - query genders file SYNOPSIS
nodeattr [-f genders] [-q | -c | -n | -s] [-X exclude_query] query nodeattr [-f genders] [-q | -c | -n | -s] -A nodeattr [-f genders] [-v] [node] attr[=val] nodeattr [-f genders] -Q [node] query nodeattr [-f genders] -V [-U] attr nodeattr [-f genders] -l [node] nodeattr [-f genders] -k nodeattr [-f genders] -d genders nodeattr [-f genders] --expand nodeattr [-f genders] --compress DESCRIPTION
When invoked with the -q , -c , -n , or -s arguments, nodeattr reads the genders file and outputs a list of nodes that match the specified query. The nodes are listed in hostlist format, comma separated lists, newline separated lists, or space separated lists respectively. Genders queries will query the genders database for a set of nodes based on the union, intersection, difference, or complement of genders attributes and values. The set operation union is represented by two pipe symbols ('||'), intersection by two ampersand symbols ('&&'), difference by two minus symbols ('--'), and complement by a tilde ('~'). Parentheses may be used to change the order of operations. The -X argument and query can be used to exclude nodes from the resulting output. A query can be replaced with the -A option to cause nodeattr to print all the nodes listed in the genders database. When called with a node name (optional) and attribute name, nodeattr returns 0 to the environment if the node has the attribute; else 1. If -v is present, the attribute name and any value (see below) is printed on the standard output. If a node name is not specified, the local host is assumed. When called with the -Q argument, nodeattr will check if the node name (optional) is met by the attribute and value conditions specified in the query. If the conditions are met, nodeattr returns 0 to the environment; else 1. The query format is identical to the format listed above. If a node name is not specified, the local host is assumed. The -V option causes nodeattr to print all of the values that exist for a particular attribute. Also specifing -U with -V causes nodeattr to print out only unique values for the particular attribute. The -l option causes nodeattr to print the attributes of the specified node. If no node is listed, all attributes in the genders file are listed. The -k option checks the genders file for parse errors and proper formatting. If errors are found, information about the error will be output to standard error. Nodeattr will always check the default genders file, but a different genders file can be specified with the -f option. The -d option allows the specified genders database to be compared to the filename indicated by the -f option or the default genders data- base. The differences contained in the specified database will be output to standard error. The --expand option will take a genders database, expand all hostranges, and output a new genders database. The subsequent database will be identical to the first, but every node will be listed on a separate line. This option may be useful for debugging or determining the difference between databases. The --compress is opposite of the --expand option. It will output a new identical genders database with hostranges of nodes with identical attributes. Depending on the setup of your genders database, the resulting database may be longer or shorter. This option may be useful as a beginning step to compressing an existing genders database. Attribute names may optionally appear in the genders file with an equal sign followed by a value. Nodeattr ignores these values except when the -v option requests that the value, if any, be displayed; and when an attribute is specified on the command line with a value, in which case only an attribute with the specified value in the genders file will match. EXAMPLES
Retrieve a comma separated list of all login nodes: nodeattr -c login Retrieve a hostlist formatted list of all login nodes: nodeattr -q login Retrieve a comma separated list of nodes with 4 cpus: nodeattr -c cpus=4 Retrieve a comma separated list of all login and management nodes: nodeattr -c "login||mgmt" Retrieve a comma separated list of all login nodes with 4 cpus: nodeattr -c "login&&cpus=4" Retrieve a comma separated list of all nodes that are not login or management nodes: nodeattr -c "~(login||mgmt)" To use nodeattr with pdsh to run a command on all fddi nodes: pdsh -w`nodeattr -c fddi` command To use nodeattr in a ksh script to collect a list of users on login nodes: for i in `nodeattr -n login`; do rsh $i who; done To verify whether or not this node is a head node: nodeattr head && echo yes To verify whether or not this node is a head node and ntpserver: nodeattr -Q "head&&ntpserver" && echo yes FILES
/etc/genders SEE ALSO
libgenders(3) LLNL
August 2003 NODEATTR(1)
All times are GMT -4. The time now is 09:36 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy