Sponsored Content
Top Forums Shell Programming and Scripting Removal of Duplicate Entries from the file Post 302254306 by radoulov on Tuesday 4th of November 2008 05:56:36 AM
Old 11-04-2008
Use nawk or /usr/xpg4/bin/awk on Solaris:

Code:
awk -F\| 'after[$1]++' infile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check host file for duplicate entries

I need a KSH script that will check a host file for duplicate IP's and/or host names and report out the errors. Anyone out there have one they would like to share? Something like: Hostname blahblah appears X times IP Address xxx.xxx.xxx.xxx appears X times TIA (4 Replies)
Discussion started by: ThreeDot
4 Replies

2. UNIX for Dummies Questions & Answers

Sort, duplicate removal - Query

Hi All, I have a problem with the sort and duplicate filter command I am using in one of my scripts. I have a '|' delimited file and want to sort and remove duplicates on the 1,2,15 fields. These fields constitute the primary key of the table I will be loading the data into. But I see that some... (4 Replies)
Discussion started by: novice1324
4 Replies

3. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi All, I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like Code: wcars42g wcars89j wcars42g wcars42b wcars42b Will duplicate entries... (1 Reply)
Discussion started by: akash_mahakode
1 Replies

4. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi, I forgot how to start a new thread. :( Can somebody please guide me? I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like wcars42g... (2 Replies)
Discussion started by: akash_mahakode
2 Replies

5. Shell Programming and Scripting

Counting duplicate entries in a file using awk

Hi, I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d The file looks like 10.1.1.1 10.1.1.1 10.1.1.1 10.1.2.4 10.1.2.4 12.1.5.6 . . . . and so on.... There are duplicate/multiple entries for some IP... (3 Replies)
Discussion started by: sajal.bhatia
3 Replies

6. Shell Programming and Scripting

Duplicate line removal matching some columns only

I'm looking to remove duplicate rows from a CSV file with a twist. The first row is a header. There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies

7. UNIX for Advanced & Expert Users

Duplicate removal

I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record . Based on 5 fields the duplicates has to be removed . Kindly request to help me in writing a Unix Script. Thanks Asim (11 Replies)
Discussion started by: duplicate
11 Replies

8. Shell Programming and Scripting

String removal from file

Dear all From below mention input file I needed op file as show below. I am using below code but not worked. I/p file BSCBCH1 EXAL-1-4 WO* SMPS MAINS FAIL BSCBCH1 EXAL-1-5 WO* SMPS RECTIFIER FAIL BSCBCH1 EXAL-1-6 WO* SMPS MAJOR ALARM BSCBCH2 EXAL-1-10 WO* ... (5 Replies)
Discussion started by: jaydeep_sadaria
5 Replies

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

10. Shell Programming and Scripting

How to check duplicate entries in file ? (Solaris-9)

Hi, There are duplicate entries in file, but uniq will not see because first field is different. How will I catch all lines, which are having duplicate IPs ? bash-2.05# cat db.file | grep 172.30.133.11 dsrq-ctrl1-prod A 172.30.133.11 e911q-db1-nxge0 A 172.30.133.11... (4 Replies)
Discussion started by: solaris_1977
4 Replies
aclcheck(3SEC)					       File Access Control Library Functions					    aclcheck(3SEC)

NAME
aclcheck - check the validity of an ACL SYNOPSIS
cc [ flag... ] file... -lsec [ library... ] #include <sys/acl.h> int aclcheck(aclent_t *aclbufp, int nentries, int *which); DESCRIPTION
The aclcheck() function checks the validity of an ACL pointed to by aclbufp. The nentries argument is the number of entries contained in the buffer. The which parameter returns the index of the first entry that is invalid. The function verifies that an ACL pointed to by aclbufp is valid according to the following rules: o There must be exactly one GROUP_OBJ ACL entry. o There must be exactly one USER_OBJ ACL entry. o There must be exactly one OTHER_OBJ ACL entry. o If there are any GROUP ACL entries, then the group ID in each group ACL entry must be unique. o If there are any USER ACL entries, then the user ID in each user ACL entry must be unique. o If there are any GROUP or USER ACL entries, then there must be exactly one CLASS_OBJ (ACL mask) entry. o If there are any default ACL entries, then the following apply: o There must be exactly one default GROUP_OBJ ACL entry. o There must be exactly one default OTHER_OBJ ACL entry. o There must be exactly one default USER_OBJ ACL entry. o If there are any DEF_GROUP entries, then the group ID in each DEF_GROUP ACL entry must be unique. o If there are any DEF_USER entries, then the user ID in each DEF_USER ACL entry must be unique. o If there are any DEF_GROUP or DEF_USER entries, then there must be exactly one DEF_CLASS_OBJ (default ACL mask) entry. o If any of the above rules are violated, then the function fails with errno set to EINVAL. RETURN VALUES
If the ACL is valid, alcheck() will return 0. Otherwise errno is set to EINVAL and return code is set to one of the following: GRP_ERROR There is more than one GROUP_OBJ or DEF_GROUP_OBJ ACL entry. USER_ERROR There is more than one USER_OBJ or DEF_USER_OBJ ACL entry. CLASS_ERROR There is more than one CLASS_OBJ (ACL mask) or DEF_CLASS_OBJ (default ACL mask) entry. OTHER_ERROR There is more than one OTHER_OBJ or DEF_OTHER_OBJ ACL entry. DUPLICATE_ERROR Duplicate entries of USER, GROUP, DEF_USER, or DEF_GROUP. ENTRY_ERROR The entry type is invalid. MISS_ERROR Missing an entry. The which parameter returns -1 in this case. MEM_ERROR The system cannot allocate any memory. The which parameter returns -1 in this case. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Interface Stability |Evolving | +-----------------------------+-----------------------------+ |MT-Level |Unsafe | +-----------------------------+-----------------------------+ SEE ALSO
acl(2), aclsort(3SEC), attributes(5) SunOS 5.10 10 Dec 2001 aclcheck(3SEC)
All times are GMT -4. The time now is 11:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy