Sponsored Content
Top Forums UNIX for Dummies Questions & Answers deleteing duplicate lines sing uniq while ignoring a column Post 302429829 by japaneseguitars on Tuesday 15th of June 2010 05:38:24 PM
Old 06-15-2010
deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example


Code:
10 chr1 ASF 30
15 chr1 ASF 20
5 chr1 ASF 30
6 chr2 EBC 15
4 chr2 EBC 30
...

I want to know if I can delete duplicate lines while ignoring column 1, so the results will look like (I will of course sort, etc before I use uniq)

Code:
10 chr1 ASF 30
15 chr1 ASF 20
6 chr2 EBC 15
4 chr2 EBC 30
...

the 3rd line deleted since the information in column 2, 3, 4 has a duplicate somewhere else in the file.

I know that there's a command with uniq that lets you ignore the first n characters, but I don't have a set n, so I cannot use that. Thanks.

Last edited by Scott; 06-15-2010 at 06:47 PM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

oneliner:sing SED on a specific column

is this possible a one liner sed command. I have a text file ex. from : xxx yyy ZZZ /test/devl/aasdasd.log1 xxx yyy ZZZ /test/devl/aasdasd.log2 to : xxx yyy ZZZ /test/prod/aasdasd.log1 xxx yyy ZZZ /test/prod/aasdasd.log2 and I want to sed only the fourth column sed 's/devl/prod/g' ... (8 Replies)
Discussion started by: chaseeem
8 Replies

2. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

3. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

4. Shell Programming and Scripting

Replace a column with a value by ignoring the header lines

i have a file in the gz format , the content of the file is as follow. gzcat f1.gz # 1.name # 2.location # 3.age # 4.dob . . . . . . . . . # 43.hobbies < Aravind,33,chennai,09091980, , , , , , , surfing> (5 Replies)
Discussion started by: aravindj80
5 Replies

5. UNIX for Dummies Questions & Answers

awk to sum column field from duplicate row/lines

Hello, I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file Name m/c usage abc xxx 10 abc xxx 20 abc xxx 5 xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies

6. UNIX for Dummies Questions & Answers

awk solution to duplicate lines based on column

Hi experts, I have a tab-delimited file with one column containing values separated by a comma. I wish to duplicate the entire line for every value in that comma-delimited field. For example: $cat file 4444 4444 4444 4444 9990 2222,7777 6666 2222 ... (3 Replies)
Discussion started by: torchij
3 Replies

7. Shell Programming and Scripting

Count duplicate lines ignoring certain columns

I have this structure: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt 2 xxx 38 aaa yyy 1 xxx 38 aaa yyy I need to collapse duplicate lines ignoring column 1 and add values of duplicate lines (col1) so it will look like this: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt ... (3 Replies)
Discussion started by: coppuca
3 Replies

8. Shell Programming and Scripting

Remove duplicate lines after ignoring case and spaces between

Oracle Linux 6.5 $ cat someStrings.txt GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on SCOTT.emp to JOHN; grant select on scott.emp to john; grant select on scott.dept to hr;If you ignore the case and the empty space between the... (6 Replies)
Discussion started by: kraljic
6 Replies

9. Shell Programming and Scripting

Removing duplicate lines on first column based with pipe delimiter

Hi, I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines Command : sort -t'|' -nuk1 file.txt Input : 38376KZ|09/25/15|1.057 38376KZ|09/25/15|1.057 02006YB|09/25/15|0.859 12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies

10. UNIX for Beginners Questions & Answers

Find lines with duplicate values in a particular column

I have a file with 5 columns. I want to pull out all records where the value in column 4 is not unique. For example in the sample below, I would want it to print out all lines except for the last two. 40991764 2419 724 47182 Cand A 40992936 3591 724 47182 Cand B 40993016 3671 724 47182 Cand C... (5 Replies)
Discussion started by: kaktus
5 Replies
asecure(1M)															       asecure(1M)

NAME
asecure - control access to Audio on a workstation (OBSOLETED) SYNOPSIS
[] [] [] [] [] [] [] [] [] DESCRIPTION
On Series 700 workstations, audio is secured so that only the user on the local workstation can access audio. You use the command to mod- ify audio security. This command does not apply to X stations; on an X station, access to audio is unrestricted. To modify audio security, become root on the local workstation where you want make a change. Then, use as follows: When prompted, enter any meaningful password. Issuing creates the Audio Security File (ASF). The ASF contains information that determines which hosts and users can access the Aserver, and which users (other than the superuser) can modify the ASF. If needed, you can allow unrestricted access to audio on this workstation. To remove audio security, issue this command: If instead, you wish to modify security, you use to make changes to the information in the ASF. (Because the ASF is a binary file, we do not recommend using an editor on this file.) You can use to make these types of changes: o Allow all clients from a remote host to access the server. o Allow specific users from all other hosts to access the server. o Allow a specific user from a specific host to access the server. o Disable access control, allowing complete unrestricted access to the server, but leaving the ASF intact. Every operation that creates, reinitializes, or changes the contents of the ASF is logged in the file, so that you can track any changes to the ASF. Options supports the following options: Add/delete hostname,username pair. You must be either superuser or a to do this. You can supply more than one hostname,username pair separated by blanks. To use either the or options, you MUST supply at least one hostname,username pair. This option will not work without a pair. Create a new ASF file, called the file. Access control default is enabled with no entries in the access list. Aserver can now be accessed only by local users on the host machine. If an file already exists, it is re-initialized. You must be superuser to execute this option. This option is mutually-exclusive of all other options. This option requires a password. This is an extra layer of protection for the contents of the ASF. It is designed to prevent surreptitious manipulation of the ASF. If you are creating a new ASF, you are prompted for a password and an encrypted copy of that password is stored in the new ASF. If the ASF already exists, you are prompted for the password. If your password matches the password stored in the ASF, the ASF is then re-initialized. Disable access control to the Aserver. This allows unrestricted access by all clients. Enable access control to the Aserver. This restricts access to clients listed in the ASF. Enabled is the default state. Add/delete hostnames for ALL users. You must be either superuser or a to do this. You can supply more than one hostname sepa- rated by blanks. List the contents of the ASF. This option shows a list of the hostnames and/or usernames that have access to the Aserver. Change password for file. You must be superuser to do this. You are prompted once for the old password, then prompted twice for the new password. Add/delete You must be superuser to do this and must enter the password given when the ASF was created (see option). To see a list of privileged users, you must be superuser and use the option. Add/delete usernames for ALL hosts. You must be either superuser or a to do this. You can supply more than one username sepa- rated by blanks. EXAMPLES
List entries in access list. Disable access control. This means anyone can connect to Aserver without restriction. Add host for all users to access list. Remove host for all users from access list. Add user for hosts and to access list. Add user to access list for all hosts. Remove users and from access list for all hosts. Create new access list. AUTHOR
was developed by HP. FILES
asecure log pathname ASF pathname SEE ALSO
attributes(1), convert(1), send_sound(1), asecure(1M), aserver(1M), audio(5). OBSOLETED asecure(1M)
All times are GMT -4. The time now is 07:36 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy