Sponsored Content
Full Discussion: question about "uniq"
Top Forums UNIX for Dummies Questions & Answers question about "uniq" Post 302340946 by zac100 on Tuesday 4th of August 2009 05:50:56 PM
Old 08-04-2009
question about "uniq"

I'm sure this is very basic, but I'm just a beginner, so please be kind. :-)

I think I understand how to "sort" a text file, and how to save the unique lines to a new file:

sort file.txt | uniq > newfile.txt

Now, I want to do something similar, but only the first few entries of the line must be unique in order to be saved to the new file.

I have a text file with data that looks like this:
1.2 2.1 0 1 2 3 4 5 6 7 8 9 11 10
1.2 2.1 0 1 2 3 4 5 6 7 8 10 11 9
1.2 2.1 0 1 2 3 4 5 6 7 8 10 9 11
1.3 3.1 0 1 2 3 4 5 6 7 8 11 10 9
1.2 2.1 0 1 2 3 4 5 6 7 9 8 10 11
1.2 2.1 0 1 2 3 4 5 6 7 9 11 10 8
1.3 3.1 0 1 2 3 4 5 6 7 10 11 8 9
1.3 3.1 1.5 0 1 2 3 4 5 6 7 10 9 8 11
1.4 4.1 0 1 2 3 4 5 6 7 11 10 9 8

It's a series of decimal numbers (1.2 2.1, though there are not necessarily only two decimal numbers per line) followed by 12 integers, always beginning with 0.

What I want to do is to sort all of these (easy!), and then keep only the lines whose series of DECIMAL number series is unique. From this list, there are only two entries here to keep: the line beginning with 1.3 3.1 1.5 (second to last) and the line beginning with 1.4 4.1 (last line).

Do any of you know how to do this? I can unix "sort" the lines, but then to eliminate lines that have partial duplication as I've described above?

Thank you VERY much for your help!
Zac
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. UNIX and Linux Applications

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or question of my own) is: Oracle tns listener, "CT_LISTENER", and the enterprise manager (EM) of the instance, which is uniq instance and called... (0 Replies)
Discussion started by: talipk
0 Replies

3. UNIX for Advanced & Expert Users

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have a problem about the Oracle related components. I'm not able to find any answer yet, and waiting for your responses... Here is the configuration of my system: * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or... (1 Reply)
Discussion started by: talipk
1 Replies

4. UNIX for Dummies Questions & Answers

Difference between plain "uniq" and "uniq -u"

Dear all, It's not entirely clear to me from manpage the difference between them. Why we still need "-u" flag? - monkfan (3 Replies)
Discussion started by: monkfan
3 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

Show "uniq -c" results only for more than X occurrences

Say I have test.txt with the following data: user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user3 mailbox31 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1... (4 Replies)
Discussion started by: striker4o
4 Replies

7. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

8. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

9. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies
UNIQ(1) 						      General Commands Manual							   UNIQ(1)

NAME
uniq - report repeated lines in a file SYNOPSIS
uniq [ -udc [ +n ] [ -n ] ] [ input [ output ] ] DESCRIPTION
Uniq reads the input file comparing adjacent lines. In the normal case, the second and succeeding copies of repeated lines are removed; the remainder is written on the output file. Note that repeated lines must be adjacent in order to be found; see sort(1). If the -u flag is used, just the lines that are not repeated in the original file are output. The -d option specifies that one copy of just the repeated lines is to be written. The normal mode output is the union of the -u and -d mode outputs. The -c option supersedes -u and -d and generates an output report in default style but with each line preceded by a count of the number of times it occurred. The n arguments specify skipping an initial portion of each line in the comparison: -n The first n fields together with any blanks before each are ignored. A field is defined as a string of non-space, non-tab charac- ters separated by tabs and spaces from its neighbors. +n The first n characters are ignored. Fields are skipped before characters. SEE ALSO
sort(1), comm(1) UNIQ(1)
All times are GMT -4. The time now is 04:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy