Sponsored Content
Full Discussion: question about "uniq"
Top Forums UNIX for Dummies Questions & Answers question about "uniq" Post 302340946 by zac100 on Tuesday 4th of August 2009 05:50:56 PM
Old 08-04-2009
question about "uniq"

I'm sure this is very basic, but I'm just a beginner, so please be kind. :-)

I think I understand how to "sort" a text file, and how to save the unique lines to a new file:

sort file.txt | uniq > newfile.txt

Now, I want to do something similar, but only the first few entries of the line must be unique in order to be saved to the new file.

I have a text file with data that looks like this:
1.2 2.1 0 1 2 3 4 5 6 7 8 9 11 10
1.2 2.1 0 1 2 3 4 5 6 7 8 10 11 9
1.2 2.1 0 1 2 3 4 5 6 7 8 10 9 11
1.3 3.1 0 1 2 3 4 5 6 7 8 11 10 9
1.2 2.1 0 1 2 3 4 5 6 7 9 8 10 11
1.2 2.1 0 1 2 3 4 5 6 7 9 11 10 8
1.3 3.1 0 1 2 3 4 5 6 7 10 11 8 9
1.3 3.1 1.5 0 1 2 3 4 5 6 7 10 9 8 11
1.4 4.1 0 1 2 3 4 5 6 7 11 10 9 8

It's a series of decimal numbers (1.2 2.1, though there are not necessarily only two decimal numbers per line) followed by 12 integers, always beginning with 0.

What I want to do is to sort all of these (easy!), and then keep only the lines whose series of DECIMAL number series is unique. From this list, there are only two entries here to keep: the line beginning with 1.3 3.1 1.5 (second to last) and the line beginning with 1.4 4.1 (last line).

Do any of you know how to do this? I can unix "sort" the lines, but then to eliminate lines that have partial duplication as I've described above?

Thank you VERY much for your help!
Zac
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. UNIX and Linux Applications

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or question of my own) is: Oracle tns listener, "CT_LISTENER", and the enterprise manager (EM) of the instance, which is uniq instance and called... (0 Replies)
Discussion started by: talipk
0 Replies

3. UNIX for Advanced & Expert Users

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have a problem about the Oracle related components. I'm not able to find any answer yet, and waiting for your responses... Here is the configuration of my system: * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or... (1 Reply)
Discussion started by: talipk
1 Replies

4. UNIX for Dummies Questions & Answers

Difference between plain "uniq" and "uniq -u"

Dear all, It's not entirely clear to me from manpage the difference between them. Why we still need "-u" flag? - monkfan (3 Replies)
Discussion started by: monkfan
3 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

Show "uniq -c" results only for more than X occurrences

Say I have test.txt with the following data: user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user3 mailbox31 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1... (4 Replies)
Discussion started by: striker4o
4 Replies

7. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

8. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

9. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies
uniq(1) 						      General Commands Manual							   uniq(1)

NAME
uniq - Removes or lists repeated lines in a file SYNOPSIS
Current Syntax uniq [-cdu] [-f fields] [-s chars] [input-file [output-file]] Obsolescent Syntax uniq [-cdu] [-fields] [+chars] [input-file [output-file]] The uniq command reads from the specified input_file, compares adjacent lines, removes the second and succeeding occurrences of a line, and writes to standard output. STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: uniq: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
Precedes each output line with a count of the number of times each line appears in the file. This option supersedes the -d and -u options. Displays repeated lines only. Ignores the first fields fields on each input line when doing comparisons, where fields is a positive deci- mal integer. A field is the maximal string matched by the basic regular expression: [[:blank:]]*[^[:blank:]]* If the fields argument specifies more fields than appear on an input line, a null string is used for comparisons. Ignores the spec- ified number of characters when doing comparisons. The chars argument is a positive decimal integer. If specified with the -f option, the first chars characters after the first fields fields are ignored. If the chars argument speci- fies more characters than remain on an input line, uniq uses a null string for comparison. Displays unique lines only. Equivalent to -f fields. (Obsolescent) Equivalent to -s chars. (Obsolescent) OPERANDS
A pathname for the input file. If this operand is omitted or specified as -, then standard input is read. A pathname for the output file. If this operand is omitted, then standard output is written. DESCRIPTION
The input_file and output_file arguments must be different files. If the input_file operand is not specified, or if it is -, uniq uses standard input. Repeated lines must be on consecutive lines to be found. You can arrange them with the sort command before processing. EXAMPLES
To delete repeated lines in the following file called fruit and save it to a file named newfruit, enter: uniq fruit newfruit The file fruit contains the following lines: apples apples bananas cherries cherries peaches pears The file newfruit contains the following lines: apples bananas cherries peaches pears EXIT STATUS
The following exit values are returned: Successful completion. An error occurred. ENVIRONMENT VARIABLES
The following environment variables affect the execution of uniq: Provides a default value for the internationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari- ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments). Determines the locale for the for- mat and contents of diagnostic messages written to standard error. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: comm(1), sort(1) Standards: standards(5) uniq(1)
All times are GMT -4. The time now is 03:25 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy