08-04-2009
question about "uniq"
I'm sure this is very basic, but I'm just a beginner, so please be kind. :-)
I think I understand how to "sort" a text file, and how to save the unique lines to a new file:
sort file.txt | uniq > newfile.txt
Now, I want to do something similar, but only the first few entries of the line must be unique in order to be saved to the new file.
I have a text file with data that looks like this:
1.2 2.1 0 1 2 3 4 5 6 7 8 9 11 10
1.2 2.1 0 1 2 3 4 5 6 7 8 10 11 9
1.2 2.1 0 1 2 3 4 5 6 7 8 10 9 11
1.3 3.1 0 1 2 3 4 5 6 7 8 11 10 9
1.2 2.1 0 1 2 3 4 5 6 7 9 8 10 11
1.2 2.1 0 1 2 3 4 5 6 7 9 11 10 8
1.3 3.1 0 1 2 3 4 5 6 7 10 11 8 9
1.3 3.1 1.5 0 1 2 3 4 5 6 7 10 9 8 11
1.4 4.1 0 1 2 3 4 5 6 7 11 10 9 8
It's a series of decimal numbers (1.2 2.1, though there are not necessarily only two decimal numbers per line) followed by 12 integers, always beginning with 0.
What I want to do is to sort all of these (easy!), and then keep only the lines whose series of DECIMAL number series is unique. From this list, there are only two entries here to keep: the line beginning with 1.3 3.1 1.5 (second to last) and the line beginning with 1.4 4.1 (last line).
Do any of you know how to do this? I can unix "sort" the lines, but then to eliminate lines that have partial duplication as I've described above?
Thank you VERY much for your help!
Zac
9 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi Friends,
Can any of you explain me about the below line of code?
mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`
Im not able to understand, what exactly it is doing :confused:
Any help would be useful for me.
Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies
2. UNIX and Linux Applications
hi,
I have
* an IBM P550 machine,
* an AIX 5.3 running on it and
* an oracle database, already installed on it.
The problem (or question of my own) is:
Oracle tns listener, "CT_LISTENER", and the enterprise manager (EM) of the instance, which is uniq instance and called... (0 Replies)
Discussion started by: talipk
0 Replies
3. UNIX for Advanced & Expert Users
hi,
I have a problem about the Oracle related components. I'm not able to find any answer yet, and waiting for your responses...
Here is the configuration of my system:
* an IBM P550 machine,
* an AIX 5.3 running on it and
* an oracle database, already installed on it.
The problem (or... (1 Reply)
Discussion started by: talipk
1 Replies
4. UNIX for Dummies Questions & Answers
Dear all,
It's not entirely clear to me from manpage the difference between them.
Why we still need "-u" flag?
- monkfan (3 Replies)
Discussion started by: monkfan
3 Replies
5. Shell Programming and Scripting
Hi,
I have line in input file as below:
3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL
My expected output for line in the file must be :
"1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL"
Can someone... (7 Replies)
Discussion started by: shis100
7 Replies
6. Shell Programming and Scripting
Say I have test.txt with the following data:
user1 mailbox11 IP1
user1 mailbox12 IP2
user2 mailbox21 IP1
user3 mailbox31 IP1
user1 mailbox11 IP1
user1 mailbox11 IP1
user1 mailbox11 IP1
user1 mailbox12 IP2
user2 mailbox21 IP1
user2 mailbox21 IP1
user2 mailbox21 IP1
user2 mailbox21 IP1... (4 Replies)
Discussion started by: striker4o
4 Replies
7. UNIX for Dummies Questions & Answers
How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address
and column 3 contains “cc” e-mail address to include with same email.
Sample input file, email.txt
Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies
8. Shell Programming and Scripting
Hello.
System : opensuse leap 42.3
I have a bash script that build a text file.
I would like the last command doing :
print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt
where :
print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies
9. AIX
Hi 2 all,
i have had AIX 7.2
:/# /usr/IBMAHS/bin/apachectl -v
Server version: Apache/2.4.12 (Unix)
Server built: May 25 2015 04:58:27
:/#:/# /usr/IBMAHS/bin/apachectl -M
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
mpm_worker_module (static)
... (3 Replies)
Discussion started by: penchev
3 Replies
uniq(1) General Commands Manual uniq(1)
NAME
uniq - Removes or lists repeated lines in a file
SYNOPSIS
Current Syntax
uniq [-cdu] [-f fields] [-s chars] [input-file [output-file]]
Obsolescent Syntax
uniq [-cdu] [-fields] [+chars] [input-file [output-file]]
The uniq command reads from the specified input_file, compares adjacent lines, removes the second and succeeding occurrences of a line, and
writes to standard output.
STANDARDS
Interfaces documented on this reference page conform to industry standards as follows:
uniq: XCU5.0
Refer to the standards(5) reference page for more information about industry standards and associated tags.
OPTIONS
Precedes each output line with a count of the number of times each line appears in the file. This option supersedes the -d and -u options.
Displays repeated lines only. Ignores the first fields fields on each input line when doing comparisons, where fields is a positive deci-
mal integer. A field is the maximal string matched by the basic regular expression:
[[:blank:]]*[^[:blank:]]*
If the fields argument specifies more fields than appear on an input line, a null string is used for comparisons. Ignores the spec-
ified number of characters when doing comparisons. The chars argument is a positive decimal integer.
If specified with the -f option, the first chars characters after the first fields fields are ignored. If the chars argument speci-
fies more characters than remain on an input line, uniq uses a null string for comparison. Displays unique lines only. Equivalent
to -f fields. (Obsolescent) Equivalent to -s chars. (Obsolescent)
OPERANDS
A pathname for the input file.
If this operand is omitted or specified as -, then standard input is read. A pathname for the output file.
If this operand is omitted, then standard output is written.
DESCRIPTION
The input_file and output_file arguments must be different files. If the input_file operand is not specified, or if it is -, uniq uses
standard input.
Repeated lines must be on consecutive lines to be found. You can arrange them with the sort command before processing.
EXAMPLES
To delete repeated lines in the following file called fruit and save it to a file named newfruit, enter: uniq fruit newfruit
The file fruit contains the following lines:
apples apples bananas cherries cherries peaches pears
The file newfruit contains the following lines:
apples bananas cherries peaches pears
EXIT STATUS
The following exit values are returned: Successful completion. An error occurred.
ENVIRONMENT VARIABLES
The following environment variables affect the execution of uniq: Provides a default value for the internationalization variables that are
unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari-
ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value,
overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes
of text data as characters (for example, single-byte as opposed to multibyte characters in arguments). Determines the locale for the for-
mat and contents of diagnostic messages written to standard error. Determines the location of message catalogues for the processing of
LC_MESSAGES.
SEE ALSO
Commands: comm(1), sort(1)
Standards: standards(5)
uniq(1)