Visit Our UNIX and Linux User Community


question about "uniq"


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers question about "uniq"
# 1  
Old 08-04-2009
question about "uniq"

I'm sure this is very basic, but I'm just a beginner, so please be kind. :-)

I think I understand how to "sort" a text file, and how to save the unique lines to a new file:

sort file.txt | uniq > newfile.txt

Now, I want to do something similar, but only the first few entries of the line must be unique in order to be saved to the new file.

I have a text file with data that looks like this:
1.2 2.1 0 1 2 3 4 5 6 7 8 9 11 10
1.2 2.1 0 1 2 3 4 5 6 7 8 10 11 9
1.2 2.1 0 1 2 3 4 5 6 7 8 10 9 11
1.3 3.1 0 1 2 3 4 5 6 7 8 11 10 9
1.2 2.1 0 1 2 3 4 5 6 7 9 8 10 11
1.2 2.1 0 1 2 3 4 5 6 7 9 11 10 8
1.3 3.1 0 1 2 3 4 5 6 7 10 11 8 9
1.3 3.1 1.5 0 1 2 3 4 5 6 7 10 9 8 11
1.4 4.1 0 1 2 3 4 5 6 7 11 10 9 8

It's a series of decimal numbers (1.2 2.1, though there are not necessarily only two decimal numbers per line) followed by 12 integers, always beginning with 0.

What I want to do is to sort all of these (easy!), and then keep only the lines whose series of DECIMAL number series is unique. From this list, there are only two entries here to keep: the line beginning with 1.3 3.1 1.5 (second to last) and the line beginning with 1.4 4.1 (last line).

Do any of you know how to do this? I can unix "sort" the lines, but then to eliminate lines that have partial duplication as I've described above?

Thank you VERY much for your help!
Zac
# 2  
Old 08-04-2009
You want to define the sort key - then ask sort to do a unique sort, with sort -u
something like:
Code:
# You don't have to sort the file using all the fileds first unless you really want to.
# but this does it
sort infile | sort -n -u -k1,1 -k2,2 -k3,3 -k4,4 > outfile
# without the extra sort
sort -n -u -k1,1 -k2,2 -k3,3 -k4,4 infile > newfile

 

Previous Thread | Next Thread
Test Your Knowledge in Computers #429
Difficulty: Medium
JavaScript is officially managed by Mozilla Foundation.
True or False?

9 More Discussions You Might Find Interesting

1. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies

2. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

3. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

4. Shell Programming and Scripting

Show "uniq -c" results only for more than X occurrences

Say I have test.txt with the following data: user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user3 mailbox31 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox11 IP1 user1 mailbox12 IP2 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1 user2 mailbox21 IP1... (4 Replies)
Discussion started by: striker4o
4 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. UNIX for Dummies Questions & Answers

Difference between plain "uniq" and "uniq -u"

Dear all, It's not entirely clear to me from manpage the difference between them. Why we still need "-u" flag? - monkfan (3 Replies)
Discussion started by: monkfan
3 Replies

7. UNIX for Advanced & Expert Users

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have a problem about the Oracle related components. I'm not able to find any answer yet, and waiting for your responses... Here is the configuration of my system: * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or... (1 Reply)
Discussion started by: talipk
1 Replies

8. UNIX and Linux Applications

A question/problem about oracle "tns listener" and "enterprise manager"

hi, I have * an IBM P550 machine, * an AIX 5.3 running on it and * an oracle database, already installed on it. The problem (or question of my own) is: Oracle tns listener, "CT_LISTENER", and the enterprise manager (EM) of the instance, which is uniq instance and called... (0 Replies)
Discussion started by: talipk
0 Replies

9. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

Featured Tech Videos