Sponsored Content
Full Discussion: Uniq and sort -u
Top Forums UNIX for Dummies Questions & Answers Uniq and sort -u Post 302927647 by senhia83 on Thursday 4th of December 2014 09:47:28 AM
Old 12-04-2014
Uniq and sort -u

Hello all,

Need to pick your brains,

I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters.

So I want to do a

Code:
sort -u file

Will it be considerably faster or slower to use a uniq before piping it to sort -u ? Does consolidating all the neighbors first help in this case?

Code:
cat file | uniq | sort -u

My file looks like

Code:
name1
name1
name1
name2
name2
name2
name1
....

name50

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies

2. UNIX for Dummies Questions & Answers

Help with Last,uniq, sort and cut

Using the last, uniq, sort and cut commands, determine how many times the different users have logged in. I know how to use the last command and cut command... i came up with last | cut -f1 -d" " | uniq i dont know if this is right, can someone please help me... thanks (1 Reply)
Discussion started by: jay1228
1 Replies

3. Shell Programming and Scripting

sort and uniq in perl

Does anyone have a quick and dirty way of performing a sort and uniq in perl? How an array with data like: this is bkupArr BOLADVICE_VN this is bkupArr MLT6800PROD2A this is bkupArr MLT6800PROD2A this is bkupArr BOLADVICE_VN_7YR this is bkupArr MLT6800PROD2A I want to sort it... (4 Replies)
Discussion started by: reggiej
4 Replies

4. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script... (5 Replies)
Discussion started by: Amruta Pitkar
5 Replies

5. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies

6. Shell Programming and Scripting

sort | uniq question

Hello, I have a large data file: 1234 8888 bbb 2745 8888 bbb 9489 8888 bbb 1234 8888 aaa 4838 8888 aaa 3977 8888 aaa I need to remove duplicate lines (where the first column is the duplicate). I have been using: sort file.txt | uniq -w4 > newfile.txt However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies

7. Shell Programming and Scripting

Sort and uniq after comparision

Hi All, I have a text file with the format shown below. Some of the records are duplicated with the only exception being date (Field 15). I want to compare all duplicate records using subscriber number (field 7) and keep only those records with greater date. ... (1 Reply)
Discussion started by: nua7
1 Replies

8. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

9. Shell Programming and Scripting

Sort uniq or awk

Hi again, I have files with the following contents datetime,ip1,port1,ip2,port2,number How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up? Please mind the file may contain 100k lines. (8 Replies)
Discussion started by: LDHB2012
8 Replies

10. Shell Programming and Scripting

Sort & Uniq -u

Hi All, Below the actual file which i like to sort and Uniq -u /opt/oracle/work/Antony/Shell_Script> cat emp.1st 2233|a.k. shukula |g.m. |sales |12/12/52 |6000 1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies
clusters(4)						     Sun Cluster File Formats						       clusters(4)

NAME
clusters - cluster names database SYNOPSIS
/etc/clusters DESCRIPTION
The clusters file contains information regarding the known clusters in the local naming domain. For each cluster a single line should be present with the following information: clustername whitespace-delimited list of hosts Expansion is recursive if a name on the right hand side is tagged with the expansion marker: ``*''. Items are separated by any number of blanks and/or TAB characters. A `#' indicates the beginning of a comment. Characters up to the end of the line are not interpreted by routines which search the file. Cluster names may contain any printable character other than an upper case character, a field delimiter, NEWLINE, or comment character. The maximum length of a cluster name is 32 characters. This information is used by Sun Cluster system administration tools, like cconsole(1M) to specify a group of nodes to administer. The names used in this database must be host names, as used in the hosts database. The database is available from either NIS or NIS+ maps or a local file. Lookup order can be specified in the /etc/nsswitch.conf file. The default order is nis files. EXAMPLES
Example 1 A Sample /etc/clusters File Here is a typical /etc/clusters file: bothclusters *planets *wine planets mercury venus wine zinfandel merlot chardonnay riesling Here is a typical /etc/nsswitch.conf entry: clusters: nis files FILES
/etc/clusters /etc/nsswitch.conf ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWsczu | +-----------------------------+-----------------------------+ |Interface Stability |Uncommitted | +-----------------------------+-----------------------------+ SEE ALSO
cconsole(1M), chosts(1M), serialports(4), nsswitch.conf(4), attributes(5) Sun Cluster 3.2 26 Jun 2006 clusters(4)
All times are GMT -4. The time now is 04:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy