Sponsored Content
Full Discussion: Script to remove duplicates
Top Forums Shell Programming and Scripting Script to remove duplicates Post 302417091 by Franklin52 on Wednesday 28th of April 2010 01:00:19 PM
Old 04-28-2010
Try this:
Code:
awk -F\. '!a[$1]++' file

BTW: per forum rules, and the benefit of all users, please search the network and the forums before posting a question.

You can easily search the forums using our internal Google search engine or our advanced internal search engine. You can also search our huge UNIX and Linux database by user generated tags or search the database for unanswered questions and provide an answer.

Thank you.

The UNIX and Linux Forums
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

2. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies

3. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

4. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

5. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

6. UNIX for Dummies Questions & Answers

script to remove duplicates per line

Hello experts! I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file: example input (56)(63) (56)(70)(56)(70)(24) (25)(78) (12)(33)(12) (10) (10) desired output (56)(63) (56)(70)(24) (25)(78)... (5 Replies)
Discussion started by: torchij
5 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies
REINDEXDB(1)						  PostgreSQL Client Applications					      REINDEXDB(1)

NAME
reindexdb - reindex a PostgreSQL database SYNOPSIS
reindexdb [ connection-option... ] [ --table | -t table ] [ --index | -i index ] [ dbname ] reindexdb [ connection-option... ] [ --all | -a ] reindexdb [ connection-option... ] [ --system | -s ] [ dbname ] DESCRIPTION
reindexdb is a utility for rebuilding indexes in a PostgreSQL database. reindexdb is a wrapper around the SQL command REINDEX [reindex(7)]. There is no effective difference between reindexing databases via this utility and via other methods for accessing the server. OPTIONS
reindexdb accepts the following command-line arguments: -a --all Reindex all databases. [-d] dbname [--dbname] dbname Specifies the name of the database to be reindexed. If this is not specified and -a (or --all) is not used, the database name is read from the environment variable PGDATABASE. If that is not set, the user name specified for the connection is used. -e --echo Echo the commands that reindexdb generates and sends to the server. -i index --index index Recreate index only. -q --quiet Do not display progress messages. -s --system Reindex database's system catalogs. -t table --table table Reindex table only. reindexdb also accepts the following command-line arguments for connection parameters: -h host --host host Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. -p port --port port Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. -U username --username username User name to connect as. -w --no-password Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password. -W --password Force reindexdb to prompt for a password before connecting to a database. This option is never essential, since reindexdb will automatically prompt for a password if the server demands password authentica- tion. However, reindexdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt. ENVIRONMENT
PGDATABASE PGHOST PGPORT PGUSER Default connection parameters This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see in the documentation). DIAGNOSTICS
In case of difficulty, see REINDEX [reindex(7)] and psql(1) for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply. NOTES
reindexdb might need to connect several times to the PostgreSQL server, asking for a password each time. It is convenient to have a ~/.pgpass file in such cases. See in the documentation for more information. EXAMPLES
To reindex the database test: $ reindexdb test To reindex the table foo and the index bar in a database named abcd: $ reindexdb --table foo --index bar abcd SEE ALSO
REINDEX [reindex(7)] Application 2010-05-14 REINDEXDB(1)
All times are GMT -4. The time now is 10:32 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy