Sponsored Content
Top Forums Shell Programming and Scripting Find duplicates in the first column of text file Post 302432822 by alister on Sunday 27th of June 2010 10:03:59 AM
Old 06-27-2010
A single-pass version (increased ram requirement since all lines of the file are stored for END use):
Code:
 awk '{a[NR]=$0; a[NR,"k"]=$1; k[$1]++} END {for (i=1; i<=NR; i++) if (k[a[i,"k"]] > 1) print a[i]}' data

Regards,
Alister
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find the number of column in the text file...?

Hi, i have text file with ~ seperated columns. it is very huge size of file, in the file sompulsary supposed to has 20 columns with ~ seperated. so how can i find if the file has 20 column in the all rows...? Sample file: APA+VU~10~~~~~03~101~101~~~APA.N O 20081017 120.00... (1 Reply)
Discussion started by: psiva_arul
1 Replies

2. UNIX for Dummies Questions & Answers

Remove duplicates based on a column in fixed width file

Hi, How to output the duplicate record to another file. We say the record is duplicate based on a column whose position is from 2 and its length is 11 characters. The file is a fixed width file. ex of Record: DTYU12333567opert tjhi kkklTRG9012 The data in bold is the key on which... (1 Reply)
Discussion started by: Qwerty123
1 Replies

3. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ... (13 Replies)
Discussion started by: script_op2a
13 Replies

4. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

5. Red Hat

How to find a garbage entry in a column wise text file in Linux?

Suppose I have a file containing :- 1 Apple $50 2 Orange $30 3 Banana $10 4 Guava $25 5 Pine@apple $12 6 Strawberry $21 7 Grapes $12 In the 5th row, @ character inserted. I want through sort command or by any other way this row should either on top or bottom. By sort command garbage... (1 Reply)
Discussion started by: Dipankar Mitra
1 Replies

6. Shell Programming and Scripting

Find duplicates in column 1 and merge their lines (awk?)

Hi, I have a file (sorted by sort) with 8 tab delimited columns. The first column contains duplicated fields and I need to merge all these identical lines. My input file: comp100002 aaa bbb ccc ddd eee fff ggg comp100003 aba aba aba aba aba aba aba comp100003 fff fff fff fff fff fff fff... (5 Replies)
Discussion started by: falcox
5 Replies

7. Shell Programming and Scripting

Find duplicates in 2 & 3rd column and their ID

with below given format, I have been trying to find out all IDs for those entries with duplicate names in 2nd and 3rd columns and their count like how many time duplication happened for any name if any, 0.237788 Aaban Aahva 0.291066 Aabheer Aahlaad 0.845814 Aabid Aahan 0.152208 Aadam... (6 Replies)
Discussion started by: busyboy
6 Replies

8. Shell Programming and Scripting

awk to Sum columns when other column has duplicates and append one column value to another with Care

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (1 Reply)
Discussion started by: as7951
1 Replies

9. UNIX for Beginners Questions & Answers

Find duplicates in file with line numbers

Hello All, This is a noob question. I tried searching for the answer but the answer found did not help me . I have a file that can have duplicates. 100 200 300 400 100 150 the number 100 is duplicated twice. I want to find the duplicate along with the line number. expected... (4 Replies)
Discussion started by: vatigers
4 Replies
libapache2-mod-perl2-2.0.7::docs::api::ModPerl::Global(3User Contributed Perl Documentalibapache2-mod-perl2-2.0.7::docs::api::ModPerl::Global(3pm)

NAME
ModPerl::Global -- Perl API for manipulating special Perl lists Synopsis use ModPerl::Global (); my $package = 'MyApache2::Package'; # make the END blocks of this package special ModPerl::Global::special_list_register(END => $package); # Execute all encoutered END blocks from this package now ModPerl::Global::special_list_call( END => $package); # delete the list of END blocks ModPerl::Global::special_list_clear( END => $package); Description "ModPerl::Global" provides an API to manipulate special perl lists. At the moment only the "END" blocks list is supported. This API allows you to change the normal Perl behavior, and execute special lists when you need to. For example "ModPerl::RegistryCooker" uses it to run "END" blocks in the scripts at the end of each request. Before loading a module containing package $package, you need to register it, so the special blocks will be intercepted by mod_perl and not given to Perl. "special_list_register" does that. Later on when you want to execute the special blocks, "special_list_call" should be called. Unless you want to call the list more than once, clear the list with "special_list_clear". API
"ModPerl::Global" provides the following methods: "special_list_call" Call the special list $ok = special_list_call($key => $package); arg1: $key ( string ) The name of the special list. At the moment only 'END' is supported. arg2: $package ( string ) what package to special list is for ret: $ok ( boolean ) true value if $key is known, false otherwise. since: 2.0.00 "special_list_clear" Clear the special list $ok = special_list_clear($key => $package); arg1: $key ( string ) The name of the special list. At the moment only 'END' is supported. arg2: $package ( string ) what package to special list is for ret: $ok ( boolean ) true value if $key is known, false otherwise. since: 2.0.00 "special_list_register" Register the special list $ok = special_list_call($key => $package); arg1: $key ( string ) The name of the special list. At the moment only 'END' is supported. arg2: $package ( string ) what package to special list is for ret: $ok ( boolean ) true value if $key is known, false otherwise. since: 2.0.00 Notice that you need to register the package before it is loaded. If you register it after, Perl has already compiled the "END" blocks and there are no longer under your control. See Also mod_perl 2.0 documentation. Copyright mod_perl 2.0 and its core modules are copyrighted under The Apache Software License, Version 2.0. Authors The mod_perl development team and numerous contributors. perl v5.14.2 2011-02-08 libapache2-mod-perl2-2.0.7::docs::api::ModPerl::Global(3pm)
All times are GMT -4. The time now is 04:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy