Remove lines with duplicate first field


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines with duplicate first field
# 8  
Old 03-18-2012
Code:
awk 'END{print RS} p==$1 && !/^2012/{i++; next} i{print " (" i ")"; i=0} NR>1{print RS}{p=$1}1' ORS= infile


Last edited by Scrutinizer; 03-18-2012 at 01:12 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

2. UNIX for Dummies Questions & Answers

Using awk to remove duplicate line if field is empty

Hi all, I've got a file that has 12 fields. I've merged 2 files and there will be some duplicates in the following: FILE: 1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100 2. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, (EMPTY) 3. CDC, 54321, TEST3,... (4 Replies)
Discussion started by: tugar
4 Replies

3. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

4. UNIX for Dummies Questions & Answers

awk to sum column field from duplicate row/lines

Hello, I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file Name m/c usage abc xxx 10 abc xxx 20 abc xxx 5 xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies

5. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

6. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

7. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

8. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

9. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
Login or Register to Ask a Question
Alien::Package::Rpm(3pm)				User Contributed Perl Documentation				  Alien::Package::Rpm(3pm)

NAME
Alien::Package::Rpm - an object that represents a rpm package DESCRIPTION
This is an object class that represents a rpm package. It is derived from Alien::Package. FIELDS
prefixes Relocatable rpm packages have a prefixes field. METHODS
checkfile Detect rpm files by their extention. install Install a rpm. If RPMINSTALLOPT is set in the environement, the options in it are passed to rpm on its command line. scan Implement the scan method to read a rpm file. unpack Implement the unpack method to unpack a rpm file. This is a little nasty because it has to handle relocatable rpms and has to do a bit of permissions fixing as well. prep Prepare for package building by generating the spec file. cleantree Delete the spec file. build Build a rpm. If RPMBUILDOPT is set in the environement, the options in it are passed to rpm on its command line. An optional parameter, if passed, can be used to specify the program to use to build the rpm. It defaults to rpmbuild. version Set/get version. When retreiving the version, remove any dashes in it. postinst postrm preinst prerm Set/get script fields. When retrieving a value, we have to do some truely sick mangling. Since debian/slackware scripts can be anything -- perl programs or binary files -- and rpm is limited to only shell scripts, we need to encode the files and add a scrap of shell script to make it unextract and run on the fly. When setting a value, we do some mangling too. Rpm maintainer scripts are typically shell scripts, but often lack the leading shebang line. This can confuse dpkg, so add the shebang if it looks like there is no shebang magic already in place. Additionally, it's not uncommon for rpm maintainer scripts to contain bashisms, which can be triggered when they are ran on systems where /bin/sh is not bash. To work around this, the shebang line of the scripts is changed to use bash. Also, if the rpm is relocatable, the script could refer to RPM_INSTALL_PREFIX, which is set by rpm at run time. Deal with this by adding code to the script to set RPM_INSTALL_PREFIX. arch Set/get arch field. When the arch field is set, some sanitizing is done first to convert it to the debian format used internally. When it's retreived it's converted back to rpm form from the internal form. AUTHOR
Joey Hess <joey@kitenet.net> perl v5.12.3 2011-06-11 Alien::Package::Rpm(3pm)