How to Remove duplicate value from file?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to Remove duplicate value from file?
Prev   Next
# 7  
Old 02-18-2013
This will sort your Branch codes alphabetically - not sure if you'd care about that...

Code:
crabshack:foo toki$ cat foo.txt
BI M IT056868 UNICREDI VIA LIBHIA UNCRIT2B XXX UNCR
BI M US001165 NEUBERGE LLC NEWYRK NEUBUS33 XXX NEUB
HH M IND90909 SBILIFES HNI NANANN GGGGGGGG UICC
HH M MNOOOOO 98989089 IIC UMNKSS MOHAN844 XXX KLKL
HH M MNKKKKKK 90909090 JMV MNJKMN MOHAN844 256 LOPD
HH M MKLJHJKK KIKIKIKI JKJ NMHlMM MOHAN844 456 LOPS
HH M IND90909 SBILIFES HNI NANAAN MSSMSSM 123 UIHH
HH M IND90909 SBILIFES HNI NAANAN MSSSMSM 234 UIHH
HH M IND90909 SBILIFES HNI NANANAN MSSSMSM 543 UIHH
crabshack:foo toki$ cat foo.sh
#!/bin/bash

file=foo.txt
unique_branch=$( awk '{print $7}' ${file} | sort | uniq )

for branch in ${unique_branch}; do
	grep_output=$( grep " ${branch} XXX " ${file} )
	if [ "$?" -eq "0" ]; then
		echo "${grep_output}"
	else
		grep " ${branch} " ${file}
	fi
done

exit 0
crabshack:foo toki$ ./foo.sh
HH M IND90909 SBILIFES HNI NANANN GGGGGGGG UICC
HH M MNOOOOO 98989089 IIC UMNKSS MOHAN844 XXX KLKL
HH M IND90909 SBILIFES HNI NANAAN MSSMSSM 123 UIHH
HH M IND90909 SBILIFES HNI NAANAN MSSSMSM 234 UIHH
HH M IND90909 SBILIFES HNI NANANAN MSSSMSM 543 UIHH
BI M US001165 NEUBERGE LLC NEWYRK NEUBUS33 XXX NEUB
BI M IT056868 UNICREDI VIA LIBHIA UNCRIT2B XXX UNCR

 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

2. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

3. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

4. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

5. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

6. Shell Programming and Scripting

Formatting a file - Remove Duplicate

Hi I have a file in the following format. Basically the file contains tablename and their aliases: TABLE1 TABLE1 A TABLE2 TABLE2 B TABLE3 TABLE4 TABLE4 C TABLE4 Upon formatting an sql statement I am getting such output. Problem: Whenever a tablename appears with alias, it has... (5 Replies)
Discussion started by: freakygs
5 Replies

7. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

8. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

9. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
Login or Register to Ask a Question
DH_INSTALLDEB(1)						     Debhelper							  DH_INSTALLDEB(1)

NAME
dh_installdeb - install files into the DEBIAN directory SYNOPSIS
dh_installdeb [debhelperoptions] DESCRIPTION
dh_installdeb is a debhelper program that is responsible for installing files into the DEBIAN directories in package build directories with the correct permissions. FILES
package.postinst package.preinst package.postrm package.prerm These maintainer scripts are installed into the DEBIAN directory. Inside the scripts, the token #DEBHELPER# is replaced with shell script snippets generated by other debhelper commands. package.triggers package.shlibs These control files are installed into the DEBIAN directory. Note that package.shlibs is only installed in compat level 9 and earlier. In compat 10, please use dh_makeshlibs(1). package.conffiles This control file will be installed into the DEBIAN directory. In v3 compatibility mode and higher, all files in the etc/ directory in a package will automatically be flagged as conffiles by this program, so there is no need to list them manually here. package.maintscript Lines in this file correspond to dpkg-maintscript-helper(1) commands and parameters. However, the "maint-script-parameters" should not be included as debhelper will add those automatically. Example: # Correct rm_conffile /etc/obsolete.conf 0.2~ foo # INCORRECT rm_conffile /etc/obsolete.conf 0.2~ foo -- "$@" In compat 10 or later, any shell metacharacters will be escaped, so arbitrary shell code cannot be inserted here. For example, a line such as "mv_conffile /etc/oldconffile /etc/newconffile" will insert maintainer script snippets into all maintainer scripts sufficient to move that conffile. It was also the intention to escape shell metacharacters in previous compat levels. However, it did not work properly and as such it was possible to embed arbitrary shell code in earlier compat levels. The dh_installdeb tool will do some basic validation of some of the commands listed in this file to catch common mistakes. The validation is enabled as a warning since compat 10 and as a hard error in compat 12. SEE ALSO
debhelper(7) This program is a part of debhelper. AUTHOR
Joey Hess <joeyh@debian.org> 11.1.6ubuntu2 2018-05-10 DH_INSTALLDEB(1)