Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove the duplicate content in a file
Top Forums Shell Programming and Scripting Remove the duplicate content in a file Post 302843189 by ashokvpp on Monday 12th of August 2013 10:00:29 AM
Old 08-12-2013
Remove the duplicate content in a file
Here is the contents of test.txt

Code:
Dependencies Resolved
Changes in packages about to be updated:

ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64,

- Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code)

Dependencies Resolved
Changes in packages about to be updated:

ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64,

- Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code)

ChangeLog for: openldap-2.4.23-32.el6_4.1.x86_64
* Mon Apr 22 12:00:00 2013 Jan Synáček <jsynacek@redhat.com> 2.4.23-32.1
- fix: NSS related resource leak (#954299)

Expecting result like:

Code:
Dependencies Resolved
Changes in packages about to be updated:

ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64,

- Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code)

ChangeLog for: openldap-2.4.23-32.el6_4.1.x86_64
* Mon Apr 22 12:00:00 2013 Jan Synáček <jsynacek@redhat.com> 2.4.23-32.1
- fix: NSS related resource leak (#954299)

Thanks
Last edited by ashokvpp; 08-12-2013 at 11:37 AM..
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

4. Shell Programming and Scripting

Formatting a file - Remove Duplicate

Hi I have a file in the following format. Basically the file contains tablename and their aliases: TABLE1 TABLE1 A TABLE2 TABLE2 B TABLE3 TABLE4 TABLE4 C TABLE4 Upon formatting an sql statement I am getting such output. Problem: Whenever a tablename appears with alias, it has... (5 Replies)
Discussion started by: freakygs
5 Replies

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

6. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA (2 Replies)
Discussion started by: perl_beginner
2 Replies

7. Shell Programming and Scripting

How to Remove duplicate value from file?

if different branch code is available for same BIC code and one of the branch code is XXX.only one row will be stored and with branch code as XXX .rest of the rows for the BIC code will not be stored. for example if $7 is BIC code and $8 is branch code INPUT file are following... (9 Replies)
Discussion started by: mohan sharma
9 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

LEARN ABOUT CENTOS

grub2-file

GRUB-FILE(1)						      General Commands Manual						      GRUB-FILE(1)

NAME

       grub-file -- Check if FILE is of specified type.

SYNOPSIS

       grub-file (--is-i386-xen-pae-domu | --is-x86_64-xen-domu |
		  --is-x86-xen-dom0 | --is-x86-multiboot |
		  --is-x86-multiboot2 | --is-arm-linux | --is-arm64-linux |
		  --is-ia64-linux | --is-mips-linux | --is-mipsel-linux |
		  --is-sparc64-linux | --is-powerpc-linux | --is-x86-linux |
		  --is-x86-linux32 | --is-x86-kfreebsd | --is-i386-kfreebsd |
		  --is-x86_64-kfreebsd | --is-x86-knetbsd |
		  --is-i386-knetbsd | --is-x86_64-knetbsd | --is-i386-efi |
		  --is-x86_64-efi | --is-ia64-efi | --is-arm64-efi |
		  --is-arm-efi | --is-hibernated-hiberfil | --is-x86_64-xnu |
		  --is-i386-xnu | --is-xnu-hibr | --is-x86-bios-bootsector)
		  FILE

DESCRIPTION

       grub-file is used to check if FILE is of a specified type.

OPTIONS

       --is-i386-xen-pae-domu
	      Check if FILE can be booted as i386 PAE Xen unprivileged guest kernel

       --is-x86_64-xen-domu
	      Check if FILE can be booted as x86_64 Xen unprivileged guest kernel

       --is-x86-xen-dom0
	      Check if FILE can be used as Xen x86 privileged guest kernel

       --is-x86-multiboot
	      Check if FILE can be used as x86 multiboot kernel

       --is-x86-multiboot2
	      Check if FILE can be used as x86 multiboot2 kernel

       --is-arm-linux
	      Check if FILE is ARM Linux

       --is-arm64-linux
	      Check if FILE is ARM64 Linux

       --is-ia64-linux
	      Check if FILE is IA64 Linux

       --is-mips-linux
	      Check if FILE is MIPS Linux

       --is-mipsel-linux
	      Check if FILE is MIPSEL Linux

       --is-sparc64-linux
	      Check if FILE is SPARC64 Linux

       --is-powerpc-linux
	      Check if FILE is POWERPC Linux

       --is-x86-linux
	      Check if FILE is x86 Linux

       --is-x86-linux32
	      Check if FILE is x86 Linux supporting 32-bit protocol

       --is-x86-kfreebsd
	      Check if FILE is x86 kFreeBSD

       --is-i386-kfreebsd
	      Check if FILE is i386 kFreeBSD

       --is-x86_64-kfreebsd
	      Check if FILE is x86_64 kFreeBSD

       --is-x86-knetbsd
	      Check if FILE is x86 kNetBSD

       --is-i386-knetbsd
	      Check if FILE is i386 kNetBSD

       --is-x86_64-knetbsd
	      Check if FILE is x86_64 kNetBSD

       --is-i386-efi
	      Check if FILE is i386 EFI file

       --is-x86_64-efi
	      Check if FILE is x86_64 EFI file

       --is-ia64-efi
	      Check if FILE is IA64 EFI file

       --is-arm64-efi
	      Check if FILE is ARM64 EFI file

       --is-arm-efi
	      Check if FILE is ARM EFI file

       --is-hibernated-hiberfil
	      Check if FILE is hiberfil.sys in hibernated state

       --is-x86_64-xnu
	      Check if FILE is x86_64 XNU (Mac OS X kernel)

       --is-i386-xnu
	      Check if FILE is i386 XNU (Mac OS X kernel)

       --is-xnu-hibr
	      Check if FILE is XNU (Mac OS X kernel) hibernated image

       --is-x86-bios-bootsector
	      Check if FILE is BIOS bootsector

SEE ALSO

       info grub

								  Web Feb 26 2014						      GRUB-FILE(1)
All times are GMT -4. The time now is 02:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy