Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Removing Duplicate Rows in a file
Top Forums Shell Programming and Scripting Removing Duplicate Rows in a file Post 302914062 by ekbaazigar on Thursday 21st of August 2014 03:26:43 PM
Old 08-21-2014
Removing Duplicate Rows in a file
Hello

I have a file with contents like this...

Part1 Field2 Field3 Field4 (line1)
Part2 Field2 Field3 Field4 (line2)
Part3 Field2 Field3 Field4 (line3)
Part1 Field2 Field3 Field4 (line4)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)
...

The lines are added throughout the day at different times by various programs so the listing is in the order of timestamp . At the end of the day, I want to remove the oldest values (since they are superseded). So in the example above, I want to get rid of line 1 line 2 and line 4 as there are more recent row of these Parts. Also delete the empty rows that get created during the delete of the row.

Part3 Field2 Field3 Field4 (line3)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)

Any help will be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

2. Shell Programming and Scripting

duplicate rows in a file

hi all can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row). i want to find out the numbers that are repeted in the file. eg. 123434 534 5575 4746767 347624 5575 i want 5575 please help (3 Replies)
Discussion started by: infyanurag
3 Replies

3. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

4. Shell Programming and Scripting

removing the duplicate lines in a file

Hi, I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted. eg: file1: ----- data1 value1 data2 value2 data3 value3 file2: ----- data1 value1 data4 value4 data5 value5 file3: ----- data1 value1 data4 value4 (3 Replies)
Discussion started by: Sharmila_P
3 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

6. HP-UX

How to get Duplicate rows in a file

Hi all, I have written one shell script. The output file of this script is having sql output. In that file, I want to extract the rows which are having multiple entries(duplicate rows). For example, the output file will be like the following way. ... (7 Replies)
Discussion started by: raghu.iv85
7 Replies

7. Shell Programming and Scripting

Duplicate rows in a text file

notes: i am using cygwin and notepad++ only for checking this and my OS is XP. #!/bin/bash typeset -i totalvalue=(wc -w /cygdrive/c/cygwinfiles/database.txt) typeset -i totallines=(wc -l /cygdrive/c/cygwinfiles/database.txt) typeset -i columnlines=`expr $totalvalue / $totallines` awk -F' ' -v... (5 Replies)
Discussion started by: whitecross
5 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicate rows & selecting only latest date

Gurus, From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column). Ex: Input File: Output File: I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies

9. Shell Programming and Scripting

Removing duplicate terms in a file

Hi everybody I have a .txt file that contains some assembly code for optimizing it i need to remove some replicated parts. for example I have:e_li r0,-1 e_li r25,-1 e_lis r25,0000 add r31, r31 ,r0 e_li r28,-1 e_lis r28,0000 add r31, r31 ,r0 e_li r28,-1 ... (3 Replies)
Discussion started by: Behrouzx77
3 Replies

10. UNIX for Beginners Questions & Answers

Get duplicate rows from a csv file

How can i get the duplicates rows from a file using unix, for example i have data like a,1 b,2 c,3 d,4 a,1 c,3 e,5 i want output to be like a,1 c,3 (4 Replies)
Discussion started by: ggupta
4 Replies

LEARN ABOUT HPUX

vxdestroy_lvmroot

vxdestroy_lvmroot(1M)													     vxdestroy_lvmroot(1M)

NAME

       vxdestroy_lvmroot - remove LVM root disk and associated LVM volume group

SYNOPSIS

       vxdestroy_lvmroot [-v] [-b] lvm_root_disk

DESCRIPTION

       The  vxdestroy_lvmroot  command	tears  down  and removes the LVM volume group associated with the specified physical disk. This is done by
       removing the volumes in the volume group and then removing the volume group.

OPTIONS

       -b	 Invokes the setboot command to change the primary boot disk to the VxVM root disk from which the system was booted (if  the  pri-
		 mary  boot  disk is not already set to this disk).  The alternate boot device remains unchanged.  If the -v option is also speci-
		 fied, the settings of the primary and alternate boot devices are displayed.

       -v	 Outputs verbose messages including a timestamp that indicates major operations being performed.

ARGUMENTS

       lvm_root_disk
	    Specifies the device name of the LVM root disk that is to be destroyed, along with its entire volume group.

EXAMPLES

       This example shows the vxdestroy_lvmroot command invoked in its simpliest form:

       /etc/vx/bin/vxdestroy_lvmroot c5t1d0

       # /etc/vx/bin/vxdestroy_lvmroot -v -b c3t0d0
       Are you sure about destroying c3t0d0 (imported on VG /dev/vg02) ? YES
       vxdestroy_lvmroot 06:02: Tearing down /dev/vg02 on device c3t0d0
       vxdestroy_lvmroot 06:02:
	 Unmounting and removing any volumes associated with /dev/vg02
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol1
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol2
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol3
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol4
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol5
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol6
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol7
       vxdestroy_lvmroot 06:02: removing logical volume /dev/vg02/lvol8
       vxdestroy_lvmroot 06:02: Removing Volume Group /dev/vg02
       vxdestroy_lvmroot 06:02: Removing device files in /dev/vg02
       vxdestroy_lvmroot 06:02: Removing LVM Physical Volume c3t0d0
       vxdestroy_lvmroot 06:02: Removing LVM Physical Volume c1t1d3
       vxdestroy_lvmroot 06:02: Removing LVM Physical Volume c1t1d4
       vxdestroy_lvmroot 06:02: Removing LVM Physical Volume c1t1d5
       vxdestroy_lvmroot 06:03: Current setboot values:
       vxdestroy_lvmroot 06:03: Primary:       0/0/6/0/0.0.0
       vxdestroy_lvmroot 06:03: Alternate:     0/0/1/0/0.1.0
       vxdestroy_lvmroot 06:03:
	 Making disk c0t1d0 (0/0/1/0/0.1.0) the primary boot disk
       vxdestroy_lvmroot 06:03:
	 Removal of device c3t0d0, volume group /dev/vg02 was successful

SEE ALSO

       cpio(1), dd(1), fsck(1M), setboot(1M) vxbootsetup(1M) vxcp_lvmroot(1M) vxres_lvmroot(1M)

VxVM 5.0.31.1							    24 Mar 2008 					     vxdestroy_lvmroot(1M)
All times are GMT -4. The time now is 04:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy