Sponsored Content
Top Forums Shell Programming and Scripting Removing Duplicate Rows in a file Post 302914062 by ekbaazigar on Thursday 21st of August 2014 03:26:43 PM
Old 08-21-2014
Removing Duplicate Rows in a file

Hello

I have a file with contents like this...

Part1 Field2 Field3 Field4 (line1)
Part2 Field2 Field3 Field4 (line2)
Part3 Field2 Field3 Field4 (line3)
Part1 Field2 Field3 Field4 (line4)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)
...

The lines are added throughout the day at different times by various programs so the listing is in the order of timestamp . At the end of the day, I want to remove the oldest values (since they are superseded). So in the example above, I want to get rid of line 1 line 2 and line 4 as there are more recent row of these Parts. Also delete the empty rows that get created during the delete of the row.

Part3 Field2 Field3 Field4 (line3)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)

Any help will be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

2. Shell Programming and Scripting

duplicate rows in a file

hi all can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row). i want to find out the numbers that are repeted in the file. eg. 123434 534 5575 4746767 347624 5575 i want 5575 please help (3 Replies)
Discussion started by: infyanurag
3 Replies

3. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

4. Shell Programming and Scripting

removing the duplicate lines in a file

Hi, I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted. eg: file1: ----- data1 value1 data2 value2 data3 value3 file2: ----- data1 value1 data4 value4 data5 value5 file3: ----- data1 value1 data4 value4 (3 Replies)
Discussion started by: Sharmila_P
3 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

6. HP-UX

How to get Duplicate rows in a file

Hi all, I have written one shell script. The output file of this script is having sql output. In that file, I want to extract the rows which are having multiple entries(duplicate rows). For example, the output file will be like the following way. ... (7 Replies)
Discussion started by: raghu.iv85
7 Replies

7. Shell Programming and Scripting

Duplicate rows in a text file

notes: i am using cygwin and notepad++ only for checking this and my OS is XP. #!/bin/bash typeset -i totalvalue=(wc -w /cygdrive/c/cygwinfiles/database.txt) typeset -i totallines=(wc -l /cygdrive/c/cygwinfiles/database.txt) typeset -i columnlines=`expr $totalvalue / $totallines` awk -F' ' -v... (5 Replies)
Discussion started by: whitecross
5 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicate rows & selecting only latest date

Gurus, From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column). Ex: Input File: Output File: I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies

9. Shell Programming and Scripting

Removing duplicate terms in a file

Hi everybody I have a .txt file that contains some assembly code for optimizing it i need to remove some replicated parts. for example I have:e_li r0,-1 e_li r25,-1 e_lis r25,0000 add r31, r31 ,r0 e_li r28,-1 e_lis r28,0000 add r31, r31 ,r0 e_li r28,-1 ... (3 Replies)
Discussion started by: Behrouzx77
3 Replies

10. UNIX for Beginners Questions & Answers

Get duplicate rows from a csv file

How can i get the duplicates rows from a file using unix, for example i have data like a,1 b,2 c,3 d,4 a,1 c,3 e,5 i want output to be like a,1 c,3 (4 Replies)
Discussion started by: ggupta
4 Replies
cam(1)							      General Commands Manual							    cam(1)

NAME
cam - CPU'S AUDIO MIXER for Linux SYNOPSIS
cam [-v x,y] [--volume x,y] [-b x,y] [--bass x,y] [-t x,y] [--treble x,y] [-s x,y] [--synth x,y] [-p x,y] [--pcm x,y] [-S x,y] [--speaker x,y] [-l x,y] [--line x,y] [-m x,y] [--mic x,y] [-c x,y] [--cd x,y] [-M x,y] [--mixer x,y] [-a x,y] [--altpcm x,y] [-r x,y] [--reclev x,y] [-i x,y] [--igain x,y] [-o x,y] [--ogain x,y] [-l1 x,y] [--line1 x,y] [-l2 x,y] [--line2 x,y] [-l3 x,y] [--line3 x,y] [-3d 1|0] [--3dse 1|0] [-agc 1|0] [--agc 1|0] [-h] [-?] [--help] [-get] [-save] [-getfile filename] [-savefile filename] xcam DESCRIPTION
This is an audio mixer for Linux. You have the choice to use it with an interface or at the command line. OPTIONS
When using the interface you can use the following keys : LEFT ARROW decrease the volume of the current device. RIGHT ARROW increase the volume of the current device. UP ARROW go to previous device. DOWN ARROW go to next device. L locks or unlocks : unlocked : you can control the left and right channel separately. lock : left and right channel are controlled together. R turn on/off recording mode : red lamp : recording turned on green lamp : recording turned off 3 turn on/off 3D Stereo Enhancement (3DSE) device : when turned on green indicator 3D will be lit at the left side of main volume handle. A turn on/off microphone Automatic Gain Control : when turned on green indicator AGC will be lit at the left side of microphone volume handle. S save the settings of ALL devices in a file ($HOME/.camrc) G loads the saved settings and restores them. H shows the help screen. Q/ESC exits the program and returns to shell. NOTE: you can use upper or lower case. When using the command line options the interface is not shown. The following options are available : -v, --volume x,y Set 'volume' to x (left) and y (right) -b, --bass x,y Set 'bass' to x (left) and y (right) -t, --trebble x,y Set 'treble' to x (left) and y (right) -s, --synth x,y Set 'synth' to x (left) and y (right) -p, --pcm x,y Set 'pcm' to x (left) and y (right) -S, --speaker x,y Set 'speaker' to x (left) and y (right) -l, --line x,y Set 'line in' to x (left) and y (right) -m, --mic x,y Set 'microphone' to x (left) and y (right) -c, --cd x,y Set 'CD in' to x (left) and y (right) -I, --imix x,y Set 'mixer' to x (left) and y (right) -a, --altpcm x,y Set 'altpcm' to x (left) and y (right) -r, --reclev x,y Set 'record level' to x (left) and y (right) -i, --igain x,y Set 'input gain' to x (left) and y (right) -o, --ogain x,y Set 'output gain' to x (left) and y (right) -l1, --line1 x,y Set 'line 1' to x (left) and y (right) -l2, --line2 x,y Set 'line 2' to x (left) and y (right) -l3, --line3 x,y Set 'line 3' to x (left) and y (right) -3d, --3dse 1|0 Set 3D Stereo Enhancement to 1 (on) or 0 (off) -agc, --agc 1|0 Set Automatic Gain Control for microphone to 1 (on) or 0 (off) -get restore the saved values from $HOME/.camrc -save save the values to $HOME/.camrc -getfile filename restore the saved values from file 'filename' -savefile filename save the values to file 'filename' -h, -?, --help displays help NOTE: x and y are in range from 1 to 100. Higher or lower numbers are ignored. BUGS
None known as yet. If you find one, please let me know. AUTHOR
Jan 'TWP' VANDENBERGHE <jvdbergh@uia.ua.ac.be> cam(1)
All times are GMT -4. The time now is 02:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy