08-21-2014
Removing Duplicate Rows in a file
Hello
I have a file with contents like this...
Part1 Field2 Field3 Field4 (line1)
Part2 Field2 Field3 Field4 (line2)
Part3 Field2 Field3 Field4 (line3)
Part1 Field2 Field3 Field4 (line4)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)
...
The lines are added throughout the day at different times by various programs so the listing is in the order of timestamp . At the end of the day, I want to remove the oldest values (since they are superseded). So in the example above, I want to get rid of line 1 line 2 and line 4 as there are more recent row of these Parts. Also delete the empty rows that get created during the delete of the row.
Part3 Field2 Field3 Field4 (line3)
Part4 Field2 Field3 Field4 (line5)
Part5 Field2 Field3 Field4 (line6)
Part2 Field2 Field3 Field4 (line7)
Part1 Field2 Field3 Field4 (line8)
Any help will be greatly appreciated.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi,
I am trying to remove duplicate lines from a file. For example the contents of example.txt is:
this is a test
2342
this is a test
34343
this is a test
43434
and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:
2342... (4 Replies)
Discussion started by: ocelot
4 Replies
2. Shell Programming and Scripting
hi all
can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row).
i want to find out the numbers that are repeted in the file.
eg.
123434
534
5575
4746767
347624
5575
i want 5575
please help (3 Replies)
Discussion started by: infyanurag
3 Replies
3. Shell Programming and Scripting
I have a file content like below.
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies
4. Shell Programming and Scripting
Hi,
I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted.
eg:
file1:
-----
data1 value1
data2 value2
data3 value3
file2:
-----
data1 value1
data4 value4
data5 value5
file3:
-----
data1 value1
data4 value4 (3 Replies)
Discussion started by: Sharmila_P
3 Replies
5. UNIX for Dummies Questions & Answers
Hi,
I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g.
COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT
A NEW XX22
B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies
6. HP-UX
Hi all,
I have written one shell script. The output file of this script is having sql output.
In that file, I want to extract the rows which are having multiple entries(duplicate rows).
For example, the output file will be like the following way.
... (7 Replies)
Discussion started by: raghu.iv85
7 Replies
7. Shell Programming and Scripting
notes: i am using cygwin and notepad++ only for checking this and my OS is XP.
#!/bin/bash
typeset -i totalvalue=(wc -w /cygdrive/c/cygwinfiles/database.txt)
typeset -i totallines=(wc -l /cygdrive/c/cygwinfiles/database.txt)
typeset -i columnlines=`expr $totalvalue / $totallines`
awk -F' ' -v... (5 Replies)
Discussion started by: whitecross
5 Replies
8. UNIX for Dummies Questions & Answers
Gurus,
From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column).
Ex:
Input File:
Output File:
I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies
9. Shell Programming and Scripting
Hi everybody
I have a .txt file that contains some assembly code for optimizing it i need to remove some replicated parts.
for example I have:e_li r0,-1
e_li r25,-1
e_lis r25,0000
add r31, r31 ,r0
e_li r28,-1
e_lis r28,0000
add r31, r31 ,r0
e_li r28,-1 ... (3 Replies)
Discussion started by: Behrouzx77
3 Replies
10. UNIX for Beginners Questions & Answers
How can i get the duplicates rows from a file using unix, for example i have data like
a,1
b,2
c,3
d,4
a,1
c,3
e,5
i want output to be like
a,1
c,3 (4 Replies)
Discussion started by: ggupta
4 Replies
cam(1) General Commands Manual cam(1)
NAME
cam - CPU'S AUDIO MIXER for Linux
SYNOPSIS
cam [-v x,y] [--volume x,y] [-b x,y] [--bass x,y] [-t x,y] [--treble x,y] [-s x,y] [--synth x,y] [-p x,y] [--pcm x,y] [-S x,y]
[--speaker x,y] [-l x,y] [--line x,y] [-m x,y] [--mic x,y] [-c x,y] [--cd x,y] [-M x,y] [--mixer x,y] [-a x,y] [--altpcm x,y] [-r
x,y] [--reclev x,y] [-i x,y] [--igain x,y] [-o x,y] [--ogain x,y] [-l1 x,y] [--line1 x,y] [-l2 x,y] [--line2 x,y] [-l3 x,y] [--line3
x,y] [-3d 1|0] [--3dse 1|0] [-agc 1|0] [--agc 1|0] [-h] [-?] [--help] [-get] [-save] [-getfile filename] [-savefile filename]
xcam
DESCRIPTION
This is an audio mixer for Linux. You have the choice to use it with an interface or at the command line.
OPTIONS
When using the interface you can use the following keys :
LEFT ARROW
decrease the volume of the current device.
RIGHT ARROW
increase the volume of the current device.
UP ARROW
go to previous device.
DOWN ARROW
go to next device.
L locks or unlocks :
unlocked : you can control the left and right channel
separately.
lock : left and right channel are controlled together.
R turn on/off recording mode :
red lamp : recording turned on
green lamp : recording turned off
3 turn on/off 3D Stereo Enhancement (3DSE) device :
when turned on green indicator 3D will be lit at the left side of main volume handle.
A turn on/off microphone Automatic Gain Control :
when turned on green indicator AGC will be lit at the left side of microphone volume handle.
S save the settings of ALL devices in a file ($HOME/.camrc)
G loads the saved settings and restores them.
H shows the help screen.
Q/ESC exits the program and returns to shell.
NOTE: you can use upper or lower case.
When using the command line options the interface is not shown.
The following options are available :
-v, --volume x,y
Set 'volume' to x (left) and y (right)
-b, --bass x,y
Set 'bass' to x (left) and y (right)
-t, --trebble x,y
Set 'treble' to x (left) and y (right)
-s, --synth x,y
Set 'synth' to x (left) and y (right)
-p, --pcm x,y
Set 'pcm' to x (left) and y (right)
-S, --speaker x,y
Set 'speaker' to x (left) and y (right)
-l, --line x,y
Set 'line in' to x (left) and y (right)
-m, --mic x,y
Set 'microphone' to x (left) and y (right)
-c, --cd x,y
Set 'CD in' to x (left) and y (right)
-I, --imix x,y
Set 'mixer' to x (left) and y (right)
-a, --altpcm x,y
Set 'altpcm' to x (left) and y (right)
-r, --reclev x,y
Set 'record level' to x (left) and y (right)
-i, --igain x,y
Set 'input gain' to x (left) and y (right)
-o, --ogain x,y
Set 'output gain' to x (left) and y (right)
-l1, --line1 x,y
Set 'line 1' to x (left) and y (right)
-l2, --line2 x,y
Set 'line 2' to x (left) and y (right)
-l3, --line3 x,y
Set 'line 3' to x (left) and y (right)
-3d, --3dse 1|0
Set 3D Stereo Enhancement to 1 (on) or 0 (off)
-agc, --agc 1|0
Set Automatic Gain Control for microphone to 1 (on) or 0 (off)
-get restore the saved values from $HOME/.camrc
-save save the values to $HOME/.camrc
-getfile filename
restore the saved values from file 'filename'
-savefile filename
save the values to file 'filename'
-h, -?, --help
displays help
NOTE: x and y are in range from 1 to 100. Higher or lower numbers are
ignored.
BUGS
None known as yet. If you find one, please let me know.
AUTHOR
Jan 'TWP' VANDENBERGHE <jvdbergh@uia.ua.ac.be>
cam(1)