Sponsored Content
Top Forums Shell Programming and Scripting duplicate line in a text file Post 302190184 by summer_cherry on Tuesday 29th of April 2008 02:19:21 AM
Old 04-29-2008
Code:
awk '{
  a[$0]++
}
END{
 for (i in a)
 if (a[i]=2)
 print i
}' filename

 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

2. Shell Programming and Scripting

Eleminating Duplicate IPs from a text file

Hey Guys I need to eleminate duplicate IP's from a text file using bash.Any suggestions.Appreciate your help guys. --CoolKid (4 Replies)
Discussion started by: coolkid
4 Replies

3. Shell Programming and Scripting

concatenate all duplicate line in a file.

Hi All, i have a zip file like the format 794051400123|COM|24|0|BD|R|99.98 794051413727|COM|11|0|BD|R|28.99 794051415622|COM|23|0|BD|R|28.99 883929004676|COM|0|0|BD|R|28.99 794051400123|MOM|62|0|BD|R|99.98 794051413727|MOM|4|0|BD|R|28.99 794051415622|MOM|80|0|BD|R|28.99 ... (30 Replies)
Discussion started by: vaskarbasak
30 Replies

4. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

5. Shell Programming and Scripting

Duplicate rows in a text file

notes: i am using cygwin and notepad++ only for checking this and my OS is XP. #!/bin/bash typeset -i totalvalue=(wc -w /cygdrive/c/cygwinfiles/database.txt) typeset -i totallines=(wc -l /cygdrive/c/cygwinfiles/database.txt) typeset -i columnlines=`expr $totalvalue / $totallines` awk -F' ' -v... (5 Replies)
Discussion started by: whitecross
5 Replies

6. Shell Programming and Scripting

Duplicate each field in a text file

Hello all, I am searching for a solution to the following problem: Given input such as this: I would like to find a way to output this: Thanks in advance! (4 Replies)
Discussion started by: hydrabane
4 Replies

7. Shell Programming and Scripting

Delete Duplicate line (not really) from the file

I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size). Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies

8. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

9. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

10. Shell Programming and Scripting

Removing duplicate sequences and modifying a text file

Hi. I've tried several different programs to try and solve this problem, but none of them seem to have done exactly what I want (and I need the file in a very specific format). I have a large file of DNA sequences in a multifasta file like this, with around 15 000 genes: ... (2 Replies)
Discussion started by: 4galaxy7
2 Replies
cam(1)							      General Commands Manual							    cam(1)

NAME
cam - CPU'S AUDIO MIXER for Linux SYNOPSIS
cam [-v x,y] [--volume x,y] [-b x,y] [--bass x,y] [-t x,y] [--treble x,y] [-s x,y] [--synth x,y] [-p x,y] [--pcm x,y] [-S x,y] [--speaker x,y] [-l x,y] [--line x,y] [-m x,y] [--mic x,y] [-c x,y] [--cd x,y] [-M x,y] [--mixer x,y] [-a x,y] [--altpcm x,y] [-r x,y] [--reclev x,y] [-i x,y] [--igain x,y] [-o x,y] [--ogain x,y] [-l1 x,y] [--line1 x,y] [-l2 x,y] [--line2 x,y] [-l3 x,y] [--line3 x,y] [-3d 1|0] [--3dse 1|0] [-agc 1|0] [--agc 1|0] [-h] [-?] [--help] [-get] [-save] [-getfile filename] [-savefile filename] xcam DESCRIPTION
This is an audio mixer for Linux. You have the choice to use it with an interface or at the command line. OPTIONS
When using the interface you can use the following keys : LEFT ARROW decrease the volume of the current device. RIGHT ARROW increase the volume of the current device. UP ARROW go to previous device. DOWN ARROW go to next device. L locks or unlocks : unlocked : you can control the left and right channel separately. lock : left and right channel are controlled together. R turn on/off recording mode : red lamp : recording turned on green lamp : recording turned off 3 turn on/off 3D Stereo Enhancement (3DSE) device : when turned on green indicator 3D will be lit at the left side of main volume handle. A turn on/off microphone Automatic Gain Control : when turned on green indicator AGC will be lit at the left side of microphone volume handle. S save the settings of ALL devices in a file ($HOME/.camrc) G loads the saved settings and restores them. H shows the help screen. Q/ESC exits the program and returns to shell. NOTE: you can use upper or lower case. When using the command line options the interface is not shown. The following options are available : -v, --volume x,y Set 'volume' to x (left) and y (right) -b, --bass x,y Set 'bass' to x (left) and y (right) -t, --trebble x,y Set 'treble' to x (left) and y (right) -s, --synth x,y Set 'synth' to x (left) and y (right) -p, --pcm x,y Set 'pcm' to x (left) and y (right) -S, --speaker x,y Set 'speaker' to x (left) and y (right) -l, --line x,y Set 'line in' to x (left) and y (right) -m, --mic x,y Set 'microphone' to x (left) and y (right) -c, --cd x,y Set 'CD in' to x (left) and y (right) -I, --imix x,y Set 'mixer' to x (left) and y (right) -a, --altpcm x,y Set 'altpcm' to x (left) and y (right) -r, --reclev x,y Set 'record level' to x (left) and y (right) -i, --igain x,y Set 'input gain' to x (left) and y (right) -o, --ogain x,y Set 'output gain' to x (left) and y (right) -l1, --line1 x,y Set 'line 1' to x (left) and y (right) -l2, --line2 x,y Set 'line 2' to x (left) and y (right) -l3, --line3 x,y Set 'line 3' to x (left) and y (right) -3d, --3dse 1|0 Set 3D Stereo Enhancement to 1 (on) or 0 (off) -agc, --agc 1|0 Set Automatic Gain Control for microphone to 1 (on) or 0 (off) -get restore the saved values from $HOME/.camrc -save save the values to $HOME/.camrc -getfile filename restore the saved values from file 'filename' -savefile filename save the values to file 'filename' -h, -?, --help displays help NOTE: x and y are in range from 1 to 100. Higher or lower numbers are ignored. BUGS
None known as yet. If you find one, please let me know. AUTHOR
Jan 'TWP' VANDENBERGHE <jvdbergh@uia.ua.ac.be> cam(1)
All times are GMT -4. The time now is 04:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy