Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines from a 50 MB file size Post 302577582 by vsachan on Tuesday 29th of November 2011 11:14:36 AM
Old 11-29-2011
the problem is that the file size is 100 MB - 150 MB

input file:
---------
1,2, ,TTT,DDFG,
1,2, ,TTT,DDFG,
1,2, ,TTT,DDFG,
7,8, ,TTT,DDFG,
1,2, ,TTT,DDFG,
1,2, ,TTT,DDFG,

output file should be like:
1,2, ,TTT,DDFG,
7,8, ,TTT,DDFG,
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

4. UNIX for Dummies Questions & Answers

How to delete or remove duplicate lines in a file

Hi please help me how to remove duplicate lines in any file. I have a file having huge number of lines. i want to remove selected lines in it. And also if there exists duplicate lines, I want to delete the rest & just keep one of them. Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies

5. Shell Programming and Scripting

remove duplicate lines from file linux/sh

greetings, i'm hoping there is a way to cat a file, remove duplicate lines and send that output to a new file. the file will always vary but be something similar to this: please keep in mind that the above could be eight occurrences of each hostname or it might simply have another four of an... (2 Replies)
Discussion started by: crimso
2 Replies

6. Shell Programming and Scripting

Remove duplicate lines from first file comparing second file

Hi, I have two files with below data:: file1:- 123|aaa|ppp 445|fff|yyy 999|ttt|jjj 555|hhh|hhh file2:- 445|fff|yyy 555|hhh|hhh The records present in file1, not present in file 2 should be writtent to the out put file. output:- 123|aaa|ppp 999|ttt|jjj Is there any one line... (3 Replies)
Discussion started by: gani_85
3 Replies

7. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
DBAR(1) 						      General Commands Manual							   DBAR(1)

NAME
dzen2-dbar - simple gadget that you can use in combination with dzen to display a progressbar. SYNOPSIS
dzen2-dbar [-w characters] [-s symbol] [-min minvalue] [-max maxvalue] [-l string] [-nonl] DESCRIPTION
dbar is an application that generates semi graphical progress meters, from some values you supply to it. See the usage examples for a description of the expected input format. OPTIONS
A summary of options is included below. -max Value to be considered 100% (default: 100) -min Value to be considered 0% (default: 0) -w Number of charcaters to be considered 100% in the bar (default: 25 ) -s Symbol represeting the percentage value in the bar (default: = ) -l label to be prepended to the bar (default: '' ) -nonl no new line, don't put '0 at the end of the bar (default: do print '0) USAGE EXAMPLES
Static 0% and 100% mark or single value input: Command: echo 25 | dbar -m 100 -l Sometext Output: Sometext 25% [====== ] If your 100% mark changes dynamically or 2-values input: Command: echo "50 150" | dbar Output: 33% [======== ] If your value range is not between [0, maxval] or 3-values input: Command: echo "50 -25 150" | dbar Output: 43% [=========== ] Dbar was written by Robert Manea. This manual page was written by bertagaz <bertagaz@ptitcanardnoir.org>, for the Debian project (but may be used by others). Jan 01 2008 DBAR(1)
All times are GMT -4. The time now is 08:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy