11-26-2014
Thanks Zaxxon...
Works perfect and runs very fast on a file with 6M lines!!!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.
FileA
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
2. UNIX for Dummies Questions & Answers
I have a log file "logreport" that contains several lines as seen below:
04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies
3. UNIX for Dummies Questions & Answers
Hi,
I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g.
COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT
A NEW XX22
B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies
4. UNIX for Dummies Questions & Answers
Hii Friends.. I have a huge set of data stored in a file.Which is as shown below
a.dat:
RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL
LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ... (3 Replies)
Discussion started by: reva
3 Replies
5. UNIX for Dummies Questions & Answers
hello all,
I have an input file with four columns like this with a lot of lines
and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies
6. Shell Programming and Scripting
Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file.
I wrote the following script, but does not work. What is the problem?
The output file should only contain five lines:
Later! (5 Replies)
Discussion started by: Ernst
5 Replies
7. Shell Programming and Scripting
hi,
Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies
8. Shell Programming and Scripting
Hi,
In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'.
In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies
9. Shell Programming and Scripting
I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies
10. Shell Programming and Scripting
Hi,
I have a csv file which contains some millions of lines in it.
The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line).
I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
FMT(1) BSD General Commands Manual FMT(1)
NAME
fmt -- simple text formatter
SYNOPSIS
fmt [-Cr] [goal [maximum]] [name ...]
fmt [-Cr] [-g goal] [-m maximum] [name ...]
DESCRIPTION
fmt is a simple text formatter which reads the concatenation of input files (or standard input if none are given) and produces on standard
output a version of its input with lines as close to the goal length as possible without exceeding the maximum. The goal length defaults to
65 and the maximum to 75. The spacing at the beginning of the input lines is preserved in the output, as are blank lines and interword spac-
ing. In non raw mode, lines that look like mail headers or begin with a period are not formatted.
-C instructs fmt to center the text.
-g goal New way to set the goal length.
-m maximum New way to set the maximum length.
-r Raw mode; formats all lines and does not make exceptions for lines that start with a period or look like mail headers.
fmt is meant to format mail messages prior to sending, but may also be useful for other simple tasks. For instance, within visual mode of
the ex(1) editor (e.g., vi(1)) the command
!}fmt
will reformat a paragraph, evening the lines.
SEE ALSO
mail(1), nroff(1)
HISTORY
The fmt command appeared in 3BSD.
BUGS
The program was designed to be simple and fast - for more complex operations, the standard text processors are likely to be more appropriate.
BSD
May 29, 2007 BSD