Sponsored Content
Top Forums Shell Programming and Scripting remove all duplicate lines from all files in one folder Post 302321063 by vidyadhar85 on Saturday 30th of May 2009 02:14:03 AM
Old 05-30-2009
try using commands like sort and uniq to get rid of duplicate lines refer man pages and give it a try if not possible revert back will help you

-vidya
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

4. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

6. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
rmf(1)							      General Commands Manual							    rmf(1)

NAME
rmf - remove folder (only available within the message handling system, mh) SYNOPSIS
rmf [+folder] [-help] [-[no]interactive] OPTIONS
Prints a list of the valid options to this command. Asks for confirmation before deleting a folder. By default, rmf deletes a folder and its messages without asking for confirmation. If you specify the -interactive option, rmf asks if you are sure before deleting the folder. You are advised to use this option, since when rmf deletes a folder its contents are lost irretrievably. DESCRIPTION
The rmf command removes all of the messages within the current folder, and then removes the folder itself. If there are any files within the folder which are not part of MH, they are not removed, and an error message is displayed. You can specify a folder other than the current folder by using the +folder argument. If you do not specify a folder, and rmf cannot find the current folder,rmf asks you whether you want to delete +inbox instead. If the current folder is removed, it makes +inbox current. Note that the rmf command irreversibly deletes messages that do not have other links, so use it with caution. If the folder being removed is a sub-folder, the parent folder becomes the new current folder, and rmf tells you that this has happened. This provides an easy mechanism for selecting a set of messages, operating on the list, then removing the list and returning to the current folder from which the list was extracted. Using rmf to delete a read-only folder deletes the private sequence and current message information from the file, without affecting the folder itself. If you have sub-folders within a folder, you must delete all the sub-folders before you can delete the folder itself. PROFILE COMPONENTS
Path: To determine the user's Mail directory EXAMPLES
This example shows how rmf asks for confirmation when the -interactive option is used: % rmf -interactive +test Remove folder "test"? y FILES
The user profile. SEE ALSO
rmm(1) rmf(1)
All times are GMT -4. The time now is 10:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy