Sponsored Content
Top Forums UNIX for Advanced & Expert Users In a huge file, Delete duplicate lines leaving unique lines Post 302543917 by vgersh99 on Tuesday 2nd of August 2011 11:47:57 AM
Old 08-02-2011
'man sort' yields:
Code:
       -T Directory
            Places all temporary files that are created into the directory specified by the
            Directory parameter.

specify '-T' with the directory with 'enough' space for the temp files.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete lines from huge file

I have to delete 1st 7000 lines of a file which is 12GB large. As it is so large, i can't open in vi and delete these lines. Also I found one post here which gave solution using perl, but I don't have perl installed. Also some solutions were redirecting the o/p to a different file and renaming it.... (3 Replies)
Discussion started by: rahulrathod
3 Replies

2. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies

3. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

4. UNIX for Dummies Questions & Answers

How to delete or remove duplicate lines in a file

Hi please help me how to remove duplicate lines in any file. I have a file having huge number of lines. i want to remove selected lines in it. And also if there exists duplicate lines, I want to delete the rest & just keep one of them. Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies

5. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

6. UNIX for Dummies Questions & Answers

How to delete partial duplicate lines unix

hi :) I need to delete partial duplicate lines I have this in a file sihp8027,/opt/cf20,1980182 sihp8027,/opt/oracle/10gRelIIcd,155200016 sihp8027,/opt/oracle/10gRelIIcd,155200176 sihp8027,/var/opt/ERP,10376312 and need to leave it like this: sihp8027,/opt/cf20,1980182... (2 Replies)
Discussion started by: C|KiLLeR|S
2 Replies

7. Shell Programming and Scripting

Delete lines in file containing duplicate strings, keeping longer strings

The question is not as simple as the title... I have a file, it looks like this <string name="string1">RZ-LED</string> <string name="string2">2.0</string> <string name="string2">Version 2.0</string> <string name="string3">BP</string> I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies

8. Shell Programming and Scripting

Delete duplicate lines... with a twist!

Hi, I'm sorry I'm no coder so I came here, counting on your free time and good will to beg for spoonfeeding some good code. I'll try to be quick and concise! Got file with 50k lines like this: "Heh, heh. Those darn ninjas. They're _____."*wacky The "canebrake", "timber" & "pygmy" are types... (7 Replies)
Discussion started by: shadowww
7 Replies

9. UNIX for Beginners Questions & Answers

How to delete identical lines while leaving one undeleted?

Hi, I have a file as follows. file1 Hello Hi His Hi Hi Hungry hi so I want to delete identical lines while leaving one of them undeleted. So desired output will be Hello Hi (2 Replies)
Discussion started by: beginner_99
2 Replies

10. UNIX for Beginners Questions & Answers

Delete duplicate like pattern lines

Hi I need to delete duplicate like pattern lines from a text file containing 2 duplicates only (one being subset of the other) using sed or awk preferably. Input: FM:Chicago:Development FM:Chicago:Development:Score SR:Cary:Testing:Testcases PM:Newyork:Scripting PM:Newyork:Scripting:Audit... (6 Replies)
Discussion started by: tech_frk
6 Replies
Template::Plugin::Directory(3pm)			User Contributed Perl Documentation			  Template::Plugin::Directory(3pm)

NAME
Template::Plugin::Directory - Plugin for generating directory listings SYNOPSIS
[% USE dir = Directory(dirpath) %] # files returns list of regular files [% FOREACH file = dir.files %] [% file.name %] [% file.path %] ... [% END %] # dirs returns list of sub-directories [% FOREACH subdir = dir.dirs %] [% subdir.name %] [% subdir.path %] ... [% END %] # list returns both interleaved in order [% FOREACH item = dir.list %] [% IF item.isdir %] Directory: [% item.name %] [% ELSE %] File: [% item.name %] [% END %] [% END %] # define a VIEW to display dirs/files [% VIEW myview %] [% BLOCK file %] File: [% item.name %] [% END %] [% BLOCK directory %] Directory: [% item.name %] [% item.content(myview) | indent -%] [% END %] [% END %] # display directory content using view [% myview.print(dir) %] DESCRIPTION
This Template Toolkit plugin provides a simple interface to directory listings. It is derived from the Template::Plugin::File module and uses Template::Plugin::File object instances to represent files within a directory. Sub-directories within a directory are represented by further "Template::Plugin::Directory" instances. The constructor expects a directory name as an argument. [% USE dir = Directory('/tmp') %] It then provides access to the files and sub-directories contained within the directory. # regular files (not directories) [% FOREACH file IN dir.files %] [% file.name %] [% END %] # directories only [% FOREACH file IN dir.dirs %] [% file.name %] [% END %] # files and/or directories [% FOREACH file IN dir.list %] [% file.name %] ([% file.isdir ? 'directory' : 'file' %]) [% END %] The plugin constructor will throw a "Directory" error if the specified path does not exist, is not a directory or fails to "stat()" (see Template::Plugin::File). Otherwise, it will scan the directory and create lists named '"files"' containing files, '"dirs"' containing directories and '"list"' containing both files and directories combined. The "nostat" option can be set to disable all file/directory checks and directory scanning. Each file in the directory will be represented by a Template::Plugin::File object instance, and each directory by another "Template::Plugin::Directory". If the "recurse" flag is set, then those directories will contain further nested entries, and so on. With the "recurse" flag unset, as it is by default, then each is just a place marker for the directory and does not contain any further content unless its "scan()" method is explicitly called. The "isdir" flag can be tested against files and/or directories, returning true if the item is a directory or false if it is a regular file. [% FOREACH file = dir.list %] [% IF file.isdir %] * Directory: [% file.name %] [% ELSE %] * File: [% file.name %] [% END %] [% END %] This example shows how you might walk down a directory tree, displaying content as you go. With the recurse flag disabled, as is the default, we need to explicitly call the "scan()" method on each directory, to force it to lookup files and further sub-directories contained within. [% USE dir = Directory(dirpath) %] * [% dir.path %] [% INCLUDE showdir %] [% BLOCK showdir -%] [% FOREACH file = dir.list -%] [% IF file.isdir -%] * [% file.name %] [% file.scan -%] [% INCLUDE showdir dir=file FILTER indent(4) -%] [% ELSE -%] - [% f.name %] [% END -%] [% END -%] [% END %] This example is adapted (with some re-formatting for clarity) from a test in t/directry.t which produces the following output: * test/dir - file1 - file2 * sub_one - bar - foo * sub_two - waz.html - wiz.html - xyzfile The "recurse" flag can be set (disabled by default) to cause the constructor to automatically recurse down into all sub-directories, creating a new "Template::Plugin::Directory" object for each one and filling it with any further content. In this case there is no need to explicitly call the "scan()" method. [% USE dir = Directory(dirpath, recurse=1) %] ... [% IF file.isdir -%] * [% file.name %] [% INCLUDE showdir dir=file FILTER indent(4) -%] [% ELSE -%] ... The directory plugin also provides support for views. A view can be defined as a "VIEW ... END" block and should contain "BLOCK" definitions for files ('"file"') and directories ('"directory"'). [% VIEW myview %] [% BLOCK file %] - [% item.name %] [% END %] [% BLOCK directory %] * [% item.name %] [% item.content(myview) FILTER indent %] [% END %] [% END %] The view "print()" method can then be called, passing the "Directory" object as an argument. [% USE dir = Directory(dirpath, recurse=1) %] [% myview.print(dir) %] When a directory is presented to a view, either as "[% myview.print(dir) %]" or "[% dir.present(view) %]", then the "directory" "BLOCK" within the "myview" "VIEW" is processed. The "item" variable will be set to alias the "Directory" object. [% BLOCK directory %] * [% item.name %] [% item.content(myview) FILTER indent %] [% END %] In this example, the directory name is first printed and the content(view) method is then called to present each item within the directory to the view. Further directories will be mapped to the "directory" block, and files will be mapped to the "file" block. With the recurse option disabled, as it is by default, the "directory" block should explicitly call a "scan()" on each directory. [% VIEW myview %] [% BLOCK file %] - [% item.name %] [% END %] [% BLOCK directory %] * [% item.name %] [% item.scan %] [% item.content(myview) FILTER indent %] [% END %] [% END %] [% USE dir = Directory(dirpath) %] [% myview.print(dir) %] AUTHORS
Michael Stevens wrote the original Directory plugin on which this is based. Andy Wardley split it into separate File and Directory plugins, added some extra code and documentation for "VIEW" support, and made a few other minor tweaks. COPYRIGHT
Copyright (C) 2000-2007 Michael Stevens, Andy Wardley. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Template::Plugin, Template::Plugin::File, Template::View perl v5.14.2 2011-12-20 Template::Plugin::Directory(3pm)
All times are GMT -4. The time now is 02:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy