08-02-2011
You can split the file (with "split" command), then "sort -u" the chunks separately and then merge them with "sort -m". (Of course whether you need it depends on the memory size of your system).
This User Gave Thanks to yazu For This Post:
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have to delete 1st 7000 lines of a file which is 12GB large. As it is so large, i can't open in vi and delete these lines. Also I found one post here which gave solution using perl, but I don't have perl installed. Also some solutions were redirecting the o/p to a different file and renaming it.... (3 Replies)
Discussion started by: rahulrathod
3 Replies
2. Shell Programming and Scripting
Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'"
However, on a couple of systems, they have some mount points looking like this:
/stage
/stand
/usr
/MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies
3. UNIX for Dummies Questions & Answers
OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script.
So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup
... (2 Replies)
Discussion started by: bfurlong
2 Replies
4. UNIX for Dummies Questions & Answers
Hi please help me how to remove duplicate lines in any file.
I have a file having huge number of lines.
i want to remove selected lines in it.
And also if there exists duplicate lines, I want to delete the rest & just keep one of them.
Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies
5. UNIX for Dummies Questions & Answers
Hey all, a relative bash/script newbie trying solve a problem.
I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like
2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies
6. UNIX for Dummies Questions & Answers
hi :)
I need to delete partial duplicate lines
I have this in a file
sihp8027,/opt/cf20,1980182
sihp8027,/opt/oracle/10gRelIIcd,155200016
sihp8027,/opt/oracle/10gRelIIcd,155200176
sihp8027,/var/opt/ERP,10376312
and need to leave it like this:
sihp8027,/opt/cf20,1980182... (2 Replies)
Discussion started by: C|KiLLeR|S
2 Replies
7. Shell Programming and Scripting
The question is not as simple as the title... I have a file, it looks like this
<string name="string1">RZ-LED</string>
<string name="string2">2.0</string>
<string name="string2">Version 2.0</string>
<string name="string3">BP</string>
I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies
8. Shell Programming and Scripting
Hi, I'm sorry I'm no coder so I came here, counting on your free time and good will to beg for spoonfeeding some good code. I'll try to be quick and concise!
Got file with 50k lines like this:
"Heh, heh. Those darn ninjas. They're _____."*wacky
The "canebrake", "timber" & "pygmy" are types... (7 Replies)
Discussion started by: shadowww
7 Replies
9. UNIX for Beginners Questions & Answers
Hi,
I have a file as follows.
file1
Hello
Hi
His
Hi
Hi
Hungry
hi
so I want to delete identical lines while leaving one of them undeleted.
So desired output will be
Hello
Hi (2 Replies)
Discussion started by: beginner_99
2 Replies
10. UNIX for Beginners Questions & Answers
Hi
I need to delete duplicate like pattern lines from a text file containing 2 duplicates only (one being subset of the other) using sed or awk preferably.
Input:
FM:Chicago:Development
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting
PM:Newyork:Scripting:Audit... (6 Replies)
Discussion started by: tech_frk
6 Replies
LEARN ABOUT DEBIAN
mpb-split
MPB(1) MIT Photonic-Bands Package MPB(1)
NAME
mpb-split - compute eigenmodes with MPB using multiple processes
SYNOPSIS
mpb-split NUM-SPLIT [DEFINITION]... [CTLFILE]...
DESCRIPTION
mpb-split is a parallelizing front-end to MIT Photonic Bands (MPB). For a computation with several k points, it splits the list of k
points over multiple processes. Of course, this will only benefit you on a system where different processes will run on different proces-
sors, such as an SMP or a cluster with automatic process migration (e.g. MOSIX). mpb-split is actually a trivial shell script, though, so
you can easily modify it if you need to use a special command to launch processes on other processors/machines.
MIT Photonic Bands (MPB) is a free program to compute the band structures (dispersion relations) and electromagnetic modes of periodic
dielectric structures, and is applicable both to photonic crystals (photonic band-gap materials) and a wide range of other optical prob-
lems.
More information on MPB, including a detailed manual, can be found online at the MPB home page: http://ab-initio.mit.edu/mpb/
A typical invocation of mpb-split looks like:
mpb-split num-split foo.ctl >& foo.out
This causes mpb-split to process the control file foo.ctl, divide the k points into num-split equal chunks, run each list in a separate
process with MPB, and redirect the output (in order) to foo.out. (One typically redirects output to a file, as the output is verbose and
contains a number of comma-delimited datasets that one can extract by grepping.)
Overall, the behavior and arguments are the same as for mpb except that the first argument must be the integer num-split.
What mpb-split technically does is to set the MPB variable k-split-num to num-split and k-split-index to the index (starting with 0) of the
chunk for each process. If you want, you can use these variables to divide the problem in some other way and then reset them to 1 and 0,
respectively.
BUGS
Send bug reports to S. G. Johnson, stevenj@alum.mit.edu.
AUTHORS
Written by Steven G. Johnson. Copyright (c) 1999, 2000, 2001, 2002 by the Massachusetts Institute of Technology.
SEE ALSO
mpb(1), mpb-data(1)
MPB
March 13, 2002 MPB(1)