Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Caching while spliting a large file Post 302269573 by siba.s.nayak on Thursday 18th of December 2008 04:39:38 AM
Old 12-18-2008
Caching while spliting a large file

Hi

The scenario is like this.

I need to split 5 files having size 3GB, 2GB, 4GB, 30GB and 20 GB respectively.

The machine has 15GB heap space. Before starting split proces it was showing 15gb free space. Once the split process completed it showed 100 mb free and around 12GB cached.

My doubt is why that much of memory got cached during split process. and Can I release the cached memory even if the process has been running?
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

spliting up a huge file

I have a file {filename} which contains 65000 records I need to split into 6 smaller files roughly 11000 records each. Can someone advise me of the Unix command to do so ? Many thanks (2 Replies)
Discussion started by: grinder182533
2 Replies

2. Shell Programming and Scripting

Spliting file based on condition

Hi, I have a comma separated file with millions of records in it. I have a requirement to split the file based on the value in a one of the columns. Suppose i have a text file with columns like C1, C2,C3,C4 Column C4 can hold the values either 01 or 02 03 or 04. I nned to extract... (2 Replies)
Discussion started by: Raamc
2 Replies

3. Shell Programming and Scripting

Spliting the file dynamically

i am creating the file , when this file reaches the size 2 GB, i need one message or fire (4 Replies)
Discussion started by: kingganesh04
4 Replies

4. Shell Programming and Scripting

Spliting a file and renaming it's out put

Hi friends, I am new to unix. With the help of this site I wrote a script. But it's returning some errors pls help.:confused: Here is my specifications: I have a file obsrec_31583_090212.xls at /home/lingalma/temp/cdl I want it to mv to another temp folder called /home/lingalma/temp/obsrec... (6 Replies)
Discussion started by: raaj.manoj
6 Replies

5. UNIX for Dummies Questions & Answers

spliting a file

how would i split the file "file1" into smaller files containg lines of 15 (1 Reply)
Discussion started by: JamieMurry
1 Replies

6. Shell Programming and Scripting

File Spliting problem

I have a very big log file which looks like this: I need to split this file and create files with "+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+" as the delimiter. The file names need to be the contents of the next line after the delimiter(FIRST_ITEM,SECOND_ITEM...so on..). (7 Replies)
Discussion started by: engineer
7 Replies

7. Shell Programming and Scripting

spliting up a large file

Dear All, I have a very large file which which i would like split into indvidual frames evrytime the line ends with "ENDMDL" and then name frame1.pdb frame2.pdb etc can any one give me a few sugeestions? ideally i would like to have ENDMDL at the end of each frame or not pressent at all. an... (4 Replies)
Discussion started by: Mish_99
4 Replies

8. Shell Programming and Scripting

Spliting file based field pattern

Hi all, i have file that looks like as below 2263881188,24570896,439,SOLO,SOLO_UNBEATABLE,E,+3.13,+0.00 2263881964,24339077,439,SOLO,SOLO_UNBEATABLE,F,-0.67,+0.00 2263883220,22619162,228,Bell,Bell_MONTHLY,E,-2.04,+0.00 2263883220,22619162,228,Bell,Bell_MONTHLY,F,-2.04,+0.00... (3 Replies)
Discussion started by: raghavendra.cse
3 Replies

9. Shell Programming and Scripting

Spliting log file

Hello, I want to split or cut a large size log file by year wise(eg 2009, 2010) .But the source file must not have the splited or cut lines after this process ,all of them must move to the destination folder.Does grep command have the fuctionality like cut and paste? I used grep -Ev command but... (17 Replies)
Discussion started by: jobycxa
17 Replies
MPB(1)							    MIT Photonic-Bands Package							    MPB(1)

NAME
mpb-split - compute eigenmodes with MPB using multiple processes SYNOPSIS
mpb-split NUM-SPLIT [DEFINITION]... [CTLFILE]... DESCRIPTION
mpb-split is a parallelizing front-end to MIT Photonic Bands (MPB). For a computation with several k points, it splits the list of k points over multiple processes. Of course, this will only benefit you on a system where different processes will run on different proces- sors, such as an SMP or a cluster with automatic process migration (e.g. MOSIX). mpb-split is actually a trivial shell script, though, so you can easily modify it if you need to use a special command to launch processes on other processors/machines. MIT Photonic Bands (MPB) is a free program to compute the band structures (dispersion relations) and electromagnetic modes of periodic dielectric structures, and is applicable both to photonic crystals (photonic band-gap materials) and a wide range of other optical prob- lems. More information on MPB, including a detailed manual, can be found online at the MPB home page: http://ab-initio.mit.edu/mpb/ A typical invocation of mpb-split looks like: mpb-split num-split foo.ctl >& foo.out This causes mpb-split to process the control file foo.ctl, divide the k points into num-split equal chunks, run each list in a separate process with MPB, and redirect the output (in order) to foo.out. (One typically redirects output to a file, as the output is verbose and contains a number of comma-delimited datasets that one can extract by grepping.) Overall, the behavior and arguments are the same as for mpb except that the first argument must be the integer num-split. What mpb-split technically does is to set the MPB variable k-split-num to num-split and k-split-index to the index (starting with 0) of the chunk for each process. If you want, you can use these variables to divide the problem in some other way and then reset them to 1 and 0, respectively. BUGS
Send bug reports to S. G. Johnson, stevenj@alum.mit.edu. AUTHORS
Written by Steven G. Johnson. Copyright (c) 1999, 2000, 2001, 2002 by the Massachusetts Institute of Technology. SEE ALSO
mpb(1), mpb-data(1) MPB
March 13, 2002 MPB(1)
All times are GMT -4. The time now is 06:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy