08-14-2009
How to cut some data from big file
How to cut data from big file
my file around 30 gb
I tried "head -50022172 filename > newfile.txt ,and tail -5454283 newfile.txt. It's slowy.
afer that I tried sed -n '46467831,50022172p' filename > newfile.txt ,also slow
Please recommend me , faster command to cut some data from Big file
Thanks.
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi list,
i have an orcale spool file with
SQL> select COMPMAP as SVC, ITEM, UNIT, sum(JAN), sum(FEB)
SVC ITE U SUM(JAN) SUM(FEB)
------ --- - ---------- ----------
401500 IOC Q 14 14
406200 LC Q 1 1
410124 IOC Q 5 4
410124 LC... (1 Reply)
Discussion started by: rejirajraghav
1 Replies
2. Shell Programming and Scripting
Morning guys. Another day another question. :rolleyes:
I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution:
for results in `grep "^\
... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies
3. UNIX for Dummies Questions & Answers
Hi All,
Currently i have to write a script.
For which i need to cut a block from .txt file.
I know the specific word that starts the block and ends the block.
Can we do it in shell scripting..?
Please suggest.... (6 Replies)
Discussion started by: pank29
6 Replies
4. Shell Programming and Scripting
I have a big text file. I want to cut it into 2 pieces at known point or I know the pattern of the contents from where it can separate the files. Is there any quick command/solution? (4 Replies)
Discussion started by: sandy221
4 Replies
5. Shell Programming and Scripting
Hello,
I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field.
I'm trying to sort with "sort" command and it brings me 6 minutes.
I have tried with some transformation methods in perl but it results "Out of memory". I was... (2 Replies)
Discussion started by: rubber08
2 Replies
6. Shell Programming and Scripting
Hi,
I have 2 files
format of file 1 is:
a1
b2
a2
c2
d1
f3
format of file 2 is (tab delimited):
a1 1.2 0.5 0.06 0.7 0.9 1 0.023
a3 0.91 0.007 0.12 0.34 0.45 1 0.7
a2 1.05 2.3 0.25 1 0.9 0.3 0.091
b1 1 5.4 0.3 9.2 0.3 0.2 0.1
b2 3 5 7 0.9 1 9 0 1
b3 0.001 1 2.3 4.6 8.9 10 0 1 0... (10 Replies)
Discussion started by: Lucky Ali
10 Replies
7. Shell Programming and Scripting
HI Guys,
I have file A:
Abc XyZ Abc Xyz Kal Kaloo
Abc XyZ Abc Xyz Kalpooo
Abc XyZ Abc Xyz Kloo
Abc Abc Klooo
I want file B
Abc XyZ Abc Xyz Kal Kaloo
Abc XyZ Abc Xyz Kalpooo
Abc XyZ Abc Xyz Kloo
File A is now 1 lines
Abc Abc Klooo
Cut all lines which have xyz... (2 Replies)
Discussion started by: asavaliya
2 Replies
8. Shell Programming and Scripting
Data_Consolidation_Engine_Part_2_Job2..TgtArBkt: ORA-00942: table or view does not exist
I have some thing like above in the file..
Upto this portion Data_Consolidation_Engine_Part_2_Job2..TgtArBkt: the length can be vary ..
Can some one help me in taking this portion alone ORA-00942:... (7 Replies)
Discussion started by: saj
7 Replies
9. UNIX for Dummies Questions & Answers
Hello all,
Currently I have a txt file named as a.txt with the content as:
f e100
aa bb
cc dd
ee ff
f e222
aa dd
ff gg
f e987
dd aa
f e2222
gg ff
gg aa
dd ff
ee ee
While, for some reason I want to cut a.txt into small ones, e.g. f1.txt, f2.txt, f3.txt and f4.txt. The routine is to... (6 Replies)
Discussion started by: locohd
6 Replies
LEARN ABOUT DEBIAN
file::mimeinfo::cookbook
File::MimeInfo::Cookbook(3pm) User Contributed Perl Documentation File::MimeInfo::Cookbook(3pm)
NAME
File::MimeInfo::Cookbook - various code snippets
DESCRIPTION
Some code snippets for non-basic uses of the File::MimeInfo module:
Matching an extension
A file does not have to actually exist in order to get a mimetype for it. This means that the following will work:
my $extension = '*.txt';
my $mimetype = mimetype( $extension );
Mimetyping an scalar
If you want to find the mimetype of a scalar value you need magic mimetyping; after all a scalar doesn't have a filename or inode.
What you need to do is to use IO::Scalar :
use File::MimeInfo::Magic;
use IO::Scalar;
my $io_scalar = new IO::Scalar $data;
my $mimetype = mimetype( $io_scalar );
In fact most other "IO::" will work as long as they support the "seek()" and "read()" methods. Of course if you want really obscure
things to happen you can always write your own IO object and feed it in there.
Be aware that when using a filehandle like this you need to set the ":utf8" binmode yourself if apropriate.
Mimetyping a filehandle
Regrettably for non-seekable filehandles like STDIN simply using an "IO::" object will not work. You will need to buffer enough of the
data for a proper mimetyping. For example you could mimetype data from STDIN like this:
use File::MimeInfo::Magic;
use IO::Scalar;
my $data;
read(STDIN, $data, $File::MimeInfo::Magic::max_buffer);
my $io_scalar = new IO::Scalar $data;
my $mimetype = mimetype( $io_scalar );
Be aware that when using a filehandle like this you need to set the ":utf8" binmode yourself if apropriate.
Creating a new filename
Say you have a temporary file that you want to save with a more proper filename.
use File::MimeInfo::Magic qw#mimetype extensions#;
use File::Copy;
my $tmpfile = '/tmp/foo';
my $mimetype = mimetype($tmpfile);
my $extension = extensions($mimetype);
my $newfile = 'untitled1';
$newfile .= '.'.$extension if length $extension;
move($tmpfile, $newfile);
Force the use of a certain database directory
Normally you just need to add the dir where your mime database lives to either the XDG_DATA_HOME or XDG_DATA_DIRS environment variables
for it to be found. But in some rare cases you may want to by-pass this system all together. Try one of the following:
@File::MimeInfo::DIRS = ('/home/me/share/mime');
eval 'use File::MimeInfo';
die if $@;
or:
use File::MimeInfo;
@File::MimeInfo::DIRS = ('/home/me/share/mime');
File::MimeInfo->rehash();
This can also be used for switching between databases at run time while leaving other XDG configuration stuff alone.
AUTHOR
Jaap Karssenberg <pardus@cpan.org>
Copyright (c) 2005, 2012 Jaap G Karssenberg. All rights reserved. This program is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
SEE ALSO
File::MimeInfo
perl v5.14.2 2012-01-05 File::MimeInfo::Cookbook(3pm)