Sponsored Content
Top Forums Shell Programming and Scripting Big data file - sed/grep/awk? Post 302204289 by dlam on Wednesday 11th of June 2008 07:46:35 AM
Old 06-11-2008
Thanks guys. A little bit of editing has shown it is definitely that line causing the problem, but as era says it shouldn't be the size that is casusing the problem because it's just holding one variable at a time so the array's not performing well certainly could be the reason.

I'll have a play with your script and see if I can slot it in.

Thanks again.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

filter parts of a big file using awk or sed script

I need an assistance in file generation using awk, sed or anything... I have a big file that i need to filter desired parts only. The objective is to select (and print) the report # having the string "apple" on 2 consecutive lines in every report. Please note that the "apple" line has a HEX... (1 Reply)
Discussion started by: apalex
1 Replies

2. Shell Programming and Scripting

Should I use sed/ grep/awk for wrap file?

Hi, This is my first time post a new thread. I have been trying to work on this for the past 2 days and could not find any good solution. I have 1 long long line ( EDI wrapped file) like below: NEW*SR*04411763447*279*278*Q~*ZR*AAV*SR*04511763460*SQ*21B37F04~HL*305*304*Q~K~SN1*1*1*SR*05511763461*... (6 Replies)
Discussion started by: vanda_25
6 Replies

3. Shell Programming and Scripting

sed or awk to extract data from Xml file

Hi, I want to get data from Xml file by using sed or awk command. I want to get the following result : mon titre 1;Createur1;Dossier1 mon titre 1;Createur1;Dossier1 and save it in cvs file (fichier.cvs). FROM this Xml file (test.xml): <playlist version="1"> <trackList> <track>... (1 Reply)
Discussion started by: yeclota
1 Replies

4. Shell Programming and Scripting

How to cut some data from big file

How to cut data from big file my file around 30 gb I tried "head -50022172 filename > newfile.txt ,and tail -5454283 newfile.txt. It's slowy. afer that I tried sed -n '46467831,50022172p' filename > newfile.txt ,also slow Please recommend me , faster command to cut some data from... (4 Replies)
Discussion started by: almanto
4 Replies

5. UNIX for Dummies Questions & Answers

awk and grep to search a data file

Hi everyone, I cannot figure out how I can do a search in a file that has Names, Surnames, Addresses and telephone number of a number of people. Here is an example of the data file Daisy:Hunter:490 London Road:07313196347 Richard:Murphy:983 Main Road:07002625997 Isobel:Magnusson:133 London... (1 Reply)
Discussion started by: philipisaia
1 Replies

6. Shell Programming and Scripting

formatting data file with awk or sed

Hi, I have a (quite large) data file which looks like: _____________ header part.. more header part.. x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 ... ... x59 x60 y1 y2 y3 y4... ... y100 ______________ where x1, x2,...,x60 and y1, y2,...y100 are numbers of 10 digits (so each line... (5 Replies)
Discussion started by: lego
5 Replies

7. Shell Programming and Scripting

Sort a big data file

Hello, I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field. I'm trying to sort with "sort" command and it brings me 6 minutes. I have tried with some transformation methods in perl but it results "Out of memory". I was... (2 Replies)
Discussion started by: rubber08
2 Replies

8. Shell Programming and Scripting

Router ping log extract data from it Awk/Sed/grep

Hi, I am new to this world.. Using expect i loging to router and checking ping response to my links. I need to genarate report using this output and that report contains only three file link name, packet loss, latency. my output of script is like below: -bash-3.00$ monmw/mwbkp... (2 Replies)
Discussion started by: jkmistry
2 Replies

9. Shell Programming and Scripting

awk - sed / reading from a data file and doing algebraic operations

Hi everyone, I am trying to write a bash script which reads a data file and does some algebraic operations. here is the structure of data.xml file that I have; 1 <data> 2 . 3 . 4 . 5 </data> 6 <data> 7 . 8 . 9 . 10</data> etc. Each data block contains same number of lines (say... (4 Replies)
Discussion started by: hayreter
4 Replies

10. Shell Programming and Scripting

How to grep/sed selected data from a command or file?

Below is the output of a DB2 command. Now I have 2 requirements... Database Partition 0 -- Database TESTDB1 -- Active Standby -- Up 213 days 02:33:07 -- Date 02/22/2016 17:04:50 HADR Information: Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes) Standby ... (2 Replies)
Discussion started by: rlokesh27
2 Replies
DProf(3pm)						User Contributed Perl Documentation						DProf(3pm)

NAME
Devel::DProf - a DEPRECATED Perl code profiler SYNOPSIS
perl -d:DProf test.pl ACHTUNG! "Devel::DProf" is DEPRECATED and will be removed from a future version of Perl. We strongly recommend that you install and use Devel::NYTProf instead, as it offers significantly improved profiling and reporting. DESCRIPTION
The Devel::DProf package is a Perl code profiler. This will collect information on the execution time of a Perl script and of the subs in that script. This information can be used to determine which subroutines are using the most time and which subroutines are being called most often. This information can also be used to create an execution graph of the script, showing subroutine relationships. To profile a Perl script run the perl interpreter with the -d debugging switch. The profiler uses the debugging hooks. So to profile script test.pl the following command should be used: perl -d:DProf test.pl When the script terminates (or when the output buffer is filled) the profiler will dump the profile information to a file called tmon.out. A tool like dprofpp can be used to interpret the information which is in that profile. The following command will print the top 15 subroutines which used the most time: dprofpp To print an execution graph of the subroutines in the script use the following command: dprofpp -T Consult dprofpp for other options. PROFILE FORMAT
The old profile is a text file which looks like this: #fOrTyTwO $hz=100; $XS_VERSION='DProf 19970606'; # All values are given in HZ $rrun_utime=2; $rrun_stime=0; $rrun_rtime=7 PART2 + 26 28 566822884 DynaLoader::import - 26 28 566822884 DynaLoader::import + 27 28 566822885 main::bar - 27 28 566822886 main::bar + 27 28 566822886 main::baz + 27 28 566822887 main::bar - 27 28 566822888 main::bar [....] The first line is the magic number. The second line is the hertz value, or clock ticks, of the machine where the profile was collected. The third line is the name and version identifier of the tool which created the profile. The fourth line is a comment. The fifth line contains three variables holding the user time, system time, and realtime of the process while it was being profiled. The sixth line indicates the beginning of the sub entry/exit profile section. The columns in PART2 are: sub entry(+)/exit(-) mark app's user time at sub entry/exit mark, in ticks app's system time at sub entry/exit mark, in ticks app's realtime at sub entry/exit mark, in ticks fully-qualified sub name, when possible With newer perls another format is used, which may look like this: #fOrTyTwO $hz=10000; $XS_VERSION='DProf 19971213'; # All values are given in HZ $over_utime=5917; $over_stime=0; $over_rtime=5917; $over_tests=10000; $rrun_utime=1284; $rrun_stime=0; $rrun_rtime=1284; $total_marks=6; PART2 @ 406 0 406 & 2 main bar + 2 @ 456 0 456 - 2 @ 1 0 1 & 3 main baz + 3 @ 141 0 141 + 2 @ 141 0 141 - 2 @ 1 0 1 & 4 main foo + 4 @ 142 0 142 + & Devel::DProf::write @ 5 0 5 - & Devel::DProf::write (with high value of $ENV{PERL_DPROF_TICKS}). New "$over_*" values show the measured overhead of making $over_tests calls to the profiler These values are used by the profiler to subtract the overhead from the runtimes. Lines starting with "@" mark the amount of time passed since the previous "@" line. The numbers following the "@" are integer tick counts representing user, system, and real time. Divide these numbers by the $hz value in the header to get seconds. Lines starting with "&" map subroutine identifiers (an integer) to subroutine packages and names. These should only occur once per subroutine. Lines starting with "+" or "-" mark normal entering and exit of subroutines. The number following is a reference to a subroutine identifier. Lines starting with "*" mark where subroutines are entered by "goto &subr", but note that the return will still be marked as coming from the original sub. The sequence might look like this: + 5 * 6 - 5 Lines starting with "/" is like "-" but mark where subroutines are exited by dying. Example: + 5 + 6 / 6 / 5 Finally you might find "@" time stamp marks surrounded by "+ & Devel::DProf::write" and "- & Devel::DProf::write" lines. These 3 lines are outputted when printing of the mark above actually consumed measurable time. AUTOLOAD
When Devel::DProf finds a call to an &AUTOLOAD subroutine it looks at the $AUTOLOAD variable to find the real name of the sub being called. See "Autoloading" in perlsub. ENVIRONMENT
"PERL_DPROF_BUFFER" sets size of output buffer in words. Defaults to 2**14. "PERL_DPROF_TICKS" sets number of ticks per second on some systems where a replacement for times() is used. Defaults to the value of "HZ" macro. "PERL_DPROF_OUT_FILE_NAME" sets the name of the output file. If not set, defaults to tmon.out. BUGS
Builtin functions cannot be measured by Devel::DProf. With a newer Perl DProf relies on the fact that the numeric slot of $DB::sub contains an address of a subroutine. Excessive manipulation of this variable may overwrite this slot, as in $DB::sub = 'current_sub'; ... $addr = $DB::sub + 0; will set this numeric slot to numeric value of the string "current_sub", i.e., to 0. This will cause a segfault on the exit from this subroutine. Note that the first assignment above does not change the numeric slot (it will mark it as invalid, but will not write over it). Another problem is that if a subroutine exits using goto(LABEL), last(LABEL) or next(LABEL) then perl may crash or Devel::DProf will die with the error: panic: Devel::DProf inconsistent subroutine return For example, this code will break under Devel::DProf: sub foo { last FOO; } FOO: { foo(); } A pattern like this is used by Test::More's skip() function, for example. See perldiag for more details. SEE ALSO
perl, dprofpp, times(2) perl v5.14.2 2011-08-02 DProf(3pm)
All times are GMT -4. The time now is 10:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy