Sponsored Content
Top Forums UNIX for Advanced & Expert Users Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file Post 303025070 by vgersh99 on Tuesday 23rd of October 2018 04:27:15 PM
Old 10-23-2018
Please post a sample representative chunk of a file.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to check the unique numbers in huge data

Friends, I have to write a shell script,the description is---- i Have to check the uniqueness of the numbers in a file. A file is containing 200thousand tickets and a ticket have 15 numbers in asecending order.And there is a strip that is having 6 tickets that means 90 numbers.I... (7 Replies)
Discussion started by: namishtiwari
7 Replies

2. Shell Programming and Scripting

Find percent between sum of 2 columns awk help

Hi I'm new to this forum and I'm a beginner when it comes to shell and awk programming. But I have the following problem: I have 5 csv files (data1.csv, data2.csv, etc.) and need to calculate the average between the total sum of the 1st and 7 column. csv example:... (3 Replies)
Discussion started by: sapo51
3 Replies

3. Shell Programming and Scripting

awk sum columns

can anyone help me how do i add the colums using awk seperated by character @. for eg i have 3@4 2@9 5@1 the result should be 10 14 i tried using { sum+= $1 } END { print sum } but it just gives the result 10. can anyone help me with this one thank you and best regards (7 Replies)
Discussion started by: phone_book
7 Replies

4. UNIX for Advanced & Expert Users

A variable and sum of its value in a huge data.

Hi Experts, I got a question.. In the following output of `ps -elf | grep DataFlow` I get:- 242001 A mqsiadm 2076676 1691742 0 60 20 26ad4f400 130164 * May 09 - 3:02 DataFlowEngine EAIDVBR1_BROKER 5e453de8-2001-0000-0080-fd142b9ce8cb VIPS_INQ1 0 242001 A mqsiadm... (5 Replies)
Discussion started by: varungupta
5 Replies

5. Shell Programming and Scripting

AWK/Shell script for formatting data in a file

Hi All, Need an urgent help to convert a unix file in to a particular format: **source file:** 1111111 2d2f2h2 3dfgsd3 ........... 1111111 <-- repeats in every nth line. remaining all lines will be different 123ss41 432ff45 ........... 1111111 <-- repetition qwe1234 123weq3... (1 Reply)
Discussion started by: rajivnairfis
1 Replies

6. Shell Programming and Scripting

Sum up values of columns in 4 files using shell script

I am new to shell script.I have records like below in 4 different files which have about 10000 records each, all records unique and sorted based on column 2. 1 2 3 4 5 6 --------------------------- SR|1010478|000044590|1|0|0| SR|1014759|000105790|1|0|0| SR|1016609|000108901|1|0|0|... (2 Replies)
Discussion started by: reach.sree@gmai
2 Replies

7. Shell Programming and Scripting

Awk based script to find the median of all individual columns in a data file

Hi All, I have some data like below. Step1,Param1,Param2,Param3 1,2,3,4 2,3,4,5 2,4,5,6 3,0,1,2 3,0,0,0 3,2,1,3 ........ so on Where I need to find the median(arithmetic) of each column from Param1...to..Param3 for each set of Step1 values. (Sort each specific column, if the... (5 Replies)
Discussion started by: ks_reddy
5 Replies

8. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have... (4 Replies)
Discussion started by: ks_reddy
4 Replies

9. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies

10. Shell Programming and Scripting

Sum of columns using awk

Hello everyone I am a beginner in Shell scripting. Need your help to achieve desired result. I have a file (sample format below) 001g8aX0007jxLz xxxxxxxxxxxxxxx 9213974926411 CO-COMM-133 CO-L001-DLY 7769995578239 44938 1 1 ... (1 Reply)
Discussion started by: Rohit Mallah
1 Replies
Plack::Middleware(3pm)					User Contributed Perl Documentation				    Plack::Middleware(3pm)

NAME
Plack::Middleware - Base class for easy-to-use PSGI middleware SYNOPSIS
package Plack::Middleware::Foo; use parent qw( Plack::Middleware ); sub call { my($self, $env) = @_; # Do something with $env # $self->app is the original app my $res = $self->app->($env); # Do something with $res return $res; } # then in app.psgi use Plack::Builder; my $app = sub { ... } # as usual builder { enable "Plack::Middleware::Foo"; enable "Plack::Middleware::Bar", %options; $app; }; DESCRIPTION
Plack::Middleware is a utility base class to write PSGI middleware. All you have to do is to inherit from Plack::Middleware and then implement the callback "call" method (or "to_app" method that would return the PSGI code reference) to do the actual work. You can use "$self->app" to call the original (wrapped) application. Your middleware object is created at a PSGI application compile time and is persistent during the web server life cycle (unless it is a non-persistent environment such as CGI), so you should never set or cache per-request data like $env in your middleware object. See also "OBJECT LIFECYCLE" in Plack::Component. See Plack::Builder how to actually enable middleware in your .psgi application file using the DSL. If you do not like our builder DSL, you can also use "wrap" method to wrap your application with a middleware: use Plack::Middleware::Foo; my $app = sub { ... }; $app = Plack::Middleware::Foo->wrap($app, %options); $app = Plack::Middleware::Bar->wrap($app, %options); RESPONSE CALLBACK
The typical middleware is written like this: package Plack::Middleware::Something; use parent qw(Plack::Middleware); sub call { my($self, $env) = @_; # pre-processing $env my $res = $self->app->($env); # post-processing $res return $res; } The tricky thing about post processing the response is that it could either be an immediate 3 element array ref, or a code reference that implements the delayed (streaming) interface. Dealing with these two types of response in each piece of middleware is pointless, so you're recommended to use the "response_cb" wrapper function in Plack::Util when implementing a post processing middleware. my $res = $app->($env); Plack::Util::response_cb($res, sub { my $res = shift; # do something with $res; }); The callback function gets a PSGI response as a 3 element array reference, and you can update the reference to implement the post processing. package Plack::Middleware::Always500; use parent qw(Plack::Middleware); use Plack::Util; sub call { my($self, $env) = @_; my $res = $self->app->($env); Plack::Util::response_cb($res, sub { my $res = shift; $res->[0] = 500; return; }); } In this example, the callback gets the $res and updates its first element (status code) to 500. Using "response_cb" makes sure that this works with the delayed response too. You're not required (and not recommended either) to return a new array reference - they will be simply ignored. You're suggested to explicitly return, unless you fiddle with the content filter callback (see below). Similarly, note that you have to keep the $res reference when you swap the entire response. Plack::Util::response_cb($res, sub { my $res = shift; $res = [ $new_status, $new_headers, $new_body ]; # THIS DOES NOT WORK return; }); This does not work, since assigning a new anonymous array to $res doesn't update the original PSGI response value. You should instead do: Plack::Util::response_cb($res, sub { my $res = shift; @$res = ($new_status, $new_headers, $new_body); # THIS WORKS return; }); The third element of PSGI response array ref is a body, and it could be either array ref or IO::Handle-ish object. The application could also make use of $writer object if "psgi.streaming" is in effect. Dealing with these variants is again really painful, and "response_cb" can take care of that too, by allowing you to return a content filter as a code reference. # replace all "Foo" in content body with "Bar" Plack::Util::response_cb($res, sub { my $res = shift; return sub { my $chunk = shift; return unless defined $chunk; $chunk =~ s/Foo/Bar/g; return $chunk; } }); The callback takes one argument $chunk and your callback is expected to return the updated chunk. If the given $chunk is undef, it means the stream has reached the end, so your callback should also return undef, or return the final chunk and return undef when called next time. SEE ALSO
Plack Plack::Builder Plack::Component perl v5.14.2 2011-06-22 Plack::Middleware(3pm)
All times are GMT -4. The time now is 12:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy