Sponsored Content
Top Forums UNIX for Advanced & Expert Users Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file Post 303025068 by kartikirans on Tuesday 23rd of October 2018 04:25:03 PM
Old 10-23-2018
Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file

File delimiter "|"
Need to have Sum of all columns, with column number : aggregation (summation) for each column
File not having the header

Like below -
Column 1 "Total
Column 2 : "Total
...
...
Column N : "Total :
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to check the unique numbers in huge data

Friends, I have to write a shell script,the description is---- i Have to check the uniqueness of the numbers in a file. A file is containing 200thousand tickets and a ticket have 15 numbers in asecending order.And there is a strip that is having 6 tickets that means 90 numbers.I... (7 Replies)
Discussion started by: namishtiwari
7 Replies

2. Shell Programming and Scripting

Find percent between sum of 2 columns awk help

Hi I'm new to this forum and I'm a beginner when it comes to shell and awk programming. But I have the following problem: I have 5 csv files (data1.csv, data2.csv, etc.) and need to calculate the average between the total sum of the 1st and 7 column. csv example:... (3 Replies)
Discussion started by: sapo51
3 Replies

3. Shell Programming and Scripting

awk sum columns

can anyone help me how do i add the colums using awk seperated by character @. for eg i have 3@4 2@9 5@1 the result should be 10 14 i tried using { sum+= $1 } END { print sum } but it just gives the result 10. can anyone help me with this one thank you and best regards (7 Replies)
Discussion started by: phone_book
7 Replies

4. UNIX for Advanced & Expert Users

A variable and sum of its value in a huge data.

Hi Experts, I got a question.. In the following output of `ps -elf | grep DataFlow` I get:- 242001 A mqsiadm 2076676 1691742 0 60 20 26ad4f400 130164 * May 09 - 3:02 DataFlowEngine EAIDVBR1_BROKER 5e453de8-2001-0000-0080-fd142b9ce8cb VIPS_INQ1 0 242001 A mqsiadm... (5 Replies)
Discussion started by: varungupta
5 Replies

5. Shell Programming and Scripting

AWK/Shell script for formatting data in a file

Hi All, Need an urgent help to convert a unix file in to a particular format: **source file:** 1111111 2d2f2h2 3dfgsd3 ........... 1111111 <-- repeats in every nth line. remaining all lines will be different 123ss41 432ff45 ........... 1111111 <-- repetition qwe1234 123weq3... (1 Reply)
Discussion started by: rajivnairfis
1 Replies

6. Shell Programming and Scripting

Sum up values of columns in 4 files using shell script

I am new to shell script.I have records like below in 4 different files which have about 10000 records each, all records unique and sorted based on column 2. 1 2 3 4 5 6 --------------------------- SR|1010478|000044590|1|0|0| SR|1014759|000105790|1|0|0| SR|1016609|000108901|1|0|0|... (2 Replies)
Discussion started by: reach.sree@gmai
2 Replies

7. Shell Programming and Scripting

Awk based script to find the median of all individual columns in a data file

Hi All, I have some data like below. Step1,Param1,Param2,Param3 1,2,3,4 2,3,4,5 2,4,5,6 3,0,1,2 3,0,0,0 3,2,1,3 ........ so on Where I need to find the median(arithmetic) of each column from Param1...to..Param3 for each set of Step1 values. (Sort each specific column, if the... (5 Replies)
Discussion started by: ks_reddy
5 Replies

8. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have... (4 Replies)
Discussion started by: ks_reddy
4 Replies

9. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies

10. Shell Programming and Scripting

Sum of columns using awk

Hello everyone I am a beginner in Shell scripting. Need your help to achieve desired result. I have a file (sample format below) 001g8aX0007jxLz xxxxxxxxxxxxxxx 9213974926411 CO-COMM-133 CO-L001-DLY 7769995578239 44938 1 1 ... (1 Reply)
Discussion started by: Rohit Mallah
1 Replies
RTBL(3) 						   BSD Library Functions Manual 						   RTBL(3)

NAME
rtbl_create, rtbl_destroy, rtbl_set_flags, rtbl_get_flags, rtbl_set_prefix, rtbl_set_separator, rtbl_set_column_prefix, rtbl_set_column_affix_by_id, rtbl_add_column, rtbl_add_column_by_id, rtbl_add_column_entry, rtbl_add_column_entry_by_id, rtbl_new_row, rtbl_format -- format data in simple tables LIBRARY
The roken library (libroken, -lroken) SYNOPSIS
#include <rtbl.h> int rtbl_add_column(rtbl_t table, const char *column_name, unsigned int flags); int rtbl_add_column_by_id(rtbl_t table, unsigned int column_id, const char *column_header, unsigned int flags); int rtbl_add_column_entry(rtbl_t table, const char *column_name, const char *cell_entry); int rtbl_add_column_entry_by_id(rtbl_t table, unsigned int column_id, const char *cell_entry); rtbl_t rtbl_create(void); void rtbl_destroy(rtbl_t table); int rtbl_new_row(rtbl_t table); int rtbl_set_column_affix_by_id(rtbl_t table, unsigned int column_id, const, char, *prefix", const char *suffix); int rtbl_set_column_prefix(rtbl_t table, const char *column_name, const char *prefix); unsigned int rtbl_get_flags(rtbl_t table); void rtbl_set_flags(rtbl_t table, unsigned int flags); int rtbl_set_prefix(rtbl_t table, const char *prefix); int rtbl_set_separator(rtbl_t table, const char *separator); int rtbl_format(rtbl_t table, FILE, *file"); DESCRIPTION
This set of functions assemble a simple table consisting of rows and columns, allowing it to be printed with certain options. Typical use would be output from tools such as ls(1) or netstat(1), where you have a fixed number of columns, but don't know the column widths before hand. A table is created with rtbl_create() and destroyed with rtbl_destroy(). Global flags on the table are set with rtbl_set_flags and retrieved with rtbl_get_flags. At present the only defined flag is RTBL_HEADER_STYLE_NONE which suppresses printing the header. Before adding data to the table, one or more columns need to be created. This would normally be done with rtbl_add_column_by_id(), column_id is any number of your choice (it's used only to identify columns), column_header is the header to print at the top of the column, and flags are flags specific to this column. Currently the only defined flag is RTBL_ALIGN_RIGHT, aligning column entries to the right. Columns are printed in the order they are added. There's also a way to add columns by column name with rtbl_add_column(), but this is less flexible (you need unique header names), and is considered deprecated. To add data to a column you use rtbl_add_column_entry_by_id(), where the column_id is the same as when the column was added (adding data to a non-existent column is undefined), and cell_entry is whatever string you wish to include in that cell. It should not include newlines. For columns added with rtbl_add_column() you must use rtbl_add_column_entry() instead. rtbl_new_row() fills all columns with blank entries until they all have the same number of rows. Each column can have a separate prefix and suffix, set with rtbl_set_column_affix_by_id; rtbl_set_column_prefix allows setting the prefix only by column name. In addition to this, columns may be separated by a string set with rtbl_set_separator (by default columns are not seprated by anything). The finished table is printed to file with rtbl_format. EXAMPLES
This program: #include <stdio.h> #include <rtbl.h> int main(int argc, char **argv) { rtbl_t table; table = rtbl_create(); rtbl_set_separator(table, " "); rtbl_add_column_by_id(table, 0, "Column A", 0); rtbl_add_column_by_id(table, 1, "Column B", RTBL_ALIGN_RIGHT); rtbl_add_column_by_id(table, 2, "Column C", 0); rtbl_add_column_entry_by_id(table, 0, "A-1"); rtbl_add_column_entry_by_id(table, 0, "A-2"); rtbl_add_column_entry_by_id(table, 0, "A-3"); rtbl_add_column_entry_by_id(table, 1, "B-1"); rtbl_add_column_entry_by_id(table, 2, "C-1"); rtbl_add_column_entry_by_id(table, 2, "C-2"); rtbl_add_column_entry_by_id(table, 1, "B-2"); rtbl_add_column_entry_by_id(table, 1, "B-3"); rtbl_add_column_entry_by_id(table, 2, "C-3"); rtbl_add_column_entry_by_id(table, 0, "A-4"); rtbl_new_row(table); rtbl_add_column_entry_by_id(table, 1, "B-4"); rtbl_new_row(table); rtbl_add_column_entry_by_id(table, 2, "C-4"); rtbl_new_row(table); rtbl_format(table, stdout); rtbl_destroy(table); return 0; } will output the following: Column A Column B Column C A-1 B-1 C-1 A-2 B-2 C-2 A-3 B-3 C-3 A-4 B-4 C-4 HEIMDAL
June 26, 2004 HEIMDAL
All times are GMT -4. The time now is 09:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy