Sponsored Content
Top Forums Shell Programming and Scripting awk based script to print the "mode(statistics term)" for each column in a data file Post 302662085 by ks_reddy on Tuesday 26th of June 2012 06:40:14 AM
Old 06-26-2012
This script is perfect for files with known number of columns

Hi Pamu,

Thanks for your quick reply.
This script is perfectly fine for all files which have equal number of columns.
My challenge is to find 'mode' for files(.csv type) where
1) I don't know how many columns are there unless I open each file and check(very tedious task)
2) Printing the output side by side like mode_of_col1,mode_of_col2 etc.
For the first one i can count the number of columns in each file and feed into variable and loop through.
For the second one I can use the command
Code:
tr '\n' ' '

to transpose the output.

But all these change are making my code very lengthy and also I need to process many files (some times in 1000's).
Thats why I am looking for automatic looping script to print 'mode' for each column as well as looping to random number of columns.
Can you modify your script if you can ?

Thanks
Sid
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. Shell Programming and Scripting

cat $como_file | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g'

hi All, cat file_name | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g' Can this be done by using sed or awk alone (4 Replies)
Discussion started by: harshakusam
4 Replies

3. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

4. Shell Programming and Scripting

Simplify Bash Script Using "sed" Or "awk"

Input file: 2 aux003.net3.com error12 6 awn0117.net1.com error13 84 aux008 error14 29 aux001.ha.ux.isd.com error12 209 aux002.vm.ux.isd.com error34 21 alx0027.vm.net2.com error12 227 dux001.net5.com error123 22 us008.dot.net2.com error121 13 us009.net2.com error129Expected Output: 2... (4 Replies)
Discussion started by: sQew
4 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

Choosing between repeated entries based on the "absolute values" of a column

Hello, I was looking for a way to select between the repeated entries (column1) based on the values of absolute values of column 3 (larger value). For example if the same gene id has FC value -2 and 1, I should get the output as -2. Kindly help. GeneID Description FC ... (2 Replies)
Discussion started by: Sanchari
2 Replies

7. UNIX for Dummies Questions & Answers

Replacing "." with "GG" in a certain column of a file that has heading

Hi, all, I have a file that looks like: ## XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ## YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY #AA AB AC AD AE AF AG AH AI AJ AK AL 20 60039 60039 ... (5 Replies)
Discussion started by: kush
5 Replies

8. Shell Programming and Scripting

replace the contents of first column of file "X" with second Column of file "X" in file "Y"

Hi! I am having 02 files. In first file" X" I am having 02 Columns TCP-5100 Sybase_5100 TCP-5600 Sybase_5600 Second file "Y" for example-- :services ( :AdminInfo ( :chkpf_uid ("{A2F79713-B67D-4409-83A4-A90804E983E9}") :ClassName (rule_services) ) :compound ()... (12 Replies)
Discussion started by: shahid1632
12 Replies

9. Shell Programming and Scripting

Why awk print is strange when I set FS = " " instead of FS = "\t"?

Look at the following data file(cou.data) which has four fields separated by tab. Four fields are country name, land area, population, continent where it belongs. As for country name or continent name which has two words, two words are separated by space. (Data are not accurately... (1 Reply)
Discussion started by: chihuyu
1 Replies

10. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies
ANALYZE(7)							   SQL Commands 							ANALYZE(7)

NAME
ANALYZE - collect statistics about a database SYNOPSIS
ANALYZE [ VERBOSE ] [ table [ ( column [, ...] ) ] ] DESCRIPTION
ANALYZE collects statistics about the contents of tables in the database, and stores the results in the pg_statistic system catalog. Subse- quently, the query planner uses these statistics to help determine the most efficient execution plans for queries. With no parameter, ANALYZE examines every table in the current database. With a parameter, ANALYZE examines only that table. It is further possible to give a list of column names, in which case only the statistics for those columns are collected. PARAMETERS
VERBOSE Enables display of progress messages. table The name (possibly schema-qualified) of a specific table to analyze. Defaults to all tables in the current database. column The name of a specific column to analyze. Defaults to all columns. OUTPUTS
When VERBOSE is specified, ANALYZE emits progress messages to indicate which table is currently being processed. Various statistics about the tables are printed as well. NOTES
In the default PostgreSQL configuration, in the documentation takes care of automatic analyzing of tables when they are first loaded with data, and as they change throughout regular operation. When autovacuum is disabled, it is a good idea to run ANALYZE periodically, or just after making major changes in the contents of a table. Accurate statistics will help the planner to choose the most appropriate query plan, and thereby improve the speed of query processing. A common strategy is to run VACUUM [vacuum(7)] and ANALYZE once a day during a low-usage time of day. ANALYZE requires only a read lock on the target table, so it can run in parallel with other activity on the table. The statistics collected by ANALYZE usually include a list of some of the most common values in each column and a histogram showing the approximate data distribution in each column. One or both of these can be omitted if ANALYZE deems them uninteresting (for example, in a unique-key column, there are no common values) or if the column data type does not support the appropriate operators. There is more infor- mation about the statistics in in the documentation. For large tables, ANALYZE takes a random sample of the table contents, rather than examining every row. This allows even very large tables to be analyzed in a small amount of time. Note, however, that the statistics are only approximate, and will change slightly each time ANA- LYZE is run, even if the actual table contents did not change. This might result in small changes in the planner's estimated costs shown by EXPLAIN [explain(7)]. In rare situations, this non-determinism will cause the planner's choices of query plans to change after ANALYZE is run. To avoid this, raise the amount of statistics collected by ANALYZE, as described below. The extent of analysis can be controlled by adjusting the default_statistics_target configuration variable, or on a column-by-column basis by setting the per-column statistics target with ALTER TABLE ... ALTER COLUMN ... SET STATISTICS (see ALTER TABLE [alter_table(7)]). The target value sets the maximum number of entries in the most-common-value list and the maximum number of bins in the histogram. The default target value is 100, but this can be adjusted up or down to trade off accuracy of planner estimates against the time taken for ANALYZE and the amount of space occupied in pg_statistic. In particular, setting the statistics target to zero disables collection of statistics for that column. It might be useful to do that for columns that are never used as part of the WHERE, GROUP BY, or ORDER BY clauses of queries, since the planner will have no use for statistics on such columns. The largest statistics target among the columns being analyzed determines the number of table rows sampled to prepare the statistics. Increasing the target causes a proportional increase in the time and space needed to do ANALYZE. COMPATIBILITY
There is no ANALYZE statement in the SQL standard. SEE ALSO
VACUUM [vacuum(7)], vacuumdb [vacuumdb(1)], in the documentation, in the documentation SQL - Language Statements 2010-05-14 ANALYZE(7)
All times are GMT -4. The time now is 02:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy