Sponsored Content
Top Forums Shell Programming and Scripting distinct values of all the fields Post 302461955 by vukkusila on Tuesday 12th of October 2010 09:13:49 PM
Old 10-12-2010
distinct values of all the fields

I am a beginner to scripting, please help me in this regard.

How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number of columns as a parameter to the script or the script by itself should be able to recognize the number of columns in the file based on the delimiter. The script needs to generate the output as below.

Sample data

Field1|Field2|Field3|Field4
AAA|BBB|CCC|DDD
111|222|333|777
AAA|EEE|ZZZ|EEE
111|555|333|444
AAA|EEE|CCC|DDD
111|222|555|444

For the above file, the result I am looking for would be:

Field1
AAA(3)
111(3)

Field2
BBB(1)
222(2)
EEE(2)
555(1)

Field3
ccc(2)
333(2)
zzz(1)
555(1)

Field4
DDD(2)
777(1)
EEE(1)
444(2)

Thank you in advance for your assistance.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Loop through only the distinct values in a file

Datafile has the following data seperated by : FIELD1:FIELD2:FIELD3 D1:/opt/9.1.9:Y D2:/opt/10.1.10:Y D3:/opt/9.1.9:Y D4:/opt/8.1.8:Y D5:/opt/8.1.8:Y D6:/opt/9.1.9:Y D7:/opt/9.1.9:Y D8:/opt/10.1.10:Y D9:/opt/9.1.9:Y D10:/opt/10.1.10:Y I want to do some operations only on the distinct... (2 Replies)
Discussion started by: pbekal
2 Replies

2. Shell Programming and Scripting

Awk to print distinct col values

Hi Guys... I am newbie to awk and would like a solution to probably one of the simple practical questions. I have a test file that goes as: 1,2,3,4,5,6 7,2,3,8,7,6 9,3,5,6,7,3 8,3,1,1,1,1 4,4,2,2,2,2 I would like to know how AWK can get me the distinct values say for eg: on col2... (22 Replies)
Discussion started by: anduzzi
22 Replies

3. Shell Programming and Scripting

Getting Distinct values from second field in a file....

Hi I have a pipe delimited file. I am trying to grab the DISTINCT value from the second field. The file is something like: 1233|apple|ron 1234|apple|elephant 1235|egg|man the output I am trying to get from second field is apple,egg (apple coming only once) Thanks simi (4 Replies)
Discussion started by: simi28
4 Replies

4. UNIX for Dummies Questions & Answers

Select Distinct on multiple fields

How do I create a script that provides a count of distinct values of a particular field in a file utilizing commonly available UNIX commands (sh or awk)? Field1|Field2|Field3|Field4 AAA|BBB|CCC|DDD 111|222|333|777 AAA|EEE|ZZZ|EEE 111|555|333|444 AAA|EEE|CCC|DDD 111|222|555|444 For... (2 Replies)
Discussion started by: Refresher
2 Replies

5. Shell Programming and Scripting

grep distinct values

this is a little more complex than that. I have a text file and I need to find all the distinct words that appear in a line after the word TABLESPACE when I grep for just the word tablespace, I get: how do i parse this a little better so i have a smaller file to read? This is just an... (4 Replies)
Discussion started by: guessingo
4 Replies

6. Shell Programming and Scripting

To count distinct fields in a row

I have . dat file which contains data in a specific format: 0 3 892 921 342 1 3 921 342 543 2 4 817 562 718 765 3 3 819 562 717 761 i need to compare each field in a row with another field of the same column but different row and cont the... (8 Replies)
Discussion started by: Abhik
8 Replies

7. UNIX for Dummies Questions & Answers

distinct values of all the fields

I am a beginner to scripting, please help me in this regard. How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number... (1 Reply)
Discussion started by: vukkusila
1 Replies

8. Shell Programming and Scripting

average of distinct values with awk

Hi guys, I am not an expert in shell and I need help with awk command. I have a file with values like 200 1 1 200 7 2 200 6 3 200 5 4 300 3 1 300 7 2 300 6 3 300 4 4 I need resulting file with averages of... (3 Replies)
Discussion started by: saif
3 Replies

9. Shell Programming and Scripting

Find distinct values

Hi, I have two files of the following format file1 chr1:345-456 chr2:123-456 chr2:455-678 chr3:456-789 chr3:444-555 file2 chr1:345-456 chr2:123-456 chr3:456-789 output (2 Replies)
Discussion started by: jacobs.smith
2 Replies

10. Shell Programming and Scripting

Need distinct values from command in a script

Hello, I am using below command srvctl config service -d cmdbut cmdbut_01 (P):/devoragridcn_01/app/oracle> srvctl config service -d cmdbut Service name: boms10.world Service is enabled Server pool: cmdbut_boms10.world Cardinality: 1 Disconnect: false Service role: PRIMARY Management... (7 Replies)
Discussion started by: Vishal_dba
7 Replies
PERF_3.2-SCRIPT-PERL(1) 					    perf Manual 					   PERF_3.2-SCRIPT-PERL(1)

NAME
perf-script-perl - Process trace data with a Perl script SYNOPSIS
perf script [-s [Perl]:script[.pl] ] DESCRIPTION
This perf script option is used to process perf script data using perf's built-in Perl interpreter. It reads and processes the input file and displays the results of the trace analysis implemented in the given Perl script, if any. STARTER SCRIPTS
You can avoid reading the rest of this document by running perf script -g perl in the same directory as an existing perf.data trace file. That will generate a starter script containing a handler for each of the event types in the trace file; it simply prints every available field for each event in the trace file. You can also look at the existing scripts in ~/libexec/perf-core/scripts/perl for typical examples showing how to do basic things like aggregate event data, print results, etc. Also, the check-perf-script.pl script, while not interesting for its results, attempts to exercise all of the main scripting features. EVENT HANDLERS
When perf script is invoked using a trace script, a user-defined handler function is called for each event in the trace. If there's no handler function defined for a given event type, the event is ignored (or passed to a trace_handled function, see below) and the next event is processed. Most of the event's field values are passed as arguments to the handler function; some of the less common ones aren't - those are available as calls back into the perf executable (see below). As an example, the following perf record command can be used to record all sched_wakeup events in the system: # perf record -a -e sched:sched_wakeup Traces meant to be processed using a script should be recorded with the above option: -a to enable system-wide collection. The format file for the sched_wakep event defines the following fields (see /sys/kernel/debug/tracing/events/sched/sched_wakeup/format): .ft C format: field:unsigned short common_type; field:unsigned char common_flags; field:unsigned char common_preempt_count; field:int common_pid; field:char comm[TASK_COMM_LEN]; field:pid_t pid; field:int prio; field:int success; field:int target_cpu; .ft The handler function for this event would be defined as: .ft C sub sched::sched_wakeup { my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs, $common_pid, $common_comm, $comm, $pid, $prio, $success, $target_cpu) = @_; } .ft The handler function takes the form subsystem::event_name. The $common_* arguments in the handler's argument list are the set of arguments passed to all event handlers; some of the fields correspond to the common_* fields in the format file, but some are synthesized, and some of the common_* fields aren't common enough to to be passed to every event as arguments but are available as library functions. Here's a brief description of each of the invariant event args: $event_name the name of the event as text $context an opaque 'cookie' used in calls back into perf $common_cpu the cpu the event occurred on $common_secs the secs portion of the event timestamp $common_nsecs the nsecs portion of the event timestamp $common_pid the pid of the current task $common_comm the name of the current process All of the remaining fields in the event's format file have counterparts as handler function arguments of the same name, as can be seen in the example above. The above provides the basics needed to directly access every field of every event in a trace, which covers 90% of what you need to know to write a useful trace script. The sections below cover the rest. SCRIPT LAYOUT
Every perf script Perl script should start by setting up a Perl module search path and 'use'ing a few support modules (see module descriptions below): .ft C use lib "$ENV{'PERF_EXEC_PATH'}/scripts/perl/perf-script-Util/lib"; use lib "./perf-script-Util/lib"; use Perf::Trace::Core; use Perf::Trace::Context; use Perf::Trace::Util; .ft The rest of the script can contain handler functions and support functions in any order. Aside from the event handler functions discussed above, every script can implement a set of optional functions: trace_begin, if defined, is called before any event is processed and gives scripts a chance to do setup tasks: .ft C sub trace_begin { } .ft trace_end, if defined, is called after all events have been processed and gives scripts a chance to do end-of-script tasks, such as display results: .ft C sub trace_end { } .ft trace_unhandled, if defined, is called after for any event that doesn't have a handler explicitly defined for it. The standard set of common arguments are passed into it: .ft C sub trace_unhandled { my ($event_name, $context, $common_cpu, $common_secs, $common_nsecs, $common_pid, $common_comm) = @_; } .ft The remaining sections provide descriptions of each of the available built-in perf script Perl modules and their associated functions. AVAILABLE MODULES AND FUNCTIONS
The following sections describe the functions and variables available via the various Perf::Trace::* Perl modules. To use the functions and variables from the given module, add the corresponding use Perf::Trace::XXX line to your perf script script. Perf::Trace::Core Module These functions provide some essential functions to user scripts. The flag_str and symbol_str functions provide human-readable strings for flag and symbolic fields. These correspond to the strings and values parsed from the print fmt fields of the event format files: flag_str($event_name, $field_name, $field_value) - returns the string represention corresponding to $field_value for the flag field $field_name of event $event_name symbol_str($event_name, $field_name, $field_value) - returns the string represention corresponding to $field_value for the symbolic field $field_name of event $event_name Perf::Trace::Context Module Some of the common fields in the event format file aren't all that common, but need to be made accessible to user scripts nonetheless. Perf::Trace::Context defines a set of functions that can be used to access this data in the context of the current event. Each of these functions expects a $context variable, which is the same as the $context variable passed into every event handler as the second argument. common_pc($context) - returns common_preempt count for the current event common_flags($context) - returns common_flags for the current event common_lock_depth($context) - returns common_lock_depth for the current event Perf::Trace::Util Module Various utility functions for use with perf script: nsecs($secs, $nsecs) - returns total nsecs given secs/nsecs pair nsecs_secs($nsecs) - returns whole secs portion given nsecs nsecs_nsecs($nsecs) - returns nsecs remainder given nsecs nsecs_str($nsecs) - returns printable string in the form secs.nsecs avg($total, $n) - returns average given a sum and a total number of values SEE ALSO
perf_3.2-script(1) perf 06/24/2012 PERF_3.2-SCRIPT-PERL(1)
All times are GMT -4. The time now is 08:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy