HELP - uniq values per column Post: 302984680

Sponsored Content

Top Forums Shell Programming and Scripting HELP - uniq values per column Post 302984680 by RudiC on Saturday 29th of October 2016 01:10:06 PM

10-29-2016

Registered User

Jumping in as Don Cragun seems to be logged out at the moment:

Code:

awk '
function AddFieldData(field)    {if ( (field, $field) in data) return                           # if field No. and field contents occurred (and registered) before, quit function
                                                                                                # data is an array whose elements'' contents don''t count, only its indices

                                 data[ field, FieldValue [field, ++count[field] ] = $field]     # this is quite complex (array elements are defined on first reference):
                                                                                                # register field No. and field contents as index into data array, and
                                                                                                #   at the same time, create FieldValue array element holding field contents and indexed
                                                                                                #   by field No. and an incremented counter for the field No. 
                                 if (count[field] > maxr) maxr = count[field]                   # keep max count across ALL fields (i.e. the No. of lines to be printed
                                                                                                #   e.g. for three unique values in field2 we need to print three lines)
                                 if (field > maxc) maxc = field                                 # keep max of NF across all lines
                                }

        {for(i = 1; i <= NF; i++) AddFieldData(i)                                               # main: call above function for ALL fields in ALL lines filling the necessary arrays
        }

END     {for (i = 1; i <= maxr; i++)                                                            # print maxr lines with
           for (j = 1; j <= maxc; j++)                                                          # maxc fields, each
                printf("%s%s", FieldValue[j, i], (j == maxc) ? ORS : OFS)                       # a sort of random distribution of unique field values ; after last (maxc)
                                                                                                # field print line feed (ORS) else field separator (OFS, space)
        }
'  file

This User Gave Thanks to RudiC For This Post:

RudiC

View Public Profile for RudiC

Find all posts by RudiC

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Column sum group by uniq records

Dear All, I want to get help for below case. I have a file like this. saman 1 gihan 2 saman 4 ravi 1 ravi 2 so i want to get the result, saman 5 gihan 2 ravi 3 like this. Pls help me.

2. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the...

3. UNIX for Dummies Questions & Answers

Re: How To Use UNIQ UNIX Command On single Column

Hi , Can You Please let Know How use unix uniq command on a single column for deleting records from file with Below Structure.Pipe Delimter File . Source Name | Account_Id A | 101 B...

4. UNIX for Dummies Questions & Answers

shift values in one column as header for values in another column

Hi Gurus, I have a tab separated text file with two columns. I would like to make the first column values as headings for the second column values. Ex. >value1 subjects >value2 priorities >value3 requirements ...etc and I want to have a file >value1 subjects >value2 priorities...

5. Shell Programming and Scripting

for uniq entries add values in corresponding columns

Hi, I have a file as listed below.. What I want to get is for each unique value in column 1 the corresponding values in the rest of the columns should be summed up.. AAK1 0 1 0 11 AAK1 0 0 1 1 AAK1 0 0 1 2...

6. Shell Programming and Scripting

awk uniq and longest string of a column as index

I met a challenge to filter ~70 millions of sequence rows and I want using awk with conditions: 1) longest string of each pattern in column 2, ignore any sub-string, as the index; 2) all the unique patterns after 1); 3) print the whole row; input: 1 ABCDEFGHI longest_sequence1 2 ABCDEFGH...

7. Shell Programming and Scripting

Filter uniq field values (non-substring)

Hello, I want to filter column based on string value. All substring matches are filtered out and only unique master strings are picked up. infile: 1 abcd 2 abc 3 abcd 4 cdef 5 efgh 6 efgh 7 efx 8 fgh Outfile: 1 abcd 4 cdef 5 efgh 7 efxI have tried awk '!a++; match(a, $2)>0'...

8. Shell Programming and Scripting

Uniq count second column

Hello How can I get a number of occurrence count for this file; ERR315389.1000156 CTTGAAGAAGAATTGAAAACTGTGACGAACAACTTGAAGTCACTGGAGGCTCAGGCTGAGAAGTACTCGCAGAAGGAAGACAGATATGAGGAAGAG ERR315389.1000281 ...

9. Shell Programming and Scripting

Bring values in the second column into single line (comma sep) for uniq value in the first column

I want to bring values in the second column into single line for uniq value in the first column. My input jvm01, Web 2.0 Feature Pack Library jvm01, IBM WebSphere JAX-RS jvm01, Custom01 Shared Library jvm02, Web 2.0 Feature Pack Library jvm02, IBM WebSphere JAX-RS jvm03, Web 2.0 Feature...

10. UNIX for Beginners Questions & Answers

Get first column value uniq

Hi All, I have a directory and sub-directory that having �n' number of .log file in nearly 1GB. The file is comma separated file. I need to recursively grep and uniq first column values only. I did in perl. But i wish to know more command line utilities to calculate the time for grep and...

LEARN ABOUT LINUX

uniq

UNIQ(1) 							   User Commands							   UNIQ(1)

NAME

       uniq - report or omit repeated lines

SYNOPSIS

       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION

       Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

       With no options, matching lines are merged to the first occurrence.

       Mandatory arguments to long options are mandatory for short options too.

       -c, --count
	      prefix lines by the number of occurrences

       -d, --repeated
	      only print duplicate lines

       -D, --all-repeated[=delimit-method]
	      print all duplicate lines delimit-method={none(default),prepend,separate} Delimiting is done with blank lines

       -f, --skip-fields=N
	      avoid comparing the first N fields

       -i, --ignore-case
	      ignore differences in case when comparing

       -s, --skip-chars=N
	      avoid comparing the first N characters

       -u, --unique
	      only print unique lines

       -z, --zero-terminated
	      end lines with 0 byte, not newline

       -w, --check-chars=N
	      compare no more than N characters in lines

       --help display this help and exit

       --version
	      output version information and exit

       A field is a run of blanks (usually spaces and/or TABs), then non-blank characters.  Fields are skipped before chars.

       Note:  'uniq'  does  not  detect  repeated  lines unless they are adjacent.  You may want to sort the input first, or use `sort -u' without
       `uniq'.	Also, comparisons honor the rules specified by `LC_COLLATE'.

AUTHOR

       Written by Richard M. Stallman and David MacKenzie.

REPORTING BUGS

       Report uniq bugs to bug-coreutils@gnu.org
       GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
       General help using GNU software: <http://www.gnu.org/gethelp/>
       Report uniq translation bugs to <http://translationproject.org/team/>

COPYRIGHT

       Copyright (C) 2010 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       comm(1), join(1)

       The full documentation for uniq is maintained as a Texinfo manual.  If the info and uniq programs are properly installed at your site,  the
       command

	      info coreutils 'uniq invocation'

       should give you access to the complete manual.

GNU coreutils 8.5						   February 2011							   UNIQ(1)