Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Extracting unique values of a column from a feed file
Top Forums Shell Programming and Scripting Extracting unique values of a column from a feed file Post 302894004 by Don Cragun on Sunday 23rd of March 2014 03:16:34 AM
Old 03-23-2014
You could try something like:
Code:
#!/bin/ksh
cd /usr/cft/str/  || exit 1
awk -F'|' -v id='ADM_ID' '
NR == 1 {
        for(f = 1; f <= NF; f++)
                if($f == id)
                        break
        if(f > NF) {
                printf("Column header \"%s\" not found.\n" id)
                exit 2
        }
        next
}
x[$f]++ == 0 {
        print $f
}' abc1.txt > Unique_ADM

This was tested using the Korn shell, but it will work with any shell that recognizes basic Bourne shell syntax.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

return a list of unique values of a column from csv format file

Hi all, I have a huge csv file with the following format of data, Num SNPs, 549997 Total SNPs,555352 Num Samples, 157 SNP, SampleID, Allele1, Allele2 A001,AB1,A,A A002,AB1,A,A A003,AB1,A,A ... ... ... I would like to write out a list of unique SNP (column 1). Could you... (3 Replies)
Discussion started by: phoeberunner
3 Replies

2. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

3. UNIX for Dummies Questions & Answers

Extracting rows from a text file based on numerical values of a column

I have a text file where the second column is a list of numbers going from small to large. I want to extract the rows where the second column is smaller than or equal to 0.0001. My input: rs10082730 9e-08 12 46002702 rs2544081 1e-07 12 46015487 rs1425136 1e-06 7 35396742 rs2712590... (1 Reply)
Discussion started by: evelibertine
1 Replies

4. UNIX for Dummies Questions & Answers

Extracting rows from a space delimited text file based on the values of a column

I have a space delimited text file. I want to extract rows where the third column has 0 as a value and write those rows into a new space delimited text file. How do I go about doing that? Thanks! (2 Replies)
Discussion started by: evelibertine
2 Replies

5. Shell Programming and Scripting

AWK, Perl or Shell? Unique strings and their maximum values from 3 column data file

I have a file containing data like so: 2012-01-02 GREEN 4 2012-01-02 GREEN 6 2012-01-02 GREEN 7 2012-01-02 BLUE 4 2012-01-02 BLUE 3 2012-01-02 GREEN 4 2012-01-02 RED 4 2012-01-02 RED 8 2012-01-02 GREEN 4 2012-01-02 YELLOW 5 2012-01-02 YELLOW 2 I can't always predict what the... (4 Replies)
Discussion started by: rich@ardz
4 Replies

6. Shell Programming and Scripting

Script for extracting data from csv file based on column values.

Hi all, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 5 columns having values say column 1,column 2.....column 5 as below along with their valuesm.... (3 Replies)
Discussion started by: Vivekit82
3 Replies

7. Linux

To get all the columns in a CSV file based on unique values of particular column

cat sample.csv ID,Name,no 1,AAA,1 2,BBB,1 3,AAA,1 4,BBB,1 cut -d',' -f2 sample.csv | sort | uniq this gives only the 2nd column values Name AAA BBB How to I get all the columns of CSV along with this? (1 Reply)
Discussion started by: sanvel
1 Replies

8. UNIX for Dummies Questions & Answers

Unique values in a row sum the next column in UNIX

Hi would like to ask you guys any advise regarding my problem I have this kind of data file.txt 111111111,20 111111111,50 222222222,70 333333333,40 444444444,10 444444444,20 I need to get this file1.txt 111111111,70 222222222,70 333333333,40 444444444,30 using this code I can... (6 Replies)
Discussion started by: reks
6 Replies

9. UNIX for Beginners Questions & Answers

Find unique values but only in column 1

Hi All, Does anyone have any suggestions/examples of how i could show only lines where the first field is not duplicated. If the first field is listed more than once it shouldnt be shown even if the other columns make it unique. Example file : 876,RIBDA,EC2 876,RIBDH,EX7 877,RIBDF,E28... (4 Replies)
Discussion started by: mutley2202
4 Replies

10. Shell Programming and Scripting

Count number of unique values in each column of array

What is an efficient way of counting the number of unique values in a 400 column by 1000 row array and outputting the counts per column, assuming the unique values in the array are: A, B, C, D In other words the output should look like: Value COL1 COL2 COL3 A 50 51 52... (16 Replies)
Discussion started by: Geneanalyst
16 Replies

LEARN ABOUT HPUX

sh

sh(1)							      General Commands Manual							     sh(1)

NAME

       sh - overview of various system shells

SYNOPSIS

   POSIX Shell
       option] ...  string] [arg ...]

       option] ...  string] [arg ...]

   Korn Shell
       option] ...  string] [arg ...]

       option] ...  string] [arg ...]

   C Shell
       [command_file] [argument_list ...]

   Key Shell
DESCRIPTION

   Remarks
       The POSIX .2 standard requires that, on a POSIX-compliant system, executing the command activates the POSIX shell (located in file on HP-UX
       systems), and executing the command produces an on-line manual entry that displays the syntax of the POSIX shell command-line.

       However, the command has historically been associated with the conventional Bourne shell, which could confuse some users.   To  meet  stan-
       dards  requirements  and also clarify the relationships of the various shells and where they reside on the system, this entry provides com-
       mand-line syntax and a brief description of each shell, and lists the names of the manual entries where each shell is described in  greater
       detail.

       The Bourne shell is removed from the system starting with HP-UX 11i Version 1.5.  Please use the POSIX shell as an alternative.

   Shell Descriptions
       The HP-UX operating system supports the following shells:

	      POSIX-conforming command programming language and command interpreter
			residing  in  file Can execute commands read from a terminal or a file.  This shell conforms to current POSIX standards in
			effect at the time the HP-UX system release was introduced, and is similar to the Korn shell in many respects.	Similar in
			many  respects to the Korn shell, the POSIX shell contains a history mechanism, supports job control, and provides various
			other useful features.

	      Korn-shell command programming language and commands interpreter
			residing in file Can execute commands read from a terminal or a file.  This shell, like the POSIX shell, contains  a  his-
			tory mechanism, supports job control, and provides various other useful features.

	      A command language interpreter
			that incorporates a command history buffer, C-language-like syntax, and job control facilities.

	      Restricted version of the POSIX shell command interpreter.
			Sets  up  a  login  name  and  execution  environment whose capabilities are more controlled (restricted) than normal user
			shells.

	      restricted version of the Korn-shell command interpreter
			Sets up a login name and execution environment whose capabilities  are	more  controlled  (restricted)	than  normal  user
			shells.

	      An extension of the standard Korn Shell
			that uses hierarchical softkey menus and context-sensitive help.

							  +--------------+--------------------+
							  | To obtain:	 |  Use the command:  |
							  +--------------+--------------------+
							  | POSIX Shell  |  /usr/bin/sh ...   |
							  | Korn Shell	 |  /usr/bin/ksh ...  |
							  | C Shell	 |  /usr/bin/csh ...  |
							  | Key Shell	 |  /usr/bin/keysh    |
							  +--------------+--------------------+
       These shells can also be the default invocation, depending on the entry in the file.  See also chsh(1).

WARNINGS

       Many manual entries contain descriptions of shell behavior or describe program or application behavior similar to ``the shell'' with a ref-
       erence to ``see sh(1)''.

SEE ALSO

       For more information on the various individual shells, see:

       keysh(1)       Key Shell description.

       ksh(1)	      Korn Shell description.

       sh-posix(1)    POSIX Shell description.

       csh(1)	      C Shell description.

																	     sh(1)
All times are GMT -4. The time now is 09:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy