Find columns in a file based on header and print to new file Post: 302986407

Sponsored Content

Top Forums Shell Programming and Scripting Find columns in a file based on header and print to new file Post 302986407 by LMHmedchem on Friday 25th of November 2016 12:26:41 AM

11-25-2016

Registered User

Find columns in a file based on header and print to new file

Hello,

I have to fish out some specific columns from a file based on the header value. I have the list of columns I need in a different file. I thought I could read in the list of headers I need,

Code:

# file with header names of required columns in required order
headers_file=$2

# read contents of headers_file into array
IFS=$'\n' read -a headers_list < $headers_file

and then loop through the list to pick out each column I need,

Code:

# loop on header list
for header_value in "${headers_list[@]}"
do
   # print current input file
   echo $header_value

   # look for the column in the input file
   awk -v OFS='\t' -v column_header="$header_value" 'NR==1{for (i=1; i<=NF; i++) if ($i==column_header){p=i; break}; next} {print $p}' $input_file > $output_file

done

The above awk does not work and even if it did it would overwrite the data from each previous column found. How do I find all the columns I need and then print all of them in the right order so they all end up in the output file?

The only thing I could think of was to read the header line from $input_file into another array and then loop through $headers_list making a note of the numerical position of the columns I need. In theory, I could use the list of numerical positions to cobble together a cut argument to get the columns I need. That seems like it would be horribly messy syntax and could probably be done with one line of awk from someone who knows what they are doing.

That means it's time to post and ask for help. I found allot of topics like this one, but most of them seemed to find one column by the header value and print it.

In case that makes a difference, the input files I am working have < 200 columns but may have almost any number of rows. The input file is space delimited and the output should be tab delimited, though I could replace space with tab after the fact if necessary.

Suggestions would be greatly appreciated.

LMHmedchem

LMHmedchem

View Public Profile for LMHmedchem

Find all posts by LMHmedchem

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Changing file content based on file header

Hi, I have several text files each containing some data as shown below: File1.txt >DataHeader Data... Data... File2.txt >DataHeader Data... Data... etc. What I want is to change the 'DataHeader' based on the file name. So the output should look like: File1.txt >File1 ...

2. Shell Programming and Scripting

Need to find a column from one file and print certain columns in second file

Hi, I need helping in finding some of the text in one file and some columns which have same column in file 1 EG cat file_1 aaaa bbbb cccc dddd eeee fffff gggg hhhh cat file_2 aaaa,abcd,effgh,ereref,name,age,sex,........... bbbb,efdfh,erere,afdafds,name,age,sex..............

3. Shell Programming and Scripting

Awk based script to find the median of all individual columns in a data file

Hi All, I have some data like below. Step1,Param1,Param2,Param3 1,2,3,4 2,3,4,5 2,4,5,6 3,0,1,2 3,0,0,0 3,2,1,3 ........ so on Where I need to find the median(arithmetic) of each column from Param1...to..Param3 for each set of Step1 values. (Sort each specific column, if the...

4. Shell Programming and Scripting

Remove the file content based on the Header of the file

Hi All, I want to remove the content based on the header information . Please find the example below. File1.txt Name|Last|First|Location|DepId|Depname|DepLoc naga|rr|tion|hyd|1|wer|opr Nava|ra|tin|gen|2|wera|opra I have to search for the DepId and remove the data from the...

5. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have...

6. Shell Programming and Scripting

Compare two files and find match and print the header of the second file

Hi, I have two input files; file1 and file2. I compare them based on matched values in 1 column and print selected columns of the second file (file2). I got the result but the header was not printed. i want the header of file2 to be printed together with the result. Then i did below codes:- ...

7. Shell Programming and Scripting

Make copy of text file with columns removed (based on header)

Hello, I have some tab delimited text files with a three header rows. The headers look like, (sorry the tabs look so messy). index group Name input input input input input input input input input input input...

8. Emergency UNIX and Linux Support

Average columns based on header name

Hi Friends, I have files with columns like this. This sample input below is partial. Please check below for main file link. Each file will have only two rows. ...

9. Shell Programming and Scripting

Find header in a text file and prepend it to all lines until another header is found

I've been struggling with this one for quite a while and cannot seem to find a solution for this find/replace scenario. Perhaps I'm getting rusty. I have a file that contains a number of metrics (exactly 3 fields per line) from a few appliances that are collected in parallel. To identify the...

10. UNIX for Beginners Questions & Answers

How to print multiple required columns dynamically in a file using the header name?

Hi All, i am trying to print required multiple columns dynamically from a fie. But i am able to print only one column at a time. i am new to shell script, please help me on this issue. i am using below script awk -v COLT=$1 ' NR==1 { for (i=1; i<=NF; i++) { ...

LEARN ABOUT OPENSOLARIS

sip_is_param_present

sip_get_param_value(3SIP)			   Session Initiation Protocol Library Functions			 sip_get_param_value(3SIP)

NAME

       sip_get_param_value, sip_get_params, sip_is_param_present - get parameter information for a SIP header value

SYNOPSIS

       cc [ flag ... ] file ... -lsip [ library ... ]
       #include <sip.h>

       const sip_str_t *sip_get_param_value
	    (sip_header_value_t header_value, char *param_name, int *error);

       const sip_param_t *sip_get_params
	    (sip_header_value_t header_value, int *error);;

       boolean_t sip_is_param_present
	    (const sip_param_t *param_list, char *param_name, int param_len);

DESCRIPTION

       The sip_get_param_value() function returns the value for the parameter name specified by param_name from the SIP header value header_value.

       For functions that return a pointer of type sip_str_t, sip_str_t is supplied by:

	 typedef struct sip_str {
	      char *sip_str_ptr;
	      int  sip_str_len;
	 }sip_str_t;

       The sip_str_ptr parameter points to the start of the returned value and sip_str_len supplies the length of the returned value.

       The sip_get_params() function returns the parameter list, if any, for the SIP header value header_value.

       The  sip_is_param_present()  function returns B_TRUE if the parameter specified by param_name of length supplied in param_len is present in
       the parameter list, param_list. Otherwise, it returns B_FALSE.

RETURN VALUES

       With the exception of sip_is_param_present(), these functions return the queried value on success and NULL on failure.

       The value of errno is not changed by these calls in the event of an error.

ERRORS

       If the error is non-null, one of the following values is set:

       EINVAL	 If any of the required input is NULL or if the specified SIP header value is marked deleted.

       EPROTO	 If the returned SIP header value is invalid (i.e. the parser encountered errors when parsing the value).

       On success, the value of the location pointed to by error is set to 0.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Committed 		   |
       +-----------------------------+-----------------------------+
       |MT-Level		     |MT-Safe			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       libsip(3LIB)

SunOS 5.11							    25 Jan 2007 					 sip_get_param_value(3SIP)