Sponsored Content
Top Forums Shell Programming and Scripting Find columns in a file based on header and print to new file Post 302986507 by drl on Saturday 26th of November 2016 08:16:28 AM
Old 11-26-2016
Hi.

This issue seems to be resolved, but for completeness on the sub-topic of csvtool, there are options for specifying the input and output field separators (see -t and -u below). I should have presented this version first:
Code:
#!/usr/bin/env bash

# @(#) s2       Demonstrate extraction of fields, csvtool.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C dixf sed pass-fail
pe
dixf csvtool

FILE=${1-data1}
E=expected-output.txt
H=headers

pl " Input data file $FILE:"
head $FILE

pl " Header name file $H, name list:"
head $H
h1=$( paste -s -d, $H )
pe " Header list = $h1"

pl " Expected output:"
cat $E

pl " Results, extract spaced-separated fields, write TAB-spaced:"
csvtool namedcol $h1 -t " " -u TAB $FILE |
tee f1

pl " Verify results if possible:"
C=$HOME/bin/pass-fail
[ -f $C ] && $C || ( pe; pe " Results cannot be verified." ) >&2

exit 0

produces the same output as in post #5.

In addition, there are some facilities that handle embedded problematic characters in field strings in csv-formatted files with perl modules. For example, a small code (about 15 lines of script, but surrounded by 100 lines of supporting code for error detection, argument processing, etc. plus comments) adds quotes to fields, as in:
Code:
#!/usr/bin/env bash

# @(#) s3       Demonstrate short perl code to add quotes to csv files.

LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C dixf
pe
dixf csvaq

FILE=${1-data3}

p=$( basename $0 ) t1="$Revision: 1.14 $" v=${t1//[!0-9.]/}
[[ $# -gt 0 ]] && [[ "$1" =~ -version ]] &&  { echo "$p (local) $v" ; exit 0 ; }

pl " Input data file $FILE:"
head $FILE

pl " Results, always add quotes:"
csvaq -q -t " " -u " " $FILE

exit 0

producing:
Code:
$ ./s3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
bash GNU bash 4.3.30
dixf (local) 1.21

csvaq   Add quotes for Comma Separated Value format files. (what)
Path    : ~/bin/csvaq
Version : 1.3
Length  : 165 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/env perl
Help    : probably available with [     ]-h,--help
Modules : (for perl codes)
 warnings       1.23
 strict 1.08
 diagnostics    1.34
 English        1.09
 Carp   1.3301
 Data::Dumper   2.151_01
 Getopt::Long   2.42
 feature        1.36_01
 experimental   0.007
 Text::CSV      1.32

-----
 Input data file data3:
cn0 name
cn,1 1,1,4,4-tetrabutylpiperazine
cn,2 N,N,N',N'-tetramethylguanidine
cn,3 1-(2-aminoethyl)piperazine

-----
 Results, always add quotes:
"cn0" "name"
"cn,1" "1,1,4,4-tetrabutylpiperazine"
"cn,2" "N,N,N',N'-tetramethylguanidine"
"cn,3" "1-(2-aminoethyl)piperazine"

Best wishes ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Changing file content based on file header

Hi, I have several text files each containing some data as shown below: File1.txt >DataHeader Data... Data... File2.txt >DataHeader Data... Data... etc. What I want is to change the 'DataHeader' based on the file name. So the output should look like: File1.txt >File1 ... (1 Reply)
Discussion started by: Fahmida
1 Replies

2. Shell Programming and Scripting

Need to find a column from one file and print certain columns in second file

Hi, I need helping in finding some of the text in one file and some columns which have same column in file 1 EG cat file_1 aaaa bbbb cccc dddd eeee fffff gggg hhhh cat file_2 aaaa,abcd,effgh,ereref,name,age,sex,........... bbbb,efdfh,erere,afdafds,name,age,sex.............. (1 Reply)
Discussion started by: jpkumar10
1 Replies

3. Shell Programming and Scripting

Awk based script to find the median of all individual columns in a data file

Hi All, I have some data like below. Step1,Param1,Param2,Param3 1,2,3,4 2,3,4,5 2,4,5,6 3,0,1,2 3,0,0,0 3,2,1,3 ........ so on Where I need to find the median(arithmetic) of each column from Param1...to..Param3 for each set of Step1 values. (Sort each specific column, if the... (5 Replies)
Discussion started by: ks_reddy
5 Replies

4. Shell Programming and Scripting

Remove the file content based on the Header of the file

Hi All, I want to remove the content based on the header information . Please find the example below. File1.txt Name|Last|First|Location|DepId|Depname|DepLoc naga|rr|tion|hyd|1|wer|opr Nava|ra|tin|gen|2|wera|opra I have to search for the DepId and remove the data from the... (5 Replies)
Discussion started by: i150371485
5 Replies

5. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have... (4 Replies)
Discussion started by: ks_reddy
4 Replies

6. Shell Programming and Scripting

Compare two files and find match and print the header of the second file

Hi, I have two input files; file1 and file2. I compare them based on matched values in 1 column and print selected columns of the second file (file2). I got the result but the header was not printed. i want the header of file2 to be printed together with the result. Then i did below codes:- ... (3 Replies)
Discussion started by: redse171
3 Replies

7. Shell Programming and Scripting

Make copy of text file with columns removed (based on header)

Hello, I have some tab delimited text files with a three header rows. The headers look like, (sorry the tabs look so messy). index group Name input input input input input input input input input input input... (9 Replies)
Discussion started by: LMHmedchem
9 Replies

8. Emergency UNIX and Linux Support

Average columns based on header name

Hi Friends, I have files with columns like this. This sample input below is partial. Please check below for main file link. Each file will have only two rows. ... (8 Replies)
Discussion started by: jacobs.smith
8 Replies

9. Shell Programming and Scripting

Find header in a text file and prepend it to all lines until another header is found

I've been struggling with this one for quite a while and cannot seem to find a solution for this find/replace scenario. Perhaps I'm getting rusty. I have a file that contains a number of metrics (exactly 3 fields per line) from a few appliances that are collected in parallel. To identify the... (3 Replies)
Discussion started by: verdepollo
3 Replies

10. UNIX for Beginners Questions & Answers

How to print multiple required columns dynamically in a file using the header name?

Hi All, i am trying to print required multiple columns dynamically from a fie. But i am able to print only one column at a time. i am new to shell script, please help me on this issue. i am using below script awk -v COLT=$1 ' NR==1 { for (i=1; i<=NF; i++) { ... (2 Replies)
Discussion started by: balu1234
2 Replies
All times are GMT -4. The time now is 07:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy