Log transformations with awk


 
Thread Tools Search this Thread
# 1  
Log transformations with awk

Hi,

I have a very large csv file with almost 100,000 columns and 200 rows
Example.csv
Code:
ID,field1,field2,field3,field4
A,5,6,7,8
B,1,2,3,4
C,3,5,6,7

I'm trying to find a way, using awk, to log (base 2) all the values in the table (minus the headers and id field)

Output
Code:
ID,field1,field2,field3,field4
A,2.32,2.58,2.8,3 
B,0,1,1.58,2
C,1.58,2.32,2.58,2.8

I know to use
Code:
awk -F"," '{print log($2)/log2}' input.csv > output.csv

to log transform one column, but my script writing isn't very good yet and I have no idea how to do this for all the columns in my input file. Any help is much appreciated.
# 2  
Code:
awk -F, ' NR == 1 {
                print
} NR > 1 {
                printf $1
                for(i = 2; i <= NF; i++)
                        s = s OFS log($i)/log(2)
                print s
                s = ""
}' OFS=, input.csv

This User Gave Thanks to Yoda For This Post:
# 3  
Code:
awk -F, 'FNR>1{for(i=2;i<=NF;i++) $i=log($i)/log2}1' OFS=, CONVFMT='%.2f' myFile

What OS are you on? 'log2' is not supported by most awk-s.

Last edited by vgersh99; 03-05-2013 at 06:26 PM..
This User Gave Thanks to vgersh99 For This Post:
# 4  
Thanks to both of you! I tried them both, vgersh, yours worked a bit faster. I'm using unix. I modified to:

Code:
awk -F, 'FNR>1{for(i=2;i<=NF;i++) $i=log($i)/log(2)}1' OFS=, CONVFMT='%.4f' myFile

Thanks again!
# 5  
UNIX is not an operating system the same way the IEEE is not your wall socket. If you don't know what your operating system is, run uname in your shell.
# 6  
Sorry, linux
This User Gave Thanks to nemo246 For This Post:
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to grep log

hi expert, i Have log like : dn: EPC-Per=673,HYHGU objectClass: EPC-PerSubs ownerId: 0 groupId: 4001 shareTree: nodeName=HYHGU permissions: 15 EPC-Per: 673 dn:... (2 Replies)
Discussion started by: justbow
2 Replies

2. Shell Programming and Scripting

Need help with awk and access log

when i run this command: tail -200 /var/log/httpd/access_log | awk -F'' '{sub(/:*$/,"",$2);print $2}' i get: 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56... (3 Replies)
Discussion started by: SkySmart
3 Replies

3. Shell Programming and Scripting

Data transformations

hi all, I have oracle table which has like below example in col7 & col8 there will be values seperated , in one column it is kind of horizontal pivoting but the data is in 1 column seprated , In near future this col7 and col8 may have some more values col7 * col8 that many records it... (1 Reply)
Discussion started by: marpadga18
1 Replies

4. UNIX for Dummies Questions & Answers

awk and log files

Hello, I have a series of logs that I need to analyse. each looks something like:234.10.72.175 Mon Mar 02 20:25:00 GMT 2009 226.91.87.86 Thu Mar 05 03:50:26 GMT 2009 226.91.87.86 Thu Mar 05 04:06:07 GMT 2009 Using awk, so far I have been able to count the lines in a... (5 Replies)
Discussion started by: Freaky
5 Replies

5. Shell Programming and Scripting

Matching and replacing text with case transformations

Hi, I am trying to edit an XML file automatically but my regex and shell script knowledge is very limited. I would appreciate your help fellows. The XML file has this structure: <?xml version="1.0" encoding="UTF-8"?> <map map_file="maps/world.swf" zoom="350%" zoom_x="-115%"... (1 Reply)
Discussion started by: boonymagique
1 Replies

6. UNIX for Dummies Questions & Answers

log to columns using AWK

How can one take the log of a column using AWK?:( (4 Replies)
Discussion started by: cosmologist
4 Replies

7. Shell Programming and Scripting

Using Awk for stream log

Hi Master, I've log in here then try parsing log become : 3GPP-IMSI,Calling-Station-Id 528111500111614,6738790448 528111500068173,6738769831 ..... it is possible to use awk? cause i 've try awk ' /\Accounting\ /{f=1} f && /\3GPP-IMSI \:\ String Value\ /{imsi=$4} f &&... (1 Reply)
Discussion started by: bowbowz
1 Replies

8. UNIX for Dummies Questions & Answers

Use awk to log transform

Hello. I'm trying to use the awk command to convert certains fields to lgo base 2, not base 10. I'm trying command lines like: awk '$2 $5 $7 {print log(2)($2), log(2)($5), $7) filein.txt > fileout.txt I'm trying to make a deadline. Thanks for helping a newbie. Here's some fresh karma... (1 Reply)
Discussion started by: jmzcal
1 Replies

9. UNIX for Dummies Questions & Answers

Numerical System Transformations - Lecture

Hy, I wrote new article on Numerical Systems and Transformations yesterday evening (on my C Lessons Project webpage) and would like to share it with you. If someone can help me format my text for this forum, I would appreciate it, so I culd post full article here.. Here's intro: "Our... (0 Replies)
Discussion started by: vurdlak
0 Replies

Featured Tech Videos