Log transformations with awk


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Log transformations with awk
# 1  
Old 03-05-2013
Log transformations with awk

Hi,

I have a very large csv file with almost 100,000 columns and 200 rows
Example.csv
Code:
ID,field1,field2,field3,field4
A,5,6,7,8
B,1,2,3,4
C,3,5,6,7

I'm trying to find a way, using awk, to log (base 2) all the values in the table (minus the headers and id field)

Output
Code:
ID,field1,field2,field3,field4
A,2.32,2.58,2.8,3 
B,0,1,1.58,2
C,1.58,2.32,2.58,2.8

I know to use
Code:
awk -F"," '{print log($2)/log2}' input.csv > output.csv

to log transform one column, but my script writing isn't very good yet and I have no idea how to do this for all the columns in my input file. Any help is much appreciated.
# 2  
Old 03-05-2013
Code:
awk -F, ' NR == 1 {
                print
} NR > 1 {
                printf $1
                for(i = 2; i <= NF; i++)
                        s = s OFS log($i)/log(2)
                print s
                s = ""
}' OFS=, input.csv

This User Gave Thanks to Yoda For This Post:
# 3  
Old 03-05-2013
Code:
awk -F, 'FNR>1{for(i=2;i<=NF;i++) $i=log($i)/log2}1' OFS=, CONVFMT='%.2f' myFile

What OS are you on? 'log2' is not supported by most awk-s.

Last edited by vgersh99; 03-05-2013 at 06:26 PM..
This User Gave Thanks to vgersh99 For This Post:
# 4  
Old 03-05-2013
Thanks to both of you! I tried them both, vgersh, yours worked a bit faster. I'm using unix. I modified to:

Code:
awk -F, 'FNR>1{for(i=2;i<=NF;i++) $i=log($i)/log(2)}1' OFS=, CONVFMT='%.4f' myFile

Thanks again!
# 5  
Old 03-05-2013
UNIX is not an operating system the same way the IEEE is not your wall socket. If you don't know what your operating system is, run uname in your shell.
# 6  
Old 03-05-2013
Sorry, linux
This User Gave Thanks to nemo246 For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to grep log

hi expert, i Have log like : dn: EPC-Per=673,HYHGU objectClass: EPC-PerSubs ownerId: 0 groupId: 4001 shareTree: nodeName=HYHGU permissions: 15 EPC-Per: 673 dn:... (2 Replies)
Discussion started by: justbow
2 Replies

2. Shell Programming and Scripting

Need help with awk and access log

when i run this command: tail -200 /var/log/httpd/access_log | awk -F'' '{sub(/:*$/,"",$2);print $2}' i get: 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56 13/Aug/2012:20:56... (3 Replies)
Discussion started by: SkySmart
3 Replies

3. Shell Programming and Scripting

Data transformations

hi all, I have oracle table which has like below example in col7 & col8 there will be values seperated , in one column it is kind of horizontal pivoting but the data is in 1 column seprated , In near future this col7 and col8 may have some more values col7 * col8 that many records it... (1 Reply)
Discussion started by: marpadga18
1 Replies

4. UNIX for Dummies Questions & Answers

awk and log files

Hello, I have a series of logs that I need to analyse. each looks something like:234.10.72.175 Mon Mar 02 20:25:00 GMT 2009 226.91.87.86 Thu Mar 05 03:50:26 GMT 2009 226.91.87.86 Thu Mar 05 04:06:07 GMT 2009 Using awk, so far I have been able to count the lines in a... (5 Replies)
Discussion started by: Freaky
5 Replies

5. Shell Programming and Scripting

Matching and replacing text with case transformations

Hi, I am trying to edit an XML file automatically but my regex and shell script knowledge is very limited. I would appreciate your help fellows. The XML file has this structure: <?xml version="1.0" encoding="UTF-8"?> <map map_file="maps/world.swf" zoom="350%" zoom_x="-115%"... (1 Reply)
Discussion started by: boonymagique
1 Replies

6. UNIX for Dummies Questions & Answers

log to columns using AWK

How can one take the log of a column using AWK?:( (4 Replies)
Discussion started by: cosmologist
4 Replies

7. Shell Programming and Scripting

Using Awk for stream log

Hi Master, I've log in here then try parsing log become : 3GPP-IMSI,Calling-Station-Id 528111500111614,6738790448 528111500068173,6738769831 ..... it is possible to use awk? cause i 've try awk ' /\Accounting\ /{f=1} f && /\3GPP-IMSI \:\ String Value\ /{imsi=$4} f &&... (1 Reply)
Discussion started by: bowbowz
1 Replies

8. UNIX for Dummies Questions & Answers

Use awk to log transform

Hello. I'm trying to use the awk command to convert certains fields to lgo base 2, not base 10. I'm trying command lines like: awk '$2 $5 $7 {print log(2)($2), log(2)($5), $7) filein.txt > fileout.txt I'm trying to make a deadline. Thanks for helping a newbie. Here's some fresh karma... (1 Reply)
Discussion started by: jmzcal
1 Replies

9. UNIX for Dummies Questions & Answers

Numerical System Transformations - Lecture

Hy, I wrote new article on Numerical Systems and Transformations yesterday evening (on my C Lessons Project webpage) and would like to share it with you. If someone can help me format my text for this forum, I would appreciate it, so I culd post full article here.. Here's intro: "Our... (0 Replies)
Discussion started by: vurdlak
0 Replies
Login or Register to Ask a Question