Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Adding columns from 2 files with variable number of columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Adding columns from 2 files with variable number of columns
# 8  
Old 09-10-2018
Quote:
Originally Posted by maya3
Does the fact that I am printing it as string (printf and %s) make any problems with precision or format for later use? The entries in file1 and file2 can be either float or decimal and the precision is very important for the task, so I tried replacing %s with %d or %i to keep the output in the same format but that didn't give the result in a triangular form like it did with %s. Why is that?
well... I assumed the sample files were representative. It seems that they cover only a part of the cases.
It's doable. Please provide the more representative samples for both files.
What happens if the same entries have different floating precision from 2 files? The higher precision wins?
# 9  
Old 09-10-2018
I wrote round numbers for simplicity. So the file is actually several thousand lines/columns long and these are example lines. Yes in this case the precision of the more precise entry should be kept when adding a less precise entry to it.
And the separator is actually three whitespaces, not one, should OFS recognize that if they are consistent in both files?

file1:
Code:
3.5636651200281602E-07   7.2207650159240707E-08  -2.3855385753423701E-08   8.6154500245612697E-09   1.3474513362335099E-09
2.2183583723650399E-10   2.8850663428589998E-07   1.0632952117690300E-07  -2.8497022231591001E-08
7.6373290007977394E-09   3.0214763933556298E-09   -3.9464610062914300E-10
2.5688038136967701E-07   1.2156548011859100E-07
-6.1884355782457798E-09


Code:
2.0854016845313175E-319   0.0000000000000000E+00   0.0000000000000000E+00   0.0000000000000000E+00   0.0000000000000000E+00
0.0000000000000000E+00   0.0000000000000000E+00   0.0000000000000000E+00   0.0000000000000000E+00
7.0099790395933999E-09   0.0000000000000000E+00   0.0000000000000000E+00
3.6570785008176398E-09  -7.8575690874984004E-08
-1.7493726939535900E-08

# 10  
Old 09-10-2018
Is it safe to assume they're all floats?
# 11  
Old 09-10-2018
I thought these were all double (they're output from C)
# 12  
Old 09-10-2018
Quote:
Originally Posted by maya3
I thought these were all double (they're output from C)
Unfortunately, I won't be able to get to this for a day or so.
So if somebody else look into this...
It's a matter of parsing the format/precision of the corresponding entries in both files, coming up with the more "precise" spec and using it in the printf format string for the math...

------ Post updated at 03:58 PM ------

a bit simplified/hardwired, but could be a start:
Code:
paste file1 file2 |\
 awk '{ for(i=1;i<=NF/2;i++) printf("%.16E%s", $i+$((NF/2)+i), (i==NF/2)?ORS:OFS)}'

# 13  
Old 09-10-2018
awk works with double precision by defaut (IIRC). So why should precision be lost when adding two numbers? Try (untested)
Code:
awk '
        {getline X < F2
         n = split (X, T)
         for (i=1; i<=NF; i++) $i += T[i]
        }
1
' F2="file2" file1

# 14  
Old 09-10-2018
Quote:
Originally Posted by RudiC
awk works with double precision by defaut (IIRC). So why should precision be lost when adding two numbers? Try (untested)
Code:
awk '
        {getline X < F2
         n = split (X, T)
         for (i=1; i<=NF; i++) $i += T[i]
        }
1
' F2="file2" file1

Code:
$ awk '
         {getline X < F2
          n = split (X, T)
          for (i=1; i<=NF; i++) $i += T[i]
         }
 1
 ' F2="maya22.txt" maya11.txt
3.56367e-07 7.22077e-08 -2.38554e-08 8.61545e-09 1.34745e-09
2.21836e-10 2.88507e-07 1.0633e-07 -2.8497e-08
1.46473e-08 3.02148e-09 -3.94646e-10
2.60537e-07 4.29898e-08
-2.36822e-08

On the other hand:
Code:
 awk '
        {getline X < F2
         n = split (X, T)
         for (i=1; i<=NF; i++) $i += T[i]
        }
1
' CONVFMT='%.16E' F2="maya22.txt" maya11.txt
3.5636651200281602E-07 7.2207650159240707E-08 -2.3855385753423701E-08 8.6154500245612697E-09 1.3474513362335099E-09
2.2183583723650399E-10 2.8850663428589998E-07 1.0632952117690300E-07 -2.8497022231591001E-08
1.4647308040391139E-08 3.0214763933556298E-09 -3.9464610062914300E-10
2.6053745987049467E-07 4.2989789243606999E-08
-2.3682162517781680E-08

This has to do with converting numbers when performing math or converting numbers to strings:
Code:
CONVFMT     The conversion format for numbers, "%.6g", by default.

------ Post updated at 05:13 PM ------

you can probably generalize the solution for ANY number of the similarly formatted files to operate on:
myScript.sh file1 file2 file3 where myScript.sh is:
Code:
#!/bin/ksh

fn="${#}"
paste ${@} |\
 awk '{ for(i=1;i<=NF/fn;i++) printf("%.16E%s", $i+$((NF/fn)+i), (i==NF/fn)?ORS:OFS)}' fn="${fn}"

These 2 Users Gave Thanks to vgersh99 For This Post:
maya3 RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request: How to Parse dynamic SQL query to pad extra columns to match the fixed number of columns

Hello All, I have a requirement in which i will be given a sql query as input in a file with dynamic number of columns. For example some times i will get 5 columns, some times 8 columns etc up to 20 columns. So my requirement is to generate a output query which will have 20 columns all the... (7 Replies)
Discussion started by: vikas_trl
7 Replies

2. Shell Programming and Scripting

Adding info to end of line if two columns match from files with different separators

I have two files (csv and vcf) which look exactly like this S1.csv func,gene,start,info "exonic","AL","2309","het" "exonic","NEF","6912","hom"S1.vcf ##fileinfo #CHROM POS ID INFO chr1 4567 rs323211 1/1:84,104,99 chr4 2309 rs346742 1/1:27,213,90 chr6 5834 ... (5 Replies)
Discussion started by: Sarah_19
5 Replies

3. Shell Programming and Scripting

Adding columns with values dependent on existing columns

Hello I have a file as below chr1 start ref alt code1 code2 chr1 18884 C CAAAA 2 0 chr1 135419 TATACA T 2 0 chr1 332045 T TTG 0 2 chr1 453838 T TAC 2 0 chr1 567652 T TG 1 0 chr1 602541 ... (2 Replies)
Discussion started by: plumb_r
2 Replies

4. Shell Programming and Scripting

Adding Multiple Files via Columns

I have a number of files with multiple rows that I need to add together. Let say I have 10 files: Each file has a great number of rows and columns. I need to add these files together the following way. In other words, If, for example, file A occupies Columns 1 to 19, I want to add file B... (7 Replies)
Discussion started by: Ernst
7 Replies

5. UNIX for Dummies Questions & Answers

Adding new columns to txt files

Dear all, I have a question. I have a txt.file as below. i want to add 3 more columns: column3=conlum 2*column2; column4=(1-column2)*(1-column2); column5=1-column3-column4. Do you know how to do it? Thanks a lot! file: column1 column2 a 1 b 20 c 30 d 3 ... (2 Replies)
Discussion started by: forevertl
2 Replies

6. Shell Programming and Scripting

merging files and adding special columns

Hi everyone, I got a problem with merging files and hoped one of you would have an idea how to approach this issue. I tried it with awk, but didn't get far. This is what I have: I got 40 files looking like the ones below. All have three columns but the number of rows differs (20000 to 50000).... (6 Replies)
Discussion started by: TuAd
6 Replies

7. Shell Programming and Scripting

Adding columns of two files

Hello everyone, I have two files containing 6 columns and thousands of rows. I want to add them (i.e. first column of first file + first column of second file and so on) and print the output in a third file. Can you please help me. Thanks a lot (7 Replies)
Discussion started by: chandra321
7 Replies

8. Shell Programming and Scripting

awk adding columns from different files

Hi, I have two files and I need to add column 3 of file1 to column 3 of file 2 > file3 I also need to repeat for column 4. Thanks (1 Reply)
Discussion started by: dsstamps
1 Replies

9. Shell Programming and Scripting

comparing files - adding/subtracting/formating columns

I have two files: file1.txt: FS Total Used Free Used% /u01 10000 8000 2000 80% /u02 10000 8000 2000 80% /u03 10000 8000 2000 80% /u04 10000 8000 2000 80% /u05 10000 8000 2000 80% /u06 10000 8000 2000 80% /u07 10000 8000 2000 80% /u10 10000 5000 5000 50% file2.txt:... (7 Replies)
Discussion started by: oabdalla
7 Replies

10. Shell Programming and Scripting

Adding columns to excel files using Perl

How do I add 4 columns to an excel file using Perl? The 4 headers for those columns will all have different names? Please help and I greatly appreciate... (1 Reply)
Discussion started by: dolo21taf
1 Replies
Login or Register to Ask a Question