Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Add 1 million columns

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Add 1 million columns
# 1  
Old 09-08-2014
Add 1 million columns
Hi,

here is my problem:

I've got a file with 6 columns (file1):

Code:
a b c d e f
a b c d e f
a b c d e f
a b c d e f

I need to add 1 million columns to this file, each column needs to be a zero.
Here is how the result file (file2) should look like (for the sake of the example, I've only added 4 columns instead of the million):

Code:
a b c d e f 0 0 0 0
a b c d e f 0 0 0 0
a b c d e f 0 0 0 0
a b c d e f 0 0 0 0

I'm trying to solve this with awk. Here is what I've got so far:
Code:
awk '{for(i=7;i<=1000000;i++);$i=($i)?$i:"0"}1' file1 > file2

It doesn't really work.

Any help will be greatly appreciated!
Last edited by vbe; 09-08-2014 at 05:31 AM.. Reason: code tags please, not HTML
# 2  
Old 09-08-2014
A million fields per line is quite ambitious, and I'd wonder what it be for. This
Code:
awk '{for (i=7; i<=10000; i++) $i="0"}1' file1

is quite lengthy already, for i<=32767 it takes ages, and for i > 32768 it fails (on my system).
Try this
Code:
awk '{printf "%s", $0; for(i=7;i<=1000000;i++) printf " 0"; printf "\n"}' file1

# 3  
Old 09-08-2014
Thank you for the code.
It is quite ambitious indeed, but I need 1 million columns (it is for the genetic analysis).
I've just run your script and it took about 1 min to finish.
The output, however, it not what I would expect: it keeps the header of file1, but then everything is zero. It deleted the first six columns and replaced it with zeros. Looking at the number of fields, there are 1 million columns, not 1 million and 6 as I would expect...

many thanks for your help!
# 4  
Old 09-08-2014
You didn't mention a header in your spec nor in your input file. And in your sample code, you tried to add less than 1000000 fields to make it total 1000000. Run it from i=1 to add 1000000 fields.
This User Gave Thanks to RudiC For This Post:
zajtat
# 5  
Old 09-08-2014
@RudiC code with better performance

Code:
awk 'BEGIN {for(i=1; i<=1000000; i++) t = (t " 0")} {printf "%s%s\n", $0, t}' file1 > file2

Last edited by SriniShoo; 09-08-2014 at 08:13 AM.. Reason: Name correction
These 2 Users Gave Thanks to SriniShoo For This Post:
RudiC zajtat
# 6  
Old 09-08-2014
my apologies to RudiC for not including all the details in my description/codes

many thanks to SriniShoo for the additional code, it worked well
# 7  
Old 09-08-2014
Quote:
Originally Posted by SriniShoo
@RudiC code with better performance

Code:
awk 'BEGIN {for(i=1; i<=1000000; i++) t = (t " 0")} {printf "%s%s\n", $0, t}' file1 > file2

Sure?
Code:
time awk 'BEGIN {for(i=1;i<=100000;i++) t=t " 0"} {printf "%s%s\n", $0, t}' file1 | wc -w
1000060

real    0m4.763s
user    0m4.016s
sys    0m0.750s
time awk '{printf "%s", $0; for(i=1;i<=100000;i++) printf " 0"; printf "\n"}' file1 | wc -w
1000060

real    0m0.277s
user    0m0.328s
sys    0m0.004s

and, even faster:
Code:
time awk '{printf "%s", $0; for(i=1;i<=10000;i++) printf " 0 0 0 0 0 0 0 0 0 0"; printf "\n"}' file1 | wc -w 
1000060

real    0m0.092s
user    0m0.137s
sys    0m0.008s

 
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Group by columns and add sum in new columns

Dear Experts, I have input file which is comma separated, has 4 columns like below, BRAND,COUNTRY,MODEL,COUNT NIKE,USA,DUMMY,5 NIKE,USA,ORIGINAL,10 PUMA,FRANCE,DUMMY,20 PUMA,FRANCE,ORIGINAL,15 ADIDAS,ITALY,DUMMY,50 ADIDAS,ITALY,ORIGINAL,50 SPIKE,CHINA,DUMMY,1O And expected output add... (2 Replies)
Discussion started by: ricky1991
2 Replies

2. Shell Programming and Scripting

Add new columns based on existing columns

Hi all, I am kind of stuck with printing my desired output. Please help me if you know how it can work. My input file(tab separated): NW_0068.1 41,16 100,900 NW_0699.1 4,2,19 200,700,80 My Output file (desired): NW_0068.1 41,16 100,900 100 - 141 NW_0068.1 41,16 100,900 ... (3 Replies)
Discussion started by: sam_2921
3 Replies

3. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns satisfy the condition

HI All, I'm embedding SQL query in Script which gives following output: Assignee Group Total ABC Group1 17 PQR Group2 5 PQR Group3 6 XYZ Group1 10 XYZ Group3 5 I have saved the above output in a file. How do i sum up the contents of this output so as to get following output: ... (4 Replies)
Discussion started by: Khushbu
4 Replies

4. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

5. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns match on two rows

Hi all, I know this sounds suspiciously like a homework course; but, it is not. My goal is to take a file, and match my "ID" column to the "Date" column, if those conditions are true, add the total number of minutes worked and place it in this file, while not printing the original rows that I... (6 Replies)
Discussion started by: mtucker6784
6 Replies

6. Shell Programming and Scripting

Matching 10 Million file records with 10 Million in other file

Dear All, I have two files both containing 10 Million records each separated by comma(csv fmt). One file is input.txt other is status.txt. Input.txt-> contains fields with one unique id field (primary key we can say) Status.txt -> contains two fields only:1. unique id and 2. status ... (8 Replies)
Discussion started by: vguleria
8 Replies

7. Shell Programming and Scripting

Single command for add 2 columns and remove 2 columns in unix/performance tuning

Hi all, I have created a script which adding two columns and removing two columns for all files. Filename: Cust_information_1200_201010.txt Source Data: "1","Cust information","123","106001","street","1-203 high street" "1","Cust information","124","105001","street","1-203 high street" ... (0 Replies)
Discussion started by: onesuri
0 Replies

8. What is on Your Mind?

Pick a Number Between 0 and 20 for 1 Million Bits

Here is an easy game! I wrote a number between 0 and 20 (that can include 0 and 20) on a piece of paper. I am staring at it now, imagining the number so you can read my mind ;) Reply once, and only once, with a number from 0 to 20 and the first person to guess it wins 1,000,000 Bits. ... (24 Replies)
Discussion started by: Neo
24 Replies
Login or Register to Ask a Question