Login or Register to Ask a Question and Join Our Community


Ubuntu

Inserting a header with column number to a 1.6 GB file with special spacing

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Linux Ubuntu Inserting a header with column number to a 1.6 GB file with special spacing
# 1  
Old 11-25-2009
Inserting a header with column number to a 1.6 GB file with special spacing
Hi;

I've been searching posts to find a solution to what I'm trying to do, but I've have NOT found anything yet.

I have a file (file1) with 300K columns and 1411 rows, the columns don't have a column no. header (No header at all) and I'm trying to fetch the information from specific columns.

The only way for me to fetch the information from the file is to go by column number, I have 412 column numbers (file2).

Code:
Here is an example of file 1 (tab separated):

1  23  21  24  12  22 .......etc until column 300K
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22.
.
.
.
.
etc until row 1411

Code:
In file 1 the columns need to be assigned like this:

1  2   3   4   5   6
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22
1  23  21  24  12  22

Code:
file 2 (has only 1 column and 412 rows, each row is a column # for file 1)

123
955
1045
1184
2323
2328
2333
2756
3364
4377
5259
5351
5778
7632
8603
9399
9561
10469
.
.
.
.
until row 412

I tried transposing the large file first, to give it row numbers, BUT the file is too big and I run out of memory. So, transposing is out of the question at this point.

Thank you in advance for any help!
# 2  
Old 11-25-2009
1. translate the new line delimited file to space delimited ( if need be change space to tab. )

Code:
tr '\n' ' ' < f2 > f3

2. print the header, and file.,

Code:
sed 'r f1' f3

# 3  
Old 11-25-2009
thegeek;

It works. Thank you. Now, I'm trying to use grep to fetch only the columns from file 1 (now that has a column no. header) with the column numbers listed in file2:


file="file2"
while read line
do
grep $line file1 >> file4
done < $file

But this code does not work, it gives me an even bigger file than file1, it should be much more smaller.
Last edited by sogi; 11-26-2009 at 03:41 AM.. Reason: the code is not displaying as code
# 4  
Old 11-27-2009
above script can be replaced by :

Code:
grep -f file2 file1 >> file4

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Inserting a header with special character

Hi, I am trying to insert header row with a special character delimiter with Unicode u0109 into a file with ‘echo’, header looks like below echo –e “header1\u0109header\u0109header3\u0109header4” It just inserting as it is in the quotes but not the special character, Please suggest if am... (2 Replies)
Discussion started by: oom
2 Replies

2. UNIX for Beginners Questions & Answers

Inserting Header at different position in a file

I would like to hear your directions on how to Insert theses tag </TITLE> and <TEXT> at a given position in 1000 of text files. My Files look like as samplefile1.txt <DOC> <DOCNO>3_September_2012</DOCNO> <TITLE> ... ... ... .... ... .. .. .. ... .. .. .... </TITLE> <TEXT> .... (1 Reply)
Discussion started by: imranrasheedamu
1 Replies

3. Shell Programming and Scripting

awk command to find total number of Special character in a column

How to find total number of special character in a column? I am using awk -f "," '$col_number "*$" {print $col_number}' file.csv|wc -l but its not giving correct output. It's giving output as 1 even though i give no special character? Please use code tags next time for your code and... (4 Replies)
Discussion started by: AjitKumar
4 Replies

4. Shell Programming and Scripting

Inserting Header to another file

Need your help in appending header(file1 contains header ) to my file2. I am using KSH AIX OS. I know how to do with taking temporary files. cat file1 >temp cat file2 >>temp mv temp file2 Is there way to append directly to a file in ksh. I don't find Sed -i option on my... (10 Replies)
Discussion started by: gvkumar25
10 Replies

5. Shell Programming and Scripting

awk or sed: change the color of a column w/o screwing up column spacing

Hey folks. I wrote a little awk script that summarizes /proc/net/dev info and then pipes it to the nix column command to set up column spacing appropriately. Here's some example output: Iface RxMBytes RxPackets RxErrs RxDrop TxMBytes TxPackets TxErrs TxDrop bond0 9 83830... (3 Replies)
Discussion started by: ryran
3 Replies

6. UNIX for Dummies Questions & Answers

Rename a header column by adding another column entry to the header column name

Hi All, I have a file example.csv which looks like this GrpID,TargetID,Signal,Avg_Num CSCH74_1_1,2007,61,256 CSCH74_1_1,212007,647,679 CSCH74_1_1,12007,3,32 CSCH74_1_1,207,299,777 I want the output as GrpID,TragetID,Signal-CSCH74_1_1,Avg_Num CSCH74_1_1,2007,61,256... (1 Reply)
Discussion started by: Vavad
1 Replies

7. Shell Programming and Scripting

Rename a header column by adding another column entry to the header column name URGENT!!

Hi All, I have a file example.csv which looks like this GrpID,TargetID,Signal,Avg_Num CSCH74_1_1,2007,61,256 CSCH74_1_1,212007,647,679 CSCH74_1_1,12007,3,32 CSCH74_1_1,207,299,777 I want the output as GrpID,TragetID,Signal-CSCH74_1_1,Avg_Num CSCH74_1_1,2007,61,256... (4 Replies)
Discussion started by: Vavad
4 Replies

8. Shell Programming and Scripting

Inserting a String in a file header.

Dear all, I have a file created in the name sample.txt in UNIX with header and footer. How to insert a required string (for example "FILE1") in the header part after the file has been created. What kind of command can i use to do the same. Thanks in advance Hari (3 Replies)
Discussion started by: Hari123
3 Replies

9. Shell Programming and Scripting

Inserting a column in a file

Hi I have the following info in a file file1 ABCD DSFD sdfsd YUISA I want to add a column to the file file1 like the below. STR ABCD STR DSFD STR sdfsd STR YUISA Is there any way in sed or awk to do it. Regards Dhana (7 Replies)
Discussion started by: dhanamurthy
7 Replies

10. UNIX for Advanced & Expert Users

Inserting a new column in a file

Hey.. I'm writing a code to download some stuff from Informix database and put it on Xls. It works fine, but I have a problem fitting in a new requirement. I have currently a file which has information like below. f_name|Ronnie|Johnson|23.00| f_sal|Ronnie|Jhonson|4000.00|... (4 Replies)
Discussion started by: rosh0623
4 Replies
Login or Register to Ask a Question