How to copy a column of multiple files and paste into new excel file (next to column)?


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers How to copy a column of multiple files and paste into new excel file (next to column)?
# 22  
Old 04-13-2019
Quote:
Originally Posted by dineshkumarsrk
Its printing exactly reverse order. The last file (0rg30) columns printed first followed by org29, org28 ......org1.
There are several easy options here...

Change the order of the operands given to paste:
Code:
 > RES; for FN in $(ls *.xls | sort -t. -k1.4n); do cut -f1,5 $FN | paste RES - > TMP; mv TMP RES; done

Reverse the sort order as RudiC suggested:
Code:
 > RES; for FN in $(ls *.xls | sort -t. -k1.4nr); do cut -f1,5 $FN | paste - RES > TMP; mv TMP RES; done

In both of the above you can replace ls *.xls with printf '%s\n' *.xls to run a little bit faster (using a shell built-in instead of invoking ls).

If you know that there will always be at least 10 files to be processed and that there will never be more than 99 files to process, get rid of the ls and the sort:
Code:
 > RES; for FN in org?.xls org??.xls; do cut -f1,5 $FN | paste RES - > TMP; mv TMP RES; done

Fix the minor logic problem in what Ravinder suggested and use one invocation of awk instead of thirty invocations of paste:
Code:
awk -v OFS='\t' '
{	line[FNR] = ((NR == FNR) ? "" : line[FNR] OFS) $1 OFS $5 }
END {	for(i = 1; i <= FNR; i++) print line[i] }
' $(printf '%s\n' org*.xls | sort -t. -k1.4n) > concats.xls

or:
Code:
awk -v OFS='\t' '
{	line[FNR] = ((NR == FNR) ? "" : line[FNR] OFS) $1 OFS $5 }
END {	for(i = 1; i <= FNR; i++) print line[i] }
' org?.xls org??.xls > concats.xls

These 3 Users Gave Thanks to Don Cragun For This Post:
# 23  
Old 04-16-2019
Dear Don,
There is a minor issue with all those commands mentioned above. As I told, I have 30 excel files. among them few excel files have lesser rows than others. For example excel file 6 to 10 has only 2 rows, whereas other files has 3 rows as shown below.
Code:
org1_1	1	1	2.5	100
org1_2	1	2	5.5	98	
org1_3	1	3	7.2	88
.
.
.
org10_1	1	1	2.5	100
org10_2	2	5.5	98	98
.
.
.
org11_1	1	1	3.5	100
org11_2	1	2	8.5	77	
org11_3	1	3	7.2	88

When I execute above commands, it pasting 11th file first row along with 10th file 3rd row as given below,
Code:
org1_1	100 . . . . . org10_1	100		org11_2	77
org1_2	98  . . . . . org10_2	98		org11_3	88
org1_3	88  . . . . . 		org11_1	100

Sorry for the delayed response.
# 24  
Old 04-16-2019
Quote:
Originally Posted by dineshkumarsrk
Dear Don,
There is a minor issue with all those commands mentioned above. As I told, I have 30 excel files. among them few excel files have lesser rows than others. For example excel file 6 to 10 has only 2 rows, whereas other files has 3 rows as shown below.
Code:
org1_1	1	1	2.5	100
org1_2	1	2	5.5	98	
org1_3	1	3	7.2	88
.
.
.
org10_1	1	1	2.5	100
org10_2	2	5.5	98	98
.
.
.
org11_1	1	1	3.5	100
org11_2	1	2	8.5	77	
org11_3	1	3	7.2	88

When I execute above commands, it pasting 11th file first row along with 10th file 3rd row as given below,
Code:
org1_1	100 . . . . . org10_1	100		org11_2	77
org1_2	98  . . . . . org10_2	98		org11_3	88
org1_3	88  . . . . . 		org11_1	100

Sorry for the delayed response.
Hi dineshkumarsrk,
There is no reason for you to apologize for a delayed response. We are trying to help you solve your problem. If you aren't in a hurry to get something that works, it doesn't make any difference to us.

We might note, however, that all of the suggestions I made work perfectly with every example that was provided in the first twenty-one posts in this thread. In those posts you had shown us the contents of three files and the names of twenty-seven other files. Never was anything said about the number of lines in some of the files being different than the number of lines in other files and, as you have seen, that affects the output that the script you showed us in post #1 would work and it affects the output that every single script suggested in this thread works (since we all followed your example to try to get the output you wanted).

Note that we are here to help you learn how to write code to satisfy your requirements; we are not here to act as your unpaid programming staff with continually changing requirements. Given this new information that files 6 to 10 (is it always 6 to 10, or s 6 to 10 just as an example; is it org6.xls through org10.xls inclusive or does org10.xls go back to three lines) may have different numbers of lines than other files with no prior knowledge about which files are "short" or "long", how would you propose modifying any of the scripts suggested in post #22 to get the output you want based on this new information?
This User Gave Thanks to Don Cragun For This Post:
# 25  
Old 04-16-2019
I have been using your script without any issue, until I had a data set with different row numbers (I mentioned above). So far, I had processed more than 5 groups of data sets (in each group I had 30 excel files), where in I did not face this kind of problem. Moreover, even I do not expect that, I would face this kind of issue. I thought, my data files have the same number of rows and columns. But, while processing multiple data files, I came to know that it differs. This is why I could not mention it earlier.
Code:
 Given this new information that files 6 to 10 (is it always 6 to 10, or s 6 to 10 just as an example;

It is an example only. My data set would not be same as I mentioned above. In some cases, each file differ in terms of row number (number of rows are not same for all the files). Whereas, the number of columns are same for all the files.

Code:
is it org6.xls through org10.xls inclusive or does org10.xls go back to three lines) may have different numbers of lines than other files with no prior knowledge about which files are "short" or "long", how would you propose modifying any of the scripts suggested in post #22 to get the output you want based on this new information?

I need to copy only the field/column number 1 and 5 regardless of the number of rows each file have. It would be tedious to know which file is short and which file is long when I have 50 or 60 excel files.
If it a prerequisite to know, which file is short and which file is long to write a script for the same, I would not have asked for help.
# 26  
Old 04-16-2019
Quote:
Originally Posted by dineshkumarsrk
I have been using your script without any issue, until I had a data set with different row numbers (I mentioned above). So far, I had processed more than 5 groups of data sets (in each group I had 30 excel files), where in I did not face this kind of problem. Moreover, even I do not expect that, I would face this kind of issue. I thought, my data files have the same number of rows and columns. But, while processing multiple data files, I came to know that it differs. This is why I could not mention it earlier.
Quote:
Given this new information that files 6 to 10 (is it always 6 to 10, or s 6 to 10 just as an example;
It is an example only. My data set would not be same as I mentioned above. In some cases, each file differ in terms of row number (number of rows are not same for all the files). Whereas, the number of columns are same for all the files.

Quote:
is it org6.xls through org10.xls inclusive or does org10.xls go back to three lines) may have different numbers of lines than other files with no prior knowledge about which files are "short" or "long", how would you propose modifying any of the scripts suggested in post #22 to get the output you want based on this new information?
I need to copy only the field/column number 1 and 5 regardless of the number of rows each file have. It would be tedious to know which file is short and which file is long when I have 50 or 60 excel files.
If it a prerequisite to know, which file is short and which file is long to write a script for the same, I would not have asked for help.
You never said that any of the code that I had previously suggested worked for any of your data sets. You only said that there was a problem because the number of lines in files 6 through 10 only had two lines instead of three.

One possible solution would be to add empty rows to all of the files that have fewer rows than the longest file. If you did that as a first step, all of the existing scripts might do what you wanted. And, if you knew that files 6 - 10 would always contain one line less than the other files, it would be very easy to add an empty line to those four or five files without making the other processing any more complex.

Assuming that you will always have at least one file that is named orgdigits.xls where digits is a single digit and at least one file where digits is two digits and that you will never have any files where digits is more than two digits, one could try:
Code:
awk -v OFS='\t' '
FNR == 1 {
	file_cnt++
}
{	data[file_cnt, 1, FNR] = $1
	data[file_cnt, 2, FNR] = $5
	if(FNR > line_cnt)
		line_cnt = FNR
}
END {	for(i = 1; i <= line_cnt; i++)
		for(j = 1; j <= file_cnt; j++)
			printf("%s%s%s%s",  data[j, 1, i], OFS, data[j, 2, i],
			    (j == file_cnt) ? ORS : OFS)
}
' org?.xls org??.xls > concats.xls

but I am disappointed that you were unwilling to make any attempt to solve this new wrinkle on your own.

With the other suggestions in post #22, I assume that you would be able to modify the last line of this code to handle any number of input files.

Last edited by Don Cragun; 04-16-2019 at 11:31 AM.. Reason: Fix typo: s/line this/line of this/
These 2 Users Gave Thanks to Don Cragun For This Post:
# 27  
Old 04-17-2019
Dear don,
Yes indeed, I could modify the last line of the code as per my need. Your code works perfectly fine. I have realised that, I need to fix the problem by myself. Thank you for your critical comments, suggestions and help.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Copy files if column 5 in a file contains ā€œVā€

I have number of csv files (like tmo_2019*). In these files some files have 5th column value as V. I want to copy those files having 5th column value as V to specific directory /test/V_files/. I tried to extract file names by below but not able to complete command for copy. find -type f -iname... (4 Replies)
Discussion started by: Bops
4 Replies

2. UNIX for Beginners Questions & Answers

How to copy particular files from a multiple directories and paste in a new directory?

Dear all I have a multiple directories, say for example org1, org2, org3 ..... org100 and each directory having a file namely dnaG.fasta. I need to copy all the dnaG.fasta file from each directory and paste in another directory fastconcatg. Therefore, my script has to copy dnaG.fasta file from... (5 Replies)
Discussion started by: dineshkumarsrk
5 Replies

3. UNIX for Beginners Questions & Answers

How to insert data into black column( Secound Column ) in excel (.XLSX) file using shell script?

Source Code of the original script is down below please run the script and try to solve this problem this is my data and I want it column wise 2019-03-20 13:00:00:000 2019-03-20 15:00:00:000 1 Operating System LAB 0 1 1 1 1 1 1 1 1 1 0 1 (5 Replies)
Discussion started by: Shubham1182
5 Replies

4. Shell Programming and Scripting

Locate the files in the first column and copy the files in 2nd column

#cat data.txt file1 folder1 file2 thisforfile2 file3 thisfolderforfile3 lata4 folder4 step 1: create the folder first in column 2 for i in `awk '{print $2}' data.txt` do mkdir /home/data/$i done step 2: locate the files in column1 and stored them into a file for i in... (17 Replies)
Discussion started by: kenshinhimura
17 Replies

5. Shell Programming and Scripting

Paste columns based on common column: multiple files

Hi all, I've multiple files. In this case 5. Space separated columns. Each file has 12 columns. Each file has 300-400K lines. I want to get the output such that if a value in column 2 is present in all the files then get all the columns of that value and print it side by side. Desired output... (15 Replies)
Discussion started by: genome
15 Replies

6. Shell Programming and Scripting

Problems with awk (fatal error) and paste (two variables into one column-by-column)

Hello, I have a script extracting columns of useful numbers from a data file, and manipulating the numbers with awk commands. I have problems with my script... 1. There are two lines assigning numbers to $BaseForAveraging. If I use the commented line (the first one) and let the second one... (9 Replies)
Discussion started by: vgbraymond
9 Replies

7. UNIX for Dummies Questions & Answers

Paste column from one file as column of

Any shortcuts for doing this? I need to cut the column 4 values from File1 and paste them as column4 values of File2, but only for the (first) same number of lines as File1 . All rows in File1 are contained in File2 in the exact same order, so the cut paste should work. File1 (with header and 3... (4 Replies)
Discussion started by: senhia83
4 Replies

8. UNIX for Advanced & Expert Users

Copy a column to another column in UNIX fixedwidth file

Hi All, I have a fixedwidth file of length 3000. Now i want to copy a column of 4 chars i.e( length 1678-1681) to column 1127 – 1171 to the same file. Please let me know how can i achive using a single command in fixed width file. Also source column length is 4 chars and target column length... (4 Replies)
Discussion started by: kiranparsha
4 Replies

9. Shell Programming and Scripting

Copy a column and paste to other file question

Please help me. This is simple, but urgent problem for me. :( I have a two files file1 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 ..... file2 11 12 13 14 15 11 12 13 14 15 11 12 13 14 15 ..... 1) I hope to make a new file, file 3, that consists of 2nd... (2 Replies)
Discussion started by: exsonic
2 Replies

10. Shell Programming and Scripting

paste each 10 lines of single column to several column

Hi, I need to paste each 10 lines of single column to several columns. Please, can anyone tell me how to write in awk? Input File: 22 34 36 12 17 19 15 11 89 99 56 38 29 (4 Replies)
Discussion started by: nica
4 Replies
Login or Register to Ask a Question