Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Gathering data in columns from multiple files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Gathering data in columns from multiple files
# 1  
Old 09-30-2015
Gathering data in columns from multiple files
Hello world!

I need to gather all the data from different folders and copy it in an one unique text file in columns format.

Let me explain, letīs say "a, b, c" are 3 data files of thousands and thousands lines (in the reality, I have nearly one hundred).
"a, b, c" are located in the folders A, B and C (respectively).

I want to extract "a, b and c" and joined them in the same text file "d".
Whenever I use the function "paste" or "cat" with a for loop, I get a text file like this format :
a
b
c
All the data are written in line. But I would like to get it in such format :
a b c

Is there a way to write directly in such format ?
Or maybe a way to transform from
a
b
c
to
a b c
after writing.


I hope I was clear enough...

Thanks in advance!
# 2  
Old 09-30-2015
Please use code tags as required by forum rules!

No, you weren't. Are those a, b, and c files one column text files, and you want to join them columnwise? Why doesn't paste work for you?
Please post samples of input data and desired output format based on those data.
# 3  
Old 09-30-2015
hi Rudy Smilie


Yes, this a one-column text files with... nearly 30 000 rows for only one text.

Yes, I can use
Code:
paste

, like this :

Code:
for i in *
do
paste -d " \t " $i/A/a >> finaldata
done

I obtain a big file with all the data but only in one column (I want it to be copied columnwise)

Any idea ?

EDIT : I will try to be clearer.

I have a lot of files from different time steps. For example, my folders look like :
Code:
0.001 (This is the folder for the time step t = 0,001s.)
0.002 (This is the folder for the time step t = 0,002s.)
0.003 (This is the folder for the time step t = 0,003s.)
0.004 (This is the folder for the time step t = 0,004s.)
...
1.000 (This is the folder for the time step t = 1,000s.)

In each folder, I have one data text file for every time step. The name of this data text file is always the same for every time step. For example : "pressure".
The data text file "pressure" is given to me like this for the time step
Code:
t = 0,001s

:
Code:
1.2
2.5
3
7
5.4
0.6
5.4
12

For the other time steps, I have a similar text file with different values.

When I try to gather all the data together, I have this kind of output :

Code:
1.2
2.5
3
7
5.4
0.6
5.4
12  

data from t = 0,002 s

data from t = 0,003 s

etc...

Now, I want the output to be like :

Code:
1.2                                           data from t = 0,002 s                                             data from t = 0,003s                                             etc...
2.5
3
7
5.4
0.6
5.4
12

Last edited by user10600; 09-30-2015 at 11:37 AM..
# 4  
Old 09-30-2015
Code:
cat file[123]
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
C1
C2
C3
C4
C5
C6
C7
C8
C9
paste file[123]
A1      B1      C1
A2      B2      C2
A3      B3      C3
A4      B4      C4
A5      B5      C5
A6      B6      C6
A7      B7      C7
A8      B8      C8
A9      B9      C9
A10     B10
        B11
        B12

# 5  
Old 10-01-2015
Thanks RudiC again!

How can I do that for 100 text files ? should I write :

Code:
paste file[123456..100]

? I try to figure out how to implement a
Code:
For Loop

, but I donīt get the desired output...

Any ideas ? Smilie
# 6  
Old 10-01-2015
How about experimenting, trying and testing/expanding boundaries?
This User Gave Thanks to RudiC For This Post:
user10600
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing carriage returns from multiple lines in multiple files of different number of columns

Hello Gurus, I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file. 1|ABC DEF|100|10 2|PQ RS T|200|20 3| UVWXYZ|300|30 4| GHIJKL|400|40... (7 Replies)
Discussion started by: dJHa
7 Replies

2. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

I have a series of csv files in the following format eg file1 Experiment Name,XYZ_07/28/15, Specimen Name,Specimen_001, Tube Name, Control, Record Date,7/28/2015 14:50, $OP,XYZYZ, GUID,abc, Population,#Events,%Parent All Events,10500, P1,10071,95.9 Early Apoptosis,1113,11.1 Late... (6 Replies)
Discussion started by: pawannoel
6 Replies

3. Shell Programming and Scripting

Compare multiple files with multiple number of columns

Hi, input file1 abcd 123 198 xyz1:0909090-0909091 ghij 234 999 xyz2:987654:987655 kilo 7890 7990 xyz3:12345-12357 prem 9 112 xyz5:97-1134 input file2 abcd 123 198 xyz1:0909090-0909091 -9.122 0 abed 88 98 xyz1:98989-090808 -1.234 1.345 ghij 234 999 xyz2:987654:987655 -10.87090909 5... (5 Replies)
Discussion started by: jacobs.smith
5 Replies

4. Red Hat

What is the best tools for performance data gathering and analysis?

Dear Guru, IHAC who complaint that his CentOS is getting performance issue. I have to help him out of there. Could you please tell me which tools is better to gathering the whole system performance data? -- CPU/Memory/IO(disk & Network)/swap I would like the tools could be... (6 Replies)
Discussion started by: devyfong
6 Replies

5. UNIX for Advanced & Expert Users

Gathering data using SAR

Hi everyone, I would like to ask if it is possible to gather SAR data on a specified time. let say yesterdays report, i want to get data at around 12PM and 5PM. thank you. (2 Replies)
Discussion started by: cwiggler
2 Replies

6. Shell Programming and Scripting

Averaging Data From Multiple Columns, Using Header if Possible

Hi, I have a file with multiple tab delimited columns and I would like to have the average of each column: Iteration Tree No Lh HMean 1000 1 -78.834717 -78.834717 1100 1 -77.991031 -78.624046 1200 1 -79.416055 -78.761861 1300 1 -79.280494 -78.968099 1400 1 -82.846275 -80.808696 ... (4 Replies)
Discussion started by: mikey11415
4 Replies

7. Shell Programming and Scripting

Splitting data from one row as multiple columns

Hi I have a file containing some data as follows: 11-17-2010:13:26 64 4 516414 1392258 11-17-2010:13:26 128 4 586868 695603 11-17-2010:13:26 256 4 474937 1642294 11-17-2010:13:32 64 4 378715 1357066 11-17-2010:13:32 128 4 597981 1684006 ... (17 Replies)
Discussion started by: annazpereira
17 Replies

8. UNIX for Dummies Questions & Answers

Gathering data from complex/large dataspreads .txt format

Hi, I'm working on gathering information stored in .txt files. The format of the data within the .txt files is shown in the picture uploaded with this post. Sections like the one pictured are repeated (with different data, same format) many times within each .txt file but each section is of data... (4 Replies)
Discussion started by: p43hd
4 Replies

9. Shell Programming and Scripting

Data gathering script

I am pretty new at shell scripting, but I managed to make a primative shell script to connect from my webserver (bluehost) to an external data server via ftp or sftp or whatever I need. This script is : #!/bin/sh HOST='ftp.host.com' USER='username' PASSWD='password' FILE='file' ftp -n... (7 Replies)
Discussion started by: mesoeast
7 Replies
Login or Register to Ask a Question