Stack data from multiple files into one, with variable column files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Stack data from multiple files into one, with variable column files
# 1  
Old 10-22-2015
Stack data from multiple files into one, with variable column files

Hello Gurus,
Im new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldnt find a solution to my issue. Greatly appreciate any help..

I have three csv files -> Apex_10_Latest.csv, Apex_20_Latest.csv, Apex_30_Latest.csv. Number of columns are varying in these 3 files. Typically the latest file, based on the numbering, might have some new columns appended to the end. So I want to take the latest header and stack the data from all the 3 files into a new file Apex.csv. When stacking the data from older file which might have less columns than latest file, I want the data to be populated as null with appropriate delimiters..

Also this has to be done recursively for a multiple set of files (3 each), all in the same folder.
- Apex_10_Latest.csv,Apex_20_Latest.csv,Apex_30_Latest.csv - merged into Apex.csv
- Code_10_Latest.csv,Code_20_Latest.csv,Code_30_Latest.csv - merged into Code.csv
- Trans_10_Latest.csv,Trans_20_Latest.csv,Trans_30_Latest.csv - merged into Trans.csv

Following is the format of the source files and expected target file...
SOURCE FILES:
- Apex_30_Latest.csv:
Code:
A,B,C,D 
----------- 
1,2,3,4 
2,3,4,5 
3,4,5,6

- Apex_20_Latest.csv:
Code:
A,B,C 
--------- 
4,5,6 
5,6,7 
6,7,8

- Apex_10_Latest.csv
Code:
A,B
------- 
7,8
8,9 
9,10

EXPECTED TARGET FILE:
- Apex.csv
Code:
A,B,C,D
----------
1,2,3,4
2,3,4,5
3,4,5,6
4,5,6,,
5,6,7,,
6,7,8,,
7,8,,,
8,9,,,
9,10,,,

Thanks...

Last edited by Don Cragun; 10-23-2015 at 06:48 AM.. Reason: Add CODE and ICODE tags.
# 2  
Old 10-23-2015
Please use code tags as required by forum rules!

Any attempts from your side?

---------- Post updated at 11:24 ---------- Previous update was at 11:13 ----------

Your desired output has one delimiter too many in the second and third file, or one missing i the first.
The problem is the reverse order of the input files. Try
Code:
awk 'NR==1 {MAXNF=NF} NR<=2; FNR <=2 {next} {for (i=NF+1; i<=MAXNF; i++) $i=""} 1' FS="," OFS="," $(ls -r Ape*)
A,B,C,D
-----------
1,2,3,4
2,3,4,5
3,4,5,6
4,5,6,
5,6,7,
6,7,8,
7,8,,
8,9,,
9,10,,

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to copy a column of multiple files and paste into new excel file (next to column)?

I have data of an excel files as given below, file1 org1_1 1 1 2.5 100 org1_2 1 2 5.5 98 org1_3 1 3 7.2 88 file2 org2_1 1 1 2.5 100 org2_2 1 2 5.5 56 org2_3 1 3 7.2 70 I have multiple excel files as above shown. I have to copy column 1, column 4 and paste into a new excel file as... (26 Replies)
Discussion started by: dineshkumarsrk
26 Replies

2. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

3. Shell Programming and Scripting

Select multiple column from multiple files

Hi Friends, $ cat test1.txt emeka:1438 shelley:1439 dmeyer:1440 kurtarn:1441 abdul:1442 $ cat test2.txt 1:a 2:b 3:c 4:d $ cat test3.txt cat:dog:bat man:hot:cold (5 Replies)
Discussion started by: Jewel
5 Replies

4. UNIX for Dummies Questions & Answers

Grep from multiple files by column name

I have 40 files with varying number of columns They however do have identical column names for the overlapping columns I would like to grep some info from these specific colums e.g file1 id beta se direction N 2 .5 .01 + 1000 5 -.6 .02 - 2000 file2 id ... (6 Replies)
Discussion started by: MFAB
6 Replies

5. Shell Programming and Scripting

Column extraction from multiple files to multiple files

I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file: [#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE #yr mo dy hr mn degT m/s m/s m sec ... (5 Replies)
Discussion started by: aozgaa
5 Replies

6. Shell Programming and Scripting

Compare 2 files and match column data and align data from 3 column

Hello experts, Please help me in achieving this in an easier way possible. I have 2 csv files with following data: File1 08/23/2012 12:35:47,JOB_5330 08/23/2012 12:35:47,JOB_5330 08/23/2012 12:36:09,JOB_5340 08/23/2012 12:36:14,JOB_5340 08/23/2012 12:36:22,JOB_5350 08/23/2012... (5 Replies)
Discussion started by: asnandhakumar
5 Replies

7. UNIX for Advanced & Expert Users

merge two column multiple files into one

Hi I have multiple files each with two columns and I need to combine all those file into a tab delimited file. (multiple entry with same name separated by a comma) The content of the files are as follows: --- file1.txt: name var1 aaa xx aaa gg bbb yy ddd zz --- file2.txt ... (8 Replies)
Discussion started by: mary271
8 Replies

8. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

9. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

10. Shell Programming and Scripting

Column means for multiple files

Hi all, I have multiple dat files one for each day of the year and each with two lengthy columns. For each file, I wan't to perform a mathematical operation on the data of column 2 (red), then get the mean of these values and create a new two-column file where each row shows the mean value and... (3 Replies)
Discussion started by: larrymuli
3 Replies
Login or Register to Ask a Question