Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Merge csvs with column headers
Top Forums Shell Programming and Scripting Merge csvs with column headers Post 302937247 by abh.kumar on Wednesday 4th of March 2015 11:39:36 AM
Old 03-04-2015
Merge csvs with column headers
hello gurus,

Somebody must have done this before, I couldn't find anything. Please redirect me if this was solved before, and if not please help.


To the problem now, I have multiple csv files (about 1000) which I need to concatenate by column header. The final file should have a superset of all columns of the csvs, and corresponding missing values should be blank.

All files are the same directory. Note the randomness and spaces in filenames.

File 2012 IBM 1
Code:
A,B,C
d1,d2,d3
d4,d5,d6


DELL2013 FILE_2
Code:
A,C,D
d12,d22,d32
d42,d52,d62


HP File 3
Code:
B,C,D,E
d22,d122,d132,
d32,d152,d262,142

My output should look like

Code:
A,B,C,D,E
d1,d2,d3,,
d4,d5,d6,,
d12,,d22,d32,
d42,,d52,d62,
,d22,d122,d132,
,d32,d152,d262,142


I can do this for files with common columns in the same order, but this cant incorporate new columns in the superfile

Code:
awk 'FNR==1 && NR!=1{next;}{print}' *.csv

Last edited by abh.kumar; 03-04-2015 at 12:45 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing Headers and a Column

I have a text file in unix with a layout like this Column 1 - 1-12 Column 2 - 13-39 Column 3 - 40-58 Column 4 - 59-85 Column 5 - 86-120 Columbn 6 - 121-131 The file also has a header on the first 6 lines of each page. Each page is 51 lines long. So I want to remove the header from each... (30 Replies)
Discussion started by: DerangedNick
30 Replies

2. Shell Programming and Scripting

Excel Column Headers

cat ABC.log | egrep "Error 500" >> /tmp/Logs.log egrep "<Mango>.*<.Mango>" Logs.log | sed -e "s/^.*<Mango/<Mango/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp1.xls egrep "<Apple>.*<.Apple>" Logs.log | sed -e "s/^.*<Apple/<Apple/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp2.xls print Heading1,... (1 Reply)
Discussion started by: pk_eee
1 Replies

3. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

4. UNIX for Dummies Questions & Answers

Sort by Column Headers

Hi All, I am new to UNIX can you please help me to sort a file with different columns my file looks like this $ cat gaut.txt UID PID PPID PGID SID C STIME TTY TIME CMD liveuser 3008 2892 3008 3008 0 11:58 ? 00:00:00 gnome-session liveuser 3019 1 ... (8 Replies)
Discussion started by: cgk1983
8 Replies

5. Shell Programming and Scripting

Matching words based on column headers

Hi , Pls help on this. Input file: NAME1 BSC1 TEXT ID 1 MAINSFAIL TEXT ID 2 DGON TEXT ID 3 lOADONDG NAME2 BSC2 TEXT ID 1 DGON TEXT ID 3 lOADONG (1 Reply)
Discussion started by: bha148
1 Replies

6. Shell Programming and Scripting

Transpose field names from column headers to values in one column

Hi All, I'm looking for a script which can transpose field names from column headers to values in one column. for example, the input is: IDa;IDb;IDc;PARAM1;PARAM2;PARAM3; a;b;c;p1val;p2val;p3val; d;e;f;p4val;p5val;p6val; g;h;i;p7val;p8val;p9val; into the output like this: ... (6 Replies)
Discussion started by: popesk
6 Replies

7. Shell Programming and Scripting

Merge column headers and transpose

Hello Everyone! I am new on this forum and this is my first post. I wish to apologize for my, not canonical, English. I would like to solve this problem but I have no clue of how do it!I will be grateful if someone could help me! I have a table like this: gene TF1 TF2 TF3 TF4 gene1 1 2 3 4... (5 Replies)
Discussion started by: giuliangiuseppe
5 Replies

8. Shell Programming and Scripting

Sar -u generates multiple column headers in csv file

Hi All, The below sar -u command generates multiple column headers in csv file Expected output should print column headers only once in the csv file shell script: $cat sar_cpu_EBS.sh #!/bin/bash while ; do sar -u 15 1 | awk '/^/ {print $1,$2,$4,$6,$7}' | tr -s ' ' ',' >>... (6 Replies)
Discussion started by: a1_win
6 Replies

9. Shell Programming and Scripting

Capturing column headers in an array

Hello, I am processing a tab delimited text file and need to grab all of the column headers in an array. The input looks like, num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

10. Shell Programming and Scripting

Row bind multiple csv files having different column headers

All, I guess by this time someone asked this kind of question, but sorry I am unable to find after a deep search. Here is my request I have many files out of which 2 sample files provided below. File-1 (with A,B as column headers) A,B 1,2 File-2 (with C, D as column headers) C,D 4,5 I... (7 Replies)
Discussion started by: ks_reddy
7 Replies

LEARN ABOUT DEBIAN

amc-association-auto

AMC-ASSOCIATION-AU(1)					       Auto Multiple Choice					     AMC-ASSOCIATION-AU(1)

NAME

       AMC-association-auto - automatic association between students and answer sheets for AMC multiple choice exams.

SYNOPSIS

       auto-multiple-choice association-auto --data project-data-dir --notes-id id --liste students-list.csv [--encodage-liste list-encoding]
			    --liste-key key

DESCRIPTION

       The AMC-association-auto.pl command associates students with their answer sheet (when there are no errors from students when coding their
       student number and no error during data capture). See the section called "Identification of the students" from english user documentation
       for details.

       --data project-data-dir
	   gives the directory where data files are (see for example AMC-prepare(1)).

       --notes-id id
	   gives the identification string of the code provided for student numbers (command AMCcode in the LaTeX source file).

       --liste students-list.csv
	   gives the students list.

       --encodage-liste list-encoding
	   gives the students list file encoding (default is utf-8).

       --liste-key key
	   gives the column name where to find the student number in the students list.

       --debug file.log
	   gives a file to fill with debugging information.

AUTHORS

       Alexis Bienvenue <paamc@passoire.fr>
	   Main author

       Jean Berard
	   Translation from French

       Georges Khaznadar
	   Translation from French

COPYRIGHT

       Copyright (C) 2008-2012 Alexis Bienvenue

       This document can be used according to the terms of the GNU General Public License, version 2 or later.

Auto Multiple Choice 1.1.1					    06/19/2012						     AMC-ASSOCIATION-AU(1)
All times are GMT -4. The time now is 02:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy