Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Merge csvs with column headers
Top Forums Shell Programming and Scripting Merge csvs with column headers Post 302937247 by abh.kumar on Wednesday 4th of March 2015 11:39:36 AM
Old 03-04-2015
Merge csvs with column headers
hello gurus,

Somebody must have done this before, I couldn't find anything. Please redirect me if this was solved before, and if not please help.


To the problem now, I have multiple csv files (about 1000) which I need to concatenate by column header. The final file should have a superset of all columns of the csvs, and corresponding missing values should be blank.

All files are the same directory. Note the randomness and spaces in filenames.

File 2012 IBM 1
Code:
A,B,C
d1,d2,d3
d4,d5,d6


DELL2013 FILE_2
Code:
A,C,D
d12,d22,d32
d42,d52,d62


HP File 3
Code:
B,C,D,E
d22,d122,d132,
d32,d152,d262,142

My output should look like

Code:
A,B,C,D,E
d1,d2,d3,,
d4,d5,d6,,
d12,,d22,d32,
d42,,d52,d62,
,d22,d122,d132,
,d32,d152,d262,142


I can do this for files with common columns in the same order, but this cant incorporate new columns in the superfile

Code:
awk 'FNR==1 && NR!=1{next;}{print}' *.csv

Last edited by abh.kumar; 03-04-2015 at 12:45 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing Headers and a Column

I have a text file in unix with a layout like this Column 1 - 1-12 Column 2 - 13-39 Column 3 - 40-58 Column 4 - 59-85 Column 5 - 86-120 Columbn 6 - 121-131 The file also has a header on the first 6 lines of each page. Each page is 51 lines long. So I want to remove the header from each... (30 Replies)
Discussion started by: DerangedNick
30 Replies

2. Shell Programming and Scripting

Excel Column Headers

cat ABC.log | egrep "Error 500" >> /tmp/Logs.log egrep "<Mango>.*<.Mango>" Logs.log | sed -e "s/^.*<Mango/<Mango/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp1.xls egrep "<Apple>.*<.Apple>" Logs.log | sed -e "s/^.*<Apple/<Apple/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp2.xls print Heading1,... (1 Reply)
Discussion started by: pk_eee
1 Replies

3. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

4. UNIX for Dummies Questions & Answers

Sort by Column Headers

Hi All, I am new to UNIX can you please help me to sort a file with different columns my file looks like this $ cat gaut.txt UID PID PPID PGID SID C STIME TTY TIME CMD liveuser 3008 2892 3008 3008 0 11:58 ? 00:00:00 gnome-session liveuser 3019 1 ... (8 Replies)
Discussion started by: cgk1983
8 Replies

5. Shell Programming and Scripting

Matching words based on column headers

Hi , Pls help on this. Input file: NAME1 BSC1 TEXT ID 1 MAINSFAIL TEXT ID 2 DGON TEXT ID 3 lOADONDG NAME2 BSC2 TEXT ID 1 DGON TEXT ID 3 lOADONG (1 Reply)
Discussion started by: bha148
1 Replies

6. Shell Programming and Scripting

Transpose field names from column headers to values in one column

Hi All, I'm looking for a script which can transpose field names from column headers to values in one column. for example, the input is: IDa;IDb;IDc;PARAM1;PARAM2;PARAM3; a;b;c;p1val;p2val;p3val; d;e;f;p4val;p5val;p6val; g;h;i;p7val;p8val;p9val; into the output like this: ... (6 Replies)
Discussion started by: popesk
6 Replies

7. Shell Programming and Scripting

Merge column headers and transpose

Hello Everyone! I am new on this forum and this is my first post. I wish to apologize for my, not canonical, English. I would like to solve this problem but I have no clue of how do it!I will be grateful if someone could help me! I have a table like this: gene TF1 TF2 TF3 TF4 gene1 1 2 3 4... (5 Replies)
Discussion started by: giuliangiuseppe
5 Replies

8. Shell Programming and Scripting

Sar -u generates multiple column headers in csv file

Hi All, The below sar -u command generates multiple column headers in csv file Expected output should print column headers only once in the csv file shell script: $cat sar_cpu_EBS.sh #!/bin/bash while ; do sar -u 15 1 | awk '/^/ {print $1,$2,$4,$6,$7}' | tr -s ' ' ',' >>... (6 Replies)
Discussion started by: a1_win
6 Replies

9. Shell Programming and Scripting

Capturing column headers in an array

Hello, I am processing a tab delimited text file and need to grab all of the column headers in an array. The input looks like, num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

10. Shell Programming and Scripting

Row bind multiple csv files having different column headers

All, I guess by this time someone asked this kind of question, but sorry I am unable to find after a deep search. Here is my request I have many files out of which 2 sample files provided below. File-1 (with A,B as column headers) A,B 1,2 File-2 (with C, D as column headers) C,D 4,5 I... (7 Replies)
Discussion started by: ks_reddy
7 Replies

LEARN ABOUT DEBIAN

mongoimport

MONGOIMPORT(1)							  Mongo Database						    MONGOIMPORT(1)

NAME

       mongoimport - the Mongo import tool

SYNOPSIS

       mongoimport [OPTIONS]

DESCRIPTION

       mongoimport  is a tool to import a MongoDB collection from JSON, CSV, or TSV. The query can be filtered or a list of fields to input can be
       given.

   OPTIONS
       --help show usage information

       -h, --host HOST
	      server to connect to (default HOST=localhost)

       -d, --db DATABASE
	      database to use

       -c, --c COLLECTION
	      collection to use (some commands)

       --dbpath PATH
	      directly access mongod data files in this path, instead of connecting to a mongod instance

       -v, --verbose
	      be more verbose (include multiple times for more verbosity e.g. -vvvvv)

       -f, --fields NAMES
	      comma separated list of field names e.g. -f name,age

       --fieldFile FILE
	      file with fields names - 1 per line

       --jsonArray
	      load a json array, not one item per line. Currently limited to 4MB.

       --ignoreBlanks
	      if given, empty fields in csv and tsv will be ignored

       --type TYPE
	      type of file to import.  default: json (json,csv,tsv)

       --file FILE
	      file to import from; if not specified stdin is used

       --drop drop collection first

       --headerline
	      CSV,TSV only - use first line as headers

COPYRIGHT

       Copyright 2007-2009 10gen

SEE ALSO

       For more information, please refer to the MongoDB wiki, available at http://www.mongodb.org.

AUTHOR

       Kristina Chodorow

10gen								   January 2010 						    MONGOIMPORT(1)
All times are GMT -4. The time now is 05:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy