Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Merge csvs with column headers
Top Forums Shell Programming and Scripting Merge csvs with column headers Post 302937373 by MadeInGermany on Thursday 5th of March 2015 09:43:52 AM
Old 03-05-2015
The following solution reads the input file once, storing all in memory, and prints in the END section.
Code:
awk -F, '
(FNR==1) {
  # header line
  for (i=1; i<=NF; i++) {
    if (!($i in H)) {
      # H[] is for this hash-lookup and O[] for in-order lookup
      ++nh
      H[$i]=nh
      O[nh]=$i
    }
    # column mapping for this file
    map[i]=H[$i]
  }
  next
}
{
  # store fields in F array
  ++nr
  for (i=1; i<=NF; i++) {F[nr,map[i]]=$i}
}
END {
  # print header line
  for (i=1; i<=nh; i++) {printf "%s", sep O[i]; sep=FS}
  printf RS
  # print F array
  for (n=1; n<=nr; n++) {
    # line n
    sep=""
    for (i=1; i<=nh; i++) {
      printf "%s", sep F[n,i]
      sep=FS 
    }
    printf RS
  }
}
' *.csv

Last edited by MadeInGermany; 03-05-2015 at 10:53 AM.. Reason: realized that previous solution stores only headers and mapping
These 2 Users Gave Thanks to MadeInGermany For This Post:
abh.kumar Don Cragun
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing Headers and a Column

I have a text file in unix with a layout like this Column 1 - 1-12 Column 2 - 13-39 Column 3 - 40-58 Column 4 - 59-85 Column 5 - 86-120 Columbn 6 - 121-131 The file also has a header on the first 6 lines of each page. Each page is 51 lines long. So I want to remove the header from each... (30 Replies)
Discussion started by: DerangedNick
30 Replies

2. Shell Programming and Scripting

Excel Column Headers

cat ABC.log | egrep "Error 500" >> /tmp/Logs.log egrep "<Mango>.*<.Mango>" Logs.log | sed -e "s/^.*<Mango/<Mango/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp1.xls egrep "<Apple>.*<.Apple>" Logs.log | sed -e "s/^.*<Apple/<Apple/" | cut -f2 -d">"| cut -f1 -d"<" >> /tmp/temp2.xls print Heading1,... (1 Reply)
Discussion started by: pk_eee
1 Replies

3. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

4. UNIX for Dummies Questions & Answers

Sort by Column Headers

Hi All, I am new to UNIX can you please help me to sort a file with different columns my file looks like this $ cat gaut.txt UID PID PPID PGID SID C STIME TTY TIME CMD liveuser 3008 2892 3008 3008 0 11:58 ? 00:00:00 gnome-session liveuser 3019 1 ... (8 Replies)
Discussion started by: cgk1983
8 Replies

5. Shell Programming and Scripting

Matching words based on column headers

Hi , Pls help on this. Input file: NAME1 BSC1 TEXT ID 1 MAINSFAIL TEXT ID 2 DGON TEXT ID 3 lOADONDG NAME2 BSC2 TEXT ID 1 DGON TEXT ID 3 lOADONG (1 Reply)
Discussion started by: bha148
1 Replies

6. Shell Programming and Scripting

Transpose field names from column headers to values in one column

Hi All, I'm looking for a script which can transpose field names from column headers to values in one column. for example, the input is: IDa;IDb;IDc;PARAM1;PARAM2;PARAM3; a;b;c;p1val;p2val;p3val; d;e;f;p4val;p5val;p6val; g;h;i;p7val;p8val;p9val; into the output like this: ... (6 Replies)
Discussion started by: popesk
6 Replies

7. Shell Programming and Scripting

Merge column headers and transpose

Hello Everyone! I am new on this forum and this is my first post. I wish to apologize for my, not canonical, English. I would like to solve this problem but I have no clue of how do it!I will be grateful if someone could help me! I have a table like this: gene TF1 TF2 TF3 TF4 gene1 1 2 3 4... (5 Replies)
Discussion started by: giuliangiuseppe
5 Replies

8. Shell Programming and Scripting

Sar -u generates multiple column headers in csv file

Hi All, The below sar -u command generates multiple column headers in csv file Expected output should print column headers only once in the csv file shell script: $cat sar_cpu_EBS.sh #!/bin/bash while ; do sar -u 15 1 | awk '/^/ {print $1,$2,$4,$6,$7}' | tr -s ' ' ',' >>... (6 Replies)
Discussion started by: a1_win
6 Replies

9. Shell Programming and Scripting

Capturing column headers in an array

Hello, I am processing a tab delimited text file and need to grab all of the column headers in an array. The input looks like, num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

10. Shell Programming and Scripting

Row bind multiple csv files having different column headers

All, I guess by this time someone asked this kind of question, but sorry I am unable to find after a deep search. Here is my request I have many files out of which 2 sample files provided below. File-1 (with A,B as column headers) A,B 1,2 File-2 (with C, D as column headers) C,D 4,5 I... (7 Replies)
Discussion started by: ks_reddy
7 Replies

LEARN ABOUT DEBIAN

mongoimport

MONGOIMPORT(1)							  Mongo Database						    MONGOIMPORT(1)

NAME

       mongoimport - the Mongo import tool

SYNOPSIS

       mongoimport [OPTIONS]

DESCRIPTION

       mongoimport  is a tool to import a MongoDB collection from JSON, CSV, or TSV. The query can be filtered or a list of fields to input can be
       given.

   OPTIONS
       --help show usage information

       -h, --host HOST
	      server to connect to (default HOST=localhost)

       -d, --db DATABASE
	      database to use

       -c, --c COLLECTION
	      collection to use (some commands)

       --dbpath PATH
	      directly access mongod data files in this path, instead of connecting to a mongod instance

       -v, --verbose
	      be more verbose (include multiple times for more verbosity e.g. -vvvvv)

       -f, --fields NAMES
	      comma separated list of field names e.g. -f name,age

       --fieldFile FILE
	      file with fields names - 1 per line

       --jsonArray
	      load a json array, not one item per line. Currently limited to 4MB.

       --ignoreBlanks
	      if given, empty fields in csv and tsv will be ignored

       --type TYPE
	      type of file to import.  default: json (json,csv,tsv)

       --file FILE
	      file to import from; if not specified stdin is used

       --drop drop collection first

       --headerline
	      CSV,TSV only - use first line as headers

COPYRIGHT

       Copyright 2007-2009 10gen

SEE ALSO

       For more information, please refer to the MongoDB wiki, available at http://www.mongodb.org.

AUTHOR

       Kristina Chodorow

10gen								   January 2010 						    MONGOIMPORT(1)
All times are GMT -4. The time now is 06:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy