Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Join and merge multiple files with duplicate key and fill void columns
Top Forums Shell Programming and Scripting Join and merge multiple files with duplicate key and fill void columns Post 303040708 by yjacknewton on Tuesday 5th of November 2019 03:23:22 AM
Old 11-05-2019
Join and merge multiple files with duplicate key and fill void columns by using AWK
Join and merge multiple files with duplicate key and fill void columns

Hi guys,

I have many files that I want to merge:

file1.csv:

Code:
1|abc
1|def
2|ghi
2|jkl
3|mno
3|pqr


file2.csv:

Code:
1|123|jojo
1|NULL|bibi
3|789|zaza
3|012|NULL


file3.csv:

Code:
2|1a2b|1994|US
3|3c4e|1995|MG
4|5f6g|1996|FR


My desired output is like this, conserve the duplicate key and fill it with the data of anothers files, and the void columns is filling by NULL:

Code:
1|abc|123|jojo|NULL|NULL|NULL
1|def|123|jojo|NULL|NULL|NULL
1|abc|NULL|bibi|NULL|NULL|NULL
1|def|NULL|bibi|NULL|NULL|NULL
2|ghi|NULL|NULL|1a2b|1994|US
2|jkl|NULL|NULL|1a2b|1994|US
3|mno|789|zaza|3c4e|1995|MG
3|pqr|789|zaza|3c4e|1995|MG
3|mno|012|NULL|3c4e|1995|MG
3|pqr|012|NULL|3c4e|1995|MG
4|NULL|NULL|NULL|5f6g|1996|FR

How to do that with awk, join or shell scripts?
Last edited by yjacknewton; 11-06-2019 at 05:03 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Join 2 files with multiple columns: awk/grep/join?

Hello, My apologies if this has been posted elsewhere, I have had a look at several threads but I am still confused how to use these functions. I have two files, each with 5 columns: File A: (tab-delimited) PDB CHAIN Start End Fragment 1avq A 171 176 awyfan 1avq A 172 177 wyfany 1c7k A 2 7... (3 Replies)
Discussion started by: InfoSeeker
3 Replies

2. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

3. UNIX for Dummies Questions & Answers

How to use the the join command to join multiple files by a common column

Hi, I have 20 tab delimited text files that have a common column (column 1). The files are named GSM1.txt through GSM20.txt. Each file has 3 columns (2 other columns in addition to the first common column). I want to write a script to join the files by the first common column so that in the... (5 Replies)
Discussion started by: evelibertine
5 Replies

4. Shell Programming and Scripting

Merge multiple lines in same file with common key using awk

I've been a Unix admin for nearly 30 years and never learned AWK. I've seen several similar posts here, but haven't been able to adapt the answers to my situation. AWK is so damn cryptic! ;) I have a single file with ~900 lines (CSV list). Each line starts with an ID, but with different stuff... (6 Replies)
Discussion started by: protosd
6 Replies

5. Shell Programming and Scripting

Sort and join multiple columns using awk

Is it possible to join all the values after sorting them based on 1st column key and replace empty rows with 0 like below ? input a1 0 a1 1 a1 1 a3 1 b2 1 a2 1 a4 1 a2 1 a4 1 c4 1 a3 1 d1 1 a3 1 b1 1 d1 1 a4 1 c4 1 b2 1 b1 1 b2 1 c4 1 d1 1 output... (8 Replies)
Discussion started by: quincyjones
8 Replies

6. UNIX for Dummies Questions & Answers

Merge columns from multiple files

Hi all, I've searched the web for a long time trying to figure out how to merge columns from multiple files. I know paste will append columns like so: paste file1 file2 file3 file4 file5 ... But this becomes inconvenient when you want to append a large number of files into a single file. ... (2 Replies)
Discussion started by: torchij
2 Replies

7. Shell Programming and Scripting

Merge columns from multiple files

Hello and Good day I have a lot of files with same number of rows and columns.$2 and $3 are the same in all files . I need to merge $2,$3,$6 from first file and $6 from another files. File1: $1 $2 $3 $4 $5 $6... (8 Replies)
Discussion started by: ali.seifaddini
8 Replies

8. Shell Programming and Scripting

Join two files combining multiple columns and produce mix and match output

I would like to join two files when two columns in each file matches with each other and then produce an output when taking multiple columns. Like I have file A 1234,ABCD,23,JOHN,NJ,USA 2345,ABCD,24,SAM,NY,USA 5678,GHIJ,24,TOM,NY,USA 5678,WXYZ,27,MAT,NJ,USA and file B ... (2 Replies)
Discussion started by: mady135
2 Replies

9. UNIX for Dummies Questions & Answers

Merge selective columns from files based on common key

Hi, I am trying to selectively merge two files based on keys reported in the 1st column. File1: #file1-header1 file1-header2 111 qwe rtz uio 198 asd fgh jkl 165 yxc 789 poi uzt rew 89 lkj File2: #file2-header2 file2-header2 165 ghz nko2 ... (2 Replies)
Discussion started by: dovah
2 Replies

10. Shell Programming and Scripting

Join, merge, fill NULL the void columns of multiples files like sql "LEFT JOIN" by using awk

Hello, This post is already here but want to do this with another way Merge multiples files with multiples duplicates keys by filling "NULL" the void columns for anothers joinning files file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: 1|123|jojo 1|NULL|bibi... (2 Replies)
Discussion started by: yjacknewton
2 Replies

LEARN ABOUT DEBIAN

mysqmail-dovecot-logger

mysqmail-dovecot-logger(8)				      System Manager's Manual					mysqmail-dovecot-logger(8)

NAME

       mysqmail-dovecot-logger - logs dovecot-imapd and dovecot-pop3d traffic to a mysql database

SYNOPSIS

       mysqmail-dovecot-logger subprogram [ args ...  ]

DESCRIPTION

       mysqmail-dovecot-logger reads the syslog using tail -F, and for each dovecot lines of POP3 and IMAP corresponding to a disconnection, mysq-
       mail-dovecot-logger writes the total used bandwidth corresponding to the username. Configuration is read from /etc/mysqmail.conf.

       mysqmail-dovecot-logger uses a table corresponding to this one:

       CREATE TABLE IF NOT EXISTS pop_access (
	 id varchar(32) NOT NULL default '',
	 uid int(11) NOT NULL default '65534',
	 gid int(11) NOT NULL default '65534',
	 home varchar(255) NOT NULL default '',
	 shell varchar(255) NOT NULL default '',
	 mbox_host varchar(120) NOT NULL default '',
	 crypt varchar(50) NOT NULL default '',
	 passwd varchar(50) NOT NULL default '',
	 active int(11) NOT NULL default '1',
	 start_date date NOT NULL default '0000-00-00',
	 expire_date date NOT NULL default '0000-00-00',
	 quota_size int(11) NOT NULL default '0',
	 type varchar(20) NOT NULL default 'default',
	 memo text,du bigint(20) NOT NULL default '0',
	 another_perso varchar(5) NOT NULL default 'no',
	 redirect1 varchar(255) default NULL,
	 redirect2 varchar(255) default NULL,
	 localdeliver varchar(10) NOT NULL default 'yes',
	 pop3_login_count int(9) NOT NULL default '0',
	 pop3_transfered_bytes int(14) NOT NULL default '0',
	 imap_login_count int(9) NOT NULL default '0',
	 imap_transfered_bytes int(14) NOT NULL default '0',
	 last_login int(14) NOT NULL default '0',
	 PRIMARY KEY  (id,mbox_host) ) TYPE=MyISAM

VERSION

       This documentation describes  mysqmail-dovecot-logger  version  0.4.4  or  superior.   See  http://gplhost.com/softwares-mysqmail.html  for
       updates.

SEE ALSO

       qmail-send(3), syslog(3), logger(8)

															mysqmail-dovecot-logger(8)
All times are GMT -4. The time now is 06:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy