Sponsored Content
Top Forums Shell Programming and Scripting Merging multiple lines to columns with awk, while inserting commas for missing lines Post 302953641 by RalphNY on Monday 31st of August 2015 04:24:31 AM
Old 08-31-2015
Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all,
I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person.

In the csv, a person may be listed from one to four times consecutively. Each row in which they are listed has a number of hours and a code (WE, MO, CE, ES) that indicates type of hours

I found a similar thread and a solution to a similar problem. The forum won't let me post links yet, but you can add this to the base URL /shell-programming-and-scripting/208027-merge-multiple-lines-same-file-common-key-using-awk.html to open that thread.

However, the output of that script tacks each of the optional lines on to the end of the first line, but that messes up the csv columns, so that the same data is not in the same place in the merged columns, afterward.

Here is a simplified version of the input data I have, which has a lot of fields per line after these, but where for each person, all fields except for the type-of-hours code, and number of hours are identical between lines the person is listed on:
Code:
1111,John Smith,WE,105,Fourty Two Cherry Lane,Nowhere,NY
1111,John Smith,MO,23,Fourty Two Cherry Lane,Nowhere,NY
2222,Elane Boozler,WE,70,Twelve Peach Court,Elsehere,NY
2222,Elane Boozler,MO,24,Twelve Peach Court,Elsehere,NY
2222,Elane Boozler,CE,30,Twelve Peach Court,Elsehere,NY
2222,Elane Boozler,ES,8,Twelve Peach Court,Elsehere,NY
3333,Hortense Gable,CE,25,Three Pinapple Ln,Somewhere,NY
3333,Hortense Gable,ES,14,Three Pinapple Ln,Somewhere,NY

I need the output, to be in the form
Code:
1111,John Smith,105,23,,,Fourty Two Cherry Lane,Nowhere, NY
2222,Elane Boozler,70,24,30,8,Peach Court,Elsehere,NY
3333,Hortense Gable,,,25,14,Three Pinapple Ln,Somewhere,NY

Please let me know how I might get this kind of output from this kind of input using awk. Thanks so much for your help!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging non-repeating columns of lines

Hello, I have file to work with. It has 5 columns. The first three, altogether, constitutes the position. The 4th column contains some values for downstream analysis and the fifth column contains some values that I want to add to 4th column (only if they happen to be in the same position). My... (5 Replies)
Discussion started by: menenuh
5 Replies

2. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

3. Shell Programming and Scripting

inserting multiple lines with awk

awk '/<login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient">/{p++} /<login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient">/ && p==1 {$0="Mulitple lines here\n"$0}1' login-config.xml In the above awk code inside shell script, i am having problems when... (1 Reply)
Discussion started by: sunrexstar
1 Replies

4. Shell Programming and Scripting

inserting and replacing lines with awk

Hello, I need to insert varying lines (i.e. these lines are an output of another script) between lines starting with certain fields. An example to make it more clear. This is the file where I wanna insert lines: (save it as "input.txt") ContrInMi_c_mir 2 10066 181014 200750... (12 Replies)
Discussion started by: tempestas
12 Replies

5. Shell Programming and Scripting

merging multiple lines into single line

Hi, 1. Each message starts with date 2. There is blank line between each message 3. Each message does not contain same number of lines. Any help in merging multiple lines in each message to a single line is much appreciated. AIX: Korn Shell Error log file looks like below. ... (5 Replies)
Discussion started by: bala123
5 Replies

6. Shell Programming and Scripting

Awk match multiple columns in multiple lines in single file

Hi, Input 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr21.fa chr3.fa 7488 7389 chr1.fa chr1.fa 3546 9887 chr9.fa chr5.fa 7898 7387 chrX.fa chr3.fa Desired Output 7488 7389 chr1.fa chr1.fa 2 3546 9887 chr5.fa chr9.fa 2... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

7. Shell Programming and Scripting

Merging multiple files using lines from one file

I have been working of this script for a very long time and I have searched the internet for direction but I am stuck here. I have about 3000 files with two columns each. The length of each file is 50000. Each of these files is named this way b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11, b.12... (10 Replies)
Discussion started by: iconig
10 Replies

8. Shell Programming and Scripting

Merging multiple lines

I do have a text file with multiple lines on it. I want to put the lines of text into a single line where ever there is ";" for example ert, ryt, yvig, fgr; rtyu, hjk, uio, hyu, hjo; ghj, tyu, gho, hjp, jklo, kol; The resultant file I would like to have is ert, ryt, yvig, fgr;... (2 Replies)
Discussion started by: Kanja
2 Replies

9. UNIX for Beginners Questions & Answers

Merging multiple lines into single line based on one column

I Want to merge multiple lines based on the 1st field and keep into single record. SRC File: AAA_POC_DB.TAB1 AAA_POC_DB.TAB2 AAA_POC_DB.TAB3 AAA_POC_DB.TAB4 BBB_POC_DB.TAB1 BBB_POC_DB.TAB2 CCC_POC_DB.TAB6 OUTPUT ----------------- 'AAA_POC_DB','TAB1','TAB2','TAB3','TAB4'... (10 Replies)
Discussion started by: raju2016
10 Replies

10. UNIX for Beginners Questions & Answers

Merging two lines into one (awk)

Hi, I am attempting to merge the following lines which run over two lines using awk. INITIAL OUTPUT 2019 Sep 28 10:47:24.695 hkaet9612 last message repeated 1 time 2019 Sep 28 10:47:24.695 hkaet9612 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interfa ce Ethernet1/45 is down (Interface removed)... (10 Replies)
Discussion started by: sand1234
10 Replies
INSERT(7)							   SQL Commands 							 INSERT(7)

NAME
       INSERT - create new rows in a table

SYNOPSIS
       INSERT INTO table [ ( column [, ...] ) ]
	   { DEFAULT VALUES | VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query }
	   [ RETURNING * | output_expression [ [ AS ] output_name ] [, ...] ]

DESCRIPTION
       INSERT  inserts new rows into a table.  One can insert one or more rows specified by value expressions, or zero or more rows resulting from
       a query.

       The target column names can be listed in any order. If no list of column names is given at all, the default is all the columns of the table
       in  their  declared order; or the first N column names, if there are only N columns supplied by the VALUES clause or query. The values sup-
       plied by the VALUES clause or query are associated with the explicit or implicit column list left-to-right.

       Each column not present in the explicit or implicit column list will be filled with a default value, either its declared default  value	or
       null if there is none.

       If the expression for any column is not of the correct data type, automatic type conversion will be attempted.

       The  optional  RETURNING clause causes INSERT to compute and return value(s) based on each row actually inserted.  This is primarily useful
       for obtaining values that were supplied by defaults, such as a serial sequence number. However, any expression using the table's columns is
       allowed. The syntax of the RETURNING list is identical to that of the output list of SELECT.

       You  must have INSERT privilege on a table in order to insert into it. If a column list is specified, you only need INSERT privilege on the
       listed columns.	Use of the RETURNING clause requires SELECT privilege on all columns mentioned in RETURNING.  If you use the query  clause
       to insert rows from a query, you of course need to have SELECT privilege on any table or column used in the query.

PARAMETERS
       table  The name (optionally schema-qualified) of an existing table.

       column The name of a column in table.  The column name can be qualified with a subfield name or array subscript, if needed. (Inserting into
	      only some fields of a composite column leaves the other fields null.)

       DEFAULT VALUES
	      All columns will be filled with their default values.

       expression
	      An expression or value to assign to the corresponding column.

       DEFAULT
	      The corresponding column will be filled with its default value.

       query  A query (SELECT statement) that supplies the rows to be inserted. Refer to the SELECT [select(7)] statement for a description of the
	      syntax.

       output_expression
	      An  expression to be computed and returned by the INSERT command after each row is inserted. The expression can use any column names
	      of the table.  Write * to return all columns of the inserted row(s).

       output_name
	      A name to use for a returned column.

OUTPUTS
       On successful completion, an INSERT command returns a command tag of the form

       INSERT oid count

       The count is the number of rows inserted. If count is exactly one, and the target table has OIDs, then oid  is  the  OID  assigned  to  the
       inserted row. Otherwise oid is zero.

       If the INSERT command contains a RETURNING clause, the result will be similar to that of a SELECT statement containing the columns and val-
       ues defined in the RETURNING list, computed over the row(s) inserted by the command.

EXAMPLES
       Insert a single row into table films:

       INSERT INTO films VALUES
	   ('UA502', 'Bananas', 105, '1971-07-13', 'Comedy', '82 minutes');

       In this example, the len column is omitted and therefore it will have the default value:

       INSERT INTO films (code, title, did, date_prod, kind)
	   VALUES ('T_601', 'Yojimbo', 106, '1961-06-16', 'Drama');

       This example uses the DEFAULT clause for the date columns rather than specifying a value:

       INSERT INTO films VALUES
	   ('UA502', 'Bananas', 105, DEFAULT, 'Comedy', '82 minutes');
       INSERT INTO films (code, title, did, date_prod, kind)
	   VALUES ('T_601', 'Yojimbo', 106, DEFAULT, 'Drama');

       To insert a row consisting entirely of default values:

       INSERT INTO films DEFAULT VALUES;

       To insert multiple rows using the multirow VALUES syntax:

       INSERT INTO films (code, title, did, date_prod, kind) VALUES
	   ('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
	   ('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy');

       This example inserts some rows into table films from a table tmp_films with the same column layout as films:

       INSERT INTO films SELECT * FROM tmp_films WHERE date_prod < '2004-05-07';

       This example inserts into array columns:

       -- Create an empty 3x3 gameboard for noughts-and-crosses
       INSERT INTO tictactoe (game, board[1:3][1:3])
	   VALUES (1, '{{" "," "," "},{" "," "," "},{" "," "," "}}');
       -- The subscripts in the above example aren't really needed
       INSERT INTO tictactoe (game, board)
	   VALUES (2, '{{X," "," "},{" ",O," "},{" ",X," "}}');

       Insert a single row into table distributors, returning the sequence number generated by the DEFAULT clause:

       INSERT INTO distributors (did, dname) VALUES (DEFAULT, 'XYZ Widgets')
	  RETURNING did;

COMPATIBILITY
       INSERT conforms to the SQL standard, except that the RETURNING clause is a PostgreSQL extension. Also, the case in which a column name list
       is omitted, but not all the columns are filled from the VALUES clause or query, is disallowed by the standard.

       Possible limitations of the query clause are documented under SELECT [select(7)].

SQL - Language Statements					    2010-05-14								 INSERT(7)
All times are GMT -4. The time now is 11:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy