Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Merge multiple lines into a single line

Shell Programming and Scripting


Tags
awk, sed, solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #15  
Old Unix and Linux 07-05-2017
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Linux Don, that definitely worked for all but a small portion of the log. That really helps. In a copy of the actual log, I ran the code you gave me and it did merge most lines.

I ran the od -bc command against it and I am seeing \n in some lines in the statement text= field. So when I view the log, some of the statement text= field content is still on their own line.

It sounds like the \n means new line. Would it be possible to remove the \n if it does not precede the timestamp= field if that is the case?

Last edited by dwdnet; 07-05-2017 at 05:20 PM.. Reason: Spelling mistake.
Sponsored Links
    #16  
Old Unix and Linux 07-05-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 September 2017, 6:14 PM EDT
Location: San Jose, CA, USA
Posts: 10,508
Thanks: 542
Thanked 3,675 Times in 3,135 Posts
You said originally that blank lines separate records. From what you have said in post #15 it sounds like you have some blank lines in the middle of some records. It isn't clear how you want the data represented by those blank input lines to appear in your output.

Please show us a couple of sample input records (in CODE tags) that do not work with the awk code suggested in post #12, show us the output that code is currently producing (in CODE tags), and show us the output that you want to produce from those input records (also in CODE tags).
The Following User Says Thank You to Don Cragun For This Useful Post:
dwdnet (07-10-2017)
Sponsored Links
    #17  
Old Unix and Linux 07-10-2017
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
I'll have to create a mock up of a sample of the records as I can't put up live data. Hopefully that will suffice. Thanks.

---------- Post updated at 01:58 PM ---------- Previous update was at 11:29 AM ----------

I'm not sure this will work. Some of the statement text= lines are longer than 4096 characters. Quite complex database statements. I'll have to try and approach this from a different angle. I sincerely appreciate all the help though. Thanks, you guys are great.
    #18  
Old Unix and Linux 07-10-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 September 2017, 6:14 PM EDT
Location: San Jose, CA, USA
Posts: 10,508
Thanks: 542
Thanked 3,675 Times in 3,135 Posts
If the input is a text file, you don't care that the output might not be a text file (due to line length limitations), can clearly describe the rules for combining input lines into output lines, and you can provide representative (not actual) sample input and corresponding sample output, then the fact the some of the output lines are long should not be a problem.

Some versions of the awk utility can't write more than 4K bytes at a time, but all can write a single output line by writing several partial lines as long as each segment of the line being written is less that 4K bytes.

Are you saying that your input didn't have any blank lines in the middle of any records, but instead had some records that were more than 4K bytes long?
The Following User Says Thank You to Don Cragun For This Useful Post:
dwdnet (07-10-2017)
Sponsored Links
    #19  
Old Unix and Linux 07-10-2017
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Some of the records are longer than 4K. There are no blank lines in the long record however there are some line feeds, which is what is breaking the record into different lines. I am able to manually edit the text file and remove the LF though.
Sponsored Links
    #20  
Old Unix and Linux 07-10-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 September 2017, 6:14 PM EDT
Location: San Jose, CA, USA
Posts: 10,508
Thanks: 542
Thanked 3,675 Times in 3,135 Posts
You can, of course, edit the files manually. Or, you can show us representative sample input and the corresponding output you want from that input and we might then be able to help you modify the code shown in post #12 to get what you want programmatically.
Sponsored Links
    #21  
Old Unix and Linux 07-11-2017
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Manually editing would be very time consuming so I'll attempt to explain using the example below. Running your initial function helped quite a bit and re-formated the majority of records. There were some fields, within the record, (statement text=) that still were broken up with a LF.

Sample Input

Code:
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, 
table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

Sample Output

Code:
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Need help combining txt files w/ multiple lines into csv single cell - also need data merge jetsetter UNIX for Dummies Questions & Answers 2 02-04-2013 09:12 AM
Merge a group of lines into single line prasanth_babu UNIX for Advanced & Expert Users 8 10-18-2010 01:01 PM
merge lines into single line based on symbol \t repinementer Shell Programming and Scripting 5 07-15-2009 09:16 AM
Help on Merge multi-lines into one single line happyday Shell Programming and Scripting 5 05-25-2009 07:30 AM
Merge multi-lines into one single line happyday Shell Programming and Scripting 4 05-09-2009 01:00 PM



All times are GMT -4. The time now is 04:27 AM.