Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Issue in Concatenation/Joining of lines in a dynamically generated file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Issue in Concatenation/Joining of lines in a dynamically generated file
# 1  
Old 03-24-2015
Blade Issue in Concatenation/Joining of lines in a dynamically generated file
Hi,

I have a file containing many records delimited by pipe (|).
Each record should contain 17 columnns/fields. there are some fields having fields less than 17.So i am extracting those records to a file using the below command
Code:
awk 'BEGIN {FS="|"} NF !=17 {print}' feedfile.txt >feedfilebadrecs.txt

Then i am concatenating all lines in feedfilebadrecs.txt to one single line.

Code:
tr -d '\n' < feedfilebadrecs.txt

but its not working. The concatenation is not happening.I have tried concatenating with other commands like paste,sed , awk etc. but not working(giving some random line as output from the input file)

I applied these commands for concatenating on some other manually created files and its working .

I am not able to understand why concatenation is not happening on dynamically created file using awk.

Please help me to figure this out..

Thank You
# 2  
Old 03-24-2015
Code:
awk 'BEGIN {FS="|"} NF !=17 {printf $0} END {print ""} ' feedfile.txt >feedfilebadrecs.txt

and forget about tr.
If this doesn't work, post the output of: cat -vet feedfile.txt using the code tags.
Last edited by Don Cragun; 03-24-2015 at 10:36 PM.. Reason: Fix ICODE tag.
This User Gave Thanks to vgersh99 For This Post:
TomG
# 3  
Old 03-24-2015
What operating system are you using?
How many lines are you concatenating?
After concatenating those lines, how long is your single output line supposed to be?
How have you determined that the concatenation is not working?
What output do you get if you run the commands:
Code:
awk 'BEGIN {FS="|"} NF !=17' feedfile.txt >feedfilebadrecs.txt
tr -d '\n' < feedfilebadrecs.txt > longline
wc feedfile.txt feedfilebadrecs.txt longline

P.S. If you are trying this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk.
This User Gave Thanks to Don Cragun For This Post:
TomG
# 4  
Old 03-25-2015
Hi Don Cragun,

1)I am using HP-UX
2) Depends on the requirement, right now I am concatenating only few lines as shown in the screen shot.
3)The single output line can be as long as 300 characters.
4)Its giving only single line from a group of lines in the input file as shown in the screenshot
5)Please see the screen shot
Issue in Concatenation/Joining of lines in a dynamically generated file-concatenation-issuejpg
Last edited by TomG; 03-25-2015 at 02:57 AM..
# 5  
Old 03-25-2015
Quote:
Originally Posted by vgersh99
Code:
awk 'BEGIN {FS="|"} NF !=17 {printf $0} END {print ""} ' feedfile.txt >feedfilebadrecs.txt

and forget about tr.
If this doesn't work, post the output of: cat -vet feedfile.txt using the code tags.

Hi vgersh99,

Awk is working fine, The problem is I am not able to concatenate lines in file created by awk.


cat -vet feedfile.txt


Code:
status|Date|Owner|BigDealID|OPGcode|Customer|Product|GandalfDiscoDatedd-mm-yy|Endforecastdatedd-mm-yy|Comments|ExclusioninGandalf?|Completiondate|Comments|Forecastcoomunicatedtoplanning?|Comments|Lastrefreshmentdate|Forecaststillvalid?^M$
Ok|22-10-2008|Alina^M$
Bobirca|73696227||ROYALDUTCHSHELLPLC|SD128AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDU^M$
TCHSHELLPLC|SD129AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD130AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD13 ^M$
1AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD132AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD142AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD143AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD144AA||01-12-2014||Yes||||||^M$
Ok|22-10-2008|AlinaBobirca|73696227||ROYALDUTCHSHELLPLC|SD145AA||01-12-2014||Yes||||||^M$

Last edited by Don Cragun; 03-25-2015 at 05:18 AM.. Reason: Add missing CODE tags.
# 6  
Old 03-25-2015
Everything is working just fine.

Your problems are that you have a DOS file that you're working on with UNIX utilities (get rid of the carriage return characters and you would get something closer to what you were expecting), and you want to join pairs of truncated lines from feedfile.txt; not all truncated lines.

Try (untested since you haven't given us a usable copy of feedfile.txt):
Code:
awk '
BEGIN {		FS = OFS = "|" }
{		sub(/\r/, "") }
NF < 17 {	$1 = save $1; save = $0 }
NF >= 17 {	print; save = "" }
' feedfile.txt > fixed_feedfile.txt

This User Gave Thanks to Don Cragun For This Post:
TomG
# 7  
Old 03-25-2015
Hi Don,

please find the screenshot.
I executed your command,now awk output is all records having number of fields as 17. My aim is to extract all records which are less than 17 fields into a new file and then concatenate all lines in that new file.
Also please find the attached feed file.

Thanks
Issue in Concatenation/Joining of lines in a dynamically generated file-concatenate-issuepng
feedfile.txt (1.0 KB) 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Concatenation of files with same naming patterns dynamically

Since my last threads were closed on account of spamming, keeping just this one opened! Hi, I have the following reports that get generated every 1 hour and this is my requirement: 1. 5 reports get generated every hour with the names "Report.Dddmmyy.Thhmiss.CTLR"... (5 Replies)
Discussion started by: Jesshelle David
5 Replies

2. UNIX and Linux Applications

Concatenation of different reports dynamically

Hi, I have the following reports that get generated every 1 hour and this is my requirement: 1. 5 reports get generated every hour with the names "Report.Dddmmyy.Thhmiss.CTLR" "Report.Dddmmyy.Thhmiss.ACCD" "Report.Dddmmyy.Thhmiss.BCCD" "Report.Dddmmyy.Thhmiss.CCCD"... (1 Reply)
Discussion started by: Jesshelle David
1 Replies

3. Shell Programming and Scripting

Joining lines in a file - help!

I'm looking for a way to join lines in a file; e.,g consider the following R|This is line 1 R|This is line 2 R|This is line 3 R|This is line 4 R|This is line 5 what i want to end up with is R|This is line 1 R|This is line 2 R|This is line 3 R|This is line 4 R|This is line 5 so... (15 Replies)
Discussion started by: Storms
15 Replies

4. Shell Programming and Scripting

issue while copying file dynamically whith in loop?

I need to copy the log file dynamically and that should run in loop , which means it should pick what ever the latest file is updated in that directory. I am able to display the list and copy to directly but i have no idea on how to pick the dynamically updated files. when i use this code, i... (1 Reply)
Discussion started by: johninweb
1 Replies

5. Shell Programming and Scripting

Downloading of dynamically generated URL using curl and sed

I've been attempting to use curl and sed to allow for downloading a file from a dynamically generated URL. I've been able to retrieve and save the HTML of the page that does the dynamic generation of the download URL using curl but I'm very new to sed and I seem to be stuck at this part. HTML: ... (1 Reply)
Discussion started by: schwein
1 Replies

6. Shell Programming and Scripting

deleting few lines from a file dynamically

here is the part of the code var1="replicate-ignore-db" var2="replicate-same-server-id" var3="skip-slave-start" var4="report-host" var5="master-host" var6="master-user" var7="master-password" var8="master-port" #code below deleted paramters as above if exists in my.cnf for i in 1 2 3 4... (4 Replies)
Discussion started by: vivek d r
4 Replies

7. Shell Programming and Scripting

bash - joining lines in a file

I’m writing a bash shell script and I want to join lines together where two variables on each line are the same ie. 12345variablestuff43212morevariablestuff 12345variablestuff43212morevariablestuff 34657variablestuff78945morevariablestuff 34657variablestuff78945morevariablestuff... (12 Replies)
Discussion started by: Cultcha
12 Replies

8. Shell Programming and Scripting

Issue with Joining lines from two files

Hi, I have two text files, that need their data joining/concatenation. 'Paste' works for this. But have an issue when there is mismatch in number of rows in each file. E.g. (main file) File1 - has 20 rows File2 - has 30 rows. Command 'paste file1 file2 > file3' joins all lines. I want the... (4 Replies)
Discussion started by: sharath160
4 Replies

9. Shell Programming and Scripting

Joining 2 lines in a file together

Hi guys, I've got a log file which has entries that look like this: ------------------------------------------------------------------------------- 06/08/04 07:57:57 AMQ9002: Channel program started. EXPLANATION: Channel program 'INSCCPQ1.HSMTSPQ1' started. ACTION: None. ... (3 Replies)
Discussion started by: m223464
3 Replies

10. Shell Programming and Scripting

Joining lines in log file

Hi, I need to develop a script to join multiple three lines in a log file into one line for processing with awk and grep. I looked at tr with no success. The first line contains the date time information. The second line contains the error line. The third line is a blank line. Thanks, Mike (3 Replies)
Discussion started by: bubba112557
3 Replies
Login or Register to Ask a Question