Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Reform Lines in File without blank lines and spaces

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reform Lines in File without blank lines and spaces
# 1  
Old 07-06-2012
Reform Lines in File without blank lines and spaces
Hello All,
I have a file with data as below. Each line consists of 21 fields. I am not able to load them back to the database.

Code:
50733339,"834","834            ","005010X279A1","N","Y","007977163","0001      ",30,"2110D ","EB   ","EB007 ","2    ","Conditional Required Data Element Miss
ing
",,"REJ            ",,"8    ","Segment has Data Element Errors
","2012-03-03-00.07.48.074377",
50732730,"834","834            ","005010X279A1","N","Y","007977108","0001      ",14,"2100C ","DTP  ","DTP002","1    ","Required Data Element Missing
",,"REJ            ",,"8    ","Segment has Data Element Errors
","2012-03-03-00.00.37.832866",

I need help to reform the lines in the files as below (remove all leading and trailing spaces and reform the 21 fields into a single line. I have marked the lines in different colors for clear understanding.

Code:
50733339,"834","834","005010X279A1","N","Y","007977163","0001",30,"2110D ","EB","EB007 ","2","Conditional Required Data Element Missing",,"REJ",,"8    ","Segment has Data Element Errors","2012-03-03-00.07.48.074377",
50732730,"834","834","005010X279A1","N","Y","007977108","0001",14,"2100C ","DTP","DTP002","1","Required Data Element Missing",,"REJ",,"8","Segment has Data Element Errors","2012-03-03-00.00.37.832866",

Any assistance to achieve the above would be very very helpful
Last edited by Scrutinizer; 07-06-2012 at 04:59 AM.. Reason: code tags
# 2  
Old 07-06-2012
Use:
Code:
perl -pi -e "s/([^,])\s*\n/\1/g" infile

Last edited by Klashxx; 07-06-2012 at 03:54 AM.. Reason: regex error
This User Gave Thanks to Klashxx For This Post:
Praveenkulkarni
# 3  
Old 07-06-2012
Need a bit more of help over this one. The command is not working for some of the lines. Can you please help if the lines are as below:


Code:
54961242,"780","780-HORIZ-00000","005010X223A2","Y","N","000001403","1178      ",30,"2300  ","HI   ",,,,,"REJ            ","PNASC.HIPDT.NARJ6NNJ.NEWA.D120309.T232827.0035430-H","8    ","Segment has Data Element Errors
","2012-03-10-00.07.41.438918","NM1*IL*1*FELDER*PAMELA****MI*YHQ3HZN39529840
CLM*HAH53954*4890.13***13:A:1**A*Y*Y
HI*BF:4019:::::::Y*BF:V5869:::::::E"
54961243,"780","780-HORIZ-00000","005010X223A2","Y","N","000001403","1185      ",36,"2300  ","HI   ",,,,,"REJ            ","PNASC.HIPDT.NARJ6NNJ.NEWA.D120309.T232827.0035430-H","8    ","Segment has Data Element Errors
","2012-03-10-00.07.41.499676","NM1*IL*1*MCCLURKIN*STEPHANIE****MI*NJX3HZN14544770
NM1*QC*1*MCCLURKIN*WILLIAM"

I need them as below:

Code:
54961242,"780","780-HORIZ-00000","005010X223A2","Y","N","000001403","1178",30,"2300","HI",,,,,"REJ","PNASC.HIPDT.NARJ6NNJ.NEWA.D120309.T232827.0035430-H","8","Segment has Data Element Errors","2012-03-10-00.07.41.438918", "NM1*IL*1*FELDER*PAMELA****MI*YHQ3HZN39529840CLM*HAH53954*4890.13***13:A:1**A*Y*YHI*BF:4019:::::::Y*BF:V5869:::::::E"
54961243,"780","780-HORIZ-00000","005010X223A2","Y","N","000001403","1185",36,"2300","HI",,,,,"REJ","PNASC.HIPDT.NARJ6NNJ.NEWA.D120309.T232827.0035430-H","8    ","Segment has Data Element Errors","2012-03-10-00.07.41.499676","NM1*IL*1*MCCLURKIN*STEPHANIE****MI*NJX3HZN14544770NM1*QC*1*MCCLURKIN*WILLIAM"

The data in bold represents the beginning of each line. Thanks a lot in advance

Moderator's Comments:
Mod Comment Please view this link: How to use [code]...[/code] tags
Last edited by Scrutinizer; 07-06-2012 at 05:00 AM.. Reason: Updated the sample data; mod: code tags
# 4  
Old 07-06-2012
This should be fine for the sample provided:
Code:
perl -pi -e 's/([^",])\s*\n/\1/g' infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Delete multiple lines between blank lines containing two patterns

Hi all, I'm looking for a way (sed or awk) to delete multiple lines between blank lines containing two patterns ex: user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 16... (3 Replies)
Discussion started by: ce9888
3 Replies

2. Shell Programming and Scripting

To check Blank Lines, Blank Records and Junk Characters in a File

Hi All Need Help I have a file with the below format (ABC.TXT) : ®¿¿ABCDHEJJSJJ|XCBJSKK01|M|7348974982790 HDFLJDKJSKJ|KJALKSD02|M|7378439274898 KJHSAJKHHJJ|LJDSAJKK03|F|9898982039999 (cont......) I need to write a script where it will check for : blank lines (between rows,before... (6 Replies)
Discussion started by: chatwithsaurav
6 Replies

3. UNIX for Advanced & Expert Users

Delete blank spaces and blank lines in a file

Hi Gurus, Somebody can say me how to delete blank spaces and blank lines in a file unix, please. Thank you for advanced. (10 Replies)
Discussion started by: systemoper
10 Replies

4. UNIX for Dummies Questions & Answers

Finding lines with a regular expression, replacing them with blank lines

So the tag for this forum says all newbies welcome... All I want to do is go through my file and find lines which contain a given string of characters then replace these with a blank line. I really tried to find a simple command to do this but failed. Here's what I did come up with though: ... (2 Replies)
Discussion started by: Golpette
2 Replies

5. Shell Programming and Scripting

sed show lines text between 2 blank lines

I have a file like blah blah blah blah this is the text I need, which might be between 1-4 lines, but always has a blank line above and below it, and is at the end of the text file the code tags don't show the trailing blank line. I started by deleting the last blank line with: ... (2 Replies)
Discussion started by: unclecameron
2 Replies

6. Shell Programming and Scripting

Help in replacing two blank lines with two lines of diff data

Hi.. I'm facing a trouble in replacing two blank lines in a file using shell script... I used sed to search a line and insert two blank lines after the searchd line using the following sed command. sed "/data/{G;G;}/" filename . In the file, after data tag, two lines got inserted blank lines..... (4 Replies)
Discussion started by: arjun_arippa
4 Replies

7. Shell Programming and Scripting

Delete blank lines, if blank lines are more than one using shell

Hi, Consider a file named "testfile" The contents of file are as below first line added for test second line added for test third line added for test fourth line added for test fifth line added for test (5 Replies)
Discussion started by: anil8103
5 Replies

8. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

9. UNIX for Dummies Questions & Answers

How to count lines - ignoring blank lines and commented lines

What is the command to count lines in a files, but ignore blank lines and commented lines? I have a file with 4 sections in it, and I want each section to be counted, not including the blank lines and comments... and then totalled at the end. Here is an example of what I would like my... (6 Replies)
Discussion started by: kthatch
6 Replies

10. UNIX for Dummies Questions & Answers

delete blank lines or lines with spaces only

hi there i'm trying to delete blank lines and or lines with spaces only from a series of files in an directory. to do so, i'm using this: for files in `ls /users/myname/pesop* 2>/dev/null` do grep -v ^$ $files > newfile mv newfile $files done now, this works great for blank lines but... (3 Replies)
Discussion started by: vascobrito
3 Replies
Login or Register to Ask a Question