File formatting with newlines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File formatting with newlines
# 1  
Old 04-21-2017
File formatting with newlines

Hi All -

I am in need of some help in formating the below file

Requirement -
1) replace newlines with space
2) replace '#~# ' with newline

-----------------------
sample inputfile a

Code:
I|abc|abc|aaa#~#
I|sddddd|tya|dfg

sfd

ssss#~#
I|tya1|tya2|dfg|sfd|aaa#~#
I|t#ya|a~aya4#~#

-------------------------------------------------------------------
required output file c

Code:
I|abc|abc|aaa
I|sddddd|tya|dfg sfd ssss
I|tya1|tya2|dfg|sfd|aaa
I|t#ya|a~aya4

---------------------------------------------------------------------------

I have tried regular sed commands but its not able to create newlines for step2 mentioned above

Code:
sed 's/#~# /\\\n/g' b>c

or
Code:
sed 's/#~# /'\\\n'/g' b>c

Any advise will be very much appreciated . Thanks in advance.

Last edited by Scrutinizer; 04-22-2017 at 02:20 AM.. Reason: code tags
# 2  
Old 04-22-2017
Try:
Code:
awk '{while (!/#~#/ && getline n>0) if(n!="")$0=$0 FS n; sub(/#~#/,x)}1'file


With GNU awk or mawk, if you do not mind an extra trailing newline at the end of the file, you can try this:
Code:
gawk '{$1=$1}1' RS='#~#' file

This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 04-22-2017
hi Scrutinizer -

Many thanks for your reply . I don't have access to unix box over the weekend , so haven't been able to check this on the actual output file . however i did run this on a small sample of data using a Emulator and the awk command is absolute spot on !! Thanks for suggesting this .

Would you by any chance be able to expain the option's used in the awk command ?

thanks again
# 4  
Old 04-22-2017
Depending on your sed version, this might work for you:
Code:
sed -n ':L; /#~#/ !{N;bL;}; /#~#/ {s///;s/\n\+/ /g; P;}' file
I|abc|abc|aaa
I|sddddd|tya|dfg sfd ssss
I|tya1|tya2|dfg|sfd|aaa
I|t#ya|a~aya4

This User Gave Thanks to RudiC For This Post:
# 5  
Old 04-22-2017
Hi RudiC - Thanks for your reply , would this solution be efficient enough to be run daily on files of around 20 MB size. I have an window of 5 mins to complete this formatting .
# 6  
Old 04-22-2017
I'm afraid there's no generic answer to that question. It depends on hardware performance, system load, process priority, and mayhap other factors. You'll need to test under several different conditions to come to a satisfying conclusion.
From a gut feeling, I'd say, yes, 5 mins should suffice to get 20MB done.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

UNIX file with Newlines

Hi Friends, I have a data file with new lines. How to remove the newlines and should be showed in one line. I tried using the command tr -d '\n' filename sed 's/\n//g' file name Ex: 1 abc hyd is actual record but in our scenario showing it as 1 abc hydthis record should be like... (5 Replies)
Discussion started by: victory
5 Replies

2. UNIX for Beginners Questions & Answers

Remove newlines and carriage return from a csv file using UNIX

I need to remove new lines and carriage returns from csv file. Is there anything other than sed and gwak by which we could achieve this ? Any suggestions ? (3 Replies)
Discussion started by: A_Gaddale
3 Replies

3. UNIX for Dummies Questions & Answers

Formatting data in a raw file by using another mapping file

Hi All, i have a requirement where i need to format the input RAW file ( which is CSV) by using another mapping file(also CSV file). basically i am getting feed file with dynamic headers by using mapping file (in that target field is mapped with source filed) i have to convert the raw file into... (6 Replies)
Discussion started by: ravi4informatic
6 Replies

4. Shell Programming and Scripting

Formatting file data to another file (control character related)

I have to write a program to read data from files and then format into another file. However, I face a strange problem related to control character that I can't understand and solve. The source file is compose of many lines with such format: T_NAME|P_NAME|P_CODE|DOCUMENT_PATH|REG_DATE ... (3 Replies)
Discussion started by: hk6279
3 Replies

5. Shell Programming and Scripting

'for LINE in $(cat file)' breaking at spaces, not just newlines

Hello. I'm making a (hopefully) simple shell script xml parser that outputs a file I can grep for information. I am writing it because I have yet to find a command line utility that can do this. If you know of one, please just stop now and tell me about it. Even better would be one I can input... (10 Replies)
Discussion started by: natedawg1013
10 Replies

6. UNIX for Dummies Questions & Answers

finding string in very long file without newlines

What's the best way to find a string in a very long file without newlines in Unix? The standard utility I'm aware of for finding a string in a single file is grep, but for a long file without newlines, I think the output is just going to be the input. I suppose I could use sed to replace the... (5 Replies)
Discussion started by: aaronpoley
5 Replies

7. Shell Programming and Scripting

Formatting the file

############################################################################################################# 19/Apr/2010:07:06:19 "Changed directory to dms-imrm/Fort_Wayne/Parkview" - SUCCESSFUL 19/Apr/2010:07:06:19 "Changed directory to Requisitions" - SUCCESSFUL ... (13 Replies)
Discussion started by: Tuxidow
13 Replies

8. Shell Programming and Scripting

Padding lines in a file with newlines

Hi Guys, I have a file as shown below: 55 6 77 8 88 98 7 85 45 544 1 33 32 13 13 (5 Replies)
Discussion started by: npatwardhan
5 Replies

9. Shell Programming and Scripting

Removing inserted newlines from a fileld of fixed width file.

Hi champs! I have a fixed width file in which the records appear like this 11111 <fixed spaces such as 6> description for 11111 <fixed spaces such as 6> some more field to the record of 11111 22222 <fixed spaces such as 6> description for 22222 <fixed spaces such as 6> some more field to the... (8 Replies)
Discussion started by: enigma_1
8 Replies

10. Shell Programming and Scripting

Help with formatting of file.

I have a file with following file format - DMCRH|||83000171|||14022008||0430|||8956612.23|J|||3571235|1378452|23468|6894|9234| DMCRH|||83000215|||15092007||0480|||121.33|J|||LineID003|RefNumSP003|RefNumMem003|0004|0003| What i need done is - 1. Cut the firt four digits of the date (eg 1402... (3 Replies)
Discussion started by: divz
3 Replies
Login or Register to Ask a Question