Removing cr,lf till number of fields are full


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing cr,lf till number of fields are full
# 15  
Old 12-04-2013
Seeing your real data has found a much simpler pattern, anyway. The first column should have a value like AST0000..., CI00000, etc.

Code:
awk '{gsub(/[\r\n]/,""); } (NR==1) { printf("%s",$0); next } $1 ~ /0000/ { printf "\n" ; print ; next } 1 ; END { printf("\n") }' ORS="|" test.txt > output.txt

# 16  
Old 12-04-2013
Why don't you try it vice versa - tr all pipes to <NL> and then paste serial:
Code:
tr -s '|' '\n' <file |paste -sd'|||\n'
1|2|3|4
a|b|c|d
1|2|3|4
a|b|c|d

# 17  
Old 12-04-2013
Quote:
Originally Posted by RudiC
Why don't you try it vice versa - tr all pipes to <NL> and then paste serial:
Code:
tr -s '|' '\n' <file |paste -sd'|||\n'
1|2|3|4
a|b|c|d
1|2|3|4
a|b|c|d

Because some records contain multiple newlines and this will count them as separate records.
# 18  
Old 12-04-2013
The <CR> are always at the end of "real" lines in that csv. EXCEL uses <NL> for formatting new lines within one cell. Not sure how the requestor wants to load that into a database. And, there's | in double quotes as well...
In the end, I think, sth simple might do:
Code:
awk -F\| '{while (NF < 70) {getline X; $0=$0"|"X}}1' test.txt

EDIT: Even better:
Code:
awk -F\| '{while (NF < 70) {getline X; $0=$0" "X}}1' test.txt

With this, all lines will have 70 fields except 4 which have 71 due to "...|..." constructs.

Last edited by RudiC; 12-04-2013 at 07:39 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Is there a UNIX command that can compare fields of files with differing number of fields?

Hi, Below are the sample files. x.txt is from an Excel file that is a list of users from Windows and y.txt is a list of database account. $ head -500 x.txt y.txt ==> x.txt <== TEST01 APP_USER_PROFILE USER03 APP_USER_PROFILE TEST02 APP_USER_EXP_PROFILE TEST04 APP_USER_PROFILE USER01 ... (3 Replies)
Discussion started by: newbie_01
3 Replies

2. Shell Programming and Scripting

Printing Number of Fields with the line number

Hi, How to print the number of fields in each record with the line number? Lets saw I have 3212|shipped|received| 3213|shipped|undelivered| 3214|shipped|received|delivered I tried the code awk -F '|' '{print NF}' This gives me ouput as 3 3 4 (5 Replies)
Discussion started by: machomaddy
5 Replies

3. Shell Programming and Scripting

Get the number of lines till I get line

Hi All, I have a file as below: abc.txt ****************************** * HEADER DESCRIPTION ****************************** *Supplier: Prism Customer: MNI -NIGERIA Quantity: 2 Type: PLUG-IN Profile: 70.00 *Subscription: Generic... (5 Replies)
Discussion started by: arunshankar.c
5 Replies

4. Shell Programming and Scripting

Count the number of fields in column

Hi I was going through the below thread https://www.unix.com/shell-programming-scripting/48535-how-count-number-fields-record.html I too have something similar requirement as specified in this thread but the number of columns in my case can be very high, so I am getting following error. ... (3 Replies)
Discussion started by: shekharjchandra
3 Replies

5. Shell Programming and Scripting

Number of fields handled by awk

Hi Gurus, Have a file seperated by "~" and no of fields is 104. When i try to run awk, it erros out. awk: record `B~A31~T24_STF~~~2009...' has too many fields Any idea how can i extract a specific filed with this many fields in a row. Kindly help (3 Replies)
Discussion started by: srivat79
3 Replies

6. Shell Programming and Scripting

awk sed cut? to rearrange random number of fields into 3 fields

I'm working on formatting some attendance data to meet a vendors requirements to upload to their system. With some help on the forums here, I have the data close. But they've since changed what they want. The vendor wants me to submit three fields to them. Field 1 is the studentid field,... (4 Replies)
Discussion started by: axo959
4 Replies

7. Shell Programming and Scripting

way to print all the string till we get a space and a number

Is there any way to print all the string till we get a space and a number and store it a variable for eg we have string java.io.IOException: An existing connection was forcibly closed by the remote host 12 All I want is to store "java.io.IOException: An existing connection was forcibly closed... (13 Replies)
Discussion started by: villain41
13 Replies

8. Shell Programming and Scripting

How to print lines till till a pattern is matched in loop

Dear All I have a file like this 112534554 446538656 444695656 225696966 226569744 228787874 113536566 443533535 222564552 115464656 225445345 225533234 I want to cut the file into different parts where the first two columns are '11' . The first two columns will be either... (3 Replies)
Discussion started by: anoopvraj
3 Replies

9. Shell Programming and Scripting

Removing LF and extracting two fields

I need some assistance, I am writing a script in bash. I want to do two things: 1/. I want to replace the LF at the end of the RFH    MQSTR  so I can process the file record by record using a while loop. 2/. I want to extract two fields from each record, they are identified with... (1 Reply)
Discussion started by: gugs
1 Replies

10. Shell Programming and Scripting

Removing certain fields from a file

Hi, I have a file namely 'inputs' The values inside the file are like this and seperated by a '|'. mani|21|CSE How can I extract the values from this file without the '|' symbol. Thanks in advance. (1 Reply)
Discussion started by: sendhilmani123
1 Replies
Login or Register to Ask a Question