Sponsored Content
Top Forums Shell Programming and Scripting Replace CRLF between pipe (|) delimiter with LF Post 303009328 by hishamzz on Tuesday 12th of December 2017 03:26:51 PM
Old 12-12-2017
Quote:
Originally Posted by RudiC
That's one single line, obviously. And, obviously, as anticipated, we're talking of <CR><LF> combinations. How do you tell one line from another? Do they all have the same field count? Do they all have the same <CR> count?
Yes Rudi. It is a single sample record.

The field count should be consistent. As in, if there are 7 fields there ought to be 6 pipes in the data and that is how you group a record as one.

For example :-

Code:
2016-11-30|2016-2017|2016-11-30|123|123.xlsm|Production|No|||AHB||Center||No|||2222|Unit Important|< $20 Million|Quarterly||||2011-02-24|||Some descripto|





Mandatory Fiel|xlsm|Hisham
2016-11-30|2016-2017|2016-11-30|3123|123.xlsm|Production|No|||AHB||Center||No|||2222|Unit Important|< $20 Million|Quarterly||||2011-02-24|||Some descripto|





Mandatory Fiel|xlsm|Hisham

Each record has 30 fields (29 pipes). So an entire record should contain 29 pipes for it be considered one single record.

Which is where the question comes. How can we remove the <CRLF> alone between the pipes.

Zz
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Cutting a portion of a line seperated by pipe delimiter

Hi, In the below line a|b|10065353|tefe|rhraqs|135364|5347575 dgd|rg|4333|fhra|grhrt|46423|urdsgd Here i want to cut the characters in between the second and third pipe delimiter and then between fifth and sixth delimiter and retain the rest of the line. My output should be ... (3 Replies)
Discussion started by: ragavhere
3 Replies

2. Shell Programming and Scripting

Converting hex value 7C (for pipe) to CRLF in Unix

I am trying to convert a txt file that includes one long string of data. The lines are separated with hex value 7C (for pipe). I am trying to process this file using SQR (Peoplesoft) so I thought the easiest thing to do would be to replace the eol char with a CRLF in unix so I can just... (4 Replies)
Discussion started by: sfedak
4 Replies

3. UNIX for Dummies Questions & Answers

replacing space with pipe(delimiter)

Hello All, I have a file with thousands of records: eg: |000222|123456987|||||||AARONSON| JOHN P|||PRIMARY |P |000111|567894521|||||||ATHENS| WILLIAM k|||AAAA|L Expected: |000222|123456987|||||||AARONSON| JOHN |P|||PRIMARY |P |000111|567894521|||||||ATHENS| WILLIAM |k|||AAAA|L I... (6 Replies)
Discussion started by: OSD
6 Replies

4. Shell Programming and Scripting

Replace pipe with Broken Pipe

Hi All , Is there any way to replace the pipe ( | ) with the broken pipe (0xA6) in unix (1 Reply)
Discussion started by: saj
1 Replies

5. UNIX for Dummies Questions & Answers

[Solved] How to swap PIPE seperator delimiter?

I have file like below 1|4|OR|OLAP|INT|INT||CONSTANT|2012/08/07|9999/12/31|0|0|0|0|PRL|-358.1684563||||||||||36522|55791|LNR| 2|4|OR|OLAP|CLR|CLR||CONSTANT|2012/09/07|9999/12/31|0|0|0|0|PRL|-358.1684563||||||||||36522|57891|REGS|... (2 Replies)
Discussion started by: gkskumar
2 Replies

6. Red Hat

Converting fixed width file to pipe delimiter in Linux(red-hat)

Hi, I am facing a typical scenario for AWK command . In HP- UNIX is behave as expected but in red hat linux same awk code is not give the same result. The below code is for convert the fixed width file to pipe delimiter file in HP-unix server. awk code: #!/bin/awk -f NR!=1... (11 Replies)
Discussion started by: brij_abhi
11 Replies

7. Shell Programming and Scripting

Removing duplicate lines on first column based with pipe delimiter

Hi, I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines Command : sort -t'|' -nuk1 file.txt Input : 38376KZ|09/25/15|1.057 38376KZ|09/25/15|1.057 02006YB|09/25/15|0.859 12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies

8. Shell Programming and Scripting

Problem in using cut command with pipe as a delimiter while using in a script

There is a text file in my project named as "mom.txt" in which i want to have contents like.................. LSCRM(Application Name): 1: This is my first application. 2: Today we did shell scripting automation for this app. 3: It was really a good fun in doing so. 4: Really good.| (Here i... (7 Replies)
Discussion started by: Abhijeet Anand
7 Replies

9. UNIX for Beginners Questions & Answers

Views How to replace a CRLF char from a variable length file in the middle of a string in UNIX?

My sample file is variable length, with out any field delimiters. It has min of 18 chars length and the 'CRLF' is potentially between 12-14 chars. How do I replace this with a space? I still want to keep end of record, but just want to remove these new lines chars in the middle of the data. ... (7 Replies)
Discussion started by: chandrath
7 Replies

10. Shell Programming and Scripting

Replace delimiter for a particular column in a pipe delimited file

I have an input file as below Emp1|FirstName|MiddleName|LastName|Address|Pincode|PhoneNumber 1234|FirstName1|MiddleName2|LastName3| Add1 || ADD2|123|000000000 Output : 1234|FirstName1|MiddleName2|LastName3| Add1 ,, ADD2|123|000000000 OR 1234,FirstName1,MiddleName2,LastName3, Add1 ||... (2 Replies)
Discussion started by: styris
2 Replies
Fields(3pm)						User Contributed Perl Documentation					       Fields(3pm)

NAME
Sort::Fields - Sort lines containing delimited fields SYNOPSIS
use Sort::Fields; @sorted = fieldsort [3, '2n'], @lines; @sorted = fieldsort '+', [-1, -3, 0], @lines; $sort_3_2n = make_fieldsort [3, '2n'], @lines; @sorted = $sort_3_2n->(@lines); DESCRIPTION
Sort::Fields provides a general purpose technique for efficiently sorting lists of lines that contain data separated into fields. Sort::Fields automatically imports two subroutines, "fieldsort" and "make_fieldsort", and two variants, "stable_fieldsort" and "make_sta- ble_fieldsort". "make_fieldsort" generates a sorting subroutine and returns a reference to it. "fieldsort" is a wrapper for the "make_fieldsort" subroutine. The first argument to make_fieldsort is a delimiter string, which is used as a regular expression argument for a "split" operator. The delimiter string is optional. If it is not supplied, make_fieldsort splits each line using "/s+/". The second argument is an array reference containing one or more field specifiers. The specifiers indicate what fields in the strings will be used to sort the data. The specifier "1" indicates the first field, "2" indicates the second, and so on. A negative specifier like "-2" means to sort on the second field in reverse (descending) order. To indicate a numeric rather than alphabetic comparison, append "n" to the specifier. A specifier of "0" means the entire string ("-0" means the entire string, in reverse order). The order in which the specifiers appear is the order in which they will be used to sort the data. The primary key is first, the secondary key is second, and so on. "fieldsort [1, 2], @data" is roughly equivalent to "make_fieldsort([1, 2])->(@data)". Avoid calling fieldsort repeatedly with the same sort specifiers. If you need to use a particular sort more than once, it is more efficient to call "make_fieldsort" once and reuse the subroutine it returns. "stable_fieldsort" and "make_stable_fieldsort" are like their "unstable" counterparts, except that the items that compare the same are maintained in their original order. EXAMPLES
Some sample data (in array @data): 123 asd 1.22 asdd 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 23 erww 4.21 ewet 91 fdgs 3.43 ewet 123 refs 3.22 asdd 123 refs 4.32 asdd # alpha sort on column 1 print fieldsort [1], @data; 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 23 erww 4.21 ewet 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # numeric sort on column 1 print fieldsort ['1n'], @data; 23 erww 4.21 ewet 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd # reverse numeric sort on column 1 print fieldsort ['-1n'], @data; 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 91 fdgs 3.43 ewet 51 erwt 34.2 ewet 43 rewq 2.12 ewet 32 ewq 2.32 asdd 23 erww 4.21 ewet # alpha sort on column 2, then alpha on entire line print fieldsort [2, 0], @data; 123 asd 1.22 asdd 51 erwt 34.2 ewet 23 erww 4.21 ewet 32 ewq 2.32 asdd 91 fdgs 3.43 ewet 123 refs 3.22 asdd 123 refs 4.32 asdd 43 rewq 2.12 ewet # alpha sort on column 4, then numeric on column 1, then reverse # numeric on column 3 print fieldsort [4, '1n', '-3n'], @data; 32 ewq 2.32 asdd 123 refs 4.32 asdd 123 refs 3.22 asdd 123 asd 1.22 asdd 23 erww 4.21 ewet 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # now, splitting on either literal period or whitespace # sort numeric on column 4 (fractional part of decimals) then # numeric on column 3 (whole part of decimals) print fieldsort '(?:.|s+)', ['4n', '3n'], @data; 51 erwt 34.2 ewet 43 rewq 2.12 ewet 23 erww 4.21 ewet 123 asd 1.22 asdd 123 refs 3.22 asdd 32 ewq 2.32 asdd 123 refs 4.32 asdd 91 fdgs 3.43 ewet # alpha sort on column 4, then numeric on the entire line # NOTE: produces warnings under -w print fieldsort [4, '0n'], @data; 32 ewq 2.32 asdd 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 23 erww 4.21 ewet 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # stable alpha sort on column 4 (maintains original relative order # among items that compare the same) print stable_fieldsort [4], @data; 123 asd 1.22 asdd 32 ewq 2.32 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 23 erww 4.21 ewet 91 fdgs 3.43 ewet BUGS
Some rudimentary tests now. Perhaps something should be done to catch things like: fieldsort '.', [1, 2], @lines; '.' translates to "split /./" -- probably not what you want. Passing blank lines and/or lines containing the wrong kind of data (alphas instead of numbers) can result in copious warning messages under "-w". If the regexp contains memory parentheses ("(...)" rather than "(?:...)"), split will function in "delimiter retention" mode, capturing the contents of the parentheses as well as the stuff between the delimiters. I could imagine how this could be useful, but on the other hand I could also imagine how it could be confusing if encountered unexpectedly. Caveat sortor. Not really a bug, but if you are planning to sort a large text file, consider using sort(1). Unless, of course, your operating system doesn't have sort(1). AUTHOR
Joseph N. Hall, joseph@5sigma.com SEE ALSO
perl(1). perl v5.8.8 2008-03-25 Fields(3pm)
All times are GMT -4. The time now is 10:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy