Shell Programming and Scripting

Hi! I have been struggling with a large file that has stray end of line characters.

I am working on a Mac (Lion). I mention this only because I have been mucking around with fixing my problem using sed, and I have learned far more than I wanted to know about Unix and Mac eol characters.

I can identify easily the number of tabs in each line:

awk '{print gsub(/\t/,"")}' infile > output.txt

BUT I want to selectively process the file. If the number of tabs on a line is 69, this is a legal line.

If it is less than 69, I want to remove the end of line character on that line, take the next line, append it to the end of the first line.

===
At the risk of looking stupid, but perhaps explaining the problem a bit more, I was reading this forum and was able to almost fix the file. In almost every record, a "good" line has a tab preceding the eol character. By brute force, the script below almost solves my problem:

1) changes all Unix eol to Mac eol
2) uses sed to change "tab+eol" to a string
3) uses sed to change remaining "eol" to a different string
4) reverses step 2
5) reverses step 1

I was pleased that I figured this out, but the awk command at the end made me realize that there were in fact a very small number (a few hundred in a million line file) that did not fit the pattern; they were "good" lines and had fields all to the final field. This means I am back to square one, sort of. If I could figure out the question I posed at the top, I could skip this brute force method. If I am stuck with below, I can still manually fix the remaining stray lines.

Code:

LC_CTYPE=C tr -d "\n" < test.txt > test2.txt
gsed -e 's/^I^M/#####ABCDE/g' test2.txt > test3.txt
gsed -e 's/^M/ ABCDE##### /g' test3.txt > test4.txt
gsed -e 's/#####ABCDE/^I^M/g' test4.txt > test5.txt
LC_CTYPE=C tr "\r" "\n" <test5.txt > test6.txt
awk '{print gsub(/\t/,"")}' test6.txt > test6tabs.txt

Not sure I understand your problem correctly. If you define FS, the input field separator for awk, to be TAB, NF == 70will indicate a correct line, no matter what the eol char is, and $NF=="" will indicate that "tab+eol" sequence.
So, whenever NF != 70, add the next line to your current line, e.g. by the getline function, and your requirement should be fulfilled.