Shell Programming and Scripting

Don,

You are correct, it should have been 2,$b.

The idea was to remove the "From - " line if it was on the first line, and then remove the ">" from the line that contains >From in the header.

And yes, this will modify every file, but correctly formed files will not be changed.

Comparing above mail files to what I have on my systems, the first is an "mbox" type mail, while the second seems to be or already have been converted to "thunderbird" format.
Did you try to make thunderbird read them directly?

Don and Derek,

Thanks so much for the assistance. I really appreciate the fact that your help will allow me to change mail clients again, (which is why I'm doing the conversion to mbox finally), and not lose the last couple of years of emails.

For an initial quick response, the files are email messages stored individually as separate files in a maildir hierarchy. While they are appended with the extension .eml usually in a Microsoft windows environment, in my hierarchy they're stored as files with filenames that are a string of numbers with no extension.

When I edited the first find commands that Derek suggested to search for all files, (removed the EeMmLl), it did indeed locate, and then later count, all affected files. Additionally while I originally thought I should remove the > character from the actual From header line, I found out that the conversion program works whether the > character is there or not. The important part is the removal of the first From line, (mbox postmark), when it exists. I appreciate your suggestion though of removing the > character from the actual From line, and will look to use that as well.

Finally, Rudi, the example headers I posted are pulled from two separate EML files in my hierarchy. They're not in an mbox file, (which is basically a single file with contains a number of email messages).

Thanks all,

Jason

---------- Post updated at 02:08 PM ---------- Previous update was at 08:58 AM ----------

Thanks again Don and Derek. I ended up using Don's shell script, and editing the find filename parameters to;

Code:

#!/bin/bash
bfc=0
find . -type f -name '*' | while read -r file
do      read -r f1 f2 rest < "$file"
        if [ "$f1" = "From" ] && [ "$f2" = "-" ]
        then    # Bad file found...
                bfc=$((bfc + 1))
                printf 'bad file #%d: %s\n\tFrom - %s\n' $bfc "$file" "$rest"
                ed -s "$file" <<-EOF
                        1d
                        w
                        q
                EOF
        fi
done

I ran it from the top level maildir subdirectory, and it worked perfectly, finding every file with an mbox postmark in my maildir hierarchy, displaying the filename, counting the file, and then deleting the mbox postmark from the affected file.

Total 'bad' eml files discovered, and cleaned of mbox postmark; 14,473

Thanks very much to both of you!

Jason

Jason,
I'm very glad you were able to get my script to work for you. For future reference note that:

Code:

find file... -type f -name '*'

will produce exactly the same output as:

Code:

find file... -type f

since the -name pattern * will match every file.

Don,

One last thing. Based on Derek's suggestion and his find one liners, is it possible to add to your shell script a function to replace all lines that begin with

Code:

>From

with

Code:

From

I'm not sure that is having an effect on this conversion, but I think it would be useful to clean things up from that perspective.

Thanks,

Jason

In traditional mbox format mail files, mail body lines starting with From are converted by software submitting messages to the mailer daemons to >From to keep it from being misinterpreted as a message boundary. Assuming that the lines you really want to convert are headers that start with >From and contain an email address (i.e. something containing an @ as in user@company.com), the following, in addition to what it did before, will remove the > from the start of a line that starts with >From and contains an @:

Code:

		ed -s "$file" <<-EOF
			1d
			g/^>From .*@/s/^>//
			w
			q
		EOF

If you really want to remove the > from every line containing >From , change the global substitute line above to:

Code:

			g/^>From /s//From /