What is on Your Mind?

We "completed" the migration of this vB3 site to Discourse a number of days ago. However, deeper testing by @Scrutinizer and @MadeInGermany revealed that a lot of text was mangled in the migration. We traced these bugs to two issues:

These are serious migration bugs which affect the integrity of all of the hard work our member have contributed over the years and must be corrected.

I posted a bug report over at meta-discourse on the major bug in the Ruby gem ruby-bbcode-to-md but the maintainer of that repo shut me down, deleted my bug reports, washed his hands of a repo with his name on it, and acted very unprofessional, even though the bug was easily confirmed. This was not encouraging and brought my spirits down a bit at the time.

@Scrutinizer then came to the rescue (thank you!), took a step back, and did a formal analysis of the Ruby preprocessing script and the various bbcode transformations including the:

At the same time, I was working on:

@Scrutinizer created a spreadsheet and did the analysis and determined that the mangler gem ruby-bbcode-to-md was not required.

In addition, @Scrutinizer suggested we install the discourse-bbcode plugin and test it.

With good preliminary results from discourse-bbcode, I started to learn how to modify a Discourse plugin and found that the discourse-bbcode plugin was not difficult to modify (straight forward javascript), so I set up a development environment on my desktop as follows:

From this test setup, was able to add new "preliminary" bbcode tags, for example I created two new tags which correspond (roughly) to two of our legacy tags:

This forked repo is still a work-in-progress and I am still learning to modify and test and try to add other tags. Right now, I'm having some issues with preformatted elements, but that's a discussion for another day.

GitHub - unixneo/neo-discourse-bbcode: vBulletin BBCode plugin

Although we have made progress, @Scrutinzer also discovered that the Markdown and CODE bbcode tags used by Discourse (builtin) do not permit BBCode in the fenced code blocks.

This means that our technique of using color to highlight sections of code in blocks when helping others currently has no solution, but we are working on this. Currently, in the migration script this bbcode is stripped out during migration. However, we want to find a way to preserve this capability and feature if possible.

However, there are gremlins around the corner:

Fixing this will more-than-likely require us to disable the Discourse builtin CODE tag (so far, I have not been about to override this in a plugin); and there is just about zero chance the busy folks at meta discourse will support this or even answer my question in a helpful way if I ask how to do this (disable or override their builtin code tag). Historical discussions from the meta discourse team shows a near religious passion for markdown and any deviation from that they perceive as "right and wrong" is not well received. Plus, this is a migration issue, and they are focused on the future, not the past (understandably). I see zero chance asking about this will be well received over there.

So the current options seem to be:

I don't consider removing color indicators from our code blocks and stripping out bbcode from these fenced code blocks a major issue; but there are some who will consider it a big deal, possibly. We will be "losing" this well-liked feature if we strip it out.

So, we are still looking into this.

However, I am not planning to ask on meta, because that question will surely, based on historical discussions on code tags there and my "not good" experiences with two migration bug reports over there, will not turn out good, this I am sure. Migration issues are not well received over there (they are working on building for the future, and this is understandable, honestly) and we are pretty much "own our own" on this.

That is the latest

Today, armed with a new migration script vbulletin_neo7.rb I will start a migration from scratch on the staging server for the purposes of getting raw preprocessed posts from the postgres DB and uploading them to the community site; but first I will do this on the staging site, as follow:

Yesterday, I tested the migration script vbulletin_neo7.rb and it worked OK

@Scrutinizer is also working on some other bbcode migration enhancements which we will apply when he is ready to test (he has a full time job and family demands so no hurry or worry).

I want to create a new baseline without the broken ruby-bbcode-to-md plugin which our new vbulletin_neo7.rb script and new custom bbcode plugin, neo-discourse-bbcode which has a solid new ICODE bbcode tag working.

Update:

Change in direction (too slow to keep doing the migration over and over).

Create / write a Ruby script (done):

This script above processes about a million posts in 45 minutes (much faster) and when happy with the results can rebake the raw posts into the cooked posts. Rebaking 1M posts takes about 16+ hours, so avoiding this when possible.

Ran this yesterday and found that all the bugs posted my @MadeInGermany before (mangled code, missing left square brackets) and the hard line break error reported by @Scrutinzer (where \n in code fragments were converted to hard line breaks) were fixed.

However, still more gremlins to slay, working on:

However, the main reported gremlins in code fragments appear to be fixed. Now working on other missing transformations (missing emoji, images, etc).

Making progress... slowly but surely.

All work currently done on test / staging server only.

Also, I am finding, by trial and error, that even before preprocessing with the Ruby migration scripts, there are some transformations which are easily done on the copy of the vB3 MySQL DB dump; for example:

Code:

UPDATE post SET  pagetext= LOWER(REGEXP_REPLACE(pagetext,'\\[ATTACH\\](.*)\\[\\/ATTACH\\]', '[IMG]https://www.unix.com/attachment.php?attachmentid=\\1[/IMG]'));

There was some bug in the Ruby attachment preprocessing and some attachment ids were lost, so instead of wasting time trying to find the bug in the Ruby preprocessing routine, it was easier to do the regex search and replace in the staged copy of the DB dump.

All these attachment images are automatically downloaded to the new discourse forum over time as well.

Note: Not being an expert in MariaDB REGEX_REPLACE, started with (.*) and worked my way up, and the matches only worked with the double backslashes (escapes) , would not match with single backslashes. None of the examples on the net worked (most showed no backslashes or one backslash only) in this REGEX_REPLACE expression; but I could get it to work, building it from .* up, step-by-step.

Update: This change is done and confirmed working on the staging server.

According to a quick check 35 tables (posts with tables) has been stripped or skipped from the migration:

Code:

MariaDB [vb3]> select count(postid) from post where pagetext like '%[table="head"]%';
+---------------+
| count(postid) |
+---------------+
|            35 |
+---------------+
1 row in set (5.559 sec)

MariaDB [vb3]> exit
Bye
root@discourse1-app:/shared/neo/bin# /shared/neo/bin/pg
psql (10.12 (Debian 10.12-1.pgdg100+1))
Type "help" for help.

discourse=> select count(id) from posts where raw like '%[table=�head��]%';
 count 
-------
     0
(1 row)

We can easily write code to convert these tables to markdown; but since the posts were stripped from the migration, and this would require a totally fresh migration from the start, I am inclined, as this time, to just drop these 35 posts from the new site.

We could add them at a later time, manually, as doing this manually for 35 posts would take a few hours, but redoing the migration will take a week and be painful. It would take a few hours to write and test the script, to make sure it works anyway.

So, my inclination to not worry about losing 35 posts with TABLE tags at this time (and manually add them back in the future).

Update (from one hour ago):


Updated the new community sites with latest posts, new users, likes, etc. from legacy site.

Ran an early preprocessing script against the legacy DB.

@Scrutinizer is testing a more refined version of preprocessing which will do even more migration magic. When he is ready, we will run his preprocessing script again the legacy DB and see how it looks.

Thanks for your patience.

We have already fixed the two bugs that we found in the initial launch; but are working to refine more custom bbcode issues.

Well, as a update, this earlier MariaDB REGEX was flawed, my bad.

Code:

UPDATE post SET  pagetext= LOWER(REGEXP_REPLACE(pagetext,'\\[ATTACH\\](.*)\\[\\/ATTACH\\]', '[IMG]https://www.unix.com/attachment.php?attachmentid=\\1[/IMG]'));

Should have been

Code:

UPDATE post SET  pagetext= REGEXP_REPLACE(pagetext,'\\[ATTACH\\](.*?)\\[\\/ATTACH\\]', '[IMG]https://www.unix.com/attachment.php?attachmentid=\\1[/IMG]');

Using the LOWER directive not only worked on the REGEX, but on all text in the post, moving all text to lower case (unexpectedly). That will be fixed after 12 hours of rebaking the 1M posts.

In addition, my original REGEX was too greedy and I had to add the ? to make it less greedy. Somehow, I missed that during initial testing.

Thanks to @Peasant for catching that bug quickly today.

We are "mind warped" writing and rewriting all this (new to us) Ruby migration code. The extra eyes, fresh perspectives, and bug hunting all are much appreciated and valuable contributions to the final migration success (whenever that happens, LOL)