What is on Your Mind?

03-15-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Moving along...

Code:

importing users
    62848 / 138160 ( 45.5%)  [181 items/min]

We have two servers, one for MIGRATION TEST 4 (above, current test) and a future site staged and ready for the "final migration" if when the migration and testing is done.

Don't want to get too excited until we see the results of T4, and if the ICODE and BBCODE tags migrate well this go around; and if the vB "Thanks" get staged so we can transfer all vB "Thanks" to "Likes"

It's not a matter of "if" but a matter of "when" at this point, from what I see.

It will be "quite an accomplishment" to migrate a long EOL vB to a modern, sleek, community-forum like Discourse. When I started the eval a few days ago, the very talented folks at Discourse said "you are on your own on this one, pal" (due to the fact we are on vB3) and closed my vB3 bug report with a "good luck" finale. It's too early to say "amazing work"... but it's getting close. It's actually very good at the end of T3, but I am "going for even better".

This User Gave Thanks to Neo For This Post:

Neo

03-15-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Moving right along..... no apparent errors in the modifications included in the Ruby migration script, vbulletin_neo4.rb :

Code:

importing groups..
       20 / 20 (100.0%)  [49738 items/min]    
importing users
   138160 / 138160 (100.0%)  [407 items/min]       ]  
Creating groups membership...
    Done
importing all forums  top level categories...
      119 / 119 (100.0%)  [423 items/min] 
importing topics...
   171065 / 240534 ( 71.1%)

These 2 Users Gave Thanks to Neo For This Post:

Neo

03-16-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Import is going well. My custom code for our ICODE tag and BBCODE to markdown is working fine. Everything is looking good:

... still moving long now importing posts (replies):

Code:

importing topics
   236002 / 240534 ( 98.1%)  [185691 items/min]  
importing posts...
     8597 / 651677 (  1.3%)

Did not import private messages (PMs).

Will attempt to import all attachments, but if they do not import nicely, it not a show stopper; and it will save disk and file space not importing images and files from many years ago;

The person who posts the most images as attachments is me, and I am OK if all of the do not import / migration perfectly.

The most important thing is the code, and all CODE tags and ICODE tags will migration to markdown with issue.

The only "gotcha" so far is the that it appears the vbpostid for importing the "thanks" to "likes" is not working as planned, so I will need to think how to fix this after this test run is done.

Code:

discourse=> select vbpostid from posts order by vbpostid asc limit 10;
 vbpostid 
----------
         
         
         
         
         
         
         
         
         
         
(10 rows)

Code:

discourse=> select vbpostid from posts where vbpostid > 0 order by vbpostid asc limit 10;
 vbpostid 
----------
(0 rows)

Time for some Ruby puts statements in the script to see where the bug is.

Neo

03-16-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

puts shows we are getting

Code:

vbpostid

from the mysql db ok, but it's not writing to the postgres db.

My best guest was that the structure of the posts table for discourse is defined somewhere in a lib I'm was not familiar with yet.

So, I searched and found this directory:

Code:

/var/www/discourse/script/import_scripts/base

and this script:

Code:

generic_database.rb

and added vbpostid to this code:

VBulletin 3.8 to Discourse on Docker Migration Test Take Four-screen-shot-2020-03-16-65404-pmjpg

But still no joy....

Neo

03-16-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

I have a solution to this issue, or at least a workaround:

EDIT: This method only works on posts with no bbcode because discourse preprocesses the raw posts and changes the post. Hence, the MD4 (or matching methods in general, do not work).

Code:

raw = preprocess_post_raw()

On discourse (after migrating):

Code:

discourse=> select md5(raw) from posts where raw like '%where can i get a free copy of unix? any kind... plz send %';
               md5                
----------------------------------
 e136704539982b094d8d1d4229884583
(1 row)

On vb forums:

Code:

mysql> select md5(pagetext) from post where pagetext like '%where can i get a free copy of unix? any kind... plz send %';
+----------------------------------+
| md5(pagetext)                    |
+----------------------------------+
| e136704539982b094d8d1d4229884583 |
+----------------------------------+
1 row in set (1.07 sec)

So, the MD5() function in post mysql and postgres returns the same hash on the same post in two different DBs (it's should, but always good to check).

This means that we can match the posts in both DBs without unique post ids using MD5() and the user name who created the post.

The

Code:

MD5

hash combined with the user name should does not work because the post is preprocessed for bbcode and markdown.

Neo

03-16-2020

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Hmmmm does not work consistently.... must be converting some chars in the pagetext of a vb post to postgres.

Update: Yes, the posts are preprocessed before migration for markdown and bbcode, so MD5 will not work.

Code:

discourse=> select id,md5(raw) from posts where raw like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
   id   |               md5                
--------+----------------------------------
 377402 | 04d0a536ccd72d8523309b0a4c5da55d
(1 row)

Code:

mysql> select postid,md5(pagetext) from post where pagetext like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
+-----------+----------------------------------+
| postid    | md5(pagetext)                    |
+-----------+----------------------------------+
| 303045132 | 61f0d1958e9002b69872bebb88a499b9 |
+-----------+----------------------------------+
1 row in set (0.99 sec)

Darn it.

Neo