VBulletin 3.8 to Discourse on Docker Migration Test Take Four


 
Thread Tools Search this Thread
The Lounge What is on Your Mind? VBulletin 3.8 to Discourse on Docker Migration Test Take Four
# 1  
Old 03-15-2020
VBulletin 3.8 to Discourse on Docker Migration Test Take Four

Test Build 4 on New Server, with changes identified in discourse test builds 2 and 3, primarily:
  • Insuring ruby-bbcode-to-markdown is enabled.
  • Removing line breaks from ICODE to markdown in migration script.
  • Added vbpostid to posts in discourse to setup migrating vb "thanks" to discourse "likes" (not sure if this part will work or not, first time test for this migration feature.)

Chuggin' along again, from total scratch on a new server (and with fingers crossed):

Code:
root@discourse1-app:/var/www/discourse# su discourse -c 'bundle exec ruby script/import_scripts/vbulletin_neo4.rb'
discourse:@localhost wants vb3
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...

importing groups..
       20 / 20 (100.0%)  [173261 items/min]   
importing users
     4774 / 138160 (  3.5%)  [182 items/min]

Looks like the "user part" of the migration will take about 12 hours Smilie

Entry in discourse posts table, to set up "thanks" to "likes" migration:

Code:
 vbpostid                | integer                     |           |          |

Let's see if this new vbulletin_neo4.rb code works on the first attempt (if it does, that would be a pleasant surprise).

Note: Will not be migrating private messages (PMs) from vb to discourse in this test run and probably not in the final migration either.
# 2  
Old 03-15-2020
Moving along...

Code:
importing users
    62848 / 138160 ( 45.5%)  [181 items/min]

We have two servers, one for MIGRATION TEST 4 (above, current test) and a future site staged and ready for the "final migration" if when the migration and testing is done.

Don't want to get too excited until we see the results of T4, and if the ICODE and BBCODE tags migrate well this go around; and if the vB "Thanks" get staged so we can transfer all vB "Thanks" to "Likes"

It's not a matter of "if" but a matter of "when" at this point, from what I see.

It will be "quite an accomplishment" to migrate a long EOL vB to a modern, sleek, community-forum like Discourse. When I started the eval a few days ago, the very talented folks at Discourse said "you are on your own on this one, pal" (due to the fact we are on vB3) and closed my vB3 bug report with a "good luck" finale. It's too early to say "amazing work"... but it's getting close. It's actually very good at the end of T3, but I am "going for even better".
This User Gave Thanks to Neo For This Post:
# 3  
Old 03-15-2020
Moving right along..... no apparent errors in the modifications included in the Ruby migration script, vbulletin_neo4.rb :

Code:
importing groups..
       20 / 20 (100.0%)  [49738 items/min]    
importing users
   138160 / 138160 (100.0%)  [407 items/min]       ]  
Creating groups membership...
    Done
importing all forums  top level categories...
      119 / 119 (100.0%)  [423 items/min] 
importing topics...
   171065 / 240534 ( 71.1%)

These 2 Users Gave Thanks to Neo For This Post:
# 4  
Old 03-16-2020
Import is going well. My custom code for our ICODE tag and BBCODE to markdown is working fine. Everything is looking good:


... still moving long now importing posts (replies):

Code:
importing topics
   236002 / 240534 ( 98.1%)  [185691 items/min]  
importing posts...
     8597 / 651677 (  1.3%)

Did not import private messages (PMs).

Will attempt to import all attachments, but if they do not import nicely, it not a show stopper; and it will save disk and file space not importing images and files from many years ago;

The person who posts the most images as attachments is me, and I am OK if all of the do not import / migration perfectly.

The most important thing is the code, and all CODE tags and ICODE tags will migration to markdown with issue.

The only "gotcha" so far is the that it appears the vbpostid for importing the "thanks" to "likes" is not working as planned, so I will need to think how to fix this after this test run is done.

Code:
discourse=> select vbpostid from posts order by vbpostid asc limit 10;
 vbpostid 
----------
         
         
         
         
         
         
         
         
         
         
(10 rows)


Code:
discourse=> select vbpostid from posts where vbpostid > 0 order by vbpostid asc limit 10;
 vbpostid 
----------
(0 rows)

Time for some Ruby puts statements in the script to see where the bug is.
# 5  
Old 03-16-2020
puts shows we are getting
Code:
vbpostid

from the mysql db ok, but it's not writing to the postgres db.

My best guest was that the structure of the posts table for discourse is defined somewhere in a lib I'm was not familiar with yet.

So, I searched and found this directory:

Code:
/var/www/discourse/script/import_scripts/base

and this script:

Code:
generic_database.rb

and added vbpostid to this code:

VBulletin 3.8 to Discourse on Docker Migration Test Take Four-screen-shot-2020-03-16-65404-pmjpg


But still no joy....
# 6  
Old 03-16-2020
I have a solution to this issue, or at least a workaround:

EDIT: This method only works on posts with no bbcode because discourse preprocesses the raw posts and changes the post. Hence, the MD4 (or matching methods in general, do not work).

Code:
raw = preprocess_post_raw()

On discourse (after migrating):

Code:
discourse=> select md5(raw) from posts where raw like '%where can i get a free copy of unix? any kind... plz send %';
               md5                
----------------------------------
 e136704539982b094d8d1d4229884583
(1 row)

On vb forums:

Code:
mysql> select md5(pagetext) from post where pagetext like '%where can i get a free copy of unix? any kind... plz send %';
+----------------------------------+
| md5(pagetext)                    |
+----------------------------------+
| e136704539982b094d8d1d4229884583 |
+----------------------------------+
1 row in set (1.07 sec)

So, the MD5() function in post mysql and postgres returns the same hash on the same post in two different DBs (it's should, but always good to check).

This means that we can match the posts in both DBs without unique post ids using MD5() and the user name who created the post.

The
Code:
MD5

hash combined with the user name should does not work because the post is preprocessed for bbcode and markdown.
# 7  
Old 03-16-2020
Hmmmm does not work consistently.... must be converting some chars in the pagetext of a vb post to postgres.

Update: Yes, the posts are preprocessed before migration for markdown and bbcode, so MD5 will not work.


Code:
discourse=> select id,md5(raw) from posts where raw like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
   id   |               md5                
--------+----------------------------------
 377402 | 04d0a536ccd72d8523309b0a4c5da55d
(1 row)

Code:
mysql> select postid,md5(pagetext) from post where pagetext like '%Is it possible to see the mails received by him (mail sent to him) in outlook on my Windows workstation?%';
+-----------+----------------------------------+
| postid    | md5(pagetext)                    |
+-----------+----------------------------------+
| 303045132 | 61f0d1958e9002b69872bebb88a499b9 |
+-----------+----------------------------------+
1 row in set (0.99 sec)

Darn it.
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. What is on Your Mind?

Under Consideration: Migrate the Forums to Discourse

Dear All, After being active on the Node-RED forum for the last few weeks, I have been very impressed with Discourse, and my eyes have been opened. https://www.discourse.org/ but not the paid /hosted offering, but using the open distribution: https://github.com/discourse/discourse ... (52 Replies)
Discussion started by: Neo
52 Replies

2. What is on Your Mind?

VBulletin 3.8 to Discourse on Docker Migration Test Take Two

OK. Like we all do, we learn a lot from tests, test migrations, and so forth. Today, I started from scratch on test migration 2, armed with a lot more knowledge, The main differences are as follows: Installed discourse plugin ruby-bbcode-to-md before starting the install Modified... (30 Replies)
Discussion started by: Neo
30 Replies

3. Docker

Docker learning Phase-I

Hello All, I had recently learnt a bit of Docker(which provides containerization process). Here are some of my learning points from it. Let us start first with very basic question: What is Docker: Docker is a platform for sysadmins and developers to DEPLOY, DEVELOP and RUN applications ... (7 Replies)
Discussion started by: RavinderSingh13
7 Replies

4. Shell Programming and Scripting

Problem in extracting yocto SDK for docker

Actually I was facing the following issue while building my Yocto SDK on Docker container sudo docker build --tag="akash/eclipse-che:6.5.0-1" --tag="akash/eclipse-che:latest" /home/akash/dockerimage.yocto.support/ Sending build context to Docker daemon 26.93MB Step 1/5 : FROM eclipse/cpp_gcc ... (3 Replies)
Discussion started by: Akash BHardwaj
3 Replies

5. AIX

AIX - FC Switch migration, SAN Migration question!

I'm New to AIX / VIOS We're doing a FC switch cutover on an ibm device, connected via SAN. How do I tell if one path to my remote disk is lost? (aix lvm) How do I tell when my link is down on my HBA port? Appreciate your help, very much! (4 Replies)
Discussion started by: BG_JrAdmin
4 Replies

6. Linux

Docker and pipework,ip with other subnet

Recently i found this for give to docker a "personal" ip ip addr del 10.1.1.133/24 dev eth0 ip link add link eth0 dev eth0m type macvlan mode bridge ip link set eth0m up ip addr add 10.1.1.133/24 dev eth0m route add default gw 10.1.1.1On container i did ... (0 Replies)
Discussion started by: Linusolaradm1
0 Replies

7. Web Development

Removing VBSEO for vbulletin – Reverting back to vbulletin URLs

Please note, this information was copied from vbseo.com, now showing a database error. This is posted for reference since vbSEO seems to be going out of business: If you ever need to uninstall vBSEO , you can use the following instructions. Make sure you carefully follow each step. Login... (37 Replies)
Discussion started by: Neo
37 Replies
Login or Register to Ask a Question