Login or Register to Ask a Question and Join Our Community


What is on Your Mind?

VBulletin 3.8 to Discourse on Docker Migration Test Take Two

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
The Lounge What is on Your Mind? VBulletin 3.8 to Discourse on Docker Migration Test Take Two
# 22  
Old 03-13-2020
Update:
  • The first half of migrating "thanks to likes' is done, and I have already created a matching postgres discourse user_actions table in vb mysql to transfer all user likes action to discourse postgres, so can migrate all user "thanks" from vb site to discourse.
  • Later, I will temporarily update discourse posts table and add vb postid column because that info is lost in migration (vbulletin postid != discourse posts id)
  • bbcode to markdown did not appear to work yet. Will need to fix this. This is the major obstacle at the moment. I am trying to avoid rewriting this.
  • I am thinking not to migrate PMs from vb to discourse, and start fresh.

Code:
Closing topics...

Postprocessing posts...
   924394 / 1212997 ( 76.2%)

Basically, getting closer to a great migration setup.
These 2 Users Gave Thanks to Neo For This Post:
RavinderSingh13 sea
# 23  
Old 03-14-2020
Yea!

bbcode-to-md worked.... all except for some attachments in posts.

Still working on this... looking promising

Code:
Closing topics...

Postprocessing posts...
  1212997 / 1212997 (100.0%)  
Creating Permalink File...

Updating topic status

Updating bumped_at on topics

# 24  
Old 03-14-2020
This migration test 2 is finished.

The migration has gone well, and most (not all) of the bbcode was converted to markdown.

There are two issues to fix:
  • The conversion from ICODE tags to markdown added a new line "\n" before and after the markdown tag. This was a copy-and-paste error on my part when I redesigned the migration script and it not easy to fix in the DB.
  • In order to convert the vb plugin "thanks" to likes in discourse, I need to add a new field (vbpostid) to the posts table in discourse; otherwise there is no good way to transfer "thanks" to "likes"

If there was only the ICODE tags I issue, I might leave it be; however I think it is important to transfer the "thanks" to "likes" and so I think I will attempt to fix this without starting from scratch.

Currently, I'm raking all posts to see if I can fix the the bbcode-to-markdown in a few 100K posts which were not cooked properly before I got the bbcode-to-md plugin working.

That process is humming along:

Code:
# rake posts:refresh_oneboxes 
Rebaking post markdown for 'default'
   384936 / 1212998 ( 31.7%)

Then I will check the posts again for bbcode/markdown issues.
  • IF, that goes OK.. I will try to purge the topics (threads) and posts in postgres and rebuild only these two tables (topics, posts) from scratch, adding the extra field / column to the posts table (vbpostid).
  • During that migration, I will fix the issue with ICODE tag migration.

To purge all topics and posts in discourse:

Code:
rake destroy:topics_all_categories

If that works well, we will be lookin' good.

If not, I will start over from scratch.

HOWEVER, I "think" I can do this without starting over from scratch.
# 25  
Old 03-14-2020
Have decided to go with (in progress):

Code:
rake destroy:topics_all_categories

Then add new vbpostid field to posts table in discourse and import / migrate all topics and posts and attachments again.

This will take a long time. At least two days, I am guessing.

I made a full backup of the "almost working discourse app" first, so if this fails we can revert back to the last working instance.
# 26  
Old 03-14-2020
Code:
importing posts...
   651677 / 651677 (100.0%)  [52663 items/min]    
importing attachments...
   268000 / 651677 ( 41.1%)

Some problems important attachments... trying again....


Error:

Code:
        41: from script/import_scripts/vbulletin_neo4.rb:968:in `<main>'
	40: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
	39: from script/import_scripts/vbulletin_neo4.rb:97:in `execute'
	38: from script/import_scripts/vbulletin_neo4.rb:602:in `import_attachments'
	37: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/querying.rb:21:in `find_each'
	36: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:69:in `find_each'
	35: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
	34: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `in_batches'
	33: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `loop'
	32: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:224:in `block in in_batches'
	31: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:250:in `records'
	30: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:615:in `load'
	29: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:797:in `exec_queries'
	28: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:824:in `skip_query_cache_if_necessary'
	27: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:65:in `uncached'
	26: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:398:in `scoping'
	25: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:773:in `_scoping'
	24: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:398:in `block in scoping'
	23: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/delegation.rb:65:in `block in uncached'
	22: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/query_cache.rb:21:in `uncached'
	21: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:79:in `uncached'
	20: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:825:in `block in skip_query_cache_if_necessary'
	19: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:810:in `block in exec_queries'
	18: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/querying.rb:46:in `find_by_sql'
	17: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
	16: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
	15: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:487:in `select'
	14: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
	13: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `execute_and_clear'
	12: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
	11: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:708:in `log'
	10: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
	 9: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:717:in `block in log'
	 8: from /usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
	 7: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `block (2 levels) in log'
	 6: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
	 5: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
	 4: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
	 3: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
	 2: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
	 1: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params': PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
	This probably means the server terminated abnormally
	before or while processing the request.

# 27  
Old 03-14-2020
After running all night, that did not seem to work:

Code:
discourse=# select count(1) from topics;
 count  
--------
 401096
(1 row)

discourse=# select count(1) from posts;
  count  
---------
 1213014
(1 row)

So, lets just clobber them with:

Code:
#su discourse -c 'psql discourse'

Code:
discourse=# delete from posts;

and

Code:
discourse=# delete from threads;

and see what happens...
# 28  
Old 03-14-2020
Done:

Code:
discourse=# delete from posts;
DELETE 1213014
discourse=# delete from topics;
DELETE 401096
discourse=#

LOL... that worked in a few seconds, the "old fashioned way".... no ruby raking and baking.

VBulletin 3.8 to Discourse on Docker Migration Test Take Two-screen-shot-2020-03-15-65553-amjpg


Now, I will reimport them.
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. What is on Your Mind?

Under Consideration: Migrate the Forums to Discourse

Dear All, After being active on the Node-RED forum for the last few weeks, I have been very impressed with Discourse, and my eyes have been opened. https://www.discourse.org/ but not the paid /hosted offering, but using the open distribution: https://github.com/discourse/discourse ... (52 Replies)
Discussion started by: Neo
52 Replies

2. Docker

Docker learning Phase-I

Hello All, I had recently learnt a bit of Docker(which provides containerization process). Here are some of my learning points from it. Let us start first with very basic question: What is Docker: Docker is a platform for sysadmins and developers to DEPLOY, DEVELOP and RUN applications ... (7 Replies)
Discussion started by: RavinderSingh13
7 Replies

3. UNIX for Beginners Questions & Answers

Can't pass a variable representing the output of lsb_release to a docker dontainer

I don't know why, but the rendering of my code mucks up the spacing and indentation, despite being correct in the original file. I'm having issues getting the following script to run (specifically the nested script at the end of the docker command near the end of the script; I think I'm not passing... (2 Replies)
Discussion started by: James Ray
2 Replies

4. Shell Programming and Scripting

Problem in extracting yocto SDK for docker

Actually I was facing the following issue while building my Yocto SDK on Docker container sudo docker build --tag="akash/eclipse-che:6.5.0-1" --tag="akash/eclipse-che:latest" /home/akash/dockerimage.yocto.support/ Sending build context to Docker daemon 26.93MB Step 1/5 : FROM eclipse/cpp_gcc ... (3 Replies)
Discussion started by: Akash BHardwaj
3 Replies

5. AIX

AIX - FC Switch migration, SAN Migration question!

I'm New to AIX / VIOS We're doing a FC switch cutover on an ibm device, connected via SAN. How do I tell if one path to my remote disk is lost? (aix lvm) How do I tell when my link is down on my HBA port? Appreciate your help, very much! (4 Replies)
Discussion started by: BG_JrAdmin
4 Replies

6. Linux

Docker and pipework,ip with other subnet

Recently i found this for give to docker a "personal" ip ip addr del 10.1.1.133/24 dev eth0 ip link add link eth0 dev eth0m type macvlan mode bridge ip link set eth0m up ip addr add 10.1.1.133/24 dev eth0m route add default gw 10.1.1.1On container i did ... (0 Replies)
Discussion started by: Linusolaradm1
0 Replies

7. Web Development

Removing VBSEO for vbulletin – Reverting back to vbulletin URLs

Please note, this information was copied from vbseo.com, now showing a database error. This is posted for reference since vbSEO seems to be going out of business: If you ever need to uninstall vBSEO , you can use the following instructions. Make sure you carefully follow each step. Login... (37 Replies)
Discussion started by: Neo
37 Replies
Login or Register to Ask a Question