Under Consideration: Migrate the Forums to Discourse


 
Thread Tools Search this Thread
The Lounge What is on Your Mind? Under Consideration: Migrate the Forums to Discourse
# 8  
Old 03-09-2020
OK... after seeing the results of about 40% of the main topics / threads (before the replies / posts are uploaded):

Code:
importing topics...
    95440 / 240477 ( 39.7%)  [414 items/min

I can see two problems:
  • The CODE and ICODE tags seems to not be ported into discourse.
  • The "not parent" categories of forums (child forums) are not getting moved into the new DB and case an "error".

So, current way ahead is looking like:
  • Continue to let the migration process run until finished.
  • Study the results and at least do the following:

  1. Move avatars to the DUMPED DB before DUMPING, so the AVATARS will port (maybe).
  2. DUMP the Forum DB again.
  3. Do a search and replace in the DB (DUMPED VERSION) and change all CODE tags to ``` (markdown three back ticks).
  4. Ditto for the ICODE tags, convert the all to ``` for markdown ; but I am not sure 100% this will work, but we can try.
  5. Change all forums to "parent, top level forums" in the DUMPED DB, so we I import again, they will work (not be children to parents).

Something like that.....

Looks like the CODE tags are just getting stripped out of the migration. Need to confirm when we get back up to 2019 / 2020 time frame. Right now the migrations is in the 2009 time frame.

This is what is looks like so far..... take lessons learned from this migration, make adjustments, and do it again.
These 2 Users Gave Thanks to Neo For This Post:
# 9  
Old 03-09-2020
Nice to watch and read the progress.

10 hours is a hell of a roundtrip time. Maybe 1000 or 10000 posts are enough per try ;-).
# 10  
Old 03-09-2020
Update:

Looks like the CODE tags are migrating OK, better than I thought looking at the earlier results... that is a big (and very good) surprise. I am however seeing other BBCODE tags not working (some custom and some basic) but at least they are being transferred and not stripped out, so we can deal with this easy enough later. This is very encouraging.

During topics / threads migration, there were a few errors

Code:
   201290 / 240477 ( 83.7%)  [412 items/min]  Exception while creating post thread-243715. Skipping......

Code:
   201308 / 240477 ( 83.7%)  [411 items/min]  Exception while creating post thread-243734. Skipping. .............

Code:
   201335 / 240477 ( 83.7%)  [411 items/min]  Exception while creating post thread-243762. Skipping............


and so that part of the topic migration ended up:

Code:
   239384 / 240477 ( 99.5%)  [400 items/min]

Not bad for a first run.

Now, the status is:

Code:
importing posts...
   112634 / 651396 ( 17.3%)  [607 items/min]

Estimated time to completion of posts (only posts part): 15 Hours

Example of one of the three "topic errors":

Code:
   201335 / 240477 ( 83.7%)  [411 items/min]  Exception while creating post thread-243762. Skipping.
PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `block (2 levels) in log'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:717:in `block in log'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:708:in `log'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `execute_and_clear'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:111:in `exec_delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:180:in `delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:576:in `delete_all'
/var/www/discourse/app/models/topic_link.rb:285:in `cleanup_entries'
/var/www/discourse/app/models/topic_link.rb:139:in `extract_from'
/var/www/discourse/lib/post_creator.rb:572:in `extract_links'
/var/www/discourse/lib/post_creator.rb:180:in `block in create'
/var/www/discourse/lib/post_creator.rb:359:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction'
/var/www/discourse/lib/post_creator.rb:358:in `transaction'
/var/www/discourse/lib/post_creator.rb:174:in `create'
/var/www/discourse/script/import_scripts/base.rb:567:in `create_post'
/var/www/discourse/script/import_scripts/base.rb:515:in `block in create_posts'
/var/www/discourse/script/import_scripts/base.rb:502:in `each'
/var/www/discourse/script/import_scripts/base.rb:502:in `create_posts'
script/import_scripts/vbulletin.rb:330:in `block in import_topics'
/var/www/discourse/script/import_scripts/base.rb:882:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:881:in `loop'
/var/www/discourse/script/import_scripts/base.rb:881:in `batches'
script/import_scripts/vbulletin.rb:313:in `import_topics'
script/import_scripts/vbulletin.rb:85:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/vbulletin.rb:944:in `<main>'

# 11  
Old 03-10-2020
For those interested; here is a markdown cheatsheet:

Markdown Cheatsheet . adam-p/markdown-here Wiki . GitHub

I used this cheat sheet to add some simple code to transform ICODE , MOD and NOPARSE tags, like this:

Under Consideration:  Migrate the Forums to Discourse-screen-shot-2020-03-10-112121-amjpg


But will need to run the migration again from scratch (or learn how to rake or otherwise modify the DB in discourse) after I figure out all the custom Ruby code I need to add to the migration script.

However, I'm learning Ruby a little bit now, something I never thought I would every do.
These 2 Users Gave Thanks to Neo For This Post:
# 12  
Old 03-10-2020
To me perfectly honest, this is actually pretty easy to do (technically speaking), and I think I can easily improve and customize this migration script with a few iterations.
# 13  
Old 03-10-2020
Note:

Some discourse plugins like this one might work for COLOR:

GitHub - discourse/discourse-bbcode-color: A Discourse Plugin to support BBCode color tags.

And perhaps this some for some subset of other BBCODE tags:

GitHub - discourse/discourse-bbcode: vBulletin BBCode plugin
This User Gave Thanks to Neo For This Post:
# 14  
Old 03-10-2020
A few more small errors cropped up:

Code:
   152569 / 651396 ( 23.4%)  [581 items/min]     201335 / 240477 ( 83.7%)  [411 items/min]  Exception while creating post thread-243762. Skipping.
PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.6/lib/patches/db/pg.rb:69:in `exec_params'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
   152570 / 651396 ( 23.4%)  [581 items/min]  :47:in `permit_concurrent_loads'e_support/dependencies/interlock.rb
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `block (2 levels) in log'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:717:in `block in log'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:708:in `log'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
   152571 / 651396 ( 23.4%)  [581 items/min]  ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `execute_and_clear'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:111:in `exec_delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:180:in `delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `delete'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation.rb:576:in `delete_all'
/var/www/discourse/app/models/topic_link.rb:285:in `cleanup_entries'
/var/www/discourse/app/models/topic_link.rb:139:in `extract_from'
/var/www/discourse/lib/post_creator.rb:572:in `extract_links'
/var/www/discourse/lib/post_creator.rb:180:in `block in create'
   152572 / 651396 ( 23.4%)  [581 items/min]  action'in trans
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction'
/var/www/discourse/lib/post_creator.rb:358:in `transaction'
/var/www/discourse/lib/post_creator.rb:174:in `create'

But overall, still going OK:

Code:
 200362 / 651396 ( 30.8%)  [586 items/min]

Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Migrate from FTP to SFTP

Hi,I am using following code for FTP in shell script file and it is working.Now I want to migrate from FTP to SFTP.What code changes/steps I have to perform for SFTP ? ftp -in <<FIN open $SAP_UP_SERVER user $SAP_UP_USER $SAP_UP_PASSWORD asc put... (7 Replies)
Discussion started by: Nitin Varshneya
7 Replies

2. BSD

Migrate a Hard Disk

hi Has anyone already tried to migrate a hard disk with FreeBSD using recoverdisk? (1 Reply)
Discussion started by: ccc
1 Replies

3. Linux

Mysql Migrate

Hi , I would like to (MYSQL) migrate the all the data from solari's to linux box. I have checked whether mysql is installed or not. rpm -qa | grep -i mysql I confirmed !!!! I want to know the following points. 1) How can get to know what are mysql data files and location as well.... (4 Replies)
Discussion started by: Mani_apr08
4 Replies

4. Solaris

Migrate from MPXIO to Powerpath

Here is the issue: I am building a database server using Solaris 10x86 U8. The system is jumpstarted with mpxio enabled and booting from the san. We need to have powerpath 5.3 installed and would like to have powerpath take control of the the boot san as well or have mpxio control the san... (2 Replies)
Discussion started by: nabru72
2 Replies

5. Solaris

Can you migrate UFS to ZFS ?

I have some UFS volumes (non root), that I would like to change into ZFS volumes. Is this possible ? I think the only method is to create a new zfs volume and copy the data accoss, this would take a long time for us. Is there a quicker way ? Regards (5 Replies)
Discussion started by: wjones
5 Replies

6. Solaris

Migrate oracle solaris 5.8 5.9

If I have an oracle 9 database environment on a san running solaris 5.8 as the os. Can I plug the san into a Solaris 5.9 environment and have the database work ? - as long as binaries are on the san (1 Reply)
Discussion started by: tim-carroll@com
1 Replies

7. UNIX for Dummies Questions & Answers

Pipe not taken into consideration

I'm trying to see every file which my group (staff) has in a certain directory, recursively. The pipe ls -l -R | grep staff is not working exactly as I want, as for every directory to which my user does not have access, a line like: ls: ./directory/lost+found: The file access permissions do not... (5 Replies)
Discussion started by: panchopp
5 Replies

8. UNIX for Dummies Questions & Answers

How to migrate ?

Currently, I am planning a migration between machine which under True64UNIX. The new machine will run with higher version O/S. My question is, is there any solution on migrating one machine to another which with different O/S version? My goal is keeping minimum impact to the users. Excuse my... (1 Reply)
Discussion started by: coolmans
1 Replies
Login or Register to Ask a Question