Login or Register to Ask a Question and Join Our Community


What is on Your Mind?

VBulletin 3.8 to Discourse on Docker Migration Test Take Two

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
The Lounge What is on Your Mind? VBulletin 3.8 to Discourse on Docker Migration Test Take Two
# 15  
Old 03-13-2020
I think I found the problem. In all the install directions and migration directions I read online (many) none of them mentioned setting the ENV["IMPORT"] variable

However, now that I am moving pass "complete ruby-noobie" stage, I looked at the Gemfile in discourse; and here is the issue why the bbcode to markdown plugin was not done:

Quote:
A Gemfile is a file we create which is used for describing gem dependencies for Ruby programs. A gem is a collection of Ruby code that we can extract into a “collection”¯ which we can call later.

Code:
if ENV["IMPORT"] == "1"
  gem 'mysql2'
  gem 'redcarpet'

  # NOTE: in import mode the version of sqlite can matter a lot, so we stick it to a specific one
  gem 'sqlite3', '~> 1.3', '>= 1.3.13'
  gem 'ruby-bbcode-to-md', git: 'https://github.com/nlalonde/ruby-bbcode-to-md'
  gem 'reverse_markdown'
  gem 'tiny_tds'
  gem 'csv'
end

So, I checked:

Code:
ubuntu-docker# env | grep -i IMPORT
ubuntu-docker#

Nothing.

So now I know..... LOL... next build:

Code:
export IMPORT=1

So simple when you know the problem.

Hmmm Smilie

I'm convinced now. The posts and cautions on the net about Discourse is true. Yes, it is free and open source; and yes it is built on the model of selling services (consulting, hosting, custom programming) around the Discourse ecosystem.

Again, as in an earlier post, I'm not saying this is "good" or "bad"; but I'm staying "it is what it is"..... because, for example, this one small detail export IMPORT=1 was missing from every example, README and tutorial on the net. I'm sure those Discourse paid installers know about export IMPORT=1 ....

But then when we know the issue and when I search for "IMPORT=1 discourse" on the net, now that I know the problem and how to fit it, sure enough, there is it in a post out there:

Ruby mysql2 error on migration - support - Discourse Meta

Quote:
Another tip that might be useful for anyone else coming into this thread:

Using that command given above with IMPORT=1 will trigger this part of your Gemfile:

if ENV["IMPORT"] == "1"
gem 'mysql2'
gem 'redcarpet'
gem 'sqlite3', '~> 1.3.13'
gem 'ruby-bbcode-to-md', github: 'nlalonde/ruby-bbcode-to-md'
gem 'reverse_markdown'
gem 'tiny_tds'
end
I ran into problems with the installation of tiny_tds, and then with another thing it depends on called FreeTDS. After some wasted time on this I realized these are useful only for MS SQL imports, which I won't be using, so I simply commented out the tiny_tds line in my Gemfile.

(it would be nice to have a comment in front of each of those packages saying what they are used for, if anyone knows)
Now we know..... LOL

I agree with the poster above. It would be nice if this small detail was made available to use at the beginning of the migration process; but hey, never mind... it's very nice free software and I figured it out on my own.

So, the next time I build this discourse app, it's export IMPORT=1 before building Smilie
This User Gave Thanks to Neo For This Post:
vgersh99
# 16  
Old 03-13-2020
Also:

Code:
MaxMind IP database updates require a license
Please set DISCOURSE_MAXMIND_LICENSE_KEY to one you generated at https://www.maxmind.com
MaxMind IP database updates require a license

# 17  
Old 03-13-2020
Armed with this new knowledge (export IMPORT=1), I have killed "test run 2" and have rebuilt for "test run 2b" .... and will start over with a new DB dump (but will not clear the discourse postgres db yet)

Code:
   239012 / 240522 ( 99.4%)
importing posts...
   177893 / 651614 ( 27.3%)

Let's see what "test round 2b" looks like before moving on to "test round 3".....

PS: I still don't see the bbcode to markdown plugin in the admin panel but it definitely pulled it during the build:

Code:
Using rqrcode_core 0.1.1
Using rqrcode 1.1.2
Using rtlit 0.0.5
Using ruby-bbcode-to-md 0.0.14 from https://github.com/nlalonde/ruby-bbcode-to-md (at master@3b9aaf2)
Using ruby-readability 0.7.0
Using rubyzip 2.2.0

# 18  
Old 03-13-2020
Starting to see, slowing but surely, this is gonna work and it's gonna be great.

This, when finished, will be a huge improvement for all users and the forums will be much better for all.

I am guessing the new discourse forum will be ready for member testing in less than a week from now.

importing posts...
204459 / 651614 ( 31.4%)
# 19  
Old 03-13-2020
Migration notes, tentative plans, ideas:
  • When I finishing migrating and testing and have a working discourse forum for the site I will open up the forum to all to visit and test.
  • I will not turn off the main forums anytime soon and have no plans to turn off or change www.unix.com.
  • The new forum will reside at discourse.unix.com.
  • We will keep both sites for the time being, maybe indefinitely.
  • If the testing goes well, and everyone is happy, I will set www.unix.com to read-only (reference) until sometime into the future.
  • If the testing goes well, and everyone is happy, new posts will be at the new discourse site.

More to come.... this is going to work, based on what I am seeing.
This User Gave Thanks to Neo For This Post:
RavinderSingh13
# 20  
Old 03-13-2020
I may bring this on line for testing sooner than later:

Code:
importing posts...
   264756 / 651614 ( 40.6%)

Just did a first quick cut at setting up categories based on legacy forums.

We will grant more people edit and mod privs based on more flexible discourse trust model.

Many "moderator emeritus" will have a higher trust level and will be able to edit posts in discourse.

This means we will greatly expand our leadership team based on trust levels, and many of our members who are active will be given a high trust level, by default when the new system goes live.

Edit:

Code:
importing posts...
   300396 / 651614 ( 46.1%)

# 21  
Old 03-13-2020
Update:

Code:
importing posts...
   408233 / 651614 ( 62.6%)

This is not going to be difficult.. just a lot of work.

I thought, perhaps, because the discourse db is postgres and vb was mysql, this would be difficult; but it's not. The SQL is nearly the same, only commands, but the concept is mostly the same, SQL, tables, queries.

Getting "read into" Ruby take time, but it's not that hard. It's actually interesting and I'm starting to like it.

For example, the discourse system will not let me as the overall system admin change the user "Neo" from this forum to admin. The system bot tells me that it must send me an email which never comes because email is turned off for migration.

So, I looked at the users table in discourse, found a boolean for admin, did a simple SQL query and my status is now also "admin" for user Neo. This is probably not the best way to do things, but at least I can get around when I need to in the DB.

In addition, I have been decoding where the likes are stored when a user likes a post. There is a table called user_actions, and all likes and such user actions are recorded in that table.

Code:
discourse=# \d user_actions
                                           Table "public.user_actions"
     Column      |            Type             | Collation | Nullable |                 Default                  
-----------------+-----------------------------+-----------+----------+------------------------------------------
 id              | integer                     |           | not null | nextval('user_actions_id_seq'::regclass)
 action_type     | integer                     |           | not null | 
 user_id         | integer                     |           | not null | 
 target_topic_id | integer                     |           |          | 
 target_post_id  | integer                     |           |          | 
 target_user_id  | integer                     |           |          | 
 acting_user_id  | integer                     |           |          | 
 created_at      | timestamp without time zone |           | not null | 
 updated_at      | timestamp without time zone |           | not null | 
Indexes:
    "user_actions_pkey" PRIMARY KEY, btree (id)
    "idx_unique_rows" UNIQUE, btree (action_type, user_id, target_topic_id, target_post_id, acting_user_id)
    "idx_user_actions_speed_up_user_all" btree (user_id, created_at, action_type)
    "index_user_actions_on_acting_user_id" btree (acting_user_id)
    "index_user_actions_on_action_type_and_created_at" btree (action_type, created_at)
    "index_user_actions_on_target_post_id" btree (target_post_id)
    "index_user_actions_on_target_user_id" btree (target_user_id) WHERE target_user_id IS NOT NULL
    "index_user_actions_on_user_id_and_action_type" btree (user_id, action_type)

So I decoded a like, and found:

Code:
discourse=# select * from user_actions order by created_at desc limit 10;
   id    | action_type | user_id | target_topic_id | target_post_id | target_user_id | acting_user_id |         created_at         |         updated_at         
---------+-------------+---------+-----------------+----------------+----------------+----------------+----------------------------+----------------------------
 1065308 |           2 |  138019 |          377378 |         377381 |                |              1 | 2020-03-13 13:49:39.284721 | 2020-03-13 13:49:39.380662
 1065307 |           1 |       1 |          377378 |         377381 |                |              1 | 2020-03-13 13:49:39.284721 | 2020-03-13 13:49:39.360154
 1054388 |           1 |       1 |          377400 |         706814 |                |              1 | 2020-03-13 13:32:43.396314 | 2020-03-13 13:32:43.527381
 1054389 |           2 |       2 |          377400 |         706814 |                |              1 | 2020-03-13 13:32:43.396314 | 2020-03-13 13:32:43.546778
  882882 |          11 |  138027 |          377411 |         706825 |                |              1 | 2020-03-13 09:11:37.832491 | 2020-03-13 09:11:37.832491
  844710 |          12 |      -2 |          377396 |             -1 |                |             -2 | 2020-03-13 08:12:59.394389 | 2020-03-13 08:12:59.437878
  844711 |          13 |  138027 |          377396 |             -1 |                |             -2 | 2020-03-13 08:12:59.394389 | 2020-03-13 08:12:59.448827
  844708 |          12 |      -2 |          377394 |             -1 |                |             -2 | 2020-03-13 08:12:59.053549 | 2020-03-13 08:12:59.108207
  844709 |          13 |  138026 |          377394 |             -1 |                |             -2 | 2020-03-13 08:12:59.053549 | 2020-03-13 08:12:59.153916
  844706 |          12 |      -2 |          377393 |             -1 |                |             -2 | 2020-03-13 08:12:59.01556  | 2020-03-13 08:12:59.06146
(10 rows)

that action_type 2 is a "like".

The likes from vB come from a plugin and so they are not transferred to discourse in the migration; but since I have decoded how to "like" in the discourse DB, I think I'll write.a script to transfer all the likes from mysql / vb to postgres / discourse.

The problem with this is that the vb postids are not transferred to postgres, so I will need to write my first custom migration ruby script to transfer the likes when the migration script runs.

Anyone want to do this for us, if I provide both table schemas?
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. What is on Your Mind?

Under Consideration: Migrate the Forums to Discourse

Dear All, After being active on the Node-RED forum for the last few weeks, I have been very impressed with Discourse, and my eyes have been opened. https://www.discourse.org/ but not the paid /hosted offering, but using the open distribution: https://github.com/discourse/discourse ... (52 Replies)
Discussion started by: Neo
52 Replies

2. Docker

Docker learning Phase-I

Hello All, I had recently learnt a bit of Docker(which provides containerization process). Here are some of my learning points from it. Let us start first with very basic question: What is Docker: Docker is a platform for sysadmins and developers to DEPLOY, DEVELOP and RUN applications ... (7 Replies)
Discussion started by: RavinderSingh13
7 Replies

3. UNIX for Beginners Questions & Answers

Can't pass a variable representing the output of lsb_release to a docker dontainer

I don't know why, but the rendering of my code mucks up the spacing and indentation, despite being correct in the original file. I'm having issues getting the following script to run (specifically the nested script at the end of the docker command near the end of the script; I think I'm not passing... (2 Replies)
Discussion started by: James Ray
2 Replies

4. Shell Programming and Scripting

Problem in extracting yocto SDK for docker

Actually I was facing the following issue while building my Yocto SDK on Docker container sudo docker build --tag="akash/eclipse-che:6.5.0-1" --tag="akash/eclipse-che:latest" /home/akash/dockerimage.yocto.support/ Sending build context to Docker daemon 26.93MB Step 1/5 : FROM eclipse/cpp_gcc ... (3 Replies)
Discussion started by: Akash BHardwaj
3 Replies

5. AIX

AIX - FC Switch migration, SAN Migration question!

I'm New to AIX / VIOS We're doing a FC switch cutover on an ibm device, connected via SAN. How do I tell if one path to my remote disk is lost? (aix lvm) How do I tell when my link is down on my HBA port? Appreciate your help, very much! (4 Replies)
Discussion started by: BG_JrAdmin
4 Replies

6. Linux

Docker and pipework,ip with other subnet

Recently i found this for give to docker a "personal" ip ip addr del 10.1.1.133/24 dev eth0 ip link add link eth0 dev eth0m type macvlan mode bridge ip link set eth0m up ip addr add 10.1.1.133/24 dev eth0m route add default gw 10.1.1.1On container i did ... (0 Replies)
Discussion started by: Linusolaradm1
0 Replies

7. Web Development

Removing VBSEO for vbulletin – Reverting back to vbulletin URLs

Please note, this information was copied from vbseo.com, now showing a database error. This is posted for reference since vbSEO seems to be going out of business: If you ever need to uninstall vBSEO , you can use the following instructions. Make sure you carefully follow each step. Login... (37 Replies)
Discussion started by: Neo
37 Replies
Login or Register to Ask a Question