Difference between revisions of "Opasnet update 2017"

Revision as of 11:56, 18 July 2017

This page contains notes produced while updating Opasnet server systems during the summer of 2017.

RTools server

CentOS 6.4: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

• apache 2.2.15 -> 2.2.32 or 2.4.25
  • newer branch not particularly important
• PHP 5.3.3 -> 5.6.x or 7.1.x
  • incompatibilities in mysql functions
    • "mysql" -> "mysqli" (5.5)
  • should upgrade to latest version
• MySQL 5.1 -> 5.7
• mongoDB 2.4 -> 3.2
  • should be relatively easy as we are not using sharding etc.
• rtools_server PHP application
  • starts on test server after minor changes
  • further testing needed

OpasnetBase2

Runs on Apache as a PHP web application. Updating PHP and MongoDB causes a few incompatibilities and other issues with old code:

• MongoDB driver updates
• "Mongo" PHP library deprecated, new "MongoDB" PHP library
  • Mongo-class no longer available
    • Use MongoDB\Client instead
  • Other classes changed to some extent
    • cursor does not handle query options -> everything needs to be supplied to collection.find()
    • Collections do not implement __get(), hence they cannot be chained like before. db->ident->subset->dat should be combined to db->{ident.subset.dat}.
• Use pecl to install driver
• Use composer to download library to be included in web app
• Actually, a lot of code (~1000 lines)
  • "mysql" to "mysqli" patching
    • some static functions may need new parameters e.g. OpasnetBaseActiveRecord::parse_qmarks needed a database link where one was not needed before (due to mysqli_real_escape_string requiring a database link parameter now)
  • The MongoDB drivers (both old and new) overload __get() to give shortcuts to queries and other functionality, which makes debugging difficult.
    • It might be useful to change them to use proper method calls instead.
  • "insert" to "insertOne" and "ensureIndex" to "createIndex" patching
  • "MongoDate" type changed to "MongoDB\BSON\UTCDateTime"
    • Different methods: "date('Y-M-d H:i:s',$obj[$index->ident]->sec)" to "$obj[$index->ident]->toDateTime()->format('Y-M-d H:i:s')"
  • Tracked issue with invalid times in $data to MySQL column data type issue (datetime instead of timestamp)
    • MySQL table download_sessions contains the keys, chunk counters and download times of every download ever
      • HUGE (3.5M rows), pretty much useless after download is complete. This should probably be cleaned regularly.
• Current issues:
  • Server throws no exceptions, but returns garbage when attempting to prepare uploads using unmodified OpasnetUtils.
    • Using RCurl postForm() for header works.
      • Upload, replace, append, download, include, exclude all work.
    • Return header seems fine
  • test.php upload works
  • Read APIs return errors ERROR: Load by columns: record not found!
    • MongoDB\Driver\Cursor and other driver classes methods are not exposed to PHP?
      • Cursor methods not implemented in the new PHP library -> have to do foreach loop for results and manually chunk them into files.
        • Reasonable to implement Done, performance might be worse however.
    • Collection->find(...) executes the given query which means options such as batchSize have to be included in find() as options.
      • Irrelevant with foreach loop. The driver handles everything.
• WORKS (well enough)

Opasnet-wiki server

CentOS 6.5: full support ended, but still get maintenance updates.

Key software, upgrade paths and related issues:

• apache 2.2.15 -> 2.2.32 or 2.4.25
• PHP 5.4.28 -> 5.6.x or 7.1.x
  • 5.5.9+ required by currently supported mediawiki releases
  • latest php also supported
  • incompatibilities likely to occur in 5.4->5.5, hence might as well upgrade to latest
• MySQL 5.5.37 -> 5.5.56 or 5.7.x
• MediaWiki 1.20.0 -> 1.27 (LTS) or 1.28.2 (latest)
  • need to check supported extensions
  • MyExtension.php no longer supported, need extension.json for registering extensions
  • Composer might be useful for managing PHP dependencies

RTools MediaWiki extension

• RTools special page class (now named SpecialRtools)
  • Creates instance of rtools_client
  • separate config file, which is included by SpecialRTools instead of rtools_client for some reason
  • fixed: "mysql" to "mysqli"
  • fixed: "wfMsg" to "wfMessage->text()"
• <rcode> tag
  • Detected by parser hooks
  • RToolsParser class
    • fixed: "wfMsg" to "wfMessage->text()"
    • fixed: class instead of global functions
      • create class instance in parser given by hook in order to retain variable values
    • fixed: added "wgOut->addModules( 'ext.RTools.parser' );" which enables the on-page UI
    • fixed: changed efRCodeMBInArray to a static function and changed reference in templates/inputs/checkbox.php
• rtools_client
  • handles connections and job submissions etc. to server
  • xml-rpc library
    • communicates with rtools_server API
• javascripts
  • script.aculo.us + prototype
    • UI
    • ajax
    • old versions, probably still acceptable
  • loaded by mediawiki resource loader
    • now set up in extension.json
• wiki database
  • rtools_inputs
    • added schema update hook to create the necessary table if missing
• Current issues:
  • Show code button does not work
  • Run page throws: Didn't receive 200 OK from remote server. (HTTP/1.0 500 Internal Server Error), which is a xml-rpc related error
  • Runs and shows outputs (oprint and cat), but code is not visible, embed also does not work
  • Input forms are slightly broken: conditional categories do not work, checkboxes crash the whole page
  • Rtools page does not update, shows error until refreshed once the run is over. Error on page: "jQuery is not defined", but it is loaded by mediawiki by default. Caused by some linking problem between the special page and mediawiki.
    • SpecialRTools-class problem with include of "template/index.php" in method "output" not scoping as expected. Including javascript dependencies separately resolves the first issue, but results in "mw not defined" indicating no access to mediawiki.
    • Special page includes javascript for both parser and special page, which does not happen on the current server. It is loaded as part of parser initialization on every page.
    • Works 1% of the time ...

Extensions

Our extensions

• ExternalSourceBrowser: Einari
• OpasnetBase: Einari, Juha
  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
  • wfMsg() to wfMessage()->text()
  • Issue: row counter does not always work
    • javascript autoloader does not appear to work on all included parts of the page
  • Seems to work otherwise
• OpasnetBaseImport: Einari, Juha
  • Converted to use extension.json registering (maintenance/convertExtensionToRegistration.php)
    • Parser stuff put in a new class
    • Added all classes to Autoload
    • Load configuration file at start of all class definition files
      • Added "global $wgArticlePath, $wgScript;" at config start to give access to mw globals in the new scope
    • Corrected extension documentation URL
  • wfMsg() to wfMessage()->text()
  • Current issues:
    • configuration variable $obImportUploadsPath becomes null or somehow
      • Changed config to use constants instead of global variables
  • Works
• Table2Base: Juha, Einari
  • Uses OpasnetBaseImporters OpasnetBaseUpload class
  • Converted to extension registeration
  • Created Table2BaseParser-class
  • Writes to both OpasnetBase and Wiki database
    • Need to figure out wiki database string escapes, may be unnecessary with proper use of wrapper functions. Can't use mysqli_real_escape_string due to mediawiki wrapping.
      • Wikimedia\Rdbms\Database::insert handles escaping
    • Need to add database update hook
    • OpasnetBase queries: 'mysql_real_escape_string' to 'mysqli_real_escape_string' patching
  • Current issues:
    • No index for t2b uploads in wiki base may make queries slow (affects performance on every page with t2b)?
    • Displays data normally, displays errors in data, but does not upload.
• OpasnetRater: Einari

Other Extensions

Currently installed extensions, still maintained unless specified otherwise. Almost all have been updated since last time. Nothing tested yet.

• AuthorProtect
  • Git maintained
• CategoryTree [1]
• ConfirmEdit is shipped with MediaWiki
• ContactPage [2]
  • Need to reconfigure local settings when upgrading
• Contribution Scores [3]
• Contributors [4]
• DataValues [5]
  • DataValues Geo [6]
  • DataValues Time [7]
  • DataValues Validators [8]
  • DataValues Common [9]
  • DataValues Interfaces [10]
  • Still updated
• Google Analytics Integration [11]
• ImageMap is shipped with MediaWiki
• Maps [12]
  • Very active development
  • Possible incompatibilities with RTools when upgraded
• Math [13]
• MsUpload [14]
  • Upload files from editor
    • Has this seen any use?
• PageNotice [15]
• SimpleCalendar [16]
• User Merge and Delete [17]
  • Has MediaWiki 1.29+ listed on extension page, but probably works on 1.28
• Validator [18]
• Widgets [19]
  • Works

Obsolete or otherwise dead

Not maintained anymore, very old, security risk or integrated to MediaWiki.

• Ajax Show Editors [20]
  • Archived
• CheckUser [21]
  • Integrated functionality?
• CreateBox [22]
  • Archived
  • Option: InputBox [23]
    • This is included with new installations of MediaWiki 1.21 and above
  • Used on Create_article
    • Any others?
• DynamicPageList [24]
  • Very old version, might not work on latest MediaWiki and PHP
• EmailAddressImage [25]
• Google Co-op Extension [26] ?
  • Search functionality that was never used and is now obsolete?
• MsLinks [27]
  • Used to display files on pages using "{{#l:Testfile.pdf}}"
    • Has this been used at all?
  • Very old version, likely broken
• NewPageCSS [28]
  • Very old version
  • Security risk?
• NiceCategoryList [29]
  • Security risk
• Todo Tasks [30]
  • Not maintained since 2009, expect to break.

Not listed in Special:Version

Have to check local settings to see if these are installed.

• AccessControl [31]
  • Maintained
• ArticleComments [32]
  • Archived
• CategoryStepper [33]
  • Unmaintained
• CategorySuggest [34]
  • Maintained
• Discussion [35]
  • Not maintained, download link dead.
• DiscussionThreading [36]
  • Maintained
• Header Footer [37]
  • Maintained
• Lockdown [38]
  • Maintained
• mfiles, mfiles2
• NamespacePermissions [39]
  • Suggested: Lockdown, which is also present on the server
• PageDisqus [40]
  • Still in beta, git has some recent commits ...
• PageID
  • One of ours?
• PdfHandler [41]
  • Shipped with current version of MediaWiki
• Renameuser
  • Shipped with current version of MediaWiki
• SelectCategory [42]
  • Maintained
• SimpleAntiSpam [43]
  • Merged into MediaWiki
• SpamBlacklist [44]
  • Maintained
• StubManager [45]
  • Archived
• Survey [46]
  • Unmaintained
  • Alternative if still in use: QuickSurveys [47]?
• TitleBlacklist [48]
  • Maintained
• WhoIsWatching [49]
  • Maintained
• WhosOnlineAjax [50]
  • Archived
  • Alternative: WhosOnline [51]

See Also

Opasnet:Technical_documentation