Upgrade failed - Plone 4.3.8 to Plone 5.0.2

Question

OS:Debian 8

I upgraded a Plone site from 4.3.6 to 4.3.8 using buildout. Then when I try to upgrade it from 4.3.8 to 5.0.2(copy&paste Data.fs and blobstorage), error shows as below:

Dry run selected.
Starting the migration from version: 4312
Ran upgrade step: Miscellaneous
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50alpha3 upgrade profile
Ran upgrade step: Upgrade editing control panel settings
Ran upgrade step: Upgrade maintenance control panel settings
Ran upgrade step: Upgrade navigation control panel settings
Ran upgrade step: Upgrade search control panel settings
Ran upgrade step: Upgrade site control panel settings
Role / permission map imported.
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta1 upgrade profile
Ran upgrade step: Upgrade markup control panel settings
Ran upgrade step: Upgrade mail control panel settings
Ran upgrade step: Upgrade security control panel settings
Ran upgrade step: Remove Language Tool and migrate settings
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta2 upgrade profile
Role / permission map imported.
Actions tool imported.
Control panel imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-accessibility.init.
Removed record plone.resources/mockup-patterns-accessibility.export.
Removed record plone.resources/mockup-patterns-accessibility.deps.
Removed record plone.resources/mockup-patterns-accessibility.conf.
Removed record plone.resources/mockup-patterns-accessibility.url.
Removed record plone.resources/mockup-patterns-accessibility.css.
Removed record plone.resources/mockup-patterns-accessibility.js.
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta3 upgrade profile
Role / permission map imported.
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-tablesorter.init.
Removed record plone.resources/mockup-patterns-tablesorter.export.
Removed record plone.resources/mockup-patterns-tablesorter.deps.
Removed record plone.resources/mockup-patterns-tablesorter.conf.
Removed record plone.resources/mockup-patterns-tablesorter.url.
Removed record plone.resources/mockup-patterns-tablesorter.css.
Removed record plone.resources/mockup-patterns-tablesorter.js.
Removed record plone.resources/resource-collection-css.init.
Removed record plone.resources/resource-collection-css.export.
Removed record plone.resources/resource-collection-css.deps.
Removed record plone.resources/resource-collection-css.conf.
Removed record plone.resources/resource-collection-css.url.
Removed record plone.resources/resource-collection-css.css.
Removed record plone.resources/resource-collection-css.js.
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50rc1 upgrade profile
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50rc2 upgrade profile
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/resource-plone-app-discussion-stylesheets.init.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.export.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.deps.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.conf.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.url.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.css.
Removed record plone.resources/resource-plone-app-discussion-stylesheets.js.
could not upgrade calendar_starting_year property
could not upgrade calendar_future_years_available property
Ran upgrade step: Run to50rc3 upgrade profile
Setting installed version of profile Products.PloneLanguageTool:PloneLanguageTool as unknown.
Setting installed version of profile plone.app.jquerytools:default as unknown.
Setting installed version of profile plone.app.jquery:default as unknown.
Setting installed version of profile Products.TinyMCE:TinyMCE as unknown.
Setting installed version of profile archetypes.referencebrowserwidget:default as unknown.
Setting installed version of profile plone.app.search:default as unknown.
Setting installed version of profile archetypes.querywidget:default as unknown.
Setting installed version of profile plonetheme.classic:default as unknown.
Setting installed version of profile plonetheme.sunburst:default as unknown.
Setting installed version of profile Products.CMFPlone:plone-content as unknown.
Ran upgrade step: Unmark installed profiles that are no longer available.
Ran upgrade step: Mark products as installed for installed uninstallable profiles
Ran upgrade step: Cleanup uninstalled products
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to500 upgrade profile
Ran upgrade step: Unmark installed profiles that are no longer available.
Ran upgrade step: Mark products as installed for installed uninstallable profiles
Ran upgrade step: Cleanup uninstalled products
Ran upgrade step: Cleanup the skins tool.
Ran upgrade step: Remove fake kupu tool and related settings and resources.
Role / permission map imported.
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Reindexed `getIcon` for 6 items
Ran upgrade step: Run to501 upgrade profile.
Role / permission map imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to502 upgrade profile.
End of upgrade path, main migration has finished.
Starting upgrade of core addons.
Role / permission map imported.
Done upgrading core addons.
Your Plone instance is now up-to-date.
Dry run selected, transaction aborted

When I try to view the site, error shows like:

KeyError('Interface `Products.CMFPlone.interfaces.controlpanel.ILanguageSchema` defines a field `available_languages`, for which there is no record.',) (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: Interface `Products.CMFPlone.interfaces.controlpanel.ILanguageSchema` defines a field `available_languages`, for which there is no record.)

Today(20160323), on the suggestion of maurits , I go straight to upgrade the site. Upgrade failed again:

Starting the migration from version: 4312
Ran upgrade step: Miscellaneous
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50alpha3 upgrade profile
Ran upgrade step: Upgrade editing control panel settings
Ran upgrade step: Upgrade maintenance control panel settings
Ran upgrade step: Upgrade navigation control panel settings
Ran upgrade step: Upgrade search control panel settings
Ran upgrade step: Upgrade site control panel settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta1 upgrade profile
Ran upgrade step: Upgrade markup control panel settings
Ran upgrade step: Upgrade mail control panel settings
Ran upgrade step: Upgrade security control panel settings
Ran upgrade step: Remove Language Tool and migrate settings
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta2 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Control panel imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-accessibility.init.
Removed record plone.resources/mockup-patterns-accessibility.export.
Removed record plone.resources/mockup-patterns-accessibility.deps.
Removed record plone.resources/mockup-patterns-accessibility.conf.
Removed record plone.resources/mockup-patterns-accessibility.url.
Removed record plone.resources/mockup-patterns-accessibility.css.
Removed record plone.resources/mockup-patterns-accessibility.js.
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta3 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Actions tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Removed record plone.resources/mockup-patterns-tablesorter.init.
Removed record plone.resources/mockup-patterns-tablesorter.export.
Removed record plone.resources/mockup-patterns-tablesorter.deps.
Removed record plone.resources/mockup-patterns-tablesorter.conf.
Removed record plone.resources/mockup-patterns-tablesorter.url.
Removed record plone.resources/mockup-patterns-tablesorter.css.
Removed record plone.resources/mockup-patterns-tablesorter.js.
Removed record plone.resources/resource-collection-css.init.
Removed record plone.resources/resource-collection-css.export.
Removed record plone.resources/resource-collection-css.deps.
Removed record plone.resources/resource-collection-css.conf.
Removed record plone.resources/resource-collection-css.url.
Removed record plone.resources/resource-collection-css.css.
Removed record plone.resources/resource-collection-css.js.
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Ran upgrade step: Run to50beta4 upgrade profile
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Skins tool imported.
Archetype tool imported.
Step languagetool has an invalid import handler
Role / permission map imported.
Step collective.z3cform.datetimewidget has an invalid import handler
Archetype tool imported.
Step languagetool has an invalid import handler
Upgrade aborted. Error:
Traceback (most recent call last):
  File "/opt/plone502/buildout-cache/eggs/Products.CMFPlone-5.0.2-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
    step['step'].doStep(setup)
  File "/opt/plone502/buildout-cache/eggs/Products.GenericSetup-1.8.1-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
    self.handler(tool)
  File "/opt/plone502/buildout-cache/eggs/plone.app.upgrade-1.3.20-py2.7.egg/plone/app/upgrade/v50/betas.py", line 352, in to50rc1
    qi.installProduct('plone.app.linkintegrity')
  File "/opt/plone502/buildout-cache/eggs/Products.CMFQuickInstallerTool-3.0.13-py2.7.egg/Products/CMFQuickInstallerTool/QuickInstallerTool.py", line 603, in installProduct
    blacklisted_steps=blacklistedSteps,
  File "/opt/plone502/buildout-cache/eggs/Products.GenericSetup-1.8.1-py2.7.egg/Products/GenericSetup/tool.py", line 378, in runAllImportStepsFromProfile
    dependency_strategy=dependency_strategy)
  File "/opt/plone502/buildout-cache/eggs/Products.GenericSetup-1.8.1-py2.7.egg/Products/GenericSetup/tool.py", line 1310, in _runImportStepsFromContext
    message = self._doRunImportStep(step, context)
  File "/opt/plone502/buildout-cache/eggs/Products.GenericSetup-1.8.1-py2.7.egg/Products/GenericSetup/tool.py", line 1146, in _doRunImportStep
    return handler(context)
  File "/opt/plone502/buildout-cache/eggs/plone.app.intid-1.1.0-py2.7.egg/plone/app/intid/setuphandlers.py", line 66, in registerContent
    registered, existing = register_all_content_for_intids(portal)
  File "/opt/plone502/buildout-cache/eggs/plone.app.intid-1.1.0-py2.7.egg/plone/app/intid/setuphandlers.py", line 41, in register_all_content_for_intids
    obj = brain.getObject()
  File "/opt/plone502/buildout-cache/eggs/Products.ZCatalog-3.0.2-py2.7.egg/Products/ZCatalog/CatalogBrains.py", line 108, in getObject
    return parent.restrictedTraverse(path[-1])
  File "/opt/plone502/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/OFS/Traversable.py", line 317, in restrictedTraverse
    return self.unrestrictedTraverse(path, default, restricted=True)
  File "/opt/plone502/buildout-cache/eggs/Zope2-2.13.23-py2.7.egg/OFS/Traversable.py", line 300, in unrestrictedTraverse
    raise e
NotFound: 1306115487
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed

The homepage of the site shows the same error.

Answer 1

I had the same error with a 10-digit identifier of the problematic "brain". My portal_catalog of 80,000 objects had three that cause this upgrade step to fail. To fix this, I first identified the problematic objects on the plone 5 site, then went back to plone 4 site and deleted the objects. After that the upgrade to plone 5 completed successfully. Two of the objects were discussion items, the third was a document that had one of the discussion items.

Note that I now realize that I had not migrated the discussion comments to the new system after plone 4.1 (In plone 4 discussion settings I have a warning: "You have comments that have not been migrated to the new commenting system that has been introduced in Plone 4.1. Please migrate your comments to fix this.") - perhaps that would have solved this problem as well.

To find the problematic objects, start the plone 5 instance in debug mode:

sudo bin/instance debug

and enter the following:

from Products.CMFCore.utils import getToolByName
site = app['YourSiteName']
zcatalog = getToolByName(site, 'portal_catalog')
cnt=0
brains = zcatalog.unrestrictedSearchResults()
for brain in brains:
  obj = brain._unrestrictedGetObject()
  cnt +=1

if there is a problematic object, you will see the error message. Use this to identify the object:

brains[cnt].getPath()

To check for multiple problematic objects you can modify the code snippet above.

Answer 2

The clue is at the end of the first text that you pasted:

Your Plone instance is now up-to-date.
Dry run selected, transaction aborted

So it would have worked, but you have selected a dry run, so nothing has actually changed. When you then try to view the site, things fail because the code expects that the upgrade has been run.

So: try again, but this time do not select the dry run option.