Magento 2 Migration, Integration and Deployment Problems

Magento 2.0.0 was finalized for release on November 16th, 2015. From then until July 2016 there have been 8 additional releases – the current version being 2.0.8. So far we have worked with one new Magento 2 Enterprise Edition built and have migrated a couple of websites to Magento 2 Community Edition. In both cases, […]

By Luis Amador

Magento 2 Migration, Integration and Deployment Problems

Magento 2.0.0 was finalized for release on November 16th, 2015. From then until July 2016 there have been 8 additional releases – the current version being 2.0.8. So far we have worked with one new Magento 2 Enterprise Edition built and have migrated a couple of websites to Magento 2 Community Edition. In both cases, we have encountered a number of hurdles which resulted in lost time.

First and most common Magento 2 problem we have encountered so far has been the loss of time caused by incorrect server configuration and setup from Managed Hosting providers.

Moreover, in Magento 2 there is an expected learning curve. In order to create a functional QA process, which takes at least 20 hours and 1 calendar week to address, a developer has to have experience with Magento 1 and must take the Magento 2 training courses.

There are still a number of issues due to erratic extensions and third party extension developers forgetting to test their extensions in Magento’s Production mode.

Last but not least, Magento 2 does not use semantic versioning for the 2.0 releases. That means every point release can and has resulted in backwards compatibility issues which broke some third party extensions. There are a number of issues caused by the rapid pace of changes by the Magento development team and the lack of backwards compatibility for existing extensions which were written correctly and used the Magento API appropriately only to have that API suddenly modified with a new point release of Magento.

Magento 2 Best Practices for Deployment

Use tried and tested tools.  New tools such as Git provide extremely efficient versioning control – but it is not designed for deployment of PHP application for websites. We use Rsync – an application with over 20 years of improvements and usage – https://en.wikipedia.org/wiki/Rsync. It is one of the basic applications which is always available on any Linux based system. Rsync is capable of synchronizing files and directories between your test installation and your production installation.  It is able to do this with files installed on the same server or on two different systems using either an Rsync server or over ssh. It performs extremely efficient incremental updates – not only transferring only files which have changed, but it will break those files up into small chunks and only transfer the specific chunks which have changed! It has a rich set of options allowing extremely fine-grained exclusions.  It can not only transfer files which have changed or been added, it can also delete files from the target directory which do not exist in the source directory. Lastly, it includes a smart backup system, so that any files that are updated or deleted from the target system are backed up first. This makes it incredibly simple to restore files if you discover your exclude patterns missed some critical set of files.

Automate, automate automate!  Linux has an extensive shell scripting facility which can be used to automate repetitive tasks.  There are a number shell scripting variants that are mostly the same.  We use Bash which has almost 30 years of continual usage and improvements.  It is extremely well documented, https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html and in addition there is a large online database of mini scripts available at https://www.commandlinefu.com/commands/browse.  

Shell scripting allows us to automate many kinds of procedures such as deploying a Magento website; setting up versioning using git; performing routine maintenance from the command line.   The very first time we do this we spend an exhaustive amount of time perfecting this manually via ssh to the server.  Once this is done, it is relatively simple to edit the list of all the commands we have used to create a set of programs which automate the task.  This is partly due to an extremely useful feature of Bash.  Every command that is entered in an interactive session is saved to plain text history file.  When we are done, we can simply download the file and edit as necessary.

Through the use of script variables, the shell scripts we write can be used not only for the site we initially set up – but they can be used on almost any future website by merely editing a small 20-30 line definition file.  So that the next time we have to setup some process on a new system, instead of performing another exhaustive investigation, we can get things set up in a short period of time.

By using shell scripts we have been able to automate a 90 minute interactive process of cloning a production website to a development server to under 10 minutes with a batch process.   This includes updating the configuration information stored in the database for the development server, scrubbing sensitive data such as customers and order, and updating all email addresses so there is no possibility of an e-mail being accidentally sent to a customer during testing.

Magento 2 Best Practices for Integration

Typically you have to setup a development and a QA system for integration and development. This allows you perfect new features on the Development system while performing quality assurance for collections of new features on the QA server.  Additionally this allows clients to review progress on the stable QA system.

Mainly because of the differences between production and developer mode it is a highly recommended that an additional deployment system is added.  At Shero, we maintain 3 separate websites for each customer:  A development system, a Staging system, and an internal Quality Assurance system.  The development system runs in Magento Developer mode.  This system is used for adding new extensions, new features, and for custom theme development.  While the client can have full access to this system, you should discourage them from using it to review progress.  In part, this is due to the fact that the performance of Magento 2 when in Developer mode is poor.  When new features are ready for review, we deploy them to the QA system.  The QA system runs in production mode so it’s performance is closer to the performance that will be seen on the live website. By making frequent automated deployments from development to QA these issues can be caught early.  When the new features pass internal QA process, they are then deployed to the Staging system. The Staging system also runs in production mode, with some additional performance enhancements enabled.  It is this system that you can encourage clients to use for reviewing progress and their own Quality Assurance reviews.

Magento 2 Integration Issues

Magento 2 itself has struggled through a number of issues.  The biggest problem has been that the Magento developers use a web server configuration that is not generally used by most Managed hosting providers.  Due to this they hard coded the file permissions settings into the Magento 2 code that are best for their own system. Any time Magento 2 creates a new file or directory, the permissions would have to be manually updated to work on the web server configuration most common for Managed servers.  As of Mid may – with the release of version 2.0.5 – Magento 2 had configuration options that allow the correct file and directory permissions to be set during creation.

The other major issue was a lack of backwards compatibility between versions.  This means that for each new Magento release, third party extension developers would have to check their extensions to ensure they still worked correctly.  Each time this would add up to 5 days between an upgrade of Magento 2 for the third party extensions to be updated.

Magento 2 provides two primary modes of operation: Developer and Production.   Neither mode is well suited to integration.   

In Production mode every time an extension is upgraded or changes are made to javascript, css, phtml, or xml files it requires running a sequence of commands on the system in order to create static resource.

  1. Php -f bin/magento setup:upgrade
  2. Php -f bin/magento setup:static-content:deploy
  3. Php -f bin/magento setup:di:compile-multi-tenant
  4. Php -f bin/magento cache:flush
  5. Php -f bin/magento indexer:reindex

This sequence of commands commands can take anywhere from 10 to 30 minutes.  This is completely unfeasible during integration when rapid changes are being made.

Developer mode is much more flexible.  There are only 3 commands to run which take less than 5 minutes – and frequently they are not even necessary!

  1. Php -f bin/magento setup:upgrade
  2. Php -f bin/magento cache:flush
  3. Php -f bin/magento indexer:reindex

However, Developer mode has a number of drawbacks.  Page load speed is signifacantly slower than a system in production mode.  In addition, developer mode was primarily designed for the Magento developers who are making changes to Magento itself.  As such it will frequently display error messages for trivial error messages, classified as Info and Warning.  Since Magento 2 relies heavily on Ajax – displaying error messages in the Ajax results would cause them to completely fail.  One example we ran into is when using a number of third party extensions we were completely unable to update Categories and Products through the Admin when in developer mode.

In addition, we found that many third party extension developers did not test their extensions in Production mode.  Frequently we would find that a new or upgraded extension could not be deployed to our QA system which was running production mode.

The good news is that the next release of Magento 2 will be 2.1 and there is already a release candidate for 2.1!  Many of the above issues have been fixed between 2.0.1 and 2.1  Additionally, I expect that the 2.1 version will adhere to semantic versioning – which means that only when the version changes from 2.1.x to 2.2.x will there possibly be backwards compatibility issues.

Managed Hosting Provider Issues

Managed hosting providers can be a curse or a blessing.  A managed hosting provider maintains all aspects of the server and installs software as needed by their customers. At Shero, we frequently work on and prefer Nexcess dedicated systems.  Nexcess is very responsive and very quick to fix any issues.

Other managed hosting providers can provide different levels of service and experience with Magento 2. For one of our first migrations from Magento 1 to Magento 2 the client was not using Nexcess but was using a different managed hosting provider.  There were quite a number of issues we dealt with which took us time to troubleshoot – only to find the issue was due to misconfiguration of the server by the provider. This includes extremely basic issues such as getting them to upgrade PHP to version 5.6. The provider performed the initial installation of Magento 2 and despite clear requirements for PHP 5.6 or above, they installed PHP 5.4 on the server.

The hosting provider needs to know common PHP extensions that will be needed for a Magento 2 installation, despite the fact that Magento 1 requires the same extensions and they were already installed on the old system. Some hosting providers have server configurations that do not allow for the use of SSH keys. This leads to having to manually type in passwords during cloning and deployments adds a good 20 minutes per time. Next, your developer needs to have the ability and access on the server to create or edit common Linux user configuration files in the home directory such as .bash_rc .mysql_history, and .composer.  At Shero, we automate most maintenance and deployment tasks using shell scripts – not having history files adds a considerable amount of time each time we add a set of new scripts.

Regardless of the hiccups described above, Magento 2 is a solid platform that aims to be the eCommerce software of choice for the mid-market businesses who do 5M-100M of sales online. Let’s not forget that it took 8 years for Magento 1 to get where it is today. With time, patience, and hard work from the Magento team and the Magento community of solution and technology providers, Magento 2 will get there before we know it.

Front End Developer