Developer Debugging

There are 2 options for debugging; core debugger and the Migration Devel

Add General Debugging

Full tutorial How to Print Variables using Kint in Drupal 8

$ chmod 755 /var/www/drupal/web/sites/default
$ drush -y en devel
$ drush -y en kint

# Adding the debug service
$ cp /var/www/drupal/web/sites/default/default.services.yml /var/www/drupal/web/sites/default/services.yml
        $ Change debug value
        debug:true

# Adding the debug to the template
$ nano /var/www/drupal/web/themes/contrib/carapace/templates/generated/page.html.twig
    Add under the first comment, above the first item:    {{ kint(page) }}

Install Migrate Devel

This gives a lot of feedback on a migration failure.

$ cp /var/www/drupal/
$ composer require 'drupal/migrate_devel:^2.0'
$ drush en -y    migrate_devel

When you modify a migration YML file

This messes up me all the time. So much so I created a script I just have running in the background to force all updates after they’re written. This isn’t absolutely nessisary but I find it useful.

$ drush -y fim migrate_islandora_csv

Typical Migration

It’s highly suggested to migrate by group so the order of operations work in your favor. This is a typic execution of that.

Remember that the order of operations is important (taxonomy terms like persons and subjects should go first, files, node, and lastly media) if you haven’t previously imported them. This is a problem if the dependencies are properly specified or the --execute-dependencies isn’t added to the import. The -d is for debugging. This will add a little time to the process. And you will need to include --userid=1 to this. I have experienced issues not using it.

drush migrate:import --userid=1 --execute-dependencies --group=knoxgardens_migrate_islandora_csv -d

Debugging Migration

Migrate:import has a lot of options. This is the complete list. I won’t go over most of them. To read more on drush migrate-import

migrate:import [--all] [--group GROUP] [--tag TAG] [--limit LIMIT] [--feedback FEEDBACK] [--idlist IDLIST] [--idlist-delimiter [IDLIST-DELIMITER]] [--update] [--force] [--continue-on-failure] [--execute-dependencies] [--skip-progress-bar] [--sync] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--userid USERID] [--migrate-debug [MIGRATE-DEBUG]] [--migrate-debug-pre [MIGRATE-DEBUG-PRE]] [--druplicon] [--notify [NOTIFY]] [--xh-link XH-LINK] [--] <command> [<migration_names>]

Migration test are done one migration yml at a time by specifying the migration ID. To get a list of installed migrations with the status and group name(s) run migrate status:

$ drush migrate:status
 "-----------------------------------------------------------------------------------"
  Group         Migration ID   Status Total  Imported  Unprocessed   Last Imported
  ----------------------------    ------------   ----  ------  -----  --  -----------
  pcard00_migrate_islandora_csv   pcard00_file   Idle   4224   6989   0   2020-07-10
  pcard00_migrate_islandora_csv   pcard00_node   Idle   4224   6991   0   2020-07-10
  pcard00_migrate_islandora_csv   pcard00_media  Idle   4224   6990   0   2020-07-10
 "-----------------------------------------------------------------------------------"

To only run against a specific object. Within each of the yml files a key is specified. The key is saying for each unique one of these iterate through. If the the key is to to key: PID and the PID we want to migrate/update is islandora:4223 we can the following command. Using a value with special characters gives me some concern so I suggest just adding sequential numbers as the first colunm in the CSV calling it ID to avoid possible issues. I have not personally tested using a PID as the key but I’m using it here to illustrate.

#         id of the yml file   ┐                 ┌ Respect the order of operations
$ drush migrate:import pcard00_node --execute-dependencies --migrate-debug --idlist="islandora:4223" --userid=1

This outputs a Source and a Destination variables to see how they are mapping. The Source is good to see if the values are being read. This is the Destination part of the report.

...
...
^ "---------------------------------------------------------------------"
^ "|                           $Destination                            |"
^ "---------------------------------------------------------------------"
^ array:8 [
  "circuit_breaker" => "Page"
  "name" => "Night View of Gatlinburg, Tennessee Showing Mount LeConte in the Background (GS-262)"
  "uid" => 1
  "field_media_use" => "17"
  "field_media_file" => "3074"
  "field_pid" => "pcard00:108573"
  "field_media_of" => "7095"
  "bundle" => "image"
]
^ "---------------------------------------------------------------------"
^ "|                       $DestinationIdValues                        |"
^ "---------------------------------------------------------------------"

Rolling back

This process has a small flaw. The original file is left in the Bigdata log and fills up the VM very quickly. It will eventually remove itself but I haven’t found a simple way to trigger a cron to clear it. The process can be done quickly but there is probably a fast way than I have been doing it. I suggest only rolling back when you have extra files or to remove remove the content entirely. If you plan on just updating, just use the --update switch.

$ drush migrate:rollback --userid=1 --group=knoxgardens_migrate_islandora_csv -d
# VS
$ drush migrate:import --userid=1 --group=knoxgardens_migrate_islandora_csv -d --update

Caches

# Clear Drupal cache
$ drush cr

# Karaf Cache
rm -rf /opt/karaf/data/cache/

# Migration cache. The migration caches .yml files. If you already ran a migration and updated or added .yml files, you need to rebuild the Drupal 8 cache. Otherwise, Drupal will ignore the changes in the file system.

$ drush sql:query "TRUNCATE druid_mods_file_migration_settings"

Extra

#  after running `make` pull all of the latest docker images
$ docker images | grep -v REPOSITORY | awk '{print $1}' | xargs -L1 docker pull

Deleting a migration (rare but it happens). If you’ve previously created a file like druid_mods_file_migration.yml and ran it. That configuration was imported into Drupal’s config. Deleting the module file will not remove it from Drupal’s config. You either need to overwrite it or delete it all together.

$ drush config:delete your-migration-file-name-without-extension

# example
$ drush config:delete druid_mods_file_migration

NOTES

Althought there are really straight forward migration tutorials like this one. All of the migration content on this and previous pages have been done without leveraging the more powerful part of a Drupal migration, custom PHP code to handle advanced beahviors. Here is an example of how to do this.