Github

Ingesting/Migrating

During ingests/migrations there are several items that have been covered by different sections of this document. It helps to have a working knowledge of linked data, RDF, and ontology & vocabulary basics vs repository-specific ontologies to create a comprehensive migration/ingest. Names, files, metadata along with the DOI is important to have a firm understanding of how this information is related to process it correctly.

  • Example of Islandora 7x to 8x for the same object after migrated.
    side by side Islandora versions of object display

Batch Processing/Migrating

Ingesting Files according to the documentation the Migrate API is the main way to ingest batches of data into Drupal (and because Islandora 8 is Drupal, into Islandora). The Migrate module only provides the framework, it’s up to you to create the rules that take data from a source, through a process (i.e. a mapping) to a destination. A set of these rules is called a “migration”. It has to be set up (as a Configuration Entity, either by importing a YAML file or by installing a Feature) and then it has to be ran.

In a nutshell the steps are the following:

  1. Create a CSV file.
  2. Create a yml for each type of entity (person, place, event, university, etc) needing to mint.
  3. Create a “file” yml configuration file for the files.
  4. Create a “media” yml configuration file for the files to specify their media behavior once in Drupal.
  5. Create a “node” yml configuration file for gluing the file and media to a node. It the Islandora 7 terminology this would be the “object”.

Screenshot of Migrate UI

Although this process can be done in the admin UI, it’s highly suggested to use drush commands. The UI doesn’t respect order of operations and very little information is given from the admin UI when there is an error. $ drush migrate:status will display a list of each group and the individual migration files with their current status. This example show 2 migrations completed.

migrate status in the terminal

Running the $ drush migrate:import command with the group will cause it to run the individual migrations in order as show below.

Terminal results from collection import

Important Notes

Dates

It’s highly recomended by the community to convert dates to EDTF. EDTF field type is for recording dates in Extended Date Time Format, which is a format based off of the hyphenated form of ISO 8601 (e.g. 1991-02-03 or 1991-02-03T10:00:00), but also allows expressions of different granularity and uncertainty.

Metadata edtf valid form

This doesn’t have to be done on the original metadata (MODS), it can be done in the transform process of the migration. The configuration for the Default EDTF Widget (including options for extra strict date validation, and allowing date intervals and date sets).

EDTF formatter settings

Getting Metadata into Fedora and a Triple-store

The above sections described how Drupal manages and stores metadata, but Islandora 8 provides for pushing that metadata into a Fedora 4+ repository and a triple-store. Islandora does this by using Drupal’s serialization capabilities to provide a JSON-LD serialization that can be ingested by Fedora 4+ repository and triple-stores. In response to actions taken as a result of if-then configurations in Contexts, Islandora sends notifications to the repository and triple-store that an entity available to ingest.

The JSON-LD module (written for Islandora) takes an entity and its corresponding RDF mapping (a structure defined by the contributed RDF module) to create a JSON-LD serialization. An RDF mapping for a content type, media type, or vocabulary lists its fields and the RDF predicates that should be used for them.

The JSON-LD serialization for an entity is available by appending ?_format=jsonld to the entity’s URL. This particular syntax takes advantage of the REST API module. Below is an example JSONLD document representing the RDF serialization of a Repository item node created in a standard islandora-playbook based vagrant VM:

{
  "@graph":[
    {
      "@id":"http://localhost:8000/node/1",
      "@type":[
        "http://pcdm.org/models#Object"
      ],
      "http://purl.org/dc/terms/title":[
        {
          "@value":"New York, New York. A large lobster brought in by the New England fishing boat [Fulton Fish Market]",
          "@language":"en"
        }
      ],
      "http://schema.org/author":[
        {
          "@id":"http://localhost:8000/user/1"
        }
      ],
      "http://schema.org/dateCreated":[
        {
          "@value":"2019-03-14T19:05:24+00:00",
          "@type":"http://www.w3.org/2001/XMLSchema#dateTime"
        }
      ],
      "http://schema.org/dateModified":[
        {
          "@value":"2019-03-14T19:20:51+00:00",
          "@type":"http://www.w3.org/2001/XMLSchema#dateTime"
        }
      ],
      "http://purl.org/dc/terms/date":[
        {
          "@value":"1943-05",
          "@type":"http://www.w3.org/2001/XMLSchema#string"
        },
        {
          "@value":"1943-05",
          "@type":"http://www.w3.org/2001/XMLSchema#gYearMonth"
        }
      ],
      "http://purl.org/dc/terms/extent":[
        {
          "@value":"1 negative",
          "@type":"http://www.w3.org/2001/XMLSchema#string"
        }
      ],
      "http://purl.org/dc/terms/identifier":[
        {
          "@value":"D 630714",
          "@type":"http://www.w3.org/2001/XMLSchema#string"
        }
      ],
      "http://purl.org/dc/terms/type":[
        {
          "@id":"http://localhost:8000/taxonomy/term/11"
        }
      ],
      "http://purl.org/dc/terms/rights":[
        {
          "@value":"No known restrictions. For information, see U.S. Farm Security Administration/Office of War Information Black & White Photographs(http://www.loc.gov/rr/print/res/071_fsab.html)",
          "@type":"http://www.w3.org/2001/XMLSchema#string"
        }
      ],
      "http://purl.org/dc/terms/subject":[
        {
          "@id":"http://localhost:8000/taxonomy/term/26"
        }
      ],
      "http://schema.org/sameAs":[
        {
          "@value":"http://localhost:8000/node/1"
        }
      ]
    },
    {
      "@id":"http://localhost:8000/user/1",
      "@type":[
        "http://schema.org/Person"
      ]
    },
    {
      "@id":"http://localhost:8000/taxonomy/term/11",
      "@type":[
        "http://schema.org/Thing"
      ]
    },
    {
      "@id":"http://localhost:8000/taxonomy/term/26",
      "@type":[
        "http://schema.org/Thing"
      ]
    }
  ]
}

Please see Metadata and metadata sharing for more information on this topic.

Maintaining URI reference & relationships

Migrating the URI associated with a person, place, event, etc. is an essential requirement for UTK. This process will demonstrate how to retain the URIs we have when we migrate and support complex graph relationships. To take full advantage of this, the taxonomy vocabularies (person, geographic location, subject, event, etc.) needs to be migrated or created first. Once the taxonomy term is migrated, the relationship needs to be specified on the node along with the previously created taxonomy reference URI. Islandora Defaults provides a ‘Linked Agent’ field as part of the Repository Item content type, and populates the available relationships from the MARC relators list. By utilizing this, the Entity (for example in this illustration is the person term Van Vactor) and his MARC relator is Composer.

vanvactor_example

The list of available relations for this field is configurable at ‘/admin/structure/types/manage/islandora_object/fields/node.islandora_object.field_linked_agent’. If you re-use this existing field on another content type, you can define different available relations for that instance of the field. Relations are defined in the format key|value, and the key is used in the RDF mapping (see below).

key|value RDF mapping

Unlike other fields, which can be assigned RDF predicates in RDF Mapping yaml files, a typed relation field uses a different predicate depending on the chosen “type”. These predicates are assigned using the ‘keys’ in the key|value configuration. The key must be formated namespace:predicate, e.g. relators:act. The namespace (‘relators’) must be pre-defined in code, using a hook. See the RDF Mapping section of Create / Update a Content Type for more details, including a list of available pre-defined namespaces.

For more information see Metadata and Metadata Sharing #Typed-Relation and Islandora’s Metadata Documentation

Single Objects (using the UI)

Before you start ingesting identify all of the entities (taxonomy vocabulary term reference) that needs to be created first (person, events, geographic locations, subjects, etc.) if it doesn’t already exist. You can create them in paralelle while filling out the ingest form. To create a new term go the the category it falls into (person, events, geographic locations, subjects, etc.) and click “List Terms” then “Add Term”.

{
  "@graph": [{
    "@id": "http://www.utkislandora8testing.com/taxonomy/term/34",
    "@type": [
      "http://purl.org/dc/terms/Location",
      "http://schema.org/Place"
    ],
    "http://schema.org/name": [{
      "@value": "Great Smoky Mountains",
      "@language": "en"
    }],
    "http://schema.org/dateModified": [{
      "@value": "2020-07-06T20:02:57+00:00",
      "@type": "http://www.w3.org/2001/XMLSchema#dateTime"
    }],
    "http://schema.org/sameAs": [{
      "@id": "http://sws.geonames.org/4626066"
    }]
  }]
}

For all other information on migration see Islandora 8 Migration Documentation