TRLN Discovery/POD Integration Specifications

This document defines the requirements to change the data source for current integrations to TRLN Discovery and POD from Aleph to Alma.  TRLN is the Triangle Research Libraries Network and POD is the Platform for Open Discovery through IvyPlus. POD data supports both BorrowDirect and TRLN borrowing with ReShare.

Diagrams describing data storage to support this and other Alma integrations

(CI Pipline = Continuous Integration Pipeline)

TRLN Architecture Overview Diagram

This diagram was provided by TRLN staff and describes the data architecture from a TRLN perspective.

Customizations

• duke specific https://github.com/trln/marc-to-argot/tree/main/lib/marc_to_argot/macros/duke
• general https://github.com/trln/marc-to-argot/tree/main/lib/marc_to_argot/macros/shared
• questions on microform https://github.com/trln/marc-to-argot/blob/main/lib/marc_to_argot/macros/shared/physical_media.rb

more customizations https://github.com/trln/marc-to-argot/tree/main/lib/data/duke

POD code in Github

https://github.com/pod4lib/aggregator

• Extract instance, holding, and item records from Alma that aren't suppressed from discovery and aren't marked with a "delete" tag on the record.
• Extract records that are newly suppressed and newly deleted so that they can be removed from TRLN Discovery.

See the Alma Configuration Dependencies section below for detailed integration profile documentation.

• The Aleph update-only extract ran every 30 minutes. We expect the Alma updates-only extract to run every hour; this is a configuration controlled by Ex Libris and we cannot make it run more often.

• If no records are identified during the current Aleph harvest, a 0 record file is generated, so that practice should be continued for Alma.

• Historically, a set of 5000 records is extracted every hour on the 45th minute so that eventually the entire set of inventory records in Alma will be sent to downstream systems. We do not yet know if this is needed for Alma; we have not attempted to replicate this workflow yet and won't do so  until we wknow that it's needed.

• Full extract is approx 7.15 million records

This section describes the Alma configurations required to support this integration.

OAI-PMH

An Alma integration profile is used to define the specifications for publishing records via OAI-PMH.

Documentation is in progress.

Profile details

• Name: OAI-PMH for TRLN Discovery
• Profile description: OAI-PMH for TRLN Discovery
• Publishing Parameters:
  • Status: Active
  • Scheduling: Every 1 hour(s)
  • Email notifications: no address
• Content:
  • Set name: Entire repository default set
  • Additional set name: (blank)
  • Publish the entire repository: checked
  • Filter out the data using: (blank)
  • Publish on: Bibliographic level
  • Output format: MARC21 Bibliographic
• Publishing protocol
  • FTP: (not checked)
  • OAI: checked
  • Set Spec: trln_discovery_spec
  • Set Name: trln_discovery_name
  • Metadata prefix: marc21
  • Z39.50: (not checked)

Data Enrichment

• Bibliographic normalization

  • Correct the data using normalization processes: Delete tags for TRLN extract

    (needs documentation)

  • Linked data enrichment: unchecked
• Bibliographic Enrichment
  • Add management information: checked
  • Repeatable field: 942
  • Publish suppressed records as deleted: checked
  • all other fields are empty
• Related Records Enrichment
  • Add related records information: checked
  • Relation type field:  TYPE
  • Relation record MMS ID subfield: a
  • Relation type subfield: 8
  • Related fields enrichment:
    • Related tag: 245, related subfield: a, Bib tag: RELATED, Bib subfield: a, relation type All
  • Add holdings/items of related records: checked
• Authority enrichment
  • Add authority information: unchecked
• Physical Holdings Enrichment
  • Add holdings information: checked

• Exclude suppressed records: (checked)

• Physical items enrichment
  • Add items information: checked
  • Repeatable field: 940

• Electronic Inventory Enrichment
  • Planned implementation as of August 20 2024 in the 943 

• Digital Inventory Enrichment
  • Add Digital Representation Information: Unchecked
  • Add File Information: Unchecked
  • Add Remote Representation Information: Unchecked
• Collection Enrichment
  • Add Collection Information: Unchecked

This section describes any data mapping requirements for this integration. 

Note: this section needs to be updated for Alma

Alma publishing creates MARCXML records and the downstream ingest processes for TRLN and POD map data to the discovery layer and POD. These are links to the data mappings already in use in Production with Aleph. It is assumed that the new integration with Alma will continue to use these mappings:

Mapping Tables

• Duke-specific  https://github.com/trln/marc-to-argot/tree/main/lib/translation_maps/duke
• General TRLN marc to argot https://github.com/trln/marc-to-argot/tree/main/lib/translation_maps/shared
• POD Aggregator/Datalake https://github.com/pod4lib/aggregator

Additionally, this is a working spreadsheet to track any mapping changes and example records for validation: https://duke.box.com/s/3rrlpx2soe54ir587ij3j1st3wh9b3st

Mapping spreadsheet created 3/29/23 to track MARC to TRLN mapping from Aleph and FOLIO: https://duke.box.com/s/uyebv605j1lma0ex90csrpg30qlg6gys

Argot Configuration

Defaults https://github.com/trln/marc-to-argot/tree/main/lib/data/argot

List of Duke-specific Overrides

These are attributes that Duke does not want to be parsed by the default argot configuration: https://github.com/trln/marc-to-argot/blob/main/lib/data/duke/overrides.yml

Mapping Holdings Summaries from Alma to TRLN Discovery

• Formerly, Aleph output the holdings summary to the 852$a; Now, Alma outputs the institution code to the 852$a
• For Alma, we map the 866$a and 866$z to the holding summary for display, with handling for repeatable fields.
• We do not show the 866$x if present.

Inventory of MARC fields with indication of where/how each value is set (ex: yml file, macro)

To see full list, open this link to the file in Box: https://duke.box.com/s/a98wrly961091bxnj5bj3ysnslrvqo15

Record identifiers in Books & Media - Aleph-born records and Alma-born record

When migrating records from Aleph to Alma, the Aleph bib ID is transformed to become the Alma MMS ID. See Ex Libris record number documentation: link

The format of the migrated record MMS ID is 99 + Aleph bib ID + 010 + 8501

• 99 indicates the type of record (bibliographic)
• 010 is a bibliographic library identifier added by the Ex Libris migration process
• 8501 is our Alma institutional ID

For an Alma-born record, the format of the migrated MMS ID is 99 + record identifier + 8501

• 99 indicates the type of record (bibliographic)
• 8501 is our Alma institutional ID
• The record identifier is a unique identifier for the record.

Books & Media URLs (find.library.duke.edu...)

• For Aleph born records, we strip '99' and '0108501' so that the URL does not change at Alma cutover
  • E.g., for a record with Aleph bib "006288172", the Alma MMS ID is "990062881720108501", and the Books & Media URL is https://find.library.duke.edu/catalog/DUKE006288172
• For Alma born records, the MMS ID is generated when the record is created in Alma, and it becomes the identifier in Books & Media
  • E.g., a new title is ordered in Alma and it is assigned an MMS ID of "99112812262508501". The Books & Media URL is https://find.library.duke.edu/catalog/DUKE99112812262508501

Default availability value for Rubenstein and University Archives

Because of the process we are using with Alma to obtain availability data with Summon, we have run into issues disambiguating some Rubenstein and University Archives holdings so that we know which holding record in the OAI output corresponds to which Summon record.

Because we do not have a straightforward way to disambiguate (Summon APIs do not give us collection codes,) we will default to having Rubenstein and University Archives show as available in the underlying data sent to Books & Media. When they appear in search results, the circ status API will check their availability in real time and update the display to a different  value if there is one.

The purpose of this section is to describe the validation steps needed to confirm that this integration is working successfully. Since this integration replaces the data source from Aleph to Alma, we expect the end of the data pipeline to mirror what is currently flowing from Aleph.  We want to confirm that the switch from Aleph to Alma doesn't impact how data is displayed in the TRLN catalog, POD, or Summon.  

1. Is the expected volume of records moving from Alma to TRLN and POD? 
2. For various record types, is the data from Alma displaying in the TRLN catalog as expected? 
3. For various record types, is the data from Alma mapping to POD as expected?

The following types of records should be reviewed during testing. Links to specific records of each type are maintained here: https://duke.box.com/s/3rrlpx2soe54ir587ij3j1st3wh9b3st

The following library staff should be included in validation efforts once a testing infrastructure is ready to help confirm that data sourced from Alma displays in the TRLN catalog as expected:

add_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/add_update_ingests"
delete_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/delete_ingests"
pod_transfer_url="https://pod.stanford.edu/organizations/duke/uploads"

Source: https://gitlab.oit.duke.edu/aleph/home/-/blob/23_prod/home-slash-aleph/opac/call_extract_scripts.sh
The above locations would need to be changed for testing and using a test server.

This section includes historical information about the data pipeline from Aleph to TRLN Discovery for reference. 

Historical documentation - Aleph and TRLN Discovery OPAC

This wiki page documents the Aleph to TRLN Discovery integration: Historical documentation - Aleph and TRLN Discovery OPAC

Aleph to TRLN Transformations

In the data pipeline diagram for Aleph, the gray oval representing "PERL bib data extract scripts" performs several data processes:

1. Identify which bib records should be extracted from Aleph.
2. Extract MARC items and holdings records for those bib records.
3. Extract additional item data from Aleph that isn't included in MARC for each bib ("enhanced item data").
4. Create a file containing records in MARCXML format.

Aleph-based Pipeline
add_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/add_update_ingests"
delete_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/delete_ingests"
pod_transfer_url="https://pod.stanford.edu/organizations/duke/uploads"
Source: https://gitlab.oit.duke.edu/aleph/home/-/blob/23_prod/home-slashaleph/opac/call_extract_scripts.sh

Metadata Management Imp Team documents

• Aleph alphabetic and 9xx fields: https://duke.box.com/s/2x7h54k7gctbwmf7h9egi9vdn6rvpsma
• Extract criteria used for OCLC:  /wiki/spaces/LIDS/pages/34768401
• Extract criteria used for authorities: Criteria for ACVS Bibliographic Extracts

The following table describes where each data processing task occurs in the current Aleph data pipeline so that the team can validate that each task is covered in the new Alma-sourced data pipeline to TRLN. 

• The rows contain the list of data processing tasks that are occurring against Aleph data (grey column).
• The blue columns identify where each process will be handled in the new Alma data pipeline.

Historical Enhanced item data

The following data elements were extracted from Aleph to enhance what is included in MARC

• Alma Integration Project Jiras for Epic "TRLN Discovery/POD" (Jan - October 2024)
• Alma Kanban board 
• Books & Media Kanban board 

For documents that have been approved and are in a finalized state (and locked for edits), include this change log to keep track of future changes.