TRLN Discovery/POD Integration Plan
- Julie Brannon
- Erin Nettifee
Alma to TRLN/POD Data Pipeline Diagram
Diagrams describing data storage to support this and other Alma integrations
(CI Pipline = Continuous Integration Pipeline)
TRLN Architecture Overview Diagram
This diagram was provided by TRLN staff and describes the data architecture from a TRLN perspective.
For historical reference, the diagram below describes the data pipeline architecture used from Aleph to TRLN and POD.
TRLN Discovery OPAC
This wiki page documents the current Aleph to TRLN Discovery integration: TRLN Discovery OPAC
Customizations
- duke specific https://github.com/trln/marc-to-argot/tree/main/lib/marc_to_argot/macros/duke
- general https://github.com/trln/marc-to-argot/tree/main/lib/marc_to_argot/macros/shared
- questions on microform https://github.com/trln/marc-to-argot/blob/main/lib/marc_to_argot/macros/shared/physical_media.rb
more customizations https://github.com/trln/marc-to-argot/tree/main/lib/data/duke
Aleph-based Pipeline
add_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/add_update_ingests"
delete_transfer_url="https://trln-discovery-ingest.cloud.duke.edu/delete_ingests"
pod_transfer_url="https://pod.stanford.edu/organizations/duke/uploads"
Source: https://gitlab.oit.duke.edu/aleph/home/-/blob/23_prod/home-slashaleph/opac/call_extract_scripts.sh
Metadata Management Imp Team documents
- Aleph alphabetic and 9xx fields: https://duke.box.com/s/2x7h54k7gctbwmf7h9egi9vdn6rvpsma
- Extract criteria used for OCLC: /wiki/spaces/LIDS/pages/34768401
- Extract criteria used for authorities: Criteria for ACVS Bibliographic Extracts
POD code in Github
Aleph to TRLN Transformations
In the data pipeline diagram below, the gray oval representing "PERL bib data extract scripts" performs several data processes:- Identify which bib records should be extracted from Aleph.
- Extract MARC items and holdings records for those bib records.
- Extract additional item data from Aleph that isn't included in MARC for each bib ("enhanced item data").
- Create a file containing records in MARCXML format.
The following table describes where each data processing task occurs in the current Aleph data pipeline so that the team can validate that each task is covered in the new Alma-sourced data pipeline to TRLN.
- The rows contain the list of data processing tasks that are occurring against Aleph data (grey column).
- The blue columns identify where each process will be handled in the new Alma data pipeline.
| # | Data processing in the existing Aleph to TRLN data pipeline | New Alma data pipeline - where will each process be handled? | |||||
|---|---|---|---|---|---|---|---|
| No action required | Migration of Aleph records to Alma | Alma Integration Profile | Alma to TRLN scripts | TRLN Discovery App | Other/notes | ||
| 1 | Non-unicode character cleanup | ✓ | |||||
| 2 | Aleph records are excluded from the current TRLN/POD extract based on various criteria such as:
NEED TO REVIEW THIS (Alma doesn't support suppress items) | ✓ | These records are loaded to Alma and tagged with "suppressed from discovery" flag so that they'll be excluded from Alma publishing. Note from Julie - confirm that this is true for Alma | ||||
| 3 | Drop specified local use fields such as the 029 - see notes for link to full list | ✓ | The data elements listed in the Aleph Alphabetic and 9xx Fields.xlsx spreadsheet as "Do not migrate" are not migrated on records loaded to Alma. confirm that this is true for Alma To do: Review the current Aleph to TRLN scripts to confirm that everything that is currently dropped is covered on this spreadsheet to confirm how that was handled in the Alma migration | ||||
| 4 | Identify which bib records should be extracted: Updates-only during 30 min window, skip "suppressed records" | ✓ | Follow-up with Matt/Jeff/Ayse to see if TRLN is using the same criteria as current MADS criteria for determining which records are pulled. | ||||
| 5 | Extract MARC items and holdings records for each bib record that meets the extract criteria | ✓ | |||||
| 6 | Extract additional item data that isn't included in MARC for each bib, such as location and availability | ✓ | |||||
| 7 | Extract an additional 5000 records every hour so that the entire dataset is eventually refreshed in downstream systems incrementally. | ✓ | This step enables the entire dataset to be refreshed incrementally. This was originally done as a compensatory maintenance measure for Endeca. We're not sure if it will need to be done for Alma, but are tracking it here for verification either way. | ||||
| 8 | Convert extracted data to MARCXML format | ✓ | |||||
| "New" data processing for Alma data (these processes weren't needed for Aleph data) | No action required | Migration of Aleph records to Alma | Alma Integration Profile | Alma to TRLN scripts | TRLN Discovery App | Other/notes | |
| 9 | Declare namespace at the beginning of MARC XML output, needed for Marc-to-Argot | ✓ | |||||
| 10 | Specify record type as Bibliographic in XML | ✓ | |||||
| 11 | Remove marc namespace prefixes - this was needed for FOLIO, is it needed for Alma? | ✓ | |||||
| 12 | Remove unnecessary metadata node to XML | ✓ | |||||
| 13 | Add collection node to XML | ✓ | |||||
| End of "New" data processing for Alma data | No action required | Migration of Aleph records to Alma | Alma Integration Profile | Alma to TRLN scripts | TRLN Discovery App | Other/notes | |
| 14 | Run Marc-to-Argot based off of Duke specific overrides | ✓ | Does this need to be updated for Alma? The MARC 001 field control number needs to be transformed with string 'DUKE' as prefix and first 4 digits stripped, so a FOLIO 001 containing 'in00009142400' needs tranformation to 'DUKE009142400'. Currently handled in the marc to argot scripts (added 3/29/23) | ||||
| 15 | Move data via Spofford app | ✓ | |||||
Enhanced item data
The following data elements are extracted from Aleph to enhance what is included in MARC
| Aleph Oracle Table | Data element | Description (from Aleph documentation) |
|---|---|---|
| Z30 | sublibrary | Code of the sublibrary that “owns” the item. |
| Z30 | collection | Collection code of the item |
| Z30 | call_no_type | This defines the type of location assigned in the Z30-CALL-NO field |
| Z30 | call_no | Shelving location of the item |
| Z30 | description | Description of the item (for multi-volume monographs or serial items) to help the user identify the item they are interested in. For items created through the Serials function, this field is automatically set to the enumeration and chronology description of the issue. |
| Z30 | note_opac | Note field. This note is displayed in the OPAC |
| Z30 | item_status | Status of the copy for loan purposes. |
| Z30 | item_process_status | Defines the item's processing status (e.g. on order, cancelled, binding, etc.). |
| Z30 | material | Material type. This can be VIDEO, BOOK, ISSUE, etc. "ISSUE" has special functionality within the system, all other types are used in an equal manner. |
| Z30 | barcode | Unique identifier of the item. |
| Z30 | open_date | Creation date of the copy. |
| Z30 | update_date | Date copy was last updated |
| Z30 | date_last_return | Date copy was last updated |
| Z30 | no_loans | Number of times the item was loaned. |
| Z30 | hol_doc_number_x | System number of the holdings record to which the item is attached. |
| Z30 | temp_location | This is a toggle field used to control the overriding of the location fields by the holdings record. Values are Y and N. |
| Z36 | due_date | Active due date (see also ORIGINAL-DUE-DATE and RECALLDUE-DATE). This field changes according to loan transactions (e.g. renew loan). |
| Z103 | lkr_type | Defines the type of link. ADM = Link from ADM record to BIB record. Link is build from BIB to ADM. ITM = Link between a BIB record and the items of another BIB record. |
| Z37 | status | Hold request status. |
| Z37 | end_request_date | Last date of interest for the hold request. |
| Z36 | id | Patron‟s ID |
| Z36 | recall_due_date | Due date computed as a result of the recall transaction. The date might actually be later than the due date, if the recall was generated close to the end of the loan period. It is retained for computing fine owing for late return of recalled item. |
| Z30 | rec_key 1,9 | System number of the administrative record (ADM) associated to the item. |
Need to update for Alma?
FOLIO to TRLN Transformations
LOC Local/Obsolete Data Elements Present in Harvest- 261
- 262
- 400
- 410
- 440
Local Fields Present in Harvest
- 049, 092
- 690, 692, 693, 694, 695, 697, 698
- 901, 902, 903, 904, 905, 909
- 910, 914, 915, 916
- 935, 938
- 943, 948, 949
- 951, 952, 955, 956
- 980, 981, 987
- 994, 996, 998, 999
To Do: Validate for Alma FOLIO includes the following location and call number information in the OAI-PMH harvest using this mapping:
| FOLIO Inventory app field | MARC | Additional rules |
|---|---|---|
| Effective location: Institution | 952$a | |
| Effective location: Campus | 952$b | |
| Effective location: Library | 952$c | |
| Effective location: Name | 952$d | |
| Effective call number components: call number | 952$e | |
| Effective call number components: prefix | 952$f | |
| Effective call number components: suffix | 952$g | |
| Effective call number components: type | 952$h | |
| Material type | 952$i | |
| Volume | 952$j | |
| Enumeration | 952$k | |
| Chronology | 952$l | |
| Barcode | 952$m | |
| Copy number | 952$n | |
| Electronic access: URI | 856$u | Create separate datafield for each URL. Use URLs from each record level (holding and item) |
| Electronic access: Link text | 856$y | Create separate datafield for each URL. Use URLs from each record level (holding and item) |
| Electronic access: Materials specified | 856$3 | Create separate datafield for each URL. Use URLs from each record level (holding and item) |
| Electronic access: Public note | 856$z | Create separate datafield for each URL. Use URLs from each record level (holding and item) |
| Electronic access: Relationship: No display constant generated | 856 2nd indicator 8 1st indicator 4 | |
| Electronic access: Relationship: No information provided | 856 2nd indicator blank 1st indicator 4 | Such indicator filling in works also for empty "Relationship" value |
| Electronic access: Relationship: Related resource | 856 2nd indicator 2 1st indicator 4 | |
| Electronic access: Relationship: Resource | 856 2nd indicator 0 1st indicator 4 | |
| Electronic access: Relationship: Version of resource | 856 2nd indicator 1 1st indicator 4 | |
| Electronic access: Relationship: empty value | 856 2nd indicator empty 1st indicator 4 |
Transformations from FOLIO output to match TRLN input
ruby -pi.bak -e "gsub(/<metadata>/, '')" file.xml # this finds a string and replaces it ruby -pi.bak -e "gsub(/<\/metadata>/, '')" file.xml sed -i -f - file.xml < <(sed 's/^/1i/' xml_header.txt) # insert the contents of the file at the front ruby -pi.bak -e "gsub(/marc:/, '')" file.xml ruby -pi.bak -e "gsub(/<record>/, '<record type=\"Bibliographic\">')" file.xml echo -n '</collection>' >> file.xml # insert at the end of the file
<?xml version="1.0" encoding="utf-8"?> <collection xmlns="http://www.loc.gov/MARC21/slim" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd">
Argot Configuration
Defaults https://github.com/trln/marc-to-argot/tree/main/lib/data/argot
List of Duke-specific Overrides
These are attributes that Duke does not want to be parsed by the default argot configuration: https://github.com/trln/marc-to-argot/blob/main/lib/data/duke/overrides.yml
| id |
| local_id |
| rollup_id |
| oclc_number |
| institution |
| items |
| holdings |
| names |
| url |
| access_type |
| date_cataloged |
| primary_oclc |
| physical_media |
Mapping Holdings Summaries from Alma to TRLN Discovery
Aleph output the holdings summary to the 852$a; this is wrong in Alma, which outputs the institution code to the 852$a
For Alma, we map the 866$a and 866$z to the holding summary for display, with handling for repeatable fields.
We do not show the 866$x if present.
Inventory of MARC fields with indication of where/how each value is set (ex: yml file, macro)
To see full list, open this link to the file in Box: https://duke.box.com/s/a98wrly961091bxnj5bj3ysnslrvqo15
Record identifiers in Books & Media - Aleph-born records and Alma-born record
When migrating records from Aleph to Alma, the Aleph bib ID is transformed to become the Alma MMS ID. See Ex Libris record number documentation: link
The format of the migrated record MMS ID is 99 + Aleph bib ID + 010 + 8501
- 99 indicates the type of record (bibliographic)
- 010 is a bibliographic library identifier added by the Ex Libris migration process
- 8501 is our Alma institutional ID
For an Alma-born record, the format of the migrated MMS ID is 99 + record identifier + 8501
- 99 indicates the type of record (bibliographic)
- 8501 is our Alma institutional ID
- The record identifier is a unique identifier for the record.
Books & Media URLs (find.library.duke.edu...)
- For Aleph born records, we strip '99' and '0108501' so that the URL does not change at Alma cutover
- E.g., for a record with Aleph bib "006288172", the Alma MMS ID is "990062881720108501", and the Books & Media URL is https://find.library.duke.edu/catalog/DUKE006288172
- For Alma born records, the MMS ID is generated when the record is created in Alma, and it becomes the identifier in Books & Media
- E.g., a new title is ordered in Alma and it is assigned an MMS ID of "99112812262508501". The Books & Media URL is https://find.library.duke.edu/catalog/DUKE99112812262508501
Default availability value for Rubenstein and University Archives
Because of the process we are using with Alma to obtain availability data with Summon, we have run into issues disambiguating some Rubenstein and University Archives holdings so that we know which holding record in the OAI output corresponds to which Summon record.
Because we do not have a straightforward way to disambiguate (Summon APIs do not give us collection codes,) we will default to having Rubenstein and University Archives show as available in the underlying data sent to Books & Media. When they appear in search results, the circ status API will check their availability in real time and update the display to a different value if there is one.