Organizations Data Mapping Guide: FOLIO to Alma

Purpose:

This page describes data mapping from the FOLIO Organizations app to Alma.  During the Alma implementation phase beginning in the fall of 2023, these pages serve as working documents for decision-making about data migration.  Once the migration to Alma is complete, these guides will be retained for reference.


Overview

The ExLibris automated migration extracts vendor data only once during our Alma implementation project.  Aleph vendors will be extracted on 12/21/2023 and will not be re-loaded to Alma by the ExLibris team before we go live on Alma in July, 2024.  We've been told by ExLibris staff that this approach is used to minimize the risk of potential data loss if we enhance our vendors in Alma between December and June to add information that isn't available for migration from Aleph.  This page describes the FOLIO organization data migration approach.

 Vendor Data Migration to Alma in December 2023

FOLIO Non-Aleph organizations extract process 12/15/23


  1. Use Postman to Get all FOLIO notes https://duke-okapi.folio.indexdata.com/notes?limit=1000
    • Convert results file from JSON to .xlsx using online converter
    • Import notes .xlsx into Access database
  2. Use Postman to Get all FOLIO organizations https://duke-okapi.folio.indexdata.com/organizations/organizations?limit=1000
    • Convert results file from JSON to .xlsx using online converter
    • Import notes .xlsx into Access database
  3. Use Postman to Get all FOLIO organization categories https://duke-okapi.folio.indexdata.com/organizations-storage/categories?limit=1000
    • Convert results file from JSON to .xlsx using online converter
    • Import notes .xlsx into Access database
  4. Run the Access DB query that selects only the organizations that contain “No aleph vendor code” string and joins to the notes. Use the Duke_FOLIO_Generic Field Mapping_Vendors.xls to determine which fields to display in the query.
  5. Verify the query result data set against FOLIO by spot checking a few records
  6. Export the query result into .xlsx
  7. Review the results and edit any odd-looking data in the notes and description fields (carriage returns etc). For some reason the notes truncated, so I had to copy/paste some directly from FOLIO. Removed parentheses and + from phone numbers.
  8. Review the column names and make them match what’s in the the Duke_FOLIO_Generic Field Mapping_Vendors.xls layout.
  9. Upload to the Basecamp message board by noon on 12/18


Most other types of data from Aleph and FOLIO will be re-extracted during the cutover phase which begins June 16th, 2024 but vendors are not re-extracted/migrated. This one-time data migration approach creates a data integrity risk since any changes made to vendor records in Aleph after 12/21/23 won't get updated automatically in Alma.  Staff are reluctant to manually maintain vendor data in both systems during the 6 months before implementation due to the volume of vendor adds and changes expected during that time.

To minimize the impact on staff, a DST project has been defined to describe the need for a custom script to extract Aleph vendor adds/updates and migrate them to Alma via the Alma vendor API. This update will be run as we get closer to implementation, likely just before the cutover in mid-June. 


Aleph Vendor Adds/Updates to Alma by June 2024 Cutover


The DST vendor data update project from Aleph would not include add/updates to organizations in FOLIO. There is a much smaller volume of records in FOLIO than in Aleph and DST resources likely wouldn't be available to develop a script for those updates. 

Electronic Resources & Serials Acquisitions (ERSA) staff in DUL Collections Services use FOLIO organization records to:

  • Describe license relationships
  • Manage electronic resource provider admin interface URLs and credentials
  • Manage contact people information

We'd like to migrate as much of that information to Alma as possible.  As of 12/8/2023 there are 577 organization records in FOLIO and approximately 180 of them describe providers that don't have a corresponding record in Aleph, what we call non-Aleph organizations.  We extracted those ~180 non-Aleph organizations from FOLIO on 12/15/23 and ExLibris is loading them to our Alma environment as vendor records with a type of "Licensor" that doesn't require account/payment information.


See Procedure for management of FOLIO organizations after 12/15/23 for more information about how changes should be handled from Dec 15, 2023 until June 2024.



Migration Plan for Non-Aleph vendor FOLIO Organizations

Since this is a one-time only extract process for 177 records, we used a light-weight solution to generate a file that conforms to the .csv format required by ExLibris. 

Reference: The Duke_FOLIO_Generic Field Mapping_Vendors.xls file defines the file format requirements from ExLibris and contains our mapping of fields from FOLIO to Alma.

  1. Analyze the organization records to determine whether it's clear which ones contain an Aleph vendor code and which don't.  Worked with Virginia Martin and Abby Wickes to edit a few records that either didn't contain any text to indicate whether an appropriateAleph vendor code exists or referenced several.
  2. Validate that the Aleph vendor codes that have been entered in the alias value field match those in Aleph.  (Extracted all Aleph vendor codes to compare to those extracted from FOLIO).
  3. Use Postman to Get all FOLIO notes https://duke-okapi.folio.indexdata.com/notes?limit=1000
    • Convert results file from JSON to .xlsx using online converter tool
    • Import notes .xlsx into Access database
  4. Use Postman to Get all FOLIO organizations https://duke-okapi.folio.indexdata.com/organizations/organizations?limit=1000
    • Convert results file from JSON to .xlsx using online converter
    • Import organizations.xlsx into Access database
  5. Use Postman to Get all FOLIO organization categories https://duke-okapi.folio.indexdata.com/organizations-storage/categories?limit=1000
    • Convert results file from JSON to .xlsx using online converter
    • Import notes .xlsx into Access database
  6. Create an Access table to handle data transformation of FOLIO language code value from 3 digits to 2 digits)
  7. Run an Access DB query that selects only the organizations that contain “No aleph vendor code” string in any of the 7 alias value or descriptions columns.  Set up the query to join the organization table to the notes, organization categories, and language conversion tables. Use the Duke_FOLIO_Generic Field Mapping_Vendors.xls to determine which fields to display in the query output.
  8. Verify the query result data set against FOLIO by spot-checking a few records
  9. Export the query result into .xlsx
  10. Review the results and edit any odd-looking data in the notes and description fields (carriage returns etc). For some reason, the notes were truncated, so copy/paste some directly from FOLIO as needed. Remove parentheses and "+" from phone numbers.
  11. Review the column names to match what’s in the the Duke_FOLIO_Generic Field Mapping_Vendors.xls layout.
  12. Upload to the ExLibris Basecamp message board.



Migration plan for all FOLIO Organizations Contact People and Interfaces 

We've defined a DST project to use the Alma vendor API to update the vendor records that have been created using migrated vendor data from Aleph and organization data from FOLIO for the non-Aleph organizations.  We plan to enhance those records with the additional data elements that have been captured in FOLIO for which there isn't a data element available in Aleph such as:

  • Admin interface URLs and credentials
  • Contact people information
  • Alias/alternate names - we might be able to add these to notes (there aren't dedicated fields for alternate names in Alma that we can map to directly)

FOLIO Contact People and Interfaces migrate to Alma by June 2024 Cutover