2024/01/25
We now have access to our Alma instance with our Duke Data loaded. The LSPSG has been working to validate that all the data we expected to load into Alma is in fact there. We’ve found some quirks and some clean-up projects, but all in all, our data is there and usable. We will continue to check the data until February 1. Until then, we’ve been asked to not change any of the data. We’re finding that Alma Analytics and the creation of sets of data using the Alma searching options are very helpful.
Coming up, we’ll be kicking off the agile development team on January 29 to work on the integrations for Alma that need developers to do. Those projects are being tracked in the DULDEV Atlassian instance, and in the Basecamp site Ex Libris uses for our implementation. The integration projects include (but are not limited to) publishing our data to TRLN Discovery, importing users from Identity Management, sending fines to the Bursar, creating Invoices and sending them to Accounts Payable, setting up SpineOMatic for label printing, sending bib and holdings data along with requests to CaiaSoft, and projects like that. We’ll also be configuring Z39.50 access, ReShare access, and the data exports we send to OCLC, ShareVDE, HathiTrust.
We’ve scheduled training with Ex Libris for our Duke Early Users and Trainers for the week of February 5-8. There will be morning and afternoon sessions of about 2.5 hours each. Each of the sessions has a detailed agenda of topics to cover, and is focused either toward Tech Services functionality or Circulation functionality. The Early Users will then work in our Alma instance to learn workflows, test permission sets, circulation rules as we add them. The Trainers will be working in their subject areas to review Ex Libris training resources as well as the training resources our peer institutions have shared to see what could be shared for our local training. If local documentation or training material needs to be created, the trainers will work on that as well.
2024/01/05
This week, the Alma Configuration Form, the logos for Duke’s instance, and the form to configure authentication were submitted!
2023/11/29
The first extract of Aleph data was completed on November 22. The process took 21 hours to run, and Ex Libris has transferred the data to their servers for review. We have about three weeks before the extract will be run again to fill our Alma instance with Duke's test data. We should have access to that instance in mid-January.
Coming up next week is the Kickoff for the Configuration and Integrations phase with the Ex Libris Implementation Team. We will have a few weeks to do a first pass on the Alma Configuration Form. On this form, we'll list data such as our sublibraries, location codes, patron statuses, circulation policies (loans/requests/fees/fines), circulation desks, ordering units, vendors, etc. Some of this data will be pre-populated based on the test extract that was run last week. We'll review the pre-populated information, add some more, and submit a draft of this form to have the information included in our test instance in January. we will be able to continue making changes to the configurations over the next few months. We'll submit our changes, Ex Libris will do the configuration, and we'll verify the outcome. Once we go live, our local staff will be able to change configurations.
2023/11/09
This week, we kicked off the Alma Café Series with your barista, Julie Brannon, and an overview of the Alma interface. Thank you to all who attended and made this series kickoff successful. If you missed it, you can view the recording on WarpWire (NetID login required) and the slides on Box. The next Alma Café will be on Wednesday November 15 at 9:00. The topic will be "Intro to Acquisitions". Your barista will again be Julie Brannon.
In other project work, we are finalizing migration mappings with our Ex Libris partners. We have exported Courses data from FOLIO and are working on exporting data from the Licenses app this week. Ex Libris ran a small extract of bibliographic data from Aleph so that we could see how LKR data for bound-withs would be imported to Alma so we could think about whether any data clean-up should happen in Aleph or if it would be easier to wait until after our final export is completed. Alma's "sets" feature is going to be really handy as we look at our data projects post go-live.
The LSP Steering Group also began talking about staff training and putting together a plan to make sure everyone has the training and documentation to learn how to do their work in Alma. We want to make sure everyone is confident with their new workflows before go-live. More to come on the training plan!
Finally, we are beginning the transition from working with the Ex Libris Alma Orientation and Migration teams to the Implementation Team. The Implementation team will work with us on our permission structure, our fulfillment rules, and the integrations.
2023/10/04
Check out the First Wednesday presentation!
Alma Implementation - First Wednesday 20231004.pptx
2023/09/18
...
Ex Libris has prepared "cards" (like Trello) for integrations they have implemented with Alma on a regular basis. We have added cards for Duke-specific integrations such as my.duke.edu, and My Accounts. We've sorted them into ones we'll need at go-live, and a few that we either won't need at all or will do after go-live. The card for each integration is already populated with descriptions, functional notes, and links to existing Ex Libris documentation. Projects that require local developers will be outlined and tracked in the Library's Jira instance.
...
2023/08/30
...
The LSP Steering Group has begun meeting regularly with our Ex Libris onboarding manager. As part of the onboarding process, the steering group will work with Ex Libris on milestone planning, e-learning, and project analysis. The analysis will include third party integrations, project scope, and data migration readiness. The onboarding phase is designed to make sure that we really understand Alma before we start making migration and configuration decisions. Each week in September, the steering group has been assigned a section of the "Getting to Know Alma" course. If you're interested in following along, you can find the courses at https://knowledge.exlibrisgroup.com/Alma/Training/Getting_To_Know_Alma/Getting_to_Know_Alma_-_English.
Project Overview
...
The Define phase will include more regular calls with our Ex Libris implementation team, more e-learning, and filling out the important migration and configuration forms so Ex Libris can set up our environment with the data we deliver.
The Build phase is when we will review the environment that has been loaded with our data and configured as we requested in the Define phase. We will start working on the third-party integrations and the integration of our discovery layer. This is the phase when an Ex Libris will train our trainers and we'll work on training plans for all library staff. At the end of this phase is the "cutover" when we'll stop working in Aleph and do a final data extract and load into our Alma production environment.
The Deploy phase starts with our go-live. Ex Libris will continue regular calls with Duke to help us through any functional issues and to monitor project status. At the end of the Deploy phase, we'll say goodbye to our Ex Libris implementation team and will start working with their support team.
The Life in Production phase includes continued education, and Ex Libris' "First Year Success Program". We'll start reporting issues through their support portal and keep our instance up to date with regular releases.
Our tentative schedule as defined by Ex Libris is below. We will add more Duke-defined dates such as local training as we learn more.
...
Project Activity
...
Schedule
...
Onboarding
...
Now - November
...
Implementation Project Kickoff
...
Mid-November
...
Test Load of Duke Data Start
...
Late December
...
Environment Delivery
...
Early January
...
Data Checking
...
January
...
Alma Functional Workshop (train the trainer)
...
TBD (likely March or April)
...
Workflow testing
...
January - May
...
Third Party Integrations
...
January - May
...
Cutover Start
...
Mid-June
...
Go Live
...
Mid-July
...
2023/06/12
...
| Table of Contents | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
October 14, 2024
Some very cool updates to report today.
Firstly, last week we confirmed and processed a list of ~1.3 million records to be removed from the common data stream for the TRLN catalogs. These were largely duplicate e-resource records left over from our transition away from 360KB, but it also included some DKU records. Now that these records are gone, the search results should be a bit less confusing and have less potential for incorrect information, which will be a big improvement to the user experience of our catalog. It should also fix the main remaining reason behind workaround #2: inactive records displaying in B&MC. (Other deleted and suppressed records that were originally showing up should also be gone now.)
Another big win last week - we have fixed an issue with the “View Online” buttons for ~2 million e-resource records. These records were showing in the catalog, but they were missing the access URL, so the record pages looked strangely empty and users wouldn’t know where to go to get to the resource. Now that the buttons are showing up, this should fix the primary reason behind workaround #3: no View Online button.
We’re still working as fast as we can to get the data pipeline set up so it runs on an hourly schedule, and there’s progress on that every day, but it’s all behind the scenes right now. We’re also making great progress on other quality-of-life improvements, like more automated testing so we find bugs faster. Keep watching this space for the latest information!
October 9, 2024
The records from 9/18 are all fully ingested into the common data stream, so the catalog data should now be up to date as of 9/18.
Another fun change that came with this update: you can now search by MMS ID in the Books & Media Catalog! Searching for the MMS ID (the new Alma ID number) should work either for “all fields” searches or for “ISBN/ISSN/barcode” searches.
October 7, 2024
Quick update on the data pipeline. The records harvested from Alma on 9/18/24 have just about finished their load into the TRLN common data stream. Many updates are already available in the catalog, like improvements to the New Titles results. In other cases, the updated data may just be the first step toward getting the catalog to work correctly.
We have also just published a document detailing Workarounds for Books & Media Catalog Issues. This new documentation goes into a bit more detail about the workarounds for several of the major issues still affected the Books & Media Catalog. Spoiler alert: searching via Summon is often a good alternative.
Be on the lookout for invitations to staff listening sessions related to the catalog! Thomas Crichlow will be hosting a series of meetings to hear about issues people are experiencing and offer additional support.
October 3, 2024
This update is a bit of a state-of-the-union for the Books and Media Catalog extended universe.
When something doesn’t look or work quite right in the Books and Media Catalog, the true issue could lie in a variety of places. The following summarizes recent and upcoming work related to: the underlying data, the data pipeline, the Books and Media Catalog itself, and the Catalog Request System.
Underlying Data
Context
Some of the issues related to what is showing up in the catalog can be traced back to issues with the underlying data.
Recent updates to the underlying data
Consolidated duplicate e-resource records (migrated from both Aleph and 360KB)
Collection code table in Caiasoft has been updated to allow RL to send items to LSC for accessioning
Adjusted the work order for Lilly items housed in Perkins so they can be requested
Lilly materials at remote storage put into a separate Lilly Renovation work order to facilitate identifying needed record updates (aka cleanup for Lilly items in Perkins stacks, non-Lilly materials sent to Clancy due to mis-shelving, etc.)
Fixed an issue where Lilly DVDs requested by staff were being recalled by the Lilly Renovation work order
We’ve stripped out Library of Congress Table of Contents links from print materials in Alma so they don’t end up in the catalog with a confusing “View Online” button. Regular analysis will be undertaken to remove any future LC TOCs that slip in.
Upcoming underlying data work
Continuing to consolidate duplicate e-resource records
Some DKU items and holdings remain in Alma with invalid permanent locations, so they’re still showing up in the catalog.
Ongoing work on data underlying bound-withs to prevent requesting errors
Address issue with Withdrawn items showing up in the catalog by moving them to a separate library in Alma
Withdrawn Library appears in catalog, but should be suppressed from display to prevent patron requests for materials no longer held
Address issue with Lost items showing up in the catalog by moving them to a separate library in Alma
Lost Library appears in catalog, but should be suppressed from display to prevent patron requests for materials no longer held
Items with “technical migration” status type are undergoing review to determine what Alma status type is correct. These include items that had an Item Process Status, IPS, in Aleph that was not easily mapped to Alma. IPS in this item state include those for missing, long missing, withdrawn, lost, on order, arrived, cancelled order, suppressed, etc.
Data Pipeline
Context
The data used for the catalog comes from Alma, but it takes a lot of transformation to get everything in the right format for the catalog. (Because we share our catalog and its data stream with our TRLN partner institutions, we all have to conform to an agreed-upon standard data structure.)
Our data pipeline from Alma to TRLN Discovery includes a complicated extraction process from Alma. The steps (for a full export and publish cycle) are roughly:
Setup a publishing profile in Alma to determine what metadata elements are included when we harvest records
Kick off a publishing job in Alma
Harvest all of the published records from Alma
Enrich the records with additional information not available from the harvest (e.g., availability information)
Transform the final records into the correct format for TRLN Discovery
When needed, test a subset of records in a Duke sandbox to see how they will look in the catalog before we push them into the live application
Publish the records into the TLRN Discovery data stream
A note about the stale data in the catalog: When we have to run the full pipeline on all ~8 million records that go into the catalog, the process can take almost a month to complete. Since the Alma cutover in July, we’ve had to make several changes to the pipeline, and each change meant we had to reprocess all of the records again. As the pipeline nears its final state, we are working to transition to “intermittent updates.” This means that instead of reprocessing all of our millions of records each time, we can just ask Alma for the records that have been updated since our last run. Switching to intermittent updates takes some extra coding to automate each part of the pipeline, and that work is currently underway. When we get to the point where we can switch to intermittent updates, we hope to be processing updated records every hour. In the meantime, changes in Alma will still take several weeks to appear in the catalog.
Recent updates to the data pipeline
Latest data run:
On 9/18 we began another publishing job from Alma (step 2 above), so the next update to the data in the catalog will include changes up to the morning of September 18
As of 9/25, we expect the enrichment of the records (step 5 above) to be completed by 9/30 or so. It will likely take another week (around 10/7) before we see the data from 9/18 in the catalog.
One of the data quality issues in the catalog right now involves some missing data for electronic records, and as a result the records do not show the URL to access the resource. The needed fields have now been included in the publishing profile and the data transformation scripts have been updated to get those fields into the catalog data. When the 9/18 data goes live in October, we will see the correct access URLs for those electronic records.
Another data quality issue in the catalog right now is that print records were not getting a value for the “date-cataloged” field, which is what our catalog uses to display new titles. Several widgets used throughout the library website rely on this data as well. With some new fields in the publishing profile and new logic in the transformation scripts, this issue should be resolved with the next full data refresh in October.
We also recently fixed an issue with our availability information so that it will show up correctly on TRLN partner pages and in the availability filter.
We’ve been continuing to improve the publishing profile to include the fields needed for discovery.
We’ve removed records that shouldn’t show up in the discovery layer (e.g., a large batch of deleted and suppressed records that accidentally got included in the catalog)
Modified enrichment code to request records in batches of 50, instead of one at a time.
Upcoming data pipeline work
The scripts that complete the various pipeline steps need to be configured to run on a schedule so that we can perform the entire pipeline in an automated fashion every hour or so. So far, we are ready to run the harvesting step on a schedule, but we need to complete the work for scheduling the enrichment, transformation, and publishing steps.
Explore modifying the data transformation process to build data that works better for the way the catalog expects temporary locations to work
Automate publishing our records to the Platform for Open Discovery (POD).
Trigger updates to the pipeline and catalog mappings when new libraries/locations are added in Alma
Create additional documentation about data flow from Alma to the catalog
If a previously published record becomes suppressed or deleted in Alma, create a separate process for deleting it from the catalog
Books and Media Catalog
Context
The Books and Media Catalog we host at Duke is a collaboration with our TRLN partner institutions. There is a common data stream that includes records from Duke, UNC, NC State, and NC Central. There is also a common application (“TRLN Discovery”) that sets up a basic catalog interface for all of us. NC Central uses the basic application, if you want to see what that looks like. The rest of us add customizations on top of the basic application to tailor it to our needs.
Each catalog allows users to toggle the search context from their local university (e.g., Duke) to all of TRLN. This allows users at each institution to quickly see and request items that are available at another TRLN institution. This is only possible because we are all transforming our data into a common format and publishing it to a shared data stream.
Most of the issues we’ve been addressing with work on the catalog stem from a difference in how Alma handles requesting and availability. Alma focuses on title-level requesting…
Getting our data out of Alma is also more difficult because we have to request data through APIs instead of having direct database access like we used to with Aleph. There are limits to how many times we can query the APIs each day, and the data we need for the catalog has to be blended from multiple places. To limit our usage of the APIs, we have adjusted our approach to showing availability – we now retrieve availability from Alma only at the title (or holdings) level, instead of at the item level, for everything except Rubenstein and University Archives items. For those items, users can still see item-level availability on the item detail pages in the catalog. For all print materials, though, users can click the green Request button to see availability at the item level.
Recent updates to the Books and Media Catalog
We have completed work on Duke customizations to show availability at the holdings level on both search results and item detail pages, regards of whether the user is searching just Duke’s records or all of the TRLN records.
Recently, we also updated the common catalog application so our TRLN partner institutions can also display our holdings-level availability data.
Added a clearer banner to alert patrons to ongoing issues and better match the style of the banner on our public website
Added the barcode display back in to facilitate various workflows
Improved the real-time availability check at the holdings level for search results pages and at the holdings and items levels for item detail pages.
Fixed the “show all items” functionality for long lists of items
Fixed an issue where items in temporary locations weren’t showing up correctly or linking to the correct location map. Fixing this also fixed a problem where Rubenstein records were not showing all items on the item details page.
In some cases, Alma can’t determine availability at the holdings level, and we need to point people to the Request app so they can view availability at the item level. We now display a link to the Request page labeled “Availability Details” in place of the availability information at the holdings level.
Fixed an issue where clicking on the map icon returned an unhelpful error message
Fixed an issue where more than just Rubenstein and Archives items were showing item-level availability on item detail pages
Addressed a bot attack that was slowing down performance by blocking a list of IP addresses
Updated the list of collection codes to include new locations like Bishop’s House
Analyzed 1.3M records in the catalog data stream that haven’t been updated since go-live, as those records can’t be coming from Alma. Almost all are duplicate records left over from 360KB. The remaining 2,035 have been reviewed and all but 2 can be deleted from the common data stream.
Implemented a “guest request” link for visitors to request LSC items
Upcoming Books and Media Catalog work
Improve our automated testing to make it easier to see if code changes have broken anything else
Support searching by the new Alma ID and Duke’s TRLN ID
Improve real-time availability checking for items at temporary locations
Continue to adjust display of bound-with items
Ford Kindle e-books collection not displaying properly
Complete a major underlying software upgrade from Blacklight 7 to 8
Catalog Request System
Context
The Catalog Request System is the system that drives the real-time availability updates in the catalog UI, and it is also where users go when they click on the big green Request button.
Recent updates to the Catalog Request System
Changed how items are identified to prevent collapsing of items with identical enumeration
Fixed an issue where Rubenstein items weren’t showing up if there were also LSC items
Fixed issues where Rubenstein and other items were requestable when they shouldn’t be (“In process”, “Technical-Migration”, “Acquisition”)
Fixed the ability to request bound-withs
Fixed an issue with the logic for renewing items
Improved performance of the Request app with large sets of items (greater than 50)
Fixed an issue with display of Interlibrary Loan Request (should appear if at least one item has a status other than “On order” and patron has ILR permissions)
Updated to be able to show Bishop’s House as a pick-up location
Upcoming Catalog Request System work
Sort items by holdings, then enum/chronology
Fix problems with availability and requesting for items with relations (i.e., related physical and electronic items)
Explore problems with title-level requesting that cause a loop or a failed request