Updates related to DUL Discovery/Catalog Functionality
This document includes an archive of update messages documenting work related to the Books & Media Catalog, Alma, and Related Systems and the Alma Integrations team.
March 25, 2025
Here is an overview and current status of work related to Alma integrations with the Books & Media Catalog, the Catalog Request System, and other systems.
Republish of eResources
We are currently republishing all eResources from Alma to the Books & Media Catalog. We envision completing this publishing and re-indexing by March 31. This will add View Online links to eResources that were missing them.
Alma > Discovery Sandbox
The Alma Integrations team is building an environment that will enable us to publish from the Alma premium sandbox using a sandboxed version of the pipeline that will push data to a Books & Media Catalog sandbox
We will connect these environments with a Summon preview environment and a sandbox instance of the Catalog Request System
Having each of these sandbox systems interconnected will better support developers and stakeholders as they collaborate to troubleshoot remaining problems and validate solutions
Understanding the pipeline
Erin Nettifee has posted a document in Confluence to help library staff understand what the pipeline for publishing data from Alma to the Books & Media Catalog does and how it works
Other areas of work
Catalog Request System (the CRS)
We’ve completed work to add a banner message that can be targeted to specific Alma patron groups, to support the plan to stop automatically renewing faculty loans this spring
We’ve made improvements to underlying code in the Catalog Request System to better support ongoing maintenance and development
We’ve continued building collaboration with AUXS developers through code reviews and accessibility improvements
We are doing additional development work to support specific use cases for the Medical Center Library’s pilot of Leganto in Fall 2025
We’re preparing additional scripts to support University audit requirements for regular review of Alma permissions
Upcoming work
The team is also working on fixes for various issues, including: replacing multiple identical “View Online” buttons with a single button to a services page, correctly displaying eResources as restricted to Duke that were incorrectly listed as Open Access, synchronizing the Alma harvest schedule, and improving the processing of related records
Once the Alma > Discovery Sandbox environment is completed, we will push data that can be used for testing to the sandbox
We will schedule several Test Fest sessions to test specific changes with related stakeholders; we will send Outlook invitations soon as well as instructions about how to use the sandbox for testing
After stakeholders have validated the changes, we will implement them in our production environment
We will do a Full Republish of records from Alma to the Books & Media Catalog; the timing for this has not yet been determined, but is contingent on successful completion of testing
Bonus tip
How can you tell when a record in the Books & Media Catalog was last reindexed?
You can add "/raw" to the end of the URL string to see when a record was last reindexed
This will display the underlying data from the shared TRLN Discovery Solr index that is used to display the record in the Books & Media Catalog
The last field at the end of the record will be "index_date"
Tip: Firefox displays this raw data in a format that is easier to read than either Chrome or Safari
Example: https://find.library.duke.edu/catalog/DUKE99119253801808501/raw
February 7, 2025
Alma to Books & Media Pipeline
We use a combination of processes to move data from Alma to the shared TRLN Discovery index that underpins DUL's Books & Media Catalog; these processes also transform data into a structure that is compatible with TRLN Discovery.
Recent work
Incorrect URLs: For many eResources, we have replaced incorrect URLs with correct ones
Deleted/Suppressed: We have removed some records from the Books & Media Catalog that had been suppressed or deleted in Alma
Upcoming work
How the pipeline works: We are working on a staff-friendly flow chart and description of the pipeline that we use to move and transform data from Alma to the Books & Media Catalog to help staff better understand how data flows between these two systems.
Deleted/Suppressed: We are preparing to release improvements to how we process deleted and suppressed records to fix bugs that had previously been reported
Records with multiple link modal: Some eResources currently display a modal containing multiple links when a patron clicks a record's View Online button. We are developing a change that will replace the multi-link modal with a View Online button that links to an ExLibris services page where a patron can see a list of platforms and coverage dates for that eResource.
Schedule for sending updated records to the Catalog: We will soon change the schedule for sending new and updated records from Alma to the Books & Media Catalog.
Alma currently has two publishing jobs for sending data out of Alma with one of these jobs running once per hour and the other running once every twelve hours.
Each of these publishing jobs pushes different data components from Alma, but both jobs are critical to getting needed data into the Books & Media Catalog.
To fix a bug in which some records fail to sync and to ensure compatibility with Alma's capabilities, we will soon adjust the schedule so that both publishing jobs run on the same schedule.
The new schedule will publish both jobs on a six hour schedule with processing kicking off at 5 AM, 11 AM, 5 PM, and 11 PM.
How long staff will have to wait to see changes made in Alma appear in the Books & Media Catalog will vary depending on when a change is made in Alma in relation to when the pipeline kicks off at 5 and 11 and also on how long it takes for the pipeline to complete its work.
The real-time item availability script that runs in the Books & Media Catalog will continue to function as normal after a webpage loads in a patron's browser.
The Catalog Request System will also continue to display item availability in real time.
Catalog Request System (CRS)
Recent work
For some records that have an online component, patrons will now see a DOI search field
Upcoming work
We will be updating loan sorting for My Accounts and will also work on location codes
We will work on the messaging shown when users encounter a time-out
We will require patrons to choose a valid pickup location
Other Alma Integrations Work
Related records: Matt Harrington is leading a small team to explore how we manage related records to improve compatibility with both Alma and integrated applications like the Books & Media Catalog.
This team is beginning its work with titles that are loaded onto Kindle devices at Ford and will then explore some use cases from Rubenstein Library.
They will apply lessons learned to other use cases for related records from throughout the libraries.
Leganto Course Reserves: We are working on improvements to automated course loading and are beginning work to enable the Medical Center Library to use Leganto.
Patron Accounts: We are working on improvements to how we handle expiring patron accounts and how we handle patron loader errors. Our next big test of these features will be when students graduate in May.
Subscription order data: We are working on migrating subscription order data and are testing use cases.
Problems?
Trouble accessing eResources: If a patron reports difficulty accessing eResources when using the Books & Media Catalog, help the patron use Summon to access that eResource.
Summon is currently the authoritative path for patron access to most eResources. If you spot an eResource problem in the Books & Media Catalog, there is a strong likelihood that access to that eResource will work through Summon.
The Online Journal Titles search portal allows patrons to discover both eJournals and Newspapers that are available online via search results displayed in Summon.
The eBooks search portal allows patrons to discovery eBooks via results displayed in Summon and to also browse to other platforms such as OverDrive.
After pointing a patron to Summon for access to an eResource, please report a problem observed with the Books & Media Catalog to the Alma team.
Reporting problems: You can use the DST Support Form (using the Alma support queue) at support.lib.duke.edu. Include:
A description of the problem
A URL linking to an example
A description of what the record should look like if it were displayed correctly
The information you share with us is valuable — we realize that it sometimes takes weeks or even longer for us to develop a solution to some reported problems, but the examples you have provided to us have helped us to understand what is wrong as we explore how to make it right.
Drop-in Alma Office Hours
The Alma listening sessions held with departments provided a venue for valuable informal two-way communication.
This spring we are offering two libraries-wide Office Hours sessions where staff from throughout libraries at Duke can drop-in for informal conversations or Q&A with members of the Alma Implementation team.
These sessions are scheduled for February 26 and March 10 — we will send calendar invites soon.
January 17, 2025
Incremental updates to the Books & Media Catalog are live and have been running over the last month. The process catches changes in Alma and should sync them with the catalog within a few hours of the change in Alma. We are aware, though, that some changes are still getting missed, and we're working through them as they get reported. Thanks for your patience as we fine-tune the logic of how changes move through the pipeline. If you notice problems, please continue to submit reports (using the DST Support Form, selecting the Alma option). Please include multiple examples of the problem, if you're able, and a description of what should be happening instead.
The Catalog Request System has gotten some major improvements to address holdings with long lists of items. The CRS now waits to display the items until they have all been retrieved from Alma, which means that we can sort them in a sensible manner. The interface currently includes options for patrons to sort the items by description or group by library, and additional improvements are on the way.
Similar improvements to lists of items are available in My Accounts, as well.
We have implemented a fix for an incorrect URL value that was showing up in the catalog. Affected records will need to get reprocessed, and we're currently planning that work.
Note: when we find records that need to be reprocessed, we start by "republishing" those records in Alma. The republishing process can be time consuming, and while it's going on, the hourly incremental publishing process gets paused, so any Alma changes made during that block of time will take longer to make their way into the catalog. We will try to be cautious about when we schedule republishing runs if we know they are going to disrupt incrementals for a long time.
We continue to review reports of missing, extra, or incorrect records in the Books & Media Catalog to identify the underlying issues and research possible solutions.
We have made good progress on automated course loading into Alma. We've been refining the load process on the sandbox, and we'll be working to automate the process soon.
We have made improvements to the patron loader to handle errors better so the overnight process doesn't fail.
We are working on migrating subscription log data to orders records, but we're still testing use cases in the sandbox.
These recent tasks mark an end to what we are calling Phase 1 of the Alma Implementation. Often for a large migration, we'll scope Phase 1 to get the core systems working, and then we will immediately follow that with a Phase 2 that cleans up additional smaller concerns and helps transition the system to a stable maintenance phase. Our Phase 2 started on January 15, and we've attached the summary of the kinds of project work that will be included in Phase 2 (see PDF below).
Transitioning to a new phase gives us a chance to reflect on early parts of the process, and we hope to share a summary of our Phase 1 post-mortem soon. While the Phase 2 team will continue on from the work of Phase 1, the team membership has shifted slightly. We expect additional projects and needs will continue to arise over the course of Phase 2, and those will be prioritized alongside existing issues and addressed in order of priority.
December 13, 2024
Incremental data loads are live!
We are happy to report that data are being loaded into the catalog incrementally, as of earlier this week. Data in the catalog should be current within a couple of hours.
Our data pipeline now checks for updated records in Alma once an hour. (This is a limit Alma sets, so that’s the minimum amount of time we can have between checks.)
New updates (including new records, changed records, and deleted/suppressed records) will be harvested and added to the processing pipeline on this schedule. Except when there's been a really large batch of records updated all at once, it’s likely the records will make it through the full processing pipeline and be visible in the Books & Media Catalog in about an hour.
While the incremental updates are running smoothly and have been live for several days now without issues, there is always the chance something will crop up and we will need to pause the incrementals to address another issue. We will communicate broadly if that happens.
Prioritizing remaining work
With incremental data loading live, we are working to wrap up some important documentation tasks and compile next steps from our recent staff listening sessions.
A “Phase 2” of the Alma migration is expected to begin January 15. We are in the process of detailing exactly which projects are going to be prioritized during Phase 2, but the projects will include continuing work on some large data issues that still haven’t been resolved.
The team for Phase 2 will be adjusted to match the exact projects that are being prioritized.
We will share more on the plans for Phase 2 soon.
Additional recent updates
Completed the reprocess of 2 million e-resource records, which corrected a large batch of problematic records we were tracking.
Removed additional 126k non-bibliographic records from the catalog data stream
Removed stale patron notes from 22k accounts
Replaced Twitter icon/link in the catalog footer with LinkedIn
Refined our strategy to thwart bots that aggressively crawl the catalog, degrading performance for actual users
Improved logging of parts of the data pipeline for troubleshooting purposes
Exported order log notes from Aleph database in preparation for migration to Alma
Working with Collections Services on documentation of the logic of how deleted and suppressed records in Alma should impact the Books and Media Catalog data stream.
Began development work on automating the import of courses into Alma for Leganto Course reserves
November 21, 2024
We have identified a problem with electronic resource processing, and we are re-processing over 2 million records to fix the issue. While this batch is working its way through the system, all of the computational resources will be dedicated to it, so we have paused the tests for incremental updates.
Additional explorations are underway while these records are processing, including explorations around related records and missing location and inventory information.
We have identified and deleted a few batches of leftover records that aren't in Alma, like records with only DKU holdings.
We’ve been running a lot of complicated queries of the TRLN Discovery shared data stream to try to identify sets of problematic records in the catalog
We’ve been making improvements to make it easier to troubleshoot data pipeline work
We updated the public-facing page to streamline info shared with patrons about the status of ongoing updates to the catalog; we also updated the banners on our Drupal pages and in the catalog
We’ve updated our local catalog application to pull in changes to TRLN Discovery core code
We’ve fixed accessibility issues with form fields in Advanced Search
We’ve fixed a long-standing bug that was leading to an odd layout on individual record pages following a search
We’ve patched several security vulnerabilities in software dependencies
We've been continuing work on improving large record management (sorting and paging) in the Catalog Request System
October 29, 2024
In additional to the data refresh, we’ve had several great improvements since the last update.
Availability status will now display correctly when items are in a temporary location (like New & Noteworthy).
We have implemented a new strategy to block a wide range of IP addresses belonging to Tencent Cloud, a company whose bots issue floods of simultaneous server requests, deceptively using distributed addresses to try to avoid detection. These bots were the cause of at least one catalog outage in the past week.
We are working to adjust the Online Journal Titles search to include newspapers, as well; this change should roll out this week.
Unfortunately, we have also discovered that a large number of records in the catalog (about 10%) currently do not have any access information (no items, holdings, or URLs). We're still diagnosing the problem, but we believe it is likely related to the way we extract batches of records from Alma. If you encounter records like this, please continue to use relevant workarounds like searching Summon or Alma. There is no need to report individual cases of this problem at this time. Unfortunately, we can’t provide an estimated time to resolution, since we still need to figure out the source of the problem and the complications that might be involved in fixing it.
October 14, 2024
Some very cool updates to report today.
Firstly, last week we confirmed and processed a list of ~1.3 million records to be removed from the common data stream for the TRLN catalogs. These were largely duplicate e-resource records left over from our transition away from 360KB, but it also included some DKU records. Now that these records are gone, the search results should be a bit less confusing and have less potential for incorrect information, which will be a big improvement to the user experience of our catalog. It should also fix the main remaining reason behind workaround #2: inactive records displaying in B&MC. (Other deleted and suppressed records that were originally showing up should also be gone now.)
Another big win last week - we have fixed an issue with the “View Online” buttons for ~2 million e-resource records. These records were showing in the catalog, but they were missing the access URL, so the record pages looked strangely empty and users wouldn’t know where to go to get to the resource. Now that the buttons are showing up, this should fix the primary reason behind workaround #3: no View Online button.
We’re still working as fast as we can to get the data pipeline set up so it runs on an hourly schedule, and there’s progress on that every day, but it’s all behind the scenes right now. We’re also making great progress on other quality-of-life improvements, like more automated testing so we find bugs faster. Keep watching this space for the latest information!
October 9, 2024
The records from 9/18 are all fully ingested into the common data stream, so the catalog data should now be up to date as of 9/18.
Another fun change that came with this update: you can now search by MMS ID in the Books & Media Catalog! Searching for the MMS ID (the new Alma ID number) should work either for “all fields” searches or for “ISBN/ISSN/barcode” searches.
October 7, 2024
Quick update on the data pipeline. The records harvested from Alma on 9/18/24 have just about finished their load into the TRLN common data stream. Many updates are already available in the catalog, like improvements to the New Titles results. In other cases, the updated data may just be the first step toward getting the catalog to work correctly.
We have also just published a document detailing Workarounds for Books & Media Catalog Issues. This new documentation goes into a bit more detail about the workarounds for several of the major issues still affected the Books & Media Catalog. Spoiler alert: searching via Summon is often a good alternative.
Be on the lookout for invitations to staff listening sessions related to the catalog! Thomas Crichlow will be hosting a series of meetings to hear about issues people are experiencing and offer additional support.
October 3, 2024
This update is a bit of a state-of-the-union for the Books and Media Catalog extended universe.
When something doesn’t look or work quite right in the Books and Media Catalog, the true issue could lie in a variety of places. The following summarizes recent and upcoming work related to: the underlying data, the data pipeline, the Books and Media Catalog itself, and the Catalog Request System.
Underlying Data
Context
Some of the issues related to what is showing up in the catalog can be traced back to issues with the underlying data.
Recent updates to the underlying data
Consolidated duplicate e-resource records (migrated from both Aleph and 360KB)
Collection code table in Caiasoft has been updated to allow RL to send items to LSC for accessioning
Adjusted the work order for Lilly items housed in Perkins so they can be requested
Lilly materials at remote storage put into a separate Lilly Renovation work order to facilitate identifying needed record updates (aka cleanup for Lilly items in Perkins stacks, non-Lilly materials sent to Clancy due to mis-shelving, etc.)
Fixed an issue where Lilly DVDs requested by staff were being recalled by the Lilly Renovation work order
We’ve stripped out Library of Congress Table of Contents links from print materials in Alma so they don’t end up in the catalog with a confusing “View Online” button. Regular analysis will be undertaken to remove any future LC TOCs that slip in.
Upcoming underlying data work
Continuing to consolidate duplicate e-resource records
Some DKU items and holdings remain in Alma with invalid permanent locations, so they’re still showing up in the catalog.
Ongoing work on data underlying bound-withs to prevent requesting errors
Address issue with Withdrawn items showing up in the catalog by moving them to a separate library in Alma
Withdrawn Library appears in catalog, but should be suppressed from display to prevent patron requests for materials no longer held
Address issue with Lost items showing up in the catalog by moving them to a separate library in Alma
Lost Library appears in catalog, but should be suppressed from display to prevent patron requests for materials no longer held
Items with “technical migration” status type are undergoing review to determine what Alma status type is correct. These include items that had an Item Process Status, IPS, in Aleph that was not easily mapped to Alma. IPS in this item state include those for missing, long missing, withdrawn, lost, on order, arrived, cancelled order, suppressed, etc.
Data Pipeline
Context
The data used for the catalog comes from Alma, but it takes a lot of transformation to get everything in the right format for the catalog. (Because we share our catalog and its data stream with our TRLN partner institutions, we all have to conform to an agreed-upon standard data structure.)
Our data pipeline from Alma to TRLN Discovery includes a complicated extraction process from Alma. The steps (for a full export and publish cycle) are roughly:
Setup a publishing profile in Alma to determine what metadata elements are included when we harvest records
Kick off a publishing job in Alma
Harvest all of the published records from Alma
Enrich the records with additional information not available from the harvest (e.g., availability information)
Transform the final records into the correct format for TRLN Discovery
When needed, test a subset of records in a Duke sandbox to see how they will look in the catalog before we push them into the live application
Publish the records into the TLRN Discovery data stream
A note about the stale data in the catalog: When we have to run the full pipeline on all ~8 million records that go into the catalog, the process can take almost a month to complete. Since the Alma cutover in July, we’ve had to make several changes to the pipeline, and each change meant we had to reprocess all of the records again. As the pipeline nears its final state, we are working to transition to “intermittent updates.” This means that instead of reprocessing all of our millions of records each time, we can just ask Alma for the records that have been updated since our last run. Switching to intermittent updates takes some extra coding to automate each part of the pipeline, and that work is currently underway. When we get to the point where we can switch to intermittent updates, we hope to be processing updated records every hour. In the meantime, changes in Alma will still take several weeks to appear in the catalog.
Recent updates to the data pipeline
Latest data run:
On 9/18 we began another publishing job from Alma (step 2 above), so the next update to the data in the catalog will include changes up to the morning of September 18
As of 9/25, we expect the enrichment of the records (step 5 above) to be completed by 9/30 or so. It will likely take another week (around 10/7) before we see the data from 9/18 in the catalog.
One of the data quality issues in the catalog right now involves some missing data for electronic records, and as a result the records do not show the URL to access the resource. The needed fields have now been included in the publishing profile and the data transformation scripts have been updated to get those fields into the catalog data. When the 9/18 data goes live in October, we will see the correct access URLs for those electronic records.
Another data quality issue in the catalog right now is that print records were not getting a value for the “date-cataloged” field, which is what our catalog uses to display new titles. Several widgets used throughout the library website rely on this data as well. With some new fields in the publishing profile and new logic in the transformation scripts, this issue should be resolved with the next full data refresh in October.
We also recently fixed an issue with our availability information so that it will show up correctly on TRLN partner pages and in the availability filter.
We’ve been continuing to improve the publishing profile to include the fields needed for discovery.
We’ve removed records that shouldn’t show up in the discovery layer (e.g., a large batch of deleted and suppressed records that accidentally got included in the catalog)
Modified enrichment code to request records in batches of 50, instead of one at a time.
Upcoming data pipeline work
The scripts that complete the various pipeline steps need to be configured to run on a schedule so that we can perform the entire pipeline in an automated fashion every hour or so. So far, we are ready to run the harvesting step on a schedule, but we need to complete the work for scheduling the enrichment, transformation, and publishing steps.
Explore modifying the data transformation process to build data that works better for the way the catalog expects temporary locations to work
Automate publishing our records to the Platform for Open Discovery (POD).
Trigger updates to the pipeline and catalog mappings when new libraries/locations are added in Alma
Create additional documentation about data flow from Alma to the catalog
If a previously published record becomes suppressed or deleted in Alma, create a separate process for deleting it from the catalog
Books and Media Catalog
Context
The Books and Media Catalog we host at Duke is a collaboration with our TRLN partner institutions. There is a common data stream that includes records from Duke, UNC, NC State, and NC Central. There is also a common application (“TRLN Discovery”) that sets up a basic catalog interface for all of us. NC Central uses the basic application, if you want to see what that looks like. The rest of us add customizations on top of the basic application to tailor it to our needs.
Each catalog allows users to toggle the search context from their local university (e.g., Duke) to all of TRLN. This allows users at each institution to quickly see and request items that are available at another TRLN institution. This is only possible because we are all transforming our data into a common format and publishing it to a shared data stream.
Most of the issues we’ve been addressing with work on the catalog stem from a difference in how Alma handles requesting and availability. Alma focuses on title-level requesting…
Getting our data out of Alma is also more difficult because we have to request data through APIs instead of having direct database access like we used to with Aleph. There are limits to how many times we can query the APIs each day, and the data we need for the catalog has to be blended from multiple places. To limit our usage of the APIs, we have adjusted our approach to showing availability – we now retrieve availability from Alma only at the title (or holdings) level, instead of at the item level, for everything except Rubenstein and University Archives items. For those items, users can still see item-level availability on the item detail pages in the catalog. For all print materials, though, users can click the green Request button to see availability at the item level.
Recent updates to the Books and Media Catalog
We have completed work on Duke customizations to show availability at the holdings level on both search results and item detail pages, regards of whether the user is searching just Duke’s records or all of the TRLN records.
Recently, we also updated the common catalog application so our TRLN partner institutions can also display our holdings-level availability data.
Added a clearer banner to alert patrons to ongoing issues and better match the style of the banner on our public website
Added the barcode display back in to facilitate various workflows
Improved the real-time availability check at the holdings level for search results pages and at the holdings and items levels for item detail pages.
Fixed the “show all items” functionality for long lists of items
Fixed an issue where items in temporary locations weren’t showing up correctly or linking to the correct location map. Fixing this also fixed a problem where Rubenstein records were not showing all items on the item details page.
In some cases, Alma can’t determine availability at the holdings level, and we need to point people to the Request app so they can view availability at the item level. We now display a link to the Request page labeled “Availability Details” in place of the availability information at the holdings level.
Fixed an issue where clicking on the map icon returned an unhelpful error message
Fixed an issue where more than just Rubenstein and Archives items were showing item-level availability on item detail pages
Addressed a bot attack that was slowing down performance by blocking a list of IP addresses
Updated the list of collection codes to include new locations like Bishop’s House
Analyzed 1.3M records in the catalog data stream that haven’t been updated since go-live, as those records can’t be coming from Alma. Almost all are duplicate records left over from 360KB. The remaining 2,035 have been reviewed and all but 2 can be deleted from the common data stream.
Implemented a “guest request” link for visitors to request LSC items
Upcoming Books and Media Catalog work
Improve our automated testing to make it easier to see if code changes have broken anything else
Support searching by the new Alma ID and Duke’s TRLN ID
Improve real-time availability checking for items at temporary locations
Continue to adjust display of bound-with items
Ford Kindle e-books collection not displaying properly
Complete a major underlying software upgrade from Blacklight 7 to 8
Catalog Request System
Context
The Catalog Request System is the system that drives the real-time availability updates in the catalog UI, and it is also where users go when they click on the big green Request button.
Recent updates to the Catalog Request System
Changed how items are identified to prevent collapsing of items with identical enumeration
Fixed an issue where Rubenstein items weren’t showing up if there were also LSC items
Fixed issues where Rubenstein and other items were requestable when they shouldn’t be (“In process”, “Technical-Migration”, “Acquisition”)
Fixed the ability to request bound-withs
Fixed an issue with the logic for renewing items
Improved performance of the Request app with large sets of items (greater than 50)
Fixed an issue with display of Interlibrary Loan Request (should appear if at least one item has a status other than “On order” and patron has ILR permissions)
Updated to be able to show Bishop’s House as a pick-up location
Upcoming Catalog Request System work
Sort items by holdings, then enum/chronology
Fix problems with availability and requesting for items with relations (i.e., related physical and electronic items)
Explore problems with title-level requesting that cause a loop or a failed request