Planet Cataloging

January 17, 2017

Terry's Worklog

MarcEdit’s new Logging Features

Over the years, I’ve periodically gotten requests for a much more robust logger in MarcEdit.  Currently, when the tool performs a global change, it reports the number of changes made to the user.  However, a handful of folks have been wanting much more.  Ideally, they’d like to have a log of every change the application makes, which is hard because the program isn’t built that way.  I provided the following explanation to the MarcEdit list last week.


The question that has come up a number of times since posting notes about the logger is questions about granularity.  There has been a desire to have the tool provide additional information (about the records), more information around change context, and also wondering if this will lead to a preview mode.  I think other folks wondered why this process has taken so long to develop.  Well, it stems from decisions I make around the development.  MarcEdit’s application structure can be summed up by the picture below:


In developing MarcEdit, I have made a number of very deliberate decisions, and one of those is that no one component knows what the other one does.  As you can see in this picture, the application parts of MarcEdit don’t actually talk directly to the system components.  They are referenced through a messenger, which handles all interactions between the application and the system objects.  However, the same is true of communication between the system objects themselves.  The editing library, for example, knows nothing about MARC, validation, etc. – it only knows how to parse MarcEdit’s internal file format.  Likewise, the MARC library doesn’t know anything about validation, MARC21, or linked data.  Those parts live elsewhere.  The benefit of this approach is that I can develop each component independent of the other, and avoid breaking changes because all communication runs through the messenger.  This gives me a lot of flexibility and helps to enforce MarcEdit’s agnostic view of library data.  It’s also how I’ve been able to start including support for linked data components – as far as the tool is concerned, it’s just another format to be messaged.

Of course, the challenge with an approach like this then, is that most of MarcEdit’s functions don’t have a concept of a record.  Most functions, for reasons of performance, process data much like an XML sax processor.  Fields for edit raise events to denote areas of processing, as do errors, which then push the application into a rescue mode.  While this approach allows the tool to process data very quickly, and essentially remove size restrictions for data processing – it introduces issues if, for example, I want to expose a log of the underlying changes.  Logs exist – I use them in my debugging, but they exist on a component level, and they are not attached to any particular process.  I use messaging identifiers to determine what data I want to evaluate – but these logs are not meant to record a processing history, but rather, record component actions.  They can be muddled, but they give me exactly what I need when problems arise.  The challenge with developing logging for actual users, is that they would likely want actions associated with records.  So, to do that, I’ve added an event handler in the messaging layer.  This handles all interaction with the logging subsystem and essentially tracks the internal messaging identifier and assembles data.  This means that the logger still doesn’t have a good concept of what a record is, but the messenger does, and can act as a translator.

Anyway – this is how I’ll be providing logging.  It will also let me slowly expand the logging beyond the core editing functions if there is interest.  It is also how I’ll be able to build services around the log file – to provide parsing and log enhancement, for users that want to add record specific information to a log file, that goes beyond the simple record number identifier that will be used to track changes.  This would make log files more permanent (if for example the log was enhanced with a local identifier), but due to the way MarcEdit is designed, and the general lack of a standard control number across all MARC formats (in merging for example, merging on the 001 trips checks of 9 other fields that all could store associated control data), it is my belief that providing ways to enhance the log file after run, while an extra step, will allow me the most flexibility to potentially make greater user of the processing log in the future.  It also enables me to continue to keep MARCisms out of the processing library – and focus only on handling data edits.


So that’s pretty much the work in a nut shell.  So what do you get.  Well, once you turn it on, you get lots of stuff and a few new tools.  So, let’s walk through them.

Turning on Logging:

Since Logging only captures changes made within the MarcEditor, you find the logging settings in the MarcEditor Preferences Tab:


Once enabled, the tool will generate a new session in the Log folder each time the Editor starts a new Session.  With the logs, come log management.  From within the MarcEditor or the Main window, you find the following:

Main Window:


From the MarcEditor, you’ll find in Reports:


Functionally, both areas provide the same functionality, but the MarcEditor reports entry is scoped to the current session logfile and current record file loaded into the Editor (if one is loaded).  To manage old sessions, use the entry on the Main Window.

Advanced Log Management

To of the use cases that were laid out for me were the need to be able to enhance logs and the ability to extract only the modified records from a large file.  So, I’ve included an Advanced Management tool for just these kinds of queries:


This is an example run from within the MarcEditor. 


Anyway – this is a quick write-up.  I’ll be recording a couple sessions tomorrow.  I’ll also be working to make a new plugin available.



by reeset at January 17, 2017 06:29 AM

MarcEdit update

I’ve posted a new update for all versions of MarcEdit, and it’s a large one.  It might not look like it from the outside, but it represents close to 3 1/2 months of work.  The big change is related to the inclusion of a more detailed change log.  Users can turn on logging and see, at a low level, the actual changes made to specific data elements.  I’ve also added some additional logging enhancement features to allow users to extract just changed records, or enhance the log files with additional data.  For more information, see my next post on the new logging process.

The full change log:

* Enhancement: Z39.50: Sync’ng changes made to support Z39.50 servers that are sending records missing proper encoding guidelines.  I’m seeing a lot of these from Voyager…I fixed this in one context in the last update.  This should correct it everywhere.
* Enhancement: MARCEngine: 008 processing was included when processing MARCXML records in MARC21 to update the 008, particularly the element to note when a record has been truncated.  This is causing problems when working with holdings records in MARCXML – so I’ve added code to further distinguish when this byte change is needed.
* Enhancement: MarcEdit About Page: Added copy markers to simplify capturing of the Build and Version numbers.
* Enhancement: Build New Field: The tool will only create one field per record (regardless of existing field numbers) unless the replace existing value is selected. 
* Enhancement: Swap Field Function: new option to limit swap operations if not all defined subfields are present.
* Bug Fix: MARCValidator: Potential duplicates were being flagged when records had blank fields (or empty fields) in the elements being checked. 
* Update: MarcEditor: UI responsiveness updates
* New Feature: Logging.  Logging has been added to the MarcEditor and supports all global functions currently available via the task manager. 
* New Feature:  MarcEditor – Log Manager: View and delete log files.
* New Feature:  MarcEditor – Log Manager: Advanced Toolset. Ability to enhance logs (add additional marc data) or use the logs to extract just changed records.

You can download the update directly from the website at: or you can use the automated downloader in the program.

One last note, on the downloads page, I’ve added a directly listing that will provide access to the most previous 6.2 builds.  I’m doing this partly because some of these changes are so significant, that there may be behavior changes that crop up.  If something comes up that is preventing your work – uninstall the application and pull the previous version from the archive and then let me know what isn’t working. 



by reeset at January 17, 2017 06:16 AM

January 12, 2017

OCLC Cataloging and Metadata News

How to optimize your library sales with OCLC's complete MARC record solution

Of course libraries need bibliographic records, but how do you ensure they have the best records to increase the sales, discoverability and usage of your content?

January 12, 2017 03:30 PM

January 11, 2017


The long tail of library discovery


One of our first OCLC symposium speakers was Chris Anderson, the technology writer and former editor of Wired. He spoke for us at ALA Annual back in 2005 on the subject of his famous Wired article and soon-to-be published best-seller, The Long Tail. Like many others in our profession, I found the subject to be both interesting and appropriate to libraries, as did others whom we quoted in a NextSpace article at the time.

Libraries have been collecting, preserving and promoting “long tail materials” for centuries, of course. That’s the long tail of content. But we’ve also found that, when it comes to WorldCat, there’s a long tail for discovery.

A brief history of WorldCat traffic

For the first year or so after we released in 2006, most of the traffic came through the “front door”—from what we call “organic page views.” That’s where you visit the site itself, do a search, look at the results and then find materials in a nearby library.

Today organic page views account for around 12% of our traffic.

The next largest source of page views is from “signed partners.” Call this the “warehouse door,” used for large deliveries. These are organizations that work directly with OCLC in order to get programmatic, large-scale access to library metadata. As you might expect, Google is the leader in this category.

Other large “signed partners” include Goodreads, BibMe, Citavi, CiteFast and EasyBib. These are important partnerships because they make sure that library materials are found within the online services people use the most. But what might surprise you is that even taken all together, these partnerships no longer account for even half of the site’s usage. What accounts for the rest?

The long tail of discovery.

A thousand side doors into your library

WorldCat Page Views pie chartWe refer to the last category of sites sending traffic to WorldCat as “volunteer partners.” These are websites that choose on their own to link to libraries through by taking advantage of the various tools we make available for this purpose, including APIs, Linked Data sets, widgets and the like.

Volunteer partners include some large and well-known sites. For example, because of how WorldCat centralizes library data, it is much easier for search engines to find. Bing, for example, sent 9.6 million views last year, along with Yahoo (7.8 million), HathiTrust (1.3 million), Wikipedia (1.1 million) and others.

But the story doesn’t end with large web services. Because OCLC makes it easy to link to libraries through, many smaller organizations can provide discovery access to libraries, too. For OCLC, these sites represent the “long tail” of library discovery.

Knit one, purl two-hundred-thousand

At an OCLC Member Forum in Boston last year, I shared the fact that the knitting site,, is one of the most active examples of our third category of referrers: volunteer partners. Last year Ravelry users viewed library materials on around 200,000 times. That’s 200,000 opportunities for members of this tight-knit community (sorry, couldn’t help myself) to share knowledge and connect to local libraries and other fans.

More than half of all page views come from volunteer partners.
Click To Tweet

The reaction from librarians at the forum was very positive. Many had heard of or even used or recommended Ravelry. They were also pleased that an online community like this was able to easily send traffic to library materials through WorldCat. Some other interesting partners in this category include:

All together? From mighty search engines to knitters, volunteer partners now account for 58% of our page views on WorldCat.

Regardless of the size of the partner, the point is that easy tools and a variety of opportunities help OCLC democratize discovery and access to library materials. Meaning that the long tail now helps get people into libraries as well as give them content options once they’re there.

Visit the OCLC Developer Network to learn more about how you can incorporate WorldCat data into your applications and services.

The post The long tail of library discovery appeared first on OCLC Next.

by Chip Nilges at January 11, 2017 06:09 PM

January 06, 2017

Thingology (LibraryThing's ideas blog)

Introducing Syndetics Unbound


Short Version

Today we’re going public with a new product for libraries, jointly developed by LibraryThing and ProQuest. It’s called Syndetics Unbound, and it makes library catalogs better, with catalog enrichments that provide information about each item, and jumping-off points for exploring the catalog.

To see it in action, check out the Hartford Public Library in Hartford, CT. Here are some sample links:

We’ve also got a press release and a nifty marketing site.

UPDATE: Webinars Every Week!

We’re now having weekly webinars, in which you can learn all about Syndetics Unbound, and ask us questions. Visit ProQuest’s WebEx portal to see the schedule and sign up!

Long Version


The Basic Idea

Syndetics Unbound aims to make patrons happier and increase circulation. It works by enhancing discovery within your OPAC, giving patrons useful information about books, movies, music, and video games, and helping them find other things they like. This means adding elements like cover images, summaries, recommendations, series, tags, and both professional and user reviews.

In one sense, Syndetics Unbound combines products—the ProQuest product Syndetics Plus and the LibraryThing products LibraryThing for Libraries and Book Display Widgets. In a more important sense, however, it leaps forward from these products to something new, simple, and powerful. New elements were invented. Static elements have become newly dynamic. Buttons provide deep-dives into your library’s collection. And—we think—everything looks better than anything Syndetics or LibraryThing have done before! (That’s one of only two exclamation points in this blog post, so we mean it.)


Syndetics Unbound is a complete and unified solution, not a menu of options spread across one or even multiple vendors.

This simplicity starts with the design, which is made to look good out of the box, already configured for your OPAC and look.

The installation requirements for Syndetics Unbound are minimal. If you already have Syndetics Plus or LibraryThing for Libraries, you’re all set. If you’ve never been a customer, you only need to add a line of HTML to your OPAC, and to upload your holdings.

Although it’s simple, we didn’t neglect options. Libraries can reorder elements, or drop them entirely. We expect libraries will pick and choose, and evaluate elements according to patron needs, or feedback from our detailed usage stats. Libraries can also tweak the look and feel with custom CSS stylesheets.

And simplicity is cheap. To assemble a not-quite-equivalent bundle from ProQuest’s and LibraryThing’s separate offerings would cost far more. We want everyone who has Syndetics Unbound to have it in its full glory.

Comprehensiveness and Enrichments

Syndetics Unbound enriches your catalog with some sixteen enrichments, but the number is less important than the options they encompass. These include both professional and user-generated content, information about the item you’re looking at, and jumping-off points to explore similar items.

Quick descriptions of the enrichments:

Boilterplate covers for items without covers.

Premium Cover Service. Syndetics offers the most comprehensive cover database in existence for libraries—over 25 million full-color cover images for books, videos, DVDs, and CDs, with thousands of new covers added every week.

For Syndetics Unbound, we added boilerplate covers for items that don’t have a cover, which include the title, author, and media type.

Summaries. Over 18 million essential summaries and annotations, so patrons know what the book’s about.

About the Author. This section includes the author biography and a small shelf of other items by the author. The section is also adorned by a small author photo—a first in the catalog, although familiar elsewhere on the web.

Look Inside. Includes three previous Syndetics enrichments—first chapters or excerpts, table of contents and large-size covers—newly presented as a “peek inside the book” feature.

Series. Shows a book’s series, including reading order. If the library is missing part of the series, those covers are shown but grayed out.

You May Also Like. Provides sharp, on-the-spot readers advisory in your catalog, with the option to browse a larger world of suggestions, drawn from LibraryThing members and big-data algorithms. In this and other enrichments, Syndetics Unbound only recommends items that your library owns.

The Syndetics Unbound recommendations cover far more of your collection than any similar service. For example, statistics from the Hartford Public Library show this feature on 88% of items viewed.

Professional Reviews includes more than 5.4 million reviews from Library Journal, School Library Journal, New York Times, The Guardian, The Horn Book, BookList, BookSeller + Publisher Magazine, Choice, Publisher’s Weekly, and Kirkus. A la carte review sources include Voice of Youth Advocates: VOYA, Doody’s Medical Reviews and Quill and Quire.

Reader Reviews includes more than 1.5 million vetted, reader reviews from LibraryThing members. It also allows patrons and librarians to add their own ratings and reviews, right in your catalog, and then showcase them on a library’s home page and social media.

Also Available As helps patrons find other available formats and versions of a title in your collection, including paper, audio, ebook, and translations.

Exploring the tag system

Tags rethinks LibraryThing’s celebrated tag clouds—redesigning them toward simplicity and consistency, and away from the “ransom note” look of most clouds. As data, tags are based on over 131 million tags created by LibraryThing members, and hand-vetted by our staff librarians for quality. A new exploration interface allows patrons to explore what LibraryThing calls “tag mashes”—finding books by combinations of tags—in a simple faceted way.

I’m going to be blogging about the redesign of tag clouds in the near future. Considering dozens of designs, we decided on a clean break with the past. (I expect it will get some reactions.)

Book Profile is a newly dynamic version of what Bowker has done for years—analyzing thousands of new works of fiction, short-story collections, biographies, autobiographies, and memoirs annually. Now every term is clickable, and patrons can search and browse over one million profiles.

Explore Reading Levels

Reading Level is a newly dynamic way to see and explore other books in the same age and grade range. Reading Level also includes Metametrics Lexile® Framework for Reading. Click the “more” button to get a new, super-powered reading-level explorer. This is one my favorite features! (Second and last exclamation point.)

Awards highlights the awards a title has won, and helps patrons find highly-awarded books in your collection. Includes biggies like the National Book Award and the Booker Prize, but also smaller awards like the Bram Stoker Award and Oklahoma’s Sequoyah Book Award.

Browse Shelf gives your patrons the context and serendipity of browsing a physical shelf, using your call numbers. Includes a mini shelf-browser that sits on your detail pages, and a full-screen version, launched from the detail page.

Video and Music adds summaries and other information for more than four million video and music titles including annotations, performers, track listings, release dates, genres, keywords, and themes.

Video Games provides game descriptions, ESRB ratings, star ratings, system requirements, and even screenshots.

Book Display Widgets. Finally, Syndetics Unbound isn’t limited to the catalog, but includes the LibraryThing product Book Display Widgets—virtual book displays that go on your library’s homepage, blog, LibGuides, Facebook, Twitter, Pinterest, or even in email newsletters. Display Widgets can be filled with preset content, such as popular titles, new titles, DVDs, journals, series, awards, tags, and more. Or you point them at a web page, RSS feed, or list of ISBNs, UPCs, or ISSNs. If your data is dynamic, the widget updates automatically.

Here’s a page of Book Display Widget examples.

Find out More

Made it this far? You really need to see Syndetics Unbound in action.

Check it Out. Again, here are some sample links of Syndetics Unbound at Hartford Public Library in Hartford, CT: The Raven Boys by Maggie Stiefvater, Alexander Hamilton by Ron Chernow, Faithful Place by Tana French.

Webinars. We hold webinars every Tuesday and walk you through the different elements and answer questions. To sign up for a webinar, visit this Webex page and search for “Syndetics Unbound.”

Interested in Syndetics Unbound at your library? Go here to contact a representative at ProQuest. Or read more about at the Syndetics Unbound website. Or email us at and we’ll help you find the right person or resource.

by Tim at January 06, 2017 07:22 PM

January 05, 2017

Terry's Worklog

MarcEdit Update Posted

Happy 2017!  I hope that everyone had a fine holiday season.  I spent some time over the past couple weeks away from MarcEdit and doing a little bit of writing.  This put my usual holiday update behind a bit (which I apologize about, since it included a couple bug fixes that folks were waiting for) – but it was nice to spend some time writing, building robots with my son, and otherwise taking it slow. 

With that said, I have posted an Update for the Windows/Linux versions of MarcEdit and will be working to update the Mac version (with appropriate fixes and new stuff) later this week.  I’m running slightly behind because of the above mentioned robot building.  I coach/mentor a middle school robotics team, and their regional competition is this weekend, so it seems most of my free time is being spent with the kids working on their robot and research project.

Anyway – the Changelog:

  • Update: Z39.50: When downloading records that are in UTF8, the tool will ensure that the LDR byte position related to character encoding is set if the system is supplying MARC21 or USMARC records
  • Update: MARCEngine: When translating UTF8 conversions, the tool will always check the LDR position and set when appropriate (currently, this only happens when MarcEdit actually does a character conversion).
  • Bug Fix: Print A Record Per Page — Some record data would be deleted if the last line on a page should have printed onto the next page. 
  • Bug Fix: Exporting Tab Delimited Records — the Secondary delimiter isn’t being placed correctly.  This is a regression caused when adding the ability to protect context between subfields on export.
  • Enhancement: MARCValidator — program will now do a quick record deduplication check and if found, will provide a message letting folks know that duplicate records are likely in the file.
  • Enhancement: Console Program — added -dups? switch to allow a quick check to see if duplicate records are likely in a file.
  • Enhancement: Console Program — added an -experimental switch to enable direct processing of tasks.  This improves task processing from within the Console, but is still a little experimental.  This is the best method to run task editing via Linux. 
  • Enhancement: About Windows — added the update code to the window.  This is the code (not the build number) that is most important for me when debugging.

Two specific changes I want to highlight.  One is the Console changes.  I’ve added two new switches.  The first affects running tasks via the console program.  I’ve created an –experimental switch to move to a new way of processing tasks that is faster and more consistent on all platforms (Windows/Linux).  This will eventually replace the old method, but for now, uses wanting to try the new method will need to use this flag to call it explicitly.  So, for example:

>>%MARCEDIT_PATH\cmarcedit.exe –s “.\Desktop\myfile.mrc” –d “.\Desktop\processed.mrc” –task “.\Desktop\mytask.txt” –experimental

That would initiate the new process.  Leaving off the –experimental switch would use the older method which simply shells the task to the main MarcEdit.exe application, though, run silently.  Since I’m using the console program more and more when helping users setting up automated workflows on Linux, I needed to move this process into the cmarcedit.exe stack – which also improves the speed and reliability, a lot.

The second change deals with the validator.  A question on the list this week was complicated because the source file had duplicates (when it shouldn’t have).  So, in thinking about this – I added a process to the Validator’s normal validation process that will check for likely duplicates.  When you run the validator now, if it encounters likely duplicates, you’ll see a note at the top of the log.  For example:


This won’t tell you which records are duplicates, or how many, but it will let you know, up front, that the records likely contain duplicates.  Likewise, you can get this same information from the commandline.  Run the following:

>>%MARCEDIT_PATH%\cmarcedit.exe –s “.\Desktop\myfile.mrc” –dups?

And the tool will again, give you a message letting you know if duplicates are likely:


These are small changes, but I think changes that will make a significant difference for users working with data that comes from 3rd-party sources. 

Downloads should be available via the automated updating functionality, or by getting the files directly from:


by reeset at January 05, 2017 02:53 PM

January 04, 2017


Global Council at a crossroads

Global Council Select1I recently attended the OCLC Asia Pacific Regional Council Meeting in Hong Kong where the theme was “Libraries at the Crossroads.” It was a great topic, one that is relevant to us all. As librarians, we continuously face crossroads—changing patron preferences, evolving institutions, new technologies. We had excellent discussions about how we can best move forward, together. It’s the same theme we’ll explore at the upcoming Europe, Middle East and Africa Regional Council Meeting in Germany in February. I’m greatly looking forward to continuing the discussion.

This “crossroads” analogy also has framed recent Global Council discussions and decisions. At our meeting this past November, Delegates agreed to sharpen our focus on member activities, such as OCLC regional meetings and product, user and working groups that advance the interests of our member institutions.

What is OCLC Global Council?

As an OCLC member, you have a voice in the cooperative through our Global Council, which is made up of three Regional Councils: the Americas (ARC), Europe, Middle East and Africa (EMEA) and Asia Pacific (AP). Member Delegates reflect the diversity of our membership. The 48 elected Delegates represent 20 different countries and a variety of library types. These Delegates work on your behalf to keep OCLC leadership informed on member needs and interests.

Each year, Delegates travel to the OCLC headquarters in Dublin, Ohio, USA, to share what we are hearing and learning from members. Often, these conversations raise awareness about specific needs like the importance of Unicode to the global cataloging community. One of our most important responsibilities is to elect six library members to the Board of Trustees who, along with the other eight members of the Board, answer to and represent our member institutions. We also bring members together to explore industry-wide themes and hear from thought leaders around the globe at regional meetings, Member Forums and other gatherings.

Reflecting on reflecting

Over the last year, Global Council has undergone a transformation that will strengthen Delegates’ ability to reflect the needs and interests of our members. Together, our professional expertise and understanding informs OCLC on needs within our broad-based member communities. Our diversity as a Council allows us to raise awareness of regional issues and identify important trends and opportunities based on our knowledge of the different library types in which we work.

To do this, we have agreed to take a more active role in shaping the ways in which OCLC engages with members worldwide. Over the coming months, you will have more opportunities to get to know your Delegates and learn how we can work more closely together to participate, share and learn from our colleagues and peers.

Engaging through your regional Delegates, you have a connection to OCLC and a voice in shaping cooperative participation. There are a variety of ways that any individual librarian can become more personally and professionally engaged with OCLC: everything from user and advisory groups to serving on a Regional Council. But today I want to emphasize an important and defining activity that being a member of OCLC offers: the ability to have a voice and a vote by electing Delegates to represent your interests on the Regional and Global Councils.

Elections for OCLC Global Council will be held from 1 February to 15 March 2017.
Click To Tweet

Elections for Global Council will be held from 1 February to 15 March 2017. You will be introduced to an impressive slate of candidates from your region—your peers and colleagues who have volunteered to be considered for a seat on the Council. They are leaders within their communities who will work on your behalf to bring energy, enthusiasm and expertise to the world’s largest library cooperative.

Each member institution has one vote, and one designated voting representative who will vote to elect Delegates within your region. Please make sure your voting contact’s name is current. If you don’t know who your library’s voting representative is, ask your Manager or Director, or contact us at

As President of Global Council, I am proud to be able to work so closely with this diverse group of dedicated library leaders who come together to share their viewpoints of libraries of all types and from countries all over the world. I’d like to thank these candidates, and all our members, for everything they do to help guide and propel OCLC forward.

The post Global Council at a crossroads appeared first on OCLC Next.

by Peter Sidorko at January 04, 2017 02:50 PM