Revision History

Feb 28, 2010: added Perl one-liner for comma-semicolon substitution in keywords field following export
Jan 4, 2009: addition of link to Zotero forum thread on field conversion in word processing documents; minor clarifications of prose.
August 5, 2008: correction of typos; addition of link back to post in the Zotero forum; added BE library verification command.
July 6, 2008: correction of typos
June 29, 2008: initial version

Overview

The Solution (Short Version)

The Steps Involved in Migration

The full process of moving from one bibliographic manager to another—even when just a single user will be building and making exclusive use of the database—involves a number of steps aside from installing and configuring the new software:

1. You also have to learn the ins and outs of a new interface, including the way it works (or doesn't) with your word processor.
2. You may need to install plug-ins allowing the new bibliographic software to interact with your word processor or other software.
   
3. You may need to obtain, define, or at least double-check bibliographic stylesheets for the journals or bibliographic formatting standards you typically use.
4. Most recently-developed bibliographic packages permit online search and citation download via one or more protocols (Z39.50, http, etc.). In some cases you may need to configure the software to deal with proxies or other aspects of your Internet connection.
   
5. Most importantly here, you have to export your existing bibliographic database and import it into your new manager, possibly doing some data clean-up or processing in-between.  This is the part of the migration process this document is concerned with.
6. A separate issue (not addressed here) is the possible conversion of citation fields in legacy documents.  If you have unfinished drafts or any documents that you hope to cut-and-paste from that contain embedded citation fields (constructed using your old bibliographic software), you'll need to consider whether or not having those citations in the older format is a concern. My experience is that this is the aspect of migration that has been least addressed, and very little provision is ever made for converting existing documents to new bibliographic managers (but see this thread for relevant discussion).

Considerations Specific to Bookends and Zotero

Consideration #1: Export/Import Formats

• Bookends [presumably version 9]
• Bookends (8)
• Bookends (7)
• tab-delimited text
• XML (Endnote 7)
• XML (Endnote 8 or later)
• pseudo-export as a formatted, text-based bibliography

• Refer/BibIX
• RIS
  
• BibTeX
• MODS (Metadata Object Description Schema)
• Unqualified Dublin Core RDF
• Zotero RDF

Consideration #2: Mapping Reference Types and Fields

Consideration #3: Text Encoding

One Approach to Migrating Your Library

Phase 1: Export

Download this archive, which contains a Bookends Format file (named "BEDump"), a Perl script (named "be2z") and a data file on which it depends ("be2zm.pm").  Place the dump format file in your Bookends format directory (typically something like <home-directory>/Library/Application Support/Bookends/Custom Formats/). Writing out your entire Bookends library using the BEDump format will ensure that every field—regardless of reference type or customizations—gets written out to a text file (UTF-16 encoded, under MacOS X).  This file can then be parsed and manipulated much more simply using text editors such as TextWrangler or text-savvy tools such as Perl. You can also construct a BEDump format file yourself (which would allow you, for example, to employ this solution with Bookends version 10).  Simply specify the following in the  Primary Order field (for all 27 reference types), ensuring no additional wrapping takes place if you cut-and-paste:

$%y$ y¬$%a$ a¬$%b$ b¬$%d$ d¬$%e$ e¬$%f$ f¬$%h$ h¬$%i$ i¬$%j$ j¬$%k$ k¬$%l$ l¬$%n$ n¬$%p$ p¬$%p-$ p-¬$%s$ s¬$%t$ t¬$%u$ u¬$%v$ v¬$%u1$ u1¬$%u2$ u2¬$%u3$ u3¬$%u4$ u4¬$%u5$ u5¬$%u6$ u6¬$%u7$ u7¬$%u8$ u8¬$%u9$ u9¬$%u10$ u10¬$%u11$ u11¬$%u12$ u12¬$%u13$ u13¬$%u14$ u14¬$%u15$ u15¬$%u16$ u16¬$%z$ z¬$%@$ @¬$%#$ #¬$%!$ !¬$%%$¬

You needn't specify a Secondary Order, and authors' and editors' names can in general be first-name-first or last-name-first (Zotero appears to import either). However, the Punctuation:Between names and Punctuation:Between last names must be a semicolon. See p. 126 of the Bookends User Manual, version 9, for details on the process of creating a file format.

Once you have the format file in place, ensure it's available by selecting Biblio->Formats Manager..., checking the BEDump checkbox, and clicking Done.

At this point you will likely want to do any final clean-up of your Bookends references (e.g. consolidation of keywords; removal of returns from all but the author, editor, keyword, and note fields; insertion of semicolons where necessary between multiple keywords).  Perform a final library verification using File->Database Maintenance->Verify... (see p. 89 of the Users' Manual) and back up the result.

Ensure all your references are showing (Window->List View->All or CMD-L) and select all the records in your library (Refs->Mark->Mark All References or SHIFT-CMD-A), or select some subset if you don't wish to export/import them all.  Go to the Bibliography Formatter (Biblio->Bibliography Formatter... or SHIFT-CMD-B), select BEDump and as UTF-8 from the output format pull-down menus, and select Send the bibiography to Disk. Click Make Bib and you'll be prompted to name the export file. Export may take several minutes depending on the speed of your Mac and the size of your library.

Tip: if you have a lot of records with keywords separated by commas, you can either do a find-and-replace in Bookends, or your can export the file first and run the following command on the result before going on to Phase 2:

[lane@localhost]$ perl -0x0D -pe 's/,/;/g if /^\%k /' < [exported_file.in] > [exported_file.out]

This will substitute semicolons for all commas in any line in exported_file.in that begins with %k (i.e. the keyword field of each record), writing the result to exported_file.out. Note that the "-0x0D" flag tells Perl to assume that lines in the input file end with a carriage return (\r) rather than a newline (\n). In fact, this command could be used to perform arbitrary substitutions within any given field (see the other Bookends codes above and use one of them rather than %k).

Phase 2: Processing

• A field_map (defined in be2zm.pm) allows the user to customize the mapping from Bookends fields to RIS/Zotero fields.
• Bookends keywords (as well as any other user-specified field) get imported as Zotero tags, and Bookends notes (as well as any other user-specified field) get imported as attached Zotero notes.
• While keywords is a single field in Bookends, keywords separated by semicolons get translated into separate Zotero tags by be2z.  The same is true for multiple attachments (if separated by semicolons)—see the thread on this problem in the forum.
• Multiple authors and editors placed in a single Bookends field (but separated by semicolons) get translated into separate Zotero authors/editors.
  
• A type_map (defined in be2zm.pm) allows the user to customize the mapping from Bookends reference types to RIS/Zotero reference types.
  
• Translation to the expected IBM-850 encoding is performed where possible (preserving special characters).
• An encoding_map (defined in be2zm.pm) allows the user to specify substitutions for characters that are not directly translatable.
• Links to attached documents are preserved, even when attachments have special characters in their filenames such as a space or ampersand (assuming all attachments are in a single folder accessible by be2z and Zotero at import).
• Attached document types (HTML, PDF, or image) are differentiated by filename extension, and get annotated accordingly by Zotero.
• While there are several cases in Bookends in which a reference with an empty author field makes sense (e.g. an edited volume), in Zotero (for some reference types) non-empty author tags that appear after empty ones in the same reference can cause a failure on import. be2z reorders author tags to avoid this.
  

Phase 3: Import

Requirements and Limitations

be2z requires Perl (it's been tested successfully on versions 5.8.6 and 5.8.8) as well as a couple of the modules that come standard with Perl (Encode and PerlIO).

Note that be2z will include periods in imported keywords, and splits up authors, editors, keywords and any fields mapped to those fields on semicolons and newlines, not spaces or periods. Thus the following in the keyword field of a Bookends reference:

Weimar Republic -- Germany. Optics.