Improved Quest NMSP migration jobs for standard Notes Document Library to SharePoint (Online)

  • Posted by Marten Vosmer
  • March 19, 2019
  • Quest NMSP

When migrating a Notes document library to SharePoint (Online), Quest Notes Migrator for SharePoint (NMSP) provides out of the box several pre-configured migration jobs. Depending on the Notes database content (Rich Text, attachments, responses) and the intended use of the migrated data (archiving, SharePoint employment), you can choose a job that best fits your needs. In our blog ‘Notes Rich Text Migration to SharePoint Online’, we discussed some SharePoint targets to migrate Rich Text to (Custom List, WikiPage, Document Set, ShareFlex), and explained which option works best for which scenarios.

In summary:

  • Migrate to a custom list if the document library doesn’t contain any files or objects
  • Migrate to a file library if the Rich Text fields contain files/objects only
  • Use any standard Quest migration job when:
    • The migrated data is for archive/read-only use, AND
    • There are no readers fields used in Notes, AND
    • There is no response structure in Notes

The vast majority of the document libraries contain Rich Text fields storing a mixture of content: attachments, pictures, objects, tables, formatted text, etc. Also there may be responses to documents, or some Notes documents may be allowed limited access. In all these scenarios, we recommend to use the Document Set as the migration target type.

Document Set

A Document Set is a special type of folder that stores metadata that applies to all of its content. In the migration job specification, you can freely define what you want to include in the migrated Document Set. We use the following approach:

  • A Document Set is created with the name based on the subject of the Notes document
  • All attachments and objects in the Notes document are migrated to the root of the Document Set
  • a PDF document is created, displaying the Notes Rich Text Body field:
    • images are embedded in the PDF;
    • attachment icons are displayed on the PDF. When clicking the icon, the linked attachment in the Document Set is opened;
    • all other Rich Text content (like tables, formatted text) is displayed like in Lotus Notes.
  • Additional Notes field content is stored as Metadata of the Document Set.

When using a standard Quest job to migrate a Notes document library to a SharePoint Document Set library, you will notice that some aspects are not handled well. In this blog, I will pinpoint these issues, and show you how we handle these in our improved Quest jobs.

Demonstration database

For demonstration purpose, we use a small Notes document library that contains 2 main documents, a response and a response-to-response document.

  • The ‘Goats in a tree’ document is a main document without responses, and has one assigned category: ‘Animals’.
  • The ‘Circus’ document is a main document with a response and a response-to-response document. This document has 2 assigned categories: ‘Locations\Circus’ and ‘Animals’. In the ‘By Category’ view, you can see that this set of documents is displayed under both categories.

All documents contain a Rich Text Body field with content of any kind, like attachments, pictures, tables, formatted text, etc. See below the document with subject ‘Circus’

Challenge 1: Notes Document-Response Structure

Like in a Notes Discussion Board and Team Room, documents in a Notes Document Library may be organized in a hierarchical Document-Response structure. When using a standard Quest job for migration, this structure is not migrated: each Notes document is migrated to a single SharePoint list item without any reference to a possible parent/child item.

Of course you can migrate the Universal ID and $REF properties and use these to build an interface that connects these items. But you will need extra coding in SharePoint to do this.

Our improved migration job evaluates the existing response hierarchy in the source definition, so that you can use it when writing the content to the target.

Solution: parent and all responses to 1 Document Set

In our improved NMSP migration jobs, each main Notes document is migrated to one Document Set. Additionally, all descendants (responses and response-to-responses) are migrated to the same Document Set where the parent lives.

As you can see below, the Notes parent document ‘Circus’ with 2 responses is migrated to a document set called ‘Circus’ that contains 3 Rich Text Body PDF’s, and all attachments. To keep track of the response hierarchy, the Rich Text PDF filename holds the value of the parent document: … (response to …)

The ((Rich Text Body)) naming of the PDF file is deliberate, to make sure that these PDF Rich Text files always end up at the top of the file list. This is especially useful when a Notes document contains many attachments that are all migrated to the same location.

Below is a display of the ((RichText Body)) Circus PDF. Note that the ‘To Do.docx’ attachment is just an icon with a link pointing to the migrated To Do file. You can also choose to migrate attachments directly in the PDF itself, however this may cause slow response times when opening a PDF that contains many/large attachments.

This migration approach uses only 1 job to migrate all documents, responses, and response-to-responses at once, and it works fine as long as the number of responses and hierarchical response levels are limited. If there are many responses and many sub-levels, the content of the Document Set will become very complicated and unreadable. Therefore, we have developed an alternative migration solution.

Solution: response folder hierarchy within Document Set

Some Notes document libraries contains many responses and an extensive response hierarchy. For these libraries we developed a solution to migrate responses to a folder hierarchy within a document set. 2 Migration jobs should run in sequence: one job for main documents, and one job for response documents:

  • The Notes main documents are migrated to a Document Set. One Document Set for each Notes document.
  • The Notes response documents are migrated to folders within the parent Document Set: each folder contains the content of one response document, including attachments etc.

When using this migration job, take the following into account:

  • Only information from the main Notes document is migrated to metadata of the Document Set.
  • Depending on the extent of folder sublevels, the url may become too long. We created a workable solution to calculate and shorten the overall length of the url.
  • Because the response folders are created within the list item, these folders are not displayed in the navigator.

Challenge 2: Broken response structure caused by user actions

Elaborating on the Notes response structure, a user with sufficient permissions can perform various actions in a Notes document library, that will break existing or create new response structures:

  1. delete or cut/paste a main document. This results in previous associated response documents being separated from their parent, i.e. children without parent documents (orphan responses);
  2. copy/paste documents from the same or another databases: the pasted documents will become children of the selected parent;
  3. modify the subject of the main document.

The ParentSubject of the child documents will not be updated.

When any of the before mentioned actions is executed, the standard Quest migration jobs will fail to migrate the response structure. The reason is that these jobs all use the ParentSubject value of the child documents to identify the parent. However, this value is not updated after the action was performed, and will therefore point to non-existing or even incorrect parent documents.

We have seen a lot of migrations where items were missing or migrated to wrong places, because Parent Subject instead of $REF was used to determine response structures. We also saw that attachments of many ‘orphan’ response documents were migrated to the root of a SharePoint Library, because no main document could be specified to create a document set for.

The improved NMSP migration jobs take care of broken document-response structures, ‘orphan’ responses and Subject modifications of parent documents.

  • Orphans are migrated to a folder with the name containing the subject plus ‘(response without parent)’. The added text is configurable in the job.
  • By using $REF instead of ParentSubject/OriginalSubject value, the cut/pasted/copied and modified subject documents will find their way to the correct Document Set in SharePoint.

Example 1

When I cut/paste the ‘Circus’ document, notice that all responses have disappeared:

However, when you open the ‘By Author’ view, you find that these responses are still available in the database. The hierarchical link with the main ‘Circus’ document is broken, however.

In our improved NMSP migration job, these parentless ‘orphan’ response documents will be migrated into their own Document Set, named after the response document that is highest in hierarchy (in this case: ‘Where did you get this picture’). The Document Set also indicates that this is a response without parent:

If you do not handle these orphan responses correctly, content may end up in wrong places in the SharePoint list. You may find attachments migrated to the root of the library, outside any Document Set, or added to an unrelated Document Set with the same name.

Example 2

In this example, 2 Reply documents were copied from a mail file and pasted as responses to the ‘Goats in a tree’ document. With the standard Quest job, these Notes documents will be migrated to a new Document Set, because there is no ParentSubject field. But with the improved NMSP migration job, they end up in the correct Document Set:

Example 3

When I change the subject of the ‘Circus’ document to ‘Circus and animals’, notice that – although the link on the response document still points to the parent document – the Parent Subject field is not updated.

The improved migration job uses the $REF field (and not the value of the ParentSubject field) to find parent documents. All responses are migrated to the Document Set with the new name.

Challenge 3: Notes documents with the same subject

When migrating Notes documents to Document Sets, the option ‘Unique by Document ID’ defines the Document Set target location: when set to True (default), each Notes document is migrated to its own Document Set. This works fine when you don’t have responses in your Notes database. Even when a document with the same subject is found, both are migrated to separate Document Sets: one of the Document Sets name is extended with the Universal ID of the Notes document.

Once you have responses in your database, and you’d obviously want to migrate them to the same Document Set as their parent, you need to set the ‘Unique by Document ID’ option to ‘False’.

So far so good. All responses are now migrated to the correct Document Set where the parent lives. But problems occur when the document library contains Notes main documents with the same subject. When using the standard Quest job, both Circus documents and all of their descendants will be merged into one Document Set called ‘Circus’, even though these documents are completely unrelated.

The improved migration jobs enableyou to specify a list of ‘Double Subjects’ that will be used to set the name for a Document Set. You can retrieve this list of ‘Double Subjects’ from our ‘NMSP Views’ (more information here). Copy the view ‘zz_SubjectMain’ from the NMSP Views database to the source database, and click the view action button ‘Show Double Subjects’. This will generate a line of code to identify all the ‘double subject’ documents in your Notes document library.

Apply this line of code to the _DocSetName formula in the Source Data Definition:

If the migrator finds a Notes main document with subject ‘Circus’, it will extend the Document Set name with 4 characters of the Document Unique ID (you can change the number of characters in the code). The code makes sure that response documents are migrated to the correct Document Set.

Challenge 4: Multiple Categories and Category levels

When a Notes document contains more than 1 categorythis document is displayed under each category in the Notes database. In SharePoint this is not possible. A SharePoint grouped view will display the item under 1 category only. Additionally, SharePoint can only group up to 2 hierarchical levels, whereas a category in Notes can have many more levels.

Before you start migrating Notes documents containing categories, you need to decide how to handle categories. If Notes documents have multiple categories, and you want to display the item under each category in SharePoint, you need to use a Managed Metadata column that you can use as a view navigator in your SharePoint library. A Managed Metadata column is also a good solution for displaying documents that have categories with more than 2 hierarchical levels.

Migrating to Managed Metadata takes a bit more effort than migration to standard columns, so you want to know in advance whether a Managed Metadata column is necessary. To do so, you can use 2 views from the NMSP Views database: the zz_Categories view and zz_CategoryLevels view:

The zz_Categories view shows the documents that have more than 1 category assigned. If you migrate to a normal column, only the 1st category is migrated. If you migrate to Managed Metadata column, all categories are migrated.

The zz_CategoryLevels views shows how many levels a category contains. In SharePoint, you can only use 2 grouped columns in a view (for instance Cat and SubCat). When migrating to a Managed Metadata column, you can display more than 2 levels in the navigator.

Example: Migrate a Category to 2 normal grouped columns

In this example the Category field is split up in 2 parts and mapped to a CAT01 and CAT02 column in SharePoint. Both columns will then be used as grouping columns in the SharePoint list.

Notice that only the first category is migrated. For document ‘Circus’ the 2nd category ‘Animals’ is skipped:

Because the category has 2 levels at max (‘Circus’ document), there is no problem to use a view with grouped columns. Note that although the ‘Goats in a tree’ document has a category without a sub-level, the subcategory (CAT02) is displayed anyway, being empty.

Example: Migrate Category to a Managed Metadata column

In this example, the Category field is mapped to a Managed Metadata column, called CATMM. The ‘MultiValueDisposition’ for this column is set to ‘All’.

In SharePoint, you can use this Managed Metadata column as a navigator for your list. Note that the ‘Circus’ item is displayed under both categories ‘Animals’ and ‘Locations\Circus’.

Customizable code in the Migration Job

Most of the work in our improved jobs is done at the Notes Source Data Definition, where we use formulas for Categories, Document Set Name, PDF Rich Text file name, and Subject. You can adjust these formulas at will. At the top of the formulas, variables are declared that can be adjusted. For instance, the iUNID value determines the number of characters that are added to ‘Double Subject’ Document Set names.

Why Lialis?

Lialis has been in Notes development for over 20 years and the past 7 years we increasingly focus on Notes migrations. These skills help us to use Notes formulas to fine-tune Quest migration jobs and as a consequence mimic as closely as possible the original Notes structure in SharePoint. This will reduce user impact and make the transition to another platform such as SharePoint Online smoother.

In case .

To help you make your Notes to SharePoint migration project less daunting we can give you access to our pre-built Quest NMSP migration jobs. Or our experienced consultants can assist you in the migration process.

Please contact us here in case you are interested