NDDM Technical Manual – Pipeliner

Please check this page for a demo video. Note that the video is showing an older NDDM version, currently the left navigation menu is looking differently, but the overall functionality is still the same.

Pipeliner Fundamentals

• Read-only dump of any Notes database/app to SharePoint Online
• Fast adaptation for end users because it looks just like Notes

Pipeliner supports

• Any Notes database size supported by Domino
• Any Notes database design supported by Domino
• Any number of Notes documents supported by Domino
• Any number of attachments supported by Domino
• Human effort to archive any Notes database in less than 5 minutes per Notes database (1000 nsf in 80 hr effort)
• Notes design knowledge not needed
• Central Notes document link system
• Notes response documents supported
• Conversion of OLE objects to normal files
• High speed parallel processing on multiple servers
• 100% Domino web apps may need some development

Notes view conversion example, the image below shows a Notes app view

The image below shows the SharePoint Online view, created by Pipeliner for the Notes view displayed above

Pipeliner creates SharePoint aspx page views for all available (non hidden) Notes app views.

Notes document conversion example, the image below shows a Notes document

The image below shows the SharePoint Online document, created by Pipeliner for the Notes document displayed above

A schematic drawing of how a Notes database is converted to a read-only export on SharePoint Online by Pipeliner:

The schematic drawing in the image below shows the SharePoint Online export result created by Pipeliner. Each aspx item in the root of the library represents a migrated Notes view, with similar naming. All these views are created fully automatically by Pipeliner.

The content of a migrated view is made up of links to SharePoint items that represent the Notes documents. These migrated Notes document items are stored in the folder _Notescontent. The migrated attachments are stored in the files subfolders.

When our clients have a need to archive many (>200) Notes databases to SPO, the Notes database export infrastructure can be setup according to the following diagram:

The Notes databases that are archived remain on the existing Domino servers. One or multiple Domino Pipeliner migration servers are installed. They pull the data from the source Domino servers, convert it to a structure of files and folders on disk, and then uploaded it to SPO. Everything in this process is fully automated. SPO sites are automatically created, the SPO site homepage is set to the library root, and all files are uploaded with PowerShell.

NDDM is fully integrated with Pipeliner to allow clients to archive many thousands of Notes databases to SharePoint Online fast, easy and in a short time frame.

In this setup, we have archived 10,000 Notes databases in 4 months to SPO for a USA based client. These 10,000 Notes databases where collectively holding 11M Notes documents, 20TB on data. We installed 35 Pipeliner migration servers. The Pipeliner servers work load was managed by NDDM.

Fully automated Notes database archival to SharePoint Online

• Use NDDM buttons to manage the complete archival process, no time-consuming tasks
• Opening Pipeliner is only needed in case of problems
• 20 Domino Pipe servers can archive 60 Notes databases in parallel (this can be increased)
• Fully automated process:
  • SharePoint site creation (communication site), SharePoint Library creation
  • Upload of the Notes archive to a SharePoint Library
  • SharePoint site permissions are set for the owner, users (taken from Notes) and administrators
  • SharePoint site homepage is set to the archive library
• NDDM retrieves the archive log
• NDDM button to quickly compare the Notes documents per form with the item forms on SharePoint
• NDDM matches the number of Notes database against the number of SharePoint libraries to find migration issues
• NDDM is used to migrate end user permissions from Notes to SharePoint

Archive flow managed by NDDM

• Approximately 20 Domino servers in Client Cloud dedicated for Notes database dumps to SharePoint
• These 20 Domino Servers replicate Pipeliner.nsf between each other
• NDDM creates archive task in Pipeliner.nsf on server Pipe1
• This archive task is set to be carried out by server Pipe8 (by NDDM) for example
• This archive task replicates to Domino server Pipe8
• Pipe8 pulls the Notes database from the source server, creates an export on disk and uploads this data to SharePoint via PNP PowerShell
• NDDM retrieves archive progress and log
• IT checks log and compares the number of Notes documents per form with SharePoint items (sample wise)
• NDDM mailing function can be used to report archive to owner
• NDDM compares Notes and SharePoint and reports differences

The schematic image below here shows the operations to be carried out in NDDM to archive Notes databases to SPO

NDDM has an extensive Notes to SharePoint archive quality control mechanism

NDDM can issue PowerShell commands to the SPO libraries and count the number of items and files migrated to SPO. This can be compared with the number of Notes documents and attachments in the Notes database.

In large migrations of many thousands of Notes databases and millions of Notes documents, this feature is critical to ensure a good migration quality.

The NDDM image below shows in green the Notes database statistics, in blue the Pipeliner migration results and in purple the SPO site scan statistics. If the delta docs and delta files display zero, we have a perfect match.

The archives that Pipeliner creates from the Notes databases are stored in the SharePoint Online (SPO) environment. The goal of this section is to advise on the planning of the SPO environment setup in a secure and manageable manner.

End user and owner permissions

All users and owners will get read (visitor) permissions to the SPO sites holding the Notes archives. This means that nobody can change the site content, not even the owner (who has also only read permissions to the SPO archive site). The owner can manage the read permissions to his/her SPO archive sites only via security groups. There is no way the owner can assign him/herself or other users edit permission to the SPO archive sites.

Each Notes database is archived to a dedicated SPO site. This means that 1 SPO site holds 1 SPO document library with the Notes content in aspx format.

ASPX files

The Notes database views and Notes documents are converted to ASPX files (by the Pipeliner) that are stored in SPO libraries.

Pipeliner Migration Factory

The Pipeliner software is running on Domino servers which are running on multiple Windows 2019 Migration servers.

It is recommended that the Pipeliner software uses Azure Enterprise applications to authenticate with SPO to migrate the Notes databases to SPO (see SharePoint Online permissions for details).

The Pipeliner uses PnP PowerShell to carry out the uploads of the aspx files to the SPO sites.

The whole migration factory (Pipeliner and NDDM) takes care of the entire fully automated archiving of the Notes databases. Everything is done in an automated way with a very low time effort needed by the administrators.

SPO archive site architecture

To ensure a proper quality check of the archiving results, it is recommended to archive only 1 Notes database to 1 SPO site.

For example, a Notes database with the title ‘Security Repository’ will be archived to a SPO site with the title ‘NotesA-SecurityRepository’. The ‘NotesA-‘ is added to the title, in order to enable the sorting of all sites storing Notes archives. The archive aspx files are stored in a document library with the name ‘SecurityRepository’.

In this example, the location of the Notes archive views is:
https://acme.sharepoint.com/sites/NotesA-SecurityRepository/SecurityRepository

‘NotesA-SecurityRepository’ is the SPO site. ‘SecurityRepository’ is the name of the SPO document library.
The SPO document library displays the migrated Notes views as illustrated below:

It is also possible to archive Notes databases to subsites relative to a main site in SPO. In that case, the name of the archive site is for example ‘NotesArchives’ and each Notes database is archived to a subsite based on the database title. The location of the Notes archive will then be
https://acme.sharepoint.com/sites/NotesArchives/SecurityRepository/SecurityRepository

‘SecurityRepository’ is the SPO subsite. ‘SecurityRepository’ is the name of the SPO document library.

In the subsite setup, NDDM breaks the permissions inheritance when it migrates end users from Notes to security groups.

Prior to archiving, the Pipeliner automatically creates the new site with the help of PnP PowerShell code. A new site is based on the template SITEPAGEPUBLISHING#0 which is the Communication Site (see this link for explanations of the different templates available).

To display aspx files in the user’s browser, the setting ‘DenyAddAndCustomizePages = 0’ is applied to each newly created site.

The archive owner (end user) can manage access to the SPO archive sites via security groups, for example:

Note that the owner must be a member of this group to view the SPO archive site content.

PnP PowerShell adds the security group to the SPO archive site visitors group (read only permissions). As NDDM holds the owner and the users, PnP PowerShell can create these groups and assign them to the SPO site.

SPO archive site homepage

The default homepage of a communication site is displayed below.