NDDM Technical Manual – Pipeliner
IBM Notes Domino Application Scan and Analysis tool.
Optimized for Domino Server decommission projects.
Lialis will always support your Domino Server Application Scan in short daily remote calls to ensure a perfect and swift scan.
Updated May 2023 for NDDM release 5.30
Please check this page for a demo video. Note that the video is showing an older NDDM version, currently the left navigation menu is looking differently, but the overall functionality is still the same.
Pipeliner Fundamentals
- Read-only dump of any Notes database/app to SharePoint Online
- Fast adaptation for end users because it looks just like Notes
Pipeliner supports
- Any Notes database size supported by Domino
- Any Notes database design supported by Domino
- Any number of Notes documents supported by Domino
- Any number of attachments supported by Domino
- Human effort to archive any Notes database in less than 5 minutes per Notes database (1000 nsf in 80 hr effort)
- Notes design knowledge not needed
- Central Notes document link system
- Notes response documents supported
- Conversion of OLE objects to normal files
- High speed parallel processing on multiple servers
- 100% Domino web apps may need some development
Notes view conversion example, the image below shows a Notes app view
The image below shows the SharePoint Online view, created by Pipeliner for the Notes view displayed above
Pipeliner creates SharePoint aspx page views for all available (non hidden) Notes app views.
Notes document conversion example, the image below shows a Notes document
The image below shows the SharePoint Online document, created by Pipeliner for the Notes document displayed above
A schematic drawing of how a Notes database is converted to a read-only export on SharePoint Online by Pipeliner:
The schematic drawing in the image below shows the SharePoint Online export result created by Pipeliner. Each aspx item in the root of the library represents a migrated Notes view, with similar naming. All these views are created fully automatically by Pipeliner.
The content of a migrated view is made up of links to SharePoint items that represent the Notes documents. These migrated Notes document items are stored in the folder _Notescontent. The migrated attachments are stored in the files subfolders.
When our clients have a need to archive many (>200) Notes databases to SPO, the Notes database export infrastructure can be setup according to the following diagram:
The Notes databases that are archived remain on the existing Domino servers. One or multiple Domino Pipeliner migration servers are installed. They pull the data from the source Domino servers, convert it to a structure of files and folders on disk, and then uploaded it to SPO. Everything in this process is fully automated. SPO sites are automatically created, the SPO site homepage is set to the library root, and all files are uploaded with PowerShell.
NDDM is fully integrated with Pipeliner to allow clients to archive many thousands of Notes databases to SharePoint Online fast, easy and in a short time frame.
In this setup, we have archived 10,000 Notes databases in 4 months to SPO for a USA based client. These 10,000 Notes databases where collectively holding 11M Notes documents, 20TB on data. We installed 35 Pipeliner migration servers. The Pipeliner servers work load was managed by NDDM.
Fully automated Notes database archival to SharePoint Online
- Use NDDM buttons to manage the complete archival process, no time-consuming tasks
- Opening Pipeliner is only needed in case of problems
- 20 Domino Pipe servers can archive 60 Notes databases in parallel (this can be increased)
- Fully automated process:
- SharePoint site creation (communication site), SharePoint Library creation
- Upload of the Notes archive to a SharePoint Library
- SharePoint site permissions are set for the owner, users (taken from Notes) and administrators
- SharePoint site homepage is set to the archive library
- NDDM retrieves the archive log
- NDDM button to quickly compare the Notes documents per form with the item forms on SharePoint
- NDDM matches the number of Notes database against the number of SharePoint libraries to find migration issues
- NDDM is used to migrate end user permissions from Notes to SharePoint
Archive flow managed by NDDM
- Approximately 20 Domino servers in Client Cloud dedicated for Notes database dumps to SharePoint
- These 20 Domino Servers replicate Pipeliner.nsf between each other
- NDDM creates archive task in Pipeliner.nsf on server Pipe1
- This archive task is set to be carried out by server Pipe8 (by NDDM) for example
- This archive task replicates to Domino server Pipe8
- Pipe8 pulls the Notes database from the source server, creates an export on disk and uploads this data to SharePoint via PNP PowerShell
- NDDM retrieves archive progress and log
- IT checks log and compares the number of Notes documents per form with SharePoint items (sample wise)
- NDDM mailing function can be used to report archive to owner
- NDDM compares Notes and SharePoint and reports differences
The schematic image below here shows the operations to be carried out in NDDM to archive Notes databases to SPO
NDDM has an extensive Notes to SharePoint archive quality control mechanism
NDDM can issue PowerShell commands to the SPO libraries and count the number of items and files migrated to SPO. This can be compared with the number of Notes documents and attachments in the Notes database.
In large migrations of many thousands of Notes databases and millions of Notes documents, this feature is critical to ensure a good migration quality.
The NDDM image below shows in green the Notes database statistics, in blue the Pipeliner migration results and in purple the SPO site scan statistics. If the delta docs and delta files display zero, we have a perfect match.
The archives that Pipeliner creates from the Notes databases are stored in the SharePoint Online (SPO) environment. The goal of this section is to advise on the planning of the SPO environment setup in a secure and manageable manner.
End user and owner permissions
All users and owners will get read (visitor) permissions to the SPO sites holding the Notes archives. This means that nobody can change the site content, not even the owner (who has also only read permissions to the SPO archive site). The owner can manage the read permissions to his/her SPO archive sites only via security groups. There is no way the owner can assign him/herself or other users edit permission to the SPO archive sites.
Each Notes database is archived to a dedicated SPO site. This means that 1 SPO site holds 1 SPO document library with the Notes content in aspx format.
ASPX files
The Notes database views and Notes documents are converted to ASPX files (by the Pipeliner) that are stored in SPO libraries.
Pipeliner Migration Factory
The Pipeliner software is running on Domino servers which are running on multiple Windows 2019 Migration servers.
It is recommended that the Pipeliner software uses Azure Enterprise applications to authenticate with SPO to migrate the Notes databases to SPO (see SharePoint Online permissions for details).
The Pipeliner uses PnP PowerShell to carry out the uploads of the aspx files to the SPO sites.
The whole migration factory (Pipeliner and NDDM) takes care of the entire fully automated archiving of the Notes databases. Everything is done in an automated way with a very low time effort needed by the administrators.
SPO archive site architecture
To ensure a proper quality check of the archiving results, it is recommended to archive only 1 Notes database to 1 SPO site.
For example, a Notes database with the title ‘Security Repository’ will be archived to a SPO site with the title ‘NotesA-SecurityRepository’. The ‘NotesA-‘ is added to the title, in order to enable the sorting of all sites storing Notes archives. The archive aspx files are stored in a document library with the name ‘SecurityRepository’.
In this example, the location of the Notes archive views is:
https://acme.sharepoint.com/sites/NotesA-SecurityRepository/SecurityRepository
‘NotesA-SecurityRepository’ is the SPO site. ‘SecurityRepository’ is the name of the SPO document library.
The SPO document library displays the migrated Notes views as illustrated below:
It is also possible to archive Notes databases to subsites relative to a main site in SPO. In that case, the name of the archive site is for example ‘NotesArchives’ and each Notes database is archived to a subsite based on the database title. The location of the Notes archive will then be
https://acme.sharepoint.com/sites/NotesArchives/SecurityRepository/SecurityRepository
‘SecurityRepository’ is the SPO subsite. ‘SecurityRepository’ is the name of the SPO document library.
In the subsite setup, NDDM breaks the permissions inheritance when it migrates end users from Notes to security groups.
Prior to archiving, the Pipeliner automatically creates the new site with the help of PnP PowerShell code. A new site is based on the template SITEPAGEPUBLISHING#0 which is the Communication Site (see this link for explanations of the different templates available).
To display aspx files in the user’s browser, the setting ‘DenyAddAndCustomizePages = 0’ is applied to each newly created site.
The archive owner (end user) can manage access to the SPO archive sites via security groups, for example:
Note that the owner must be a member of this group to view the SPO archive site content.
PnP PowerShell adds the security group to the SPO archive site visitors group (read only permissions). As NDDM holds the owner and the users, PnP PowerShell can create these groups and assign them to the SPO site.
SPO archive site homepage
The default homepage of a communication site is displayed below.
It is difficult for the end users to navigate to the Notes archive via site content. Therefore the Pipeliner will adjust the homepage so that the aspx Notes views of the Notes database archive library are displayed.
SharePoint Online permissions
NDDM and Pipeliner use PnP PowerShell commands to:
- create the SPO site (done by Pipeliner)
- create the library within this site (done by Pipeliner)
- upload files and folders to the library (done by Pipeliner)
- create an Azure security group, add users and owner to this group, and add this group to the site group (done by NDDM)
Pipeliner permissions method
The Pipeliner software is granted permissions via Azure Active Directory Enterprise application registration.
To prevent throttling by Microsoft, NDDM requires multiple app registrations. App registrations are created in the Azure Active Directory admin center (see image below for example).
Displayed below are the permissions that the Enterprise app requires.
App registration instructions for Pipeliner
Start PowerShell as admin
Register-PnPAzureADApp -ApplicationName “Lialis SPO Notes Migration Tool 1” -Tenant lialis.com -OutPath c:\temp -CertificatePassword (ConvertTo-SecureString -String “pipelinerpassword” -AsPlainText -Force) -devicelogin
NDDM permissions method
NDDM uses the same method as Pipeliner to access the SPO environment, create the security group for the SPO site, and add the users. To create Azure Security Groups, we use Microsoft.Graph in combination with MSAL (MS Authentication Library). 1 app registration (Pipe000) already created for Pipeliner must be extended with a client secret.
Notes link tracking
Pipeliner offers a smart Notes document link tracking system across all archived Notes databases. Pipeliner supports Notes document link resolving within one archive and across multiple archives in SPO. A central SPO site is needed to store and maintain the Pipeliner document link tracking system, for example:
https://acme.sharepoint.com/NotesLinks
All SPO users need read access to this site.
This site may never be moved or deleted. If this happens, all links will fail.
The IONET Pipeliner Notes application is responsible for archiving Notes databases to SharePoint Online.
Because Lialis supplies the Pipeliner NSF completely configured at the beginning of the project, this manual will not dive deep into the Pipeliner configuration.
This section describes how to configure a Windows server for this purpose.
It is recommended to start with 1 virtual Windows server and make it run issue-free. This server can then be used as image for the additional x migration servers. Duplicating Pipeliner migration servers is explained at the end of this section.
Run a Windows 2019 virtual computer or a more recent version. RAM 32 GB, medium CPU power and a 100 GB C-drive and a 250 GB D-drive.
Install Domino R12 and Notes R12 with all clients in the C-drive root (not in the program files folder).
The D-drive is used to store the Notes database exports before the files are uploaded to SharePoint Online.
Register the Domino server (like Pipe1/Lialis). The only settings important for the Domino server document are displayed in the image below. You can copy all other settings from another Domino server document.
Use the Domino server ID to setup the Domino server on the Windows box. Run Domino as a Windows service. The Notes clients on this Windows server are needed to research migration issues.
From each Pipe Domino server, create pull/push replication documents to a central Domino admin or hub server of your choice. The Pipe Domino servers will only replicate the Pipeliner.nsf with the central Domino server. NDDM will be connected to the Pipeliner.nsf on this central Domino server to initiate archive work and to pull the archive results from the Pipeliner. Notes databases that are going to be archived will not be replicated to this Domino server.
In the Domino Directory, create a server group document where you list all Pipeliner Domino servers. Add this group to the following fields on each Domino server document that has databases to be archived:
- Access server
- Trusted servers
- Sign or run unrestricted methods and operations
The ID used to sign the Pipeliner NSF must have manager access to all Notes databases, with all roles enabled. NDDM has a feature to make sure that all Notes database ACLs are set properly.
Lialis will provide the Pipeliner NSF with all settings already in place. We will not explain all the Pipeliner settings in this manual, because the list of settings is very long and they are subject to changes due to product improvements.
Install the PnP PowerShell module for SharePoint Online
Disable Internet Explorer, because it may cause issues
https://www.tecklyfe.com/remove-internet-explorer-ie-windows-server/
Install the Firefox browser to open aspx files directly from the server export folder
Install Notespad++ , a great text file viewer
You may want to check the Windows computer internet options and enable all settings that are currently disabled.
Duplicating Pipeliner migration servers
Once the first Pipe1 Domino server is operating smoothly and is capable of sending Notes database archives to SharePoint Online, it is time to duplicate this server and build a migration factory. How much archive servers you need depends on the archive scope. For example, one of our clients wanted to archive 10,000 Notes database, for this project we created 30 Domino servers / Windows boxes.
Go into the Domino Directory and copy/paste the Domino Pipe1 server document. On the copied document, change the Domino server name from Pipe1 to Pipe2 and change the IP address details.
Next step is to register a new Domino server ID for Pipe2. During the process you will be asked if you want to overwrite the existing entry. Say Yes. When the Domino server ID is created, double-check the Domino server document for Pipe2, just in case some settings where adjusted.
Add Pipe2 to the Pipeliner server group and create a connection document from Pipe2 to the central Domino server.
Duplicate the Windows virtual computer, give the copy a new name (Pipe2) and get the IP stuff in place. Start the Pipe2 Windows machine. The only thing left to do is to copy the Pipe2 Domino server ID file to the machine and replace Pipe1 with Pipe2 in the server’s notes.ini. Start the Domino server and make sure it runs under the Pipe2 ID.
After the Pipeliner NSF and the migration Domino server are fully configured and tested, the next step is to integrate NDDM with the Pipeliner, so that you can archive many Notes databases with minimal effort and just wait for the work to be done.
Open NDDM. Under ‘Settings\Pipeliner’ set the location of the Pipeliner NSF, as illustrated below.
The Pipeliner.nsf will replicate between a Domino Hub server and all Pipeliner servers which will do the archiving. In below setting please choose the Domino hub server where the Piperliner NSF resides. NDDM will use this replica to send work to the Pipeliner factory and retrieve archive logs from it.
Depending on the SPO archive site setup, the settings displayed above may change, but this is explained later on in this guide.
Under ‘Settings\User Mapping\Connection’, specify the Tenant target URL and the Connection settings that are used by the PowerShell script to upload the data to SPO.