NDDM Technical Manual – Pipeliner

Please check this page for a demo video. Note that the video is showing an older NDDM version, currently the left navigation menu is looking differently, but the overall functionality is still the same.

Pipeliner Fundamentals

• Read-only dump of any Notes database/app to SharePoint Online
• Fast adaptation for end users because it looks just like Notes

Pipeliner supports

• Any Notes database size supported by Domino
• Any Notes database design supported by Domino
• Any number of Notes documents supported by Domino
• Any number of attachments supported by Domino
• Human effort to archive any Notes database in less than 5 minutes per Notes database (1000 nsf in 80 hr effort)
• Notes design knowledge not needed
• Central Notes document link system
• Notes response documents supported
• Conversion of OLE objects to normal files
• High speed parallel processing on multiple servers
• 100% Domino web apps may need some development

Notes view conversion example, the image below shows a Notes app view

The image below shows the SharePoint Online view, created by Pipeliner for the Notes view displayed above

Pipeliner creates SharePoint aspx page views for all available (non hidden) Notes app views.

Notes document conversion example, the image below shows a Notes document

The image below shows the SharePoint Online document, created by Pipeliner for the Notes document displayed above

A schematic drawing of how a Notes database is converted to a read-only export on SharePoint Online by Pipeliner:

The schematic drawing in the image below shows the SharePoint Online export result created by Pipeliner. Each aspx item in the root of the library represents a migrated Notes view, with similar naming. All these views are created fully automatically by Pipeliner.

The content of a migrated view is made up of links to SharePoint items that represent the Notes documents. These migrated Notes document items are stored in the folder _Notescontent. The migrated attachments are stored in the files subfolders.

When our clients have a need to archive many (>200) Notes databases to SPO, the Notes database export infrastructure can be setup according to the following diagram:

The Notes databases that are archived remain on the existing Domino servers. One or multiple Domino Pipeliner migration servers are installed. They pull the data from the source Domino servers, convert it to a structure of files and folders on disk, and then uploaded it to SPO. Everything in this process is fully automated. SPO sites are automatically created, the SPO site homepage is set to the library root, and all files are uploaded with PowerShell.

NDDM is fully integrated with Pipeliner to allow clients to archive many thousands of Notes databases to SharePoint Online fast, easy and in a short time frame.

In this setup, we have archived 10,000 Notes databases in 4 months to SPO for a USA based client. These 10,000 Notes databases where collectively holding 11M Notes documents, 20TB on data. We installed 35 Pipeliner migration servers. The Pipeliner servers work load was managed by NDDM.

Fully automated Notes database archival to SharePoint Online

• Use NDDM buttons to manage the complete archival process, no time-consuming tasks
• Opening Pipeliner is only needed in case of problems
• 20 Domino Pipe servers can archive 60 Notes databases in parallel (this can be increased)
• Fully automated process:
  • SharePoint site creation (communication site), SharePoint Library creation
  • Upload of the Notes archive to a SharePoint Library
  • SharePoint site permissions are set for the owner, users (taken from Notes) and administrators
  • SharePoint site homepage is set to the archive library
• NDDM retrieves the archive log
• NDDM button to quickly compare the Notes documents per form with the item forms on SharePoint
• NDDM matches the number of Notes database against the number of SharePoint libraries to find migration issues
• NDDM is used to migrate end user permissions from Notes to SharePoint

Archive flow managed by NDDM

• Approximately 20 Domino servers in Client Cloud dedicated for Notes database dumps to SharePoint
• These 20 Domino Servers replicate Pipeliner.nsf between each other
• NDDM creates archive task in Pipeliner.nsf on server Pipe1
• This archive task is set to be carried out by server Pipe8 (by NDDM) for example
• This archive task replicates to Domino server Pipe8
• Pipe8 pulls the Notes database from the source server, creates an export on disk and uploads this data to SharePoint via PNP PowerShell
• NDDM retrieves archive progress and log
• IT checks log and compares the number of Notes documents per form with SharePoint items (sample wise)
• NDDM mailing function can be used to report archive to owner
• NDDM compares Notes and SharePoint and reports differences

The schematic image below here shows the operations to be carried out in NDDM to archive Notes databases to SPO

NDDM has an extensive Notes to SharePoint archive quality control mechanism

NDDM can issue PowerShell commands to the SPO libraries and count the number of items and files migrated to SPO. This can be compared with the number of Notes documents and attachments in the Notes database.

In large migrations of many thousands of Notes databases and millions of Notes documents, this feature is critical to ensure a good migration quality.

The NDDM image below shows in green the Notes database statistics, in blue the Pipeliner migration results and in purple the SPO site scan statistics. If the delta docs and delta files display zero, we have a perfect match.

The archives that Pipeliner creates from the Notes databases are stored in the SharePoint Online (SPO) environment. The goal of this section is to advise on the planning of the SPO environment setup in a secure and manageable manner.

End user and owner permissions

All users and owners will get read (visitor) permissions to the SPO sites holding the Notes archives. This means that nobody can change the site content, not even the owner (who has also only read permissions to the SPO archive site). The owner can manage the read permissions to his/her SPO archive sites only via security groups. There is no way the owner can assign him/herself or other users edit permission to the SPO archive sites.

Each Notes database is archived to a dedicated SPO site. This means that 1 SPO site holds 1 SPO document library with the Notes content in aspx format.

ASPX files

The Notes database views and Notes documents are converted to ASPX files (by the Pipeliner) that are stored in SPO libraries.

Pipeliner Migration Factory

The Pipeliner software is running on Domino servers which are running on multiple Windows 2019 Migration servers.

It is recommended that the Pipeliner software uses Azure Enterprise applications to authenticate with SPO to migrate the Notes databases to SPO (see SharePoint Online permissions for details).

The Pipeliner uses PnP PowerShell to carry out the uploads of the aspx files to the SPO sites.

The whole migration factory (Pipeliner and NDDM) takes care of the entire fully automated archiving of the Notes databases. Everything is done in an automated way with a very low time effort needed by the administrators.

SPO archive site architecture

To ensure a proper quality check of the archiving results, it is recommended to archive only 1 Notes database to 1 SPO site.

For example, a Notes database with the title ‘Security Repository’ will be archived to a SPO site with the title ‘NotesA-SecurityRepository’. The ‘NotesA-‘ is added to the title, in order to enable the sorting of all sites storing Notes archives. The archive aspx files are stored in a document library with the name ‘SecurityRepository’.

In this example, the location of the Notes archive views is:
https://acme.sharepoint.com/sites/NotesA-SecurityRepository/SecurityRepository

‘NotesA-SecurityRepository’ is the SPO site. ‘SecurityRepository’ is the name of the SPO document library.
The SPO document library displays the migrated Notes views as illustrated below:

It is also possible to archive Notes databases to subsites relative to a main site in SPO. In that case, the name of the archive site is for example ‘NotesArchives’ and each Notes database is archived to a subsite based on the database title. The location of the Notes archive will then be
https://acme.sharepoint.com/sites/NotesArchives/SecurityRepository/SecurityRepository

‘SecurityRepository’ is the SPO subsite. ‘SecurityRepository’ is the name of the SPO document library.

In the subsite setup, NDDM breaks the permissions inheritance when it migrates end users from Notes to security groups.

Prior to archiving, the Pipeliner automatically creates the new site with the help of PnP PowerShell code. A new site is based on the template SITEPAGEPUBLISHING#0 which is the Communication Site (see this link for explanations of the different templates available).

To display aspx files in the user’s browser, the setting ‘DenyAddAndCustomizePages = 0’ is applied to each newly created site.

The archive owner (end user) can manage access to the SPO archive sites via security groups, for example:

Note that the owner must be a member of this group to view the SPO archive site content.

PnP PowerShell adds the security group to the SPO archive site visitors group (read only permissions). As NDDM holds the owner and the users, PnP PowerShell can create these groups and assign them to the SPO site.

SPO archive site homepage

The default homepage of a communication site is displayed below.

It is difficult for the end users to navigate to the Notes archive via site content. Therefore the Pipeliner will adjust the homepage so that the aspx Notes views of the Notes database archive library are displayed.

SharePoint Online permissions

NDDM and Pipeliner use PnP PowerShell commands to:

1. create the SPO site (done by Pipeliner)
2. create the library within this site (done by Pipeliner)
3. upload files and folders to the library (done by Pipeliner)
4. create an Azure security group, add users and owner to this group, and add this group to the site group (done by NDDM)

Pipeliner permissions method

The Pipeliner software is granted permissions via Azure Active Directory Enterprise application registration.

To prevent throttling by Microsoft, NDDM requires multiple app registrations. App registrations are created in the Azure Active Directory admin center (see image below for example).

Displayed below are the permissions that the Enterprise app requires.

App registration instructions for Pipeliner

Start PowerShell as admin

Register-PnPAzureADApp -ApplicationName “Lialis SPO Notes Migration Tool 1” -Tenant lialis.com -OutPath c:\temp -CertificatePassword (ConvertTo-SecureString -String “pipelinerpassword” -AsPlainText -Force) -devicelogin

NDDM permissions method

NDDM uses the same method as Pipeliner to access the SPO environment, create the security group for the SPO site, and add the users. To create Azure Security Groups, we use Microsoft.Graph in combination with MSAL (MS Authentication Library). 1 app registration (Pipe000) already created for Pipeliner must be extended with a client secret.

Notes link tracking

Pipeliner offers a smart Notes document link tracking system across all archived Notes databases. Pipeliner supports Notes document link resolving within one archive and across multiple archives in SPO. A central SPO site is needed to store and maintain the Pipeliner document link tracking system, for example:

https://acme.sharepoint.com/NotesLinks

All SPO users need read access to this site.

This site may never be moved or deleted. If this happens, all links will fail.

The IONET Pipeliner Notes application is responsible for archiving Notes databases to SharePoint Online.

Because Lialis supplies the Pipeliner NSF completely configured at the beginning of the project, this manual will not dive deep into the Pipeliner configuration.

This section describes how to configure a Windows server for this purpose.

It is recommended to start with 1 virtual Windows server and make it run issue-free. This server can then be used as image for the additional x migration servers. Duplicating Pipeliner migration servers is explained at the end of this section.

Run a Windows 2019 virtual computer or a more recent version. RAM 32 GB, medium CPU power and a 100 GB C-drive and a 250 GB D-drive.

Install Domino R12 and Notes R12 with all clients in the C-drive root (not in the program files folder).

The D-drive is used to store the Notes database exports before the files are uploaded to SharePoint Online.

Register the Domino server (like Pipe1/Lialis). The only settings important for the Domino server document are displayed in the image below. You can copy all other settings from another Domino server document.

Use the Domino server ID to setup the Domino server on the Windows box. Run Domino as a Windows service. The Notes clients on this Windows server are needed to research migration issues.

From each Pipe Domino server, create pull/push replication documents to a central Domino admin or hub server of your choice. The Pipe Domino servers will only replicate the Pipeliner.nsf with the central Domino server. NDDM will be connected to the Pipeliner.nsf on this central Domino server to initiate archive work and to pull the archive results from the Pipeliner. Notes databases that are going to be archived will not be replicated to this Domino server.

In the Domino Directory, create a server group document where you list all Pipeliner Domino servers. Add this group to the following fields on each Domino server document that has databases to be archived:

• Access server
• Trusted servers
• Sign or run unrestricted methods and operations

The ID used to sign the Pipeliner NSF must have manager access to all Notes databases, with all roles enabled. NDDM has a feature to make sure that all Notes database ACLs are set properly.

Lialis will provide the Pipeliner NSF with all settings already in place. We will not explain all the Pipeliner settings in this manual, because the list of settings is very long and they are subject to changes due to product improvements.

Install the PnP PowerShell module for SharePoint Online

https://www.sharepointdiary.com/2021/02/how-to-install-pnp-powershell-module-for-sharepoint-online.html

Disable Internet Explorer, because it may cause issues

https://www.tecklyfe.com/remove-internet-explorer-ie-windows-server/

Install the Firefox browser to open aspx files directly from the server export folder

Install Notespad++ , a great text file viewer

You may want to check the Windows computer internet options and enable all settings that are currently disabled.

Duplicating Pipeliner migration servers

Once the first Pipe1 Domino server is operating smoothly and is capable of sending Notes database archives to SharePoint Online, it is time to duplicate this server and build a migration factory. How much archive servers you need depends on the archive scope. For example, one of our clients wanted to archive 10,000 Notes database, for this project we created 30 Domino servers / Windows boxes.

Go into the Domino Directory and copy/paste the Domino Pipe1 server document. On the copied document, change the Domino server name from Pipe1 to Pipe2 and change the IP address details.

Next step is to register a new Domino server ID for Pipe2. During the process you will be asked if you want to overwrite the existing entry. Say Yes. When the Domino server ID is created, double-check the Domino server document for Pipe2, just in case some settings where adjusted.

Add Pipe2 to the Pipeliner server group and create a connection document from Pipe2 to the central Domino server.

Duplicate the Windows virtual computer, give the copy a new name (Pipe2) and get the IP stuff in place. Start the Pipe2 Windows machine. The only thing left to do is to copy the Pipe2 Domino server ID file to the machine and replace Pipe1 with Pipe2 in the server’s notes.ini. Start the Domino server and make sure it runs under the Pipe2 ID.

After the Pipeliner NSF and the migration Domino server are fully configured and tested, the next step is to integrate NDDM with the Pipeliner, so that you can archive many Notes databases with minimal effort and just wait for the work to be done.

Open NDDM. Under ‘Settings\Pipeliner’ set the location of the Pipeliner NSF, as illustrated below.

The Pipeliner.nsf will replicate between a Domino Hub server and all Pipeliner servers which will do the archiving. In below setting please choose the Domino hub server where the Piperliner NSF resides. NDDM will use this replica to send work to the Pipeliner factory and retrieve archive logs from it.

Depending on the SPO archive site setup, the settings displayed above may change, but this is explained later on in this guide.

Under ‘Settings\User Mapping\Connection’, specify the Tenant target URL and the Connection settings that are used by the PowerShell script to upload the data to SPO.

The enterprise app certificates must be stored in the Work folder.
The Azure connection credentials are used for the migration of permissions to Active Directory.

NDDM is capable of migrating end user permissions from Notes to SharePoint Online. This enables the owner and the end users to access the Notes database archives.

Under ‘Settings\User Mapping\Users’, you can configure the Notes to Active Directory user mappings. This is used to resolve Notes users in AD for the purpose of assigning SharePoint Online end user permissions.

Use the ‘Actions’ button to:

• 1. ‘Import Person Documents’- imports Notes person documents from the Domino Directory
• 2. Lookup SPO account – looks up the matching SPO account of the imported Notes users, and writes them down in a text file. This action runs a PowerShell code and may take a while.
• 3. Import SPO account –  imports the SPO account from the text file into the Notes documents in this view.

Now you have mapped Notes end user names to SPO accounts. These mappings are used during the archiving process to set permissions in SharePoint.

To achieve good results, make sure to carry out the following preparations to the Notes databases that will be archived to SPO.

ACL Roles. Before the archiving of a Notes db to SPO can be started, it is important that the Pipeliner server group (in names.nsf) has manager access to the Notes db, and that this group has all roles in the Notes db ACL enabled. NDDM can carry out this task fully automated, simply by using the ‘Set Manager’ action as displayed below. To make sure the Notes admin that runs this operation has manager access to all Notes databases, he/she needs to enable the ‘Full Access Administration’ option in the Notes admin client.

Specify the Pipeliner server group in the image below. We assume you created a group where all Pipeliner Domino servers are listed in.

If the group is already present in the Notes db ACL, NDDM will check the group access, increase the access to Manager Can Delete, and assign all ACL roles to this group.

Read protected documents. A Notes db can have Notes documents that are read-protected with reader fields that only contain usernames and no ACL roles. Use NDDM to carry out Content Analysis with restricted access to identify these Notes databases. Simply carry our a content analysis with restricted access as displayed below on all Notes databases. This will identify Notes databases that have this category of read-protected documents.

The read-protected documents Notes db are listed in this view

Then use the NDDM ‘Disable Readers’ action to remove the readers fields from these Notes documents. Make sure to enable ‘Full Access Administration’ privileges in the Notes admin client.

After completion, you may want to carry out a new scan to see if the protection on these documents is removed.

Hidden designs. Some databases may have a hidden design. This NDDM column will show a blue ball when the design is hidden..

Use the ‘Unhide Hidden Design’ action in NDDM on these databases, else Pipeliner cannot archive these Notes databases.

Disable all triggered agents

Before archiving Notes databases, it is strongly recommended to stop all agents that might change the Notes db content.

OLE objects

NDDM will report if a Notes db has OLE objects. You will have to try to get these OLE objects converted to normal files, or else the Pipeliner archival will fail on these OLE objects. The number of OLE objects is calculated by the NDDM Notes db content scan and reported in the Pipeliner archival views.

Use NDDM to replace OLE objects with normal attachments. After completion, carry out a new content scan to register the new OLE and files numbers in NDDM.

NDDM has 2 means of fixing OLE objects.

Option 1 and 2 belong together. Option 3 is the new method. We suggest to only use option 3, because this option does not need the OLE source programs (like Word, PDF, Excel) to be installed on the local computer during conversion.

Before starting the OLE conversion with option 3, open the NDDM ‘Using’ document, scroll down to the bottom and excecute these details.

When option 3 is executed, the OLE column will show this value

The NDDM db record will show a log about the OLE conversion, for example:

It is recommended to do a new content scan to determine whether the OLE objects have been removed.

Notes db > 64 GB. When a Notes db exceeds the size of 64 GB, you can use NDDM to split the Notes database in 2 parts.

Note that the Notes database may increase in size, once it is archived by Pipeliner, because Pipeliner will index the Notes views prior to archival to SPO.

Read-only access. Set the ACL to read-only, so users cannot modify the content.

Perform a new ACL scan to collect users and groups that will be used for the Owner and users migration to SPO later on

Finally do a new content scan of the Notes db to be archived. We need the content details later on to match the number of Notes documents and files against the number of documents and files archived to SPO.

The actions listed below can be performed on a single or on multiple Notes databases, all from within NDDM. They should only be executed on Notes db marked as a ‘winner’. The Pipeliner factory must be up and running, and the SPO archival site design must be in place.

To be able to review the archive results, Notes administrators doing the Pipeliner archive work must be SPO site owner (information about SPO and AD permissions can be found in the ‘Pipeliner SharePoint design for Notes archive project’ document).

All our archiving projects go through a questionnaire phase. In this phase, each database will receive a status value indicating whether or not the database will be archived to SPO.

The view ‘Database Info/Status’ is the starting point for the archiving process. Open this view, sort it on the ‘Status’ column and select the Notes databases that should be archived. Then set the SP Site Collection with the action below, eg. https://lialis.sharepoint.com

Now switch to the view ‘Pipeliner Archival’ and refresh the view

Sort on the column ‘SP Site Col’, you should see something like illustrated below.

Use the buttons displayed below to set the SPO target (sub)site where the Notes db should be archived to. Depending on how the SPO archive site is designed, you can choose to archive to subsite A (Scenario 1) or subsite B (scenario 2).

Scenario 1: https://lialis.sharepoint.com/sites/NotesA-Notesdbtile – each Notes db is archive to a dedicated site
The ‘NotesA-‘ part is configured in NDDM and is useful to find all archive sites at once.

Scenario 2: https://lialis.sharepoint.com/sites/somesitename/Notesdbtile – all Notes db are archived to a subsite of ./somesitename/. The main site name (‘somesitename’) can for example be NotesArchives, in which case it’s not nessesary to add NotesA to the subsites.

If needed you can combine these 2 scenarios. However, to maintain uniformity, it is recommended to choose scenario 1 OR scenario 2.

We prefer scenario 1 by the way.

For scenario 1 you only need to press ‘Set SP Subsite A to Notes Db’.

The result:

NDDM will assign unique subsites names.

For scenario 2 you first need to press ‘Set Subsite A’.

Enter the site name

Then press ‘Set SP Subsite B to Notes Db’.

The result:

In case you made a mistake, you can clear the SPO sites settings on selected Notes documents with the ‘Clear Pipeliner Sites’ action.

Depending on the type of Notes database, an archive profile is applied to each database. NDDM can indicate whether a Notes database design falls into the group ‘Teamroom/Document Library’, see image below.

With the button “Set Profile” you may now choose which Pipeliner server will do the archive work and whether the Notes database is has a Teamroom/Document Library (kind of) design. In this example, the first 4 Notes databases are assigned to Pipe server 1 with a non-teamroom profile:

Continue this job to spread the load over your pipeliner factory, depending on how many servers you have configured (e.g. 1, 30 or 100 servers).

In the ‘Load balancing’ view, you can check how the load is spread over your factory. Keep an eye on the Notes database specifications and shift the archiving of large Notes databases to other servers to spread the load.

Next, press the button ‘Auto Assign Credentials’. The available migration credentials are distributed over all Notes databases to be archived.

The available credentials are retreived from the Pipeliner app.

When you are happy with the batch, select all documents all and press ‘Send To Pipeliner’.

After a few minutes the Pipeliner factory will start to do the work. Press the ‘Get Log Details’ button to retrieve the progress from Pipeliner and report the disk and PowerShell percentages in NDDM.

When the migration is completed, the Disk% and PS% will show 100%. If this is not the case, some issue might have occurred.

The Notes databases for which archiving has been completed, are listed in the ‘Completed’ view

The NDDM record will display the Pipeliner log on the tab ‘Pipeliner Archive’

Other fields on the ‘Pipeliner Archive’ tab:

Export Status – The status is retrieved from the Pipeliner Notes database. 05 Means that archive is completed

Export count – #Notes documents exported / #aspx files uploaded. Both numbers should be equal

Uploaded documents – #Files exported / #files uploaded to SPO by PowerShell

Disk percentage – The percentage of the Notes db exported to local disk

Upload percentage – When the disk export has been completed, PowerShell uploads all files to SPO, the upload percentage is indicated here

Pipeliner export errors – The number of errors during the export of the Notes db to disk, normally this is 0 (or empty)

Pipeliner PowerShell errors – The number of PowerShell upload errors, idealy this should be 0 (or empty). Note Pipeliner will try 5 times to bring this number down by doing new uploads

Pipeliner skipped documents – During the export of the Notes db to disk, documents are converted into aspx files. Some documents may cause problems during the conversion. These documents will not be exported, and won’t be uploaded to SPO.

Open Pipeliner document – This button opens the Pipeliner document that specifies all archiving settings for this database

Check migration forms – This button is important for admins. It opens the Notes db (in the Notes client) and the SPO site (in the browser). The Notes client opens a view that displays all databases by form. This view has been added to the Notes database by Pipeliner, and consequently has been exported to SPO as well. In SPO, the view is called ‘z. IONET Info View’ and is listed at the bottom of the library. Administrators can open this view in SPO to conveniently compare Notes records with SPO records.

Pipeliner log – The important parts of the Pipeliner log are listed here: the export to disk and the PowerShell upload. The logs can be very long, therefore we only show the start and end log of the disk export and the PowerShell uploads. Also all errors and exceptions are listed here.

Skip log – A list of Notes documents that were not exported to disk , and the reason why the export failed.

OLE status – The status of the OLE objects conversion, so the admin can see if the Notes db had an OLE fix run.

OLE log – Details on the OLE objects conversion. This log may be empty if the Notes db did not have OLE objects.

OLE NoteIDs – The Notes documents were OLE objects were found.

Permissions log – Notes accounts migrations to the SPO sites (see next section on this operation)

The fields below display data generated by the High Level/Deep Notes to SPO comparison operation (see section ‘NDDM integration with IONET Pipeliner, validate the archive quality’)

NoteID log – This log contains information of the Notes vs. SPO Deep Scan analysis, relating to the comparison of Notes documents/SPO list items.

Attachments log – This log contains information of the Notes vs. SPO Deep Scan analysis, relating to the comparison of Notes attachments/SPO files.

Nest step is to validate the archive quality, and then the permissions migration.

Various problems may occur during the conversion of Notes databases to SPO libraries. NDDM has a built-in validation procedure to make sure that SPO archive matches the Notes database content, and no issues are overlooked.

NDDM is capable in comparing the Notes databases with the SPO sites content, and it will discover missing data in SPO. This section explains the quality features NDDM offers and how you can use them.

The archival of a Notes db to SPO is a two-stage rocket:

1 – the Notes db is exported to disk, to a folders and files structure – illustrated in green in the image below

2 – with PowerShell, the folders and files are uploaded to SPO – illustrated in orange in the image below

First make sure that the Disk Errors and PowerShell Errors are empty (or low in case of large Notes db conversions).

If you encounter high error numbers there is a fundamental problem that needs to be solved.

NDDM offers a high level and deep level scan that compare the Notes db content with the SPO site content.

First, perform a high level quality scan. This scan is fairly quickly. Only when this scan reports a difference in number of documents/files between Notes and SPO, you should perform a deep scan. Do not use a deep scan when no difference is found, because the deep scan is thorough and very slow. The deep scan will tell you exactly which Notes documents and/or files are missing in SPO. Again these scans can be carried out on multiple records in NDDM, eg. multipe archive operations.

High level archive quality scan

From the ‘Pipeliner Archival\Completed’ view, you will have access to the high level scan buttons that operate on selected Notes db.

Select the NDDM Notes db records you want to do the scan for. Press the action ‘Scan SP Sites (selected)’. After some calculations, a PowerShell script is excecuted. An example is displayed below.

The high level script scans the SPO site for Readers/Visitors groups, the Pipeliner index.html file, number of aspx views, Notes documents (in aspx), files, and parent links.

The scan results are stored in a file with a fixed filename, on the local PC. The location of this file is the work directory specified in NDDM settings. When the scan is complete, the results can be imported in NDDM using below the ‘Import scan results (high level)’ action.

The results are visibible on the ‘Pipeliner Archive’ tab.

The conclusion of this scan is that 1 Notes document and 4 files are missing in SPO. There are no Readers groups found in SPO, which means that the permissions migration has not yet been done.

Deep archive quality scan

When a database contains many documents, it is extremely difficult to manually find missing documents, let alone missing files. The deep scan can do this for you.

You can choose to only scan for missing documents (NoteIDs) or scan for both missing documents and missing files (NoteIDs + attachments). Scanning for attachments takes much more time, so if the database does not contain attachments (or high level file validation equals 0), use the first option.

In this case, we choose the second option because we want to find information about 1 missing Notes document and 4 missing files in SPO.

When the PowerShell program is completed, the window will automatically close. Use menu option “Import scan results (deep scan)” to load the scan results into the NDDM record. Again, the import file has a fixed filename and a fixed location.

Now we can locate the missing Notes document in the Notes database, based on the NoteID (or Unique ID). It has to be determined if the Notes doc and files missing are relevant or not. Notes databases are famous of storing corrupted content and this is only noticed once the content is archived.

Manual investigation shows that this document is the same document that was mentioned in the Skip log. This explains the difference in number of documents. Also, this skipped document contains 4 attachments. The Attachments log does not show 4 missing attachments, because logically, a missing document can not be compared. So this explains the missing 4 attachments.

Add these findings to the ‘Validation comments’ field, so that it is clear for everyone that this issue is being taken care of.

When missing files are detected on archived documents, the Attachments log will display the NoteID of the document, and the number of attachments found in Notes as well as SP, like this:

The view ‘Pipeliner Archived\Completed’ shows all relevant data of the archived databases and their validation status.

1/4

Pipeliner Profile: This indicates which Domino server is doing the Pipeliner archive work and whether the Notes db is archived with Teamroom alike settings.

Teamr: NDDM calculates whether the Notes db has a design that is comparable to a Notes Document Library or Teamroom. This is needed to set the pipeliner profile.

Title: Notes database title

#Docs: Number of documents in the Notes database

#Att: Number of files in the Notes database

#OLE: Number of OLE objects in the Notes database

Sent to pipeliner: The date/time when the Notes db was sent to the Pipeliner factory for archival to SPO

Disk%: Pipeliner archival % ready of phase 1 export to disk

PS%: Pipeliner archival % ready of phase 2 upload to SPO

Disk Er: Pipeliner archival phase 1 export to disk – number of export errors

PS Er: Pipeliner archival phase 2 upload to SPO – number of PowerShell upload errors

#Docs read protected: Number of Notes documents in the Notes db that have Readers fields

#Repl Conflicts: Number of Notes replication/save conflicts in the Notes db

Skip Doc: Number of Notes documents skipped during Pipeliner archival phase 1

Pipe aspx uploaded: Number of aspx files uploaded to SPO, 1 aspx file equals 1 Notes document

Pipe items uploaded (aspx, fld, att): Total number of files and folders uploaded to SPO, this number cannot be compared with the Notes db statistics

2/4

SP Library: The name of the SPO library created by Pipeliner where the Notes db archive is uploaded to

#SP Index: Number of index files in the SPO library. Pipeliner creates an ‘.index.aspx’ and a ‘z. IONET Info View.aspx’ file in the SPO library.

#SP Views: Number of Notes aspx views in the SPO library

#SP Docs: Number of aspx items in the SPO library, excluding the the view and index files

#SP Files: Number of Notes attachments in the SPO library

#Par links: Number of parent links. For each file in the SPO library, Pipeliner creates a link to its parent doc aspx

#SP Reader: The number of Readers/Visitors groups listed in the SPO site groups. There should be only 1 Visitors group. Zero groups means that the Notes to AD permissions migration has not been conducted.

Delta Δ Doc: High level scan comparison between number of Notes documents and aspx files in SPO, 0 is perfect

Delta Δ File: High level scan comparison between number of Notes attachments and attachments files in SPO, 0 is perfect

SPO Validated (High level): Date when the high level comparison between Notes and SPO has been conducted

SPO Validated (Deep NoteID): Date when the deep level comparison between Notes and SPO has been conducted – Notes documents only

SPO Validated (Deep Att): Date when the deep level comparison between Notes and SPO has been conducted – Notes documents and files

NoteID/Att log (Deep): Deep level scan results, difference Notes versus SPO for Notes documents and attachments, 0/0 is perfect

Owner: The owner of the Notes db

Comments: Text listed in the NDDM record comments field

3/4

SP Site: The SharePoint site where the Notes db has been archived to, this is without tenant name

Export Status: Pipeliner archive status

OLE: Status value indicating whether OLE objects have been converted to normal files

#Owner: Number of Notes db owner names to be migrated to AD security groups

#Visitor: Number of Notes usernames to be migrated to AD security groups, these include managers, editors and so on, including Domino servers and Administrators.

#SPO Owners: Number of Notes db Owner names that will be migrated to the AD Owner group. NDDM removes user and server names that are excluded during the user activity scan, as well as Notes names that cannot be resolved to AD users.

#SPO Visitors: Number of Notes user names to be migrated to the AD Visitors group. NDDM removes user and server names that are excluded during the user activity scan, as well as Notes names that cannot be resolved to AD users.

SPO Group: Status indicating when the Notes permissions have been migrated to AD and SPO

File Name: Notes database file name on the Domino server

ACL Close Status: Status of NDDM changes to the database ACL. The status can be Read Only, Closed or Reverted. Status ‘New’ means that NDDM did not perform action on the db ACL.

Size [MB]: Notes db size

Status: NDDM status value

ACL Close Date: date on which NDDM has closed the Notes db ACL

Server: Domino server where the Notes db is located.

Winner: A ‘winner’ means that this db is used the most of all replicas. This status comes from the NDDM targeting phase.

4/4

Scan Time: Last time a NDDM scan has been conducted

Headers: If the Notes db headers have been scanned

Content: If the Notes db content has been scanned

Readers & Authors: If the Notes db readers and authors fields have been scanned

Restricted Content: If the Notes db content has been scanned for Notes document protected with readers fields

Last Error: Last NDDM Notes db scan error

When a database has been successfully migrated, the next step is to give the Owner and/or Notes users read permission to the SPO site. The Notes db ACL scan has generated a list of Notes users (Managers, Designers, Editors, Authors and Readers) that should have at least read permissions in SPO.

NDDM will use the servers and admins exclusion list to remove names from the users list. These exclusions were carefully set during the scanning phase, so you don’t need to pay much attention to this here.

Besides these generic users, Database Owner(s) (specified in the Owner field of the Status Management tab) should receive full access permissions to change the security group members created in AD later on.

Because we cannot add Notes accounts to AD groups, we first have to map Notes user names to existing accounts in the Azure Active Directory. This is done in 3 steps: import person documents in NDDM, lookup SPO accounts, and import SPO accounts in NDDM.

The SPO account lookup is done via PowerShell. The connection details (tenant url) and Azure connection credentials that PowerShell needs are specified in the Settings/User Mapping/Connection view in NDDM.

In this manual we already explained how to create the AD Enterprise applications. One of the applications has to be extended with an App secret and this app is listed in the settings above and is used to carry out the Notes to AD user mappings. Or you might want to use another Connection type displayed in the form above.

If the SPO settings are in place, go to the Settings/User Mapping/Users

In the User Mappings view you will find 4 actions, as illustrated below.

1. Import Person documents, you will be prompted to select the Domino directory from where you want to import the Notes person documents. All users will be imported

2. Lookup SPO account, a PowerShell is started to lookup the SPO user account for each imported person document. PowerShell wil use Azure connection credentials specified in the SPO settings document.

The PowerShell script should look like this

3. Import SPO account, SPO user credentials that were found in step 2 are added to the person documents in NDDM.

Now we have a mapping of Notes user ‘Tom van der Staaij/Lialis’ to SPO account Name ‘Tom van der Staaij’ / SPO account Email ‘Tom @lialis.com’

4. Retrieve SPO Mapping log

This option will fetch the pc-based log file and add it here

Once the Notes to AD users mapping is ready, you can lookup SPO logins for each NDDM record. First, use the ‘Retrieve Owners And Visitors’ action on as many selected NDDM records as you want.

This action will remove all excluded and duplicate entries from the collected Notes users list, and add the results to the Owners and Visitors field. Then it will retrieve the SPO login data for these Notes users, if a Person document exists for this user. The results are then added to the SPO login owners/visitors fields. The accounts to be added to the SPO site are listed in the image below, marked orange.

Now we are ready to start adding users to the SPO site groups.

As explained earlier there are 2 scenarios to archive Notes databases to SPO sites.

Scenario 1. Each Notes database is archived to a dedicated SPO site, like https://lialis.sharepoint.com/NotesA_ITServiceRequests

The NDDM record archiving settings are displayed below

Scenario 2. All Notes databases are archived to subsites, like https://lialis.sharepoint.com/notesarchives/ITServiceRequests

The NDDM record archiving settings are displayed below

These settings are important when we use NDDM to create the access groups after the Notes db has been archived to SPO, because the AD security group names are depending on the SPO site names that are used.

Scenario 1. Notes db is archived to a dedicated SPO site

In this scenario each SPO site is a main site, so there will be no site permission inheritance. NDDM will create an AD security group for this SPO site based on the subsite A value ‘NotesA-I_ITServiceRequest Visitors’ (note the space before Visitors), and add users and owners to this group.

Then this group is added to a SPO site group that has its name also based on this subsite A value ‘NotesA-I_ITServiceRequest-Visitors’ (note the hyphen before Visitors).

Scenario 2. Notes db are archived to subsites

This scenario uses a subsite for each Notes database. The AD and SPO group names are generated in the same way as mentioned in scenario 1, only now they are based on the subsite B value.

SPO subsites normally inherits the permissions from the main site. In scenario 2 however, NDDM will break the permission inheritance between the main site and the sub site and then NDDM will create the AD group and SPO groups based the subsite B value.

The SPO permission migration for both scenarios will look like this:

Scenario 1. Notes db is archived to a dedicated SPO site

Prior to the permissions migration by NDDM the permissions of the new site are set by SharePoint, as illustrated below.

Now select the newly and successfully migrated Notes databases and press the ‘..\Add To SPO\Owners…’ button:

This will trigger a PowerShell program that will created AD Azure security groups, which are then added to the SPO site visitor groups. Sit back and wait for the program to complete.

The SPO Visitors group is the only group that has members now, that is the AD security group

The AD security group is displayed below.

Scenario 2. Notes db are archived to subsites

Prior to the permissions migration by NDDM, the permissions of the sub site are inherited from the main site, as illustrated below.

Now select the newly and successfully migrated Notes databases and press the ‘..\Add To SPO\Owners…’ button:

This will trigger a PowerShell program that will create AD Azure security groups, which are then added to the SPO site visitors groups. Sit back and wait for the program to complete.

The PowerShell script has removed the permission inheritance and new groups are added to the SPO site.

The Visitors group has the AD security group as a member

This AD security group contains 4 members

Finally press this action to import the permissions migration log of the sites you have just processed. This log is stored on your PC.

In this example, 2 databases were processed. First all Azure Security groups are processed. After that, all SPO groups are processed.

This button uploads the log to the associated NDDM database documents.

Optionally, you can set permissions for one single site at a time. To do so, open the database info document in NDDM, in Edit mode. Then go to the tab Pipeliner Archive. When you scroll down, you will see the ‘Add Visitors to SPO’ button. Press this button:

A PowerShell is started to process permissions for the specific SPO site. When the PowerShell has completed, Lotus Notes continues to add the logging to the Permissions Log field in NDDM. You do not need to import this separately.

After the archival phase the owner needs to be informed of the archive job. For this communication the NDDM questionaires can be used. It is capable of sending emails to the owner with the link to the SPO archive and you can ask the owner to review the Notes database archive in SPO and to approve or decline it.

The mailings can be set here

An example of a questionaire mailing is displayed below. Please notice the important settings of this mailing record.

The email will be send to the Owner. Mention the Notes db title, file name and Domino server.

Select option “Add hyperlink to SPO site”. Ask 2 questions to the owner in order to get approval.

In order to send out the questionaires, you may select multiple records in the Pipeliner completed view and choose NDDM actions\Selected Documents\Send mailing

Choose the appropriate mailing and set the correct status (for example ‘700 Archive send to Owner for acknowledgement’). Then wait max 10 minutes for NDDM to send out the mailing and set the status to the 700 value.

In the previous step, questionaire emails have been sent to the owners of archived Notes databases, to acknowlegde the archive result. The email holds a link to the SPO archive site location. With the new status of the NDDM record, it is easy for the owner to find the NDDM records that should be reviewed.

In the questionaire view, sort the Status column and check if the owner replied to the questionaire, as illustrated below.

In this example, the owner approved the SPO archive, because answer Q1 is set to ‘Yes’. Now you can close the Notes db ACL (with NDDM) and set the status to “950 Archived accepted Closed”.

If the owner did not review the archive result, you can send the owner a reminder (using NDDM) and set the status to ‘701 Reminder Archive sent to Owner for acknowledgement’. When the owner does not reply within 1 week, you may consider closing the Notes db and set the status to ‘960 Archived accepted Closed Owner no response’.

When all Notes databases are closed, it is save to stop the Domino servers and the Domino decommission project is completed.