NDDM Technical Manual – Installation

NDDM Technical Manual – Installation

IBM Notes Domino Application Scan and Analysis tool.

Optimized for Domino Server decommission projects.

Lialis will always support your Domino Server Application Scan in short daily remote calls to ensure a perfect and swift scan.

Updated May 2023 for NDDM release 5.30

Download evaluation version of NDDM

Please use the form below to receive an email with a link to download an evaluation version of NDDM. We have added an limitation to the software because it is an evaluation version, hope you don’t mind. The only added limitation is that NDDM will analyze 20% of the Notes databases added to NDDM. The remaining 80% of the Notes databases are not analyzed. We highlight the Notes databases that will be analyzed in purple.

NDDM is a complex tool, please read the instructions on this page before you give it a swing. We hope you like it.

Scan steps overview and introduction

To achieve the best scanning result in the shortest possible time, we recommend that you perform the scanning sequence below.
Each step in the table below is explained in detail in this post.

Preparations
  • NDDM relies heavily on Domino server-to-server replication. Before deploying NDDM to your Domino environment, it is recommended that the current Domino Hub / Spoke server replication document is configured perfectly and that it allows fast replication of NDDM.nsf between all Domino servers in the Domino Domain. If your connection documents contain filenames, NDDM will not replicate. Please check this. A good approach is to create a dedicated Hub to Spoke server replication document for only replicating the NDDM.nsf file, with a replication interval of 15 minutes.
  • NDDM will run a Domino server-based agent that scans all Notes databases on all Domino servers. NDDM must be signed with a specific signer ID. This signer ID must be listed in all Domino server documents in the “Sign or run unrestricted methods and operations” field.
  • The signer ID must be added to the ACL of NDDM.nsf with ‘manager can delete documents’ access.
  • All Notes apps on the Domino servers to be scanned must have the hosting Domino server in the ACL, with all roles enabled. It is good practice to ensure that the LocalDomainServers group is present in all Notes database ACLs with all ACL roles enabled. NDDM has a fuction to make sure this is adjusted when needed.
  • Please review the Agent Manager task settings in the Domino server documents of the Domino servers to be scanned by NDDM. Notes content scanning can take several hours or days, depending on the size of the Notes databases on the Domino server. It is recommended to extend the agent manager runtime, for example to 36000 minutes (yes very long). If the scan takes more time, there is no problem: NDDM will continue the scan at the point where it was interrupted in the previous scan.

  • User activity recording must be enabled for all Notes databases. The scheduled servertask load statlog will take care of this. Please verify that this task exists in notes.ini or as a server program document in the Domino Directory. Check some random nsf files to make sure user activity recording is enabled. If user activity recording is not enabled, running the Domino server command load statlog will enable user activity recording. If load statlog doesn’t do anything, check the notes.ini for No_Force_Activity_Logging. If you find this setting, you can remove it or set the value to 0 (instead of  1).
  • Carefully read the “Supported Notes and Domino builds” section on this page. Certain Domino server releases on certain operating systems require a notes.ini setting LS64BITCCALLOUTPointerSupport=1.  For some 64-bit Domino servers, this setting is required to fix a Domino 32/64 bit incompatibility issue that would otherwise prevent the NDDM from performing a server-based Notes Database user activity scan.
Setup and roll out
  • Install the NDDM Notes application on the first Domino HUB server
  • Set the ACL of the NDDM Notes application, add the signer ID to the NDDM ACL with manager access, do the same for the server group that contains all the servers in your domain.
  • Assign the Admin role to all Domino admin and server groups.
  • To grant end users read access to all content in NDDM you must assign them the Admin role.
  • End users who receive questionnaire emails from NDDM should not be assigned this role. NDDM is set up so that people who receive questionnaire emails from NDDM only see the NDDM content they are entitled to. This topic is explained in more detail under section “Sending batch mailings and questionnaires to end users for information collection”.
  • Sign the NDDM Notes application with an ID that allows unrestricted agents to execute (allow restricted operations)
  • Use NDDM to create NDDM replicas to all Domino servers to be scanned in your Domino domain, or use AdminP to deploy NDDM to all Spoke servers.
Import databases
  • Instruct all Domino servers to import the Notes database filenames into NDDM
  • Wait for replication to deliver the results to NDDM on the HUB server
  • Review the exclusion list, exclude all Domino system, test and admin databases and delete them from NDDM, now they will not be imported again later on
User activity scan
  • Run the server-based user activity, on one Domino server that holds many or the most Notes databases
  • After the first run, you can exclude all discovered non-real end user accounts (e.g. admin, signer, system, server accounts).
  • Then run the activity scan a few times more. During these successive runs, accounts previously excluded will not be found. However, new non-real  end user accounts may surface. Exclude these as well.
  • Continue excluding non-real end user accounts until only real end users are found in the user activity, or the scan returns na (‘not applicable’) for certain Notes databases (this means the user activity does not hold real end users).
  • Now you are set to scan all other Domino servers for user activity. Keep on adding newly discovered non-real end user accounts to the exclude list and re-scan these Notes databases.
  • Continue this process until all Notes databases are scanned for user activity, and the scan only returns real end user accounts or na’s.
Content and design scan
  • Before starting the content scan, it is important to make sure that the Domino server groups in the ACL of each Notes database, have all ACL roles enabled. Consult this manual for how to do this with NDDM
  • Instruct all Domino servers to carry out the content and the design scan
  • Wait for replication to deliver the results to NDDM on the HUB server
Review content scan
  • A small number of Notes databases may not have been scanned by the Domino server
  • Re-scan these Notes databases using the manual scan option, with the Notes administrator client full access administration enabled
Database read protected documents
  • NDDM offers a method to identify the number of Notes documents that have readers fields without ACL roles, so these documents cannot be accessed by people not listed in the readers fields. NDDM can count the number of documents in this category per Notes database.
Target calculation
  • Make sure the Notes database user activity and content scan are completed for all Notes databases
  • Set the target calculation cut-off date, for example 1 or 2 years in the past (as per your choice)
  • Run the target calculation against all Notes databases
  • Export the results to Excel and review them in conjunction with Lialis (send us the Excel to review)
Optional: Run design keywords scan on Replace Notes databases (server/client based agent)
  • Run the ‘Scan Design on Keywords’ scan on the Notes databases that have the target ‘Replace’. This helps you find the Notes databases that are, for example, sending workflow approval emails (in this example type the keywords: @MailSend and notes://). This scan also allows you to find ODBC connections to a certain server (in this example type the specific SQL server name).
Optional: Run deep doclinks scan on Replace Notes databases (server/client based agent)
  • Run the ‘Deep Scan Doclinks’ scan on the Notes databases with the target Replace. This scan is relevant for the Notes content migration to another platform.
Optional: Run readers and authors fields scan on Replace Notes databases (client based agent)
  • Run the ‘Scan Readers/Authors fields selected docs’ scan to find out how the Notes databases are using readers and authors fields

 

Except for ‘readers/authors’ scan, all scan tasks are carried out by the Domino server. Scanning may also be done by the Notes client at much lower scan speeds though.

Supported Notes and Domino builds

Supported Notes and Domino builds

NDDM runs a scheduled agent on Domino servers to carry out the scan. Therefore NDDM has been tested extensively on the newest versions of all relevant Domino server major releases: R6, R7, R8, R9, R10, R11 and R12; on both Windows and Linux operating systems, 32 and 64 bits servers. We expect lower Domino servers versions will not pose any problems. However, if you run lower Domino server builds (R4, R5), we advise you to contact us so we can give these versions a test run. Please consult the following list with Domino versions/platforms that we tested NDDM on.

Summary of tested configurations:

Windows/32
Release 9.0.1 FP8|February 23, 2017 Build 405
Release 9.0.1 October 14, 2013 Build 405
Release 8.5.3 FP6|November 21, 2013 Build 390
Release 7.0.2 September 26, 2006 Build 265
Release 6.5.5 November 30, 2005 Build 198

Windows/64
Release 12.0.1 FP1 HF54|April 11, 2022 Build 470 *
Release 12.0.1 FP1 IF1|April 11, 2022 Build 470 *
Release 12.0.1|December 14, 2021 Build 470 *
Release 11.0.1 FP2|November 17, 2020 Build 452 *
Release 10.0.1 FP6|September 24, 2020 Build 450 *
Release 9.0.1 FP10 HF747|July 24, 2020 Build 405
Release 8.5.3 FP6 HF3175|April 28, 2017 Build 390

Linux/32
Release 9.0.1 October 14, 2013 Build 405

Linux/64
Release 12.01 (notes.ini setting NOT needed)
Release 11.0.1 FP2|October 20, 2020 Build 452 *
Release 11.0 November 25, 2019 Build 451 *
Release 10.0.1 November 29, 2018 Build 450 *
Release 9.0.1 October 14, 2013 Build 405 *

*) notes.ini setting required: LS64BITCCALLOUTPointerSupport=1

The required notes.ini settings for some of the 64-bit servers is a fix that has been applied because of 32/64 bit incompatibility. This setting is necessary for the relevant server to perform a user activity scan on a database. When an argument is passed by reference, the C function receives a 32-bit pointer to the value area. API calls from LotusScript on a 64-bit Domino server will fail (crash) when an 64-bit pointer is expected.

For example: when a variable is a pointer that is passed by reference, this works perfect on a Domino 32-bit server when this variable is defined in LotusScript as a Long. However, the same situation on a 64-bit Domino server – where the variable is defined in LotusScript as a (64-bit) Double – makes the server crash.

In order to be able to return the pointer in double, you need to set environment variable LS64BITCCALLOUTPointerSupport=1

You can set this variable by directly editing the server’s notes.ini, or by issuing a console command on the server:

set config LS64BITCCALLOUTPointerSupport=1

A server restart is not necessary after this notes.ini change. It will take effect immediately.

After adding the Notes.ini setting, the Agent Manager task has to be restarted on the Domino server.

During our extensive testing, we learned which servers this setting should apply to. If a user activity scan is initiated on such a server – and this setting is not found in the server’s notes.ini – this is reported in the scan log on the Task document. The database scan will continue its run, but the user activity scan task is skipped.

Furthermore, if a scheduled User Activity scan task is run from a server that may very well run without problems, but was not tested by us, we don’t allow the agent to run the User Activity scan anyway. Other scan tasks are allowed though. The Task document log will tell you that the User Activity scan is not possible on this server.

This also applies to non-tested 32 bit servers. As mentioned above, we tested the newest versions of all relevant Domino server major releases from R6 up to R12. Only on these Domino servers, the server-based agent is allowed to do a User Activity scan. If one of your servers is not allowed to do the User Activity scan, you can run a User Activity scan from the client (background agent).  You can also contact Lialis and pass us the Domino server release, so we can test the User Activity scan on this specific Domino version, and adjust the script.

If for whatever reason the agent is interrupted during the scan (for example by an unexpected error, or an automatic Domino server reboot or crash), the Notes database that the agent was processing at the time of the interruption is skipped when the NDDM scan agent starts again. Just in case the scan of this database has caused a server crash, omitting this database in the successive run prevents the server from crashing again. The skipped database is logged:

Domino server performance impact

Domino server performance impact

All scanning tasks are carried out by the Domino server on its own local Notes databases by a NDDM Lotus Script agent. See image below:

Domino server-based scan of the Notes database user activity. The impact on the Domino server is zero because the scan is very fast. For example scanning user activity for 1.000 Notes databases will take less than 10 minutes (depending on server speed).

Domino Server-based scan of the Notes database content. The impact on the Domino server performance is average; not high and not low. The agent will scan each Notes document in all Notes databases on the Domino server running the scan. The scans are very fast because the Domino servers scan their local Notes files. However, a single Domino server scan can take up to a few days depending on the amount of Notes files on the Domino server.

NDDM has been designed in such manner that multiple Domino servers will run their own local scans in a parallel operation, resulting in enormous scan speeds. So, scanning 1 Domino server will take the same amount of time (a day or 2) as scanning 10 or even 100 Domino servers.

The Domino server agent manager must allow long agent run times of at least 6 hours (12 hours is even better). If the Domino agent manager cuts off the NDDM scan agent there is no problem. The NDDM scan agent will pick up the Notes database scan in the new run at the exact moment it was interrupted in the previous run.

A Domino administrator can adjust the NDDM scan agent, simply by adjusting the Agent schedule. For example to set the NDDM agent to only run at night time.

 

With NDDM we have scanned many Notes databases, from small to extremely large. The database in the example below holds 115,087 Notes documents and the NSF file size is 51 GB. The Notes content scan of this particular NSF file took 30 minutes.

01/06/2021 05:57:09 PM GMT – Start Database Analysis
01/06/2021 05:57:09 PM GMT – Start scanning Design Elements
01/06/2021 05:59:25 PM GMT – Finished scanning Design Elements
01/06/2021 05:59:26 PM GMT – Start scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished Database Analysis

Here is an example of a small Notes database of 400 MB holding 876,430 Notes documents. The scan took 15 minutes.

01/06/2021 01:23:47 PM GMT – Start Database Analysis
01/06/2021 01:23:47 PM GMT – Start scanning Design Elements
01/06/2021 01:24:46 PM GMT – Finished scanning Design Elements
01/06/2021 01:24:46 PM GMT – Start scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished Database Analysis

The scan speed depends mostly on the amount of Notes documents and to a lesser extend on the Notes database file size.

The total Notes database content scan speed also depends heavily on the Notes database setup and the OS server performance. We have managed to scan 280 GB in one hour. In this one hour, 4,823,000 Notes documents where scanned by one particular Domino server. If 100 Domino servers carry out the scan, the total scanned GBs and Notes documents can by multiplied by 100.

Setup and Configuration

Setup and Configuration

When you read this technical manual, we assume you have received the NDDM Notes database file from Lialis. It is not possible to download an evaluation version.

Open the local NDDM Notes database on your Notes client and make a Notes database copy (using your Notes client) to the main IBM Domino (HUB) server that replicates with all other Domino servers. Make sure to include the documents in your copy. Do not copy the ACL.

Open the new NDDM Notes database copy on the first Domino server and continue.

ACL – First adjust the ACL. Make sure all Domino servers have manager access with delete permissions and the Admin ACL role. The same goes for the signer ID and the Administrators of NDDM. You might also want to add viewers with read permissions, or even set the default access to Reader so that any Notes user in your organization can view the content.

Signing – NDDM will run unrestricted scheduled agents on all Domino servers to be scanned. This means you have to sign NDDM with a Notes ID that is allowed to run unrestricted agents on each Domino server. In most environments, the ID of the Notes administrator has these privileges. You can also use a certain Domino server ID file to sign NDDM. If the ID does not have the required permissions, no worries: NDDM will report this so that you can adjust it later on in the scan phase.

Configuration

NDDM holds multiple Settings views listed in the left navigator where the NDDM configuration must be set prior to scanning.

Excluded Files and Folders – First exclude all Domino system files and folders from the database import. You may also want to exclude the mail folders. We already made a pre-selection of system files that we are aware of. We do this because it’s a waste of resources to scan large Domino system databases like the log.nsf. It is up to you to add additional folders and files to the exclude list prior to carry out any scanning. Because Linux servers are case-sensitive, the file/folder names entered here are case-sensitive as well.

You may use the button “Add additional files/folders” to paste files and folders to quickly setup the exclusions.

It is not a problem when some system databases were accidentally imported. You can delete these system database documents from the NDDM, and you can then improve the exclusions list after the Domino server import has been completed.

Users and servers to exclude from the user activity scan – Here you must add users and servers to exclude from the Notes database user activity scan. This step is mandatory, because filtering away all Domino servers, signer ID and names of administrators and developers will result in a much more accurate list of activity of real users. Generally, it’s not interesting to know what servers and admins are doing in a Notes database. So add as many names now. Later in this manual you will be guided how to improve this list.

Try to add as many known Domino servers and ‘unreal’ users to the exclude list. See the sample below:

You may use the button “Add additional servers/users” to paste Server and user names from Domino groups to quickly setup the exclusions.

Search keywords – The strings listed in this view will be used for the design string scan. In the setup phase we leave it for now.

NDDM Servers –  The roll-out part of NDDM allows you to create replicas of the NDDM Notes database to all (or a selection of) Domino servers in the Domino domain that you are planning to scan with NDDM. Your Notes client will create the new replicas in the Notes client background. Depending on the network bandwidth and the amount of Domino servers you are actually going to scan, this may take some time.

You may also use the Domino Adminp process to create NDDM replica’s.

You need to make sure that all Domino servers you are going to scan are setup with a so-called ‘pull push replication’ to the main or HUB Domino server. Normally this is the case in Domino environments. Check this anyway just to be sure.

If you have Domino cluster servers, you only need to install NDDM on the main Domino cluster server.

When all NDDM replicas are created with NDDM or via Adminp, please press the button “Check NDDM Replica” to make sure the NDDM file is present on each Domino server.

You may assign Keys to each group of Domino server, for example to indicate global regions where the Domino servers reside in.

Send Errors to – If scan errors occur on a database, a button will appear on the Database info document that allows you to send the error message to the specified address.

Targets – Lialis NDDM has a set of Target calculation rules that will assist you to calculate the Targets automatically when the scan is completed. Please refer to the section Target analysis on this page for details.

Pipeliner

NDDM can be integrated with the Pipeliner, a tool to archive Notes databases to SharePoint Online. You will find a lot of information on this topic in this guide.

Status Management

Here you can define the list of status values to be assigned on Notes databases to manage the life cycle. The status value can be used when carrying out communications with the NDDM questionnaire functionality, or during the archiving of the Notes database to SharePoint Online.

Mailings

NDDM offers a questionnaire tool to send out communications to your end users and collect their responses. Here you can manage the email templates and questions to be send.

User Mapping

This part of NDDM is used in the Notes database archival to SharePoint Online process.

The Connection view contains the SharePoint Online Tenant and SPO credentials. You can also specify the local temporary folder where data is saved before it is uploaded to the SPO site.

The Users view contains the mappings of Notes user names with AD names, so NDDM can add the users to the SharePoint site permissions.

Scheduled Agents

NDDM runs 4 scheduled agents, of which the agent ‘ProcessMailingQueue’ need adjustments in the Notes Designer Client prior to running.

You don’t need to make changes to the first 3 scheduled agents.

Agent DBDISCOVERYANALYSISONSERVER is the agent that does all of the import and analysis work. This agent needs to be able to perform restricted operations with full administration rights. This will prevent a lot of database access errors. No changes needed.

Agent EXPORTONSERVER creates a CSV export of all databases that are included in the NDDM tool. To enable the agent to save the CSV file on a drive, this agent should be allowed to perform restricted operations. No changes needed.

Agent ProcessMailingQueue sends out email messages to database owners. To make sure the message is only sent once from one server, and to provide a single reply address, this agent must run on one specified server. Please adjust the Domino server where the agent should run on, for instance the HUB Domino server.

Please also adjust the run on behalf of the reply user:

This is also described in section ‘Sending batch mailings and questionnaires to end users for information collection’.