NDDM Technical Manual

Please use the form below to receive an email with a link to download an evaluation version of NDDM. We have added an limitation to the software because it is an evaluation version, hope you don’t mind. The only added limitation is that NDDM will analyze 20% of the Notes databases added to NDDM. The remaining 80% of the Notes databases are not analyzed. We highlight the Notes databases that will be analyzed in purple.

NDDM is a complex tool, please read the instructions on this page before you give it a swing. We hope you like it.

To achieve the best scanning result in the shortest possible time, we recommend that you perform the scanning sequence below.
Each step in the table below is explained in detail in this post.

Preparations	NDDM relies heavily on Domino server-to-server replication. Before deploying NDDM to your Domino environment, it is recommended that the current Domino Hub / Spoke server replication document is configured perfectly and that it allows fast replication of NDDM.nsf between all Domino servers in the Domino Domain. If your connection documents contain filenames, NDDM will not replicate. Please check this. A good approach is to create a dedicated Hub to Spoke server replication document for only replicating the NDDM.nsf file, with a replication interval of 15 minutes. NDDM will run a Domino server-based agent that scans all Notes databases on all Domino servers. NDDM must be signed with a specific signer ID. This signer ID must be listed in all Domino server documents in the “Sign or run unrestricted methods and operations” field. The signer ID must be added to the ACL of NDDM.nsf with ‘manager can delete documents’ access. All Notes apps on the Domino servers to be scanned must have the hosting Domino server in the ACL, with all roles enabled. It is good practice to ensure that the LocalDomainServers group is present in all Notes database ACLs with all ACL roles enabled. NDDM has a fuction to make sure this is adjusted when needed. Please review the Agent Manager task settings in the Domino server documents of the Domino servers to be scanned by NDDM. Notes content scanning can take several hours or days, depending on the size of the Notes databases on the Domino server. It is recommended to extend the agent manager runtime, for example to 36000 minutes (yes very long). If the scan takes more time, there is no problem: NDDM will continue the scan at the point where it was interrupted in the previous scan. User activity recording must be enabled for all Notes databases. The scheduled servertask load statlog will take care of this. Please verify that this task exists in notes.ini or as a server program document in the Domino Directory. Check some random nsf files to make sure user activity recording is enabled. If user activity recording is not enabled, running the Domino server command load statlog will enable user activity recording. If load statlog doesn’t do anything, check the notes.ini for No_Force_Activity_Logging. If you find this setting, you can remove it or set the value to 0 (instead of  1). Carefully read the “Supported Notes and Domino builds” section on this page. Certain Domino server releases on certain operating systems require a notes.ini setting LS64BITCCALLOUTPointerSupport=1.  For some 64-bit Domino servers, this setting is required to fix a Domino 32/64 bit incompatibility issue that would otherwise prevent the NDDM from performing a server-based Notes Database user activity scan.
Setup and roll out	Install the NDDM Notes application on the first Domino HUB server Set the ACL of the NDDM Notes application, add the signer ID to the NDDM ACL with manager access, do the same for the server group that contains all the servers in your domain. Assign the Admin role to all Domino admin and server groups. To grant end users read access to all content in NDDM you must assign them the Admin role. End users who receive questionnaire emails from NDDM should not be assigned this role. NDDM is set up so that people who receive questionnaire emails from NDDM only see the NDDM content they are entitled to. This topic is explained in more detail under section “Sending batch mailings and questionnaires to end users for information collection”. Sign the NDDM Notes application with an ID that allows unrestricted agents to execute (allow restricted operations) Use NDDM to create NDDM replicas to all Domino servers to be scanned in your Domino domain, or use AdminP to deploy NDDM to all Spoke servers.
Import databases	Instruct all Domino servers to import the Notes database filenames into NDDM Wait for replication to deliver the results to NDDM on the HUB server Review the exclusion list, exclude all Domino system, test and admin databases and delete them from NDDM, now they will not be imported again later on
User activity scan	Run the server-based user activity, on one Domino server that holds many or the most Notes databases After the first run, you can exclude all discovered non-real end user accounts (e.g. admin, signer, system, server accounts). Then run the activity scan a few times more. During these successive runs, accounts previously excluded will not be found. However, new non-real  end user accounts may surface. Exclude these as well. Continue excluding non-real end user accounts until only real end users are found in the user activity, or the scan returns na (‘not applicable’) for certain Notes databases (this means the user activity does not hold real end users). Now you are set to scan all other Domino servers for user activity. Keep on adding newly discovered non-real end user accounts to the exclude list and re-scan these Notes databases. Continue this process until all Notes databases are scanned for user activity, and the scan only returns real end user accounts or na’s.
Content and design scan	Before starting the content scan, it is important to make sure that the Domino server groups in the ACL of each Notes database, have all ACL roles enabled. Consult this manual for how to do this with NDDM Instruct all Domino servers to carry out the content and the design scan Wait for replication to deliver the results to NDDM on the HUB server
Review content scan	A small number of Notes databases may not have been scanned by the Domino server Re-scan these Notes databases using the manual scan option, with the Notes administrator client full access administration enabled
Database read protected documents	NDDM offers a method to identify the number of Notes documents that have readers fields without ACL roles, so these documents cannot be accessed by people not listed in the readers fields. NDDM can count the number of documents in this category per Notes database.
Target calculation	Make sure the Notes database user activity and content scan are completed for all Notes databases Set the target calculation cut-off date, for example 1 or 2 years in the past (as per your choice) Run the target calculation against all Notes databases Export the results to Excel and review them in conjunction with Lialis (send us the Excel to review)
Optional: Run design keywords scan on Replace Notes databases (server/client based agent)	Run the ‘Scan Design on Keywords’ scan on the Notes databases that have the target ‘Replace’. This helps you find the Notes databases that are, for example, sending workflow approval emails (in this example type the keywords: @MailSend and notes://). This scan also allows you to find ODBC connections to a certain server (in this example type the specific SQL server name).
Optional: Run deep doclinks scan on Replace Notes databases (server/client based agent)	Run the ‘Deep Scan Doclinks’ scan on the Notes databases with the target Replace. This scan is relevant for the Notes content migration to another platform.
Optional: Run readers and authors fields scan on Replace Notes databases (client based agent)	Run the ‘Scan Readers/Authors fields selected docs’ scan to find out how the Notes databases are using readers and authors fields

 

Except for ‘readers/authors’ scan, all scan tasks are carried out by the Domino server. Scanning may also be done by the Notes client at much lower scan speeds though.

Supported Notes and Domino builds

NDDM runs a scheduled agent on Domino servers to carry out the scan. Therefore NDDM has been tested extensively on the newest versions of all relevant Domino server major releases: R6, R7, R8, R9, R10, R11 and R12; on both Windows and Linux operating systems, 32 and 64 bits servers. We expect lower Domino servers versions will not pose any problems. However, if you run lower Domino server builds (R4, R5), we advise you to contact us so we can give these versions a test run. Please consult the following list with Domino versions/platforms that we tested NDDM on.

Summary of tested configurations:

Windows/32
Release 9.0.1 FP8|February 23, 2017 Build 405
Release 9.0.1 October 14, 2013 Build 405
Release 8.5.3 FP6|November 21, 2013 Build 390
Release 7.0.2 September 26, 2006 Build 265
Release 6.5.5 November 30, 2005 Build 198

Windows/64
Release 12.0.1 FP1 HF54|April 11, 2022 Build 470 *
Release 12.0.1 FP1 IF1|April 11, 2022 Build 470 *
Release 12.0.1|December 14, 2021 Build 470 *
Release 11.0.1 FP2|November 17, 2020 Build 452 *
Release 10.0.1 FP6|September 24, 2020 Build 450 *
Release 9.0.1 FP10 HF747|July 24, 2020 Build 405
Release 8.5.3 FP6 HF3175|April 28, 2017 Build 390

Linux/32
Release 9.0.1 October 14, 2013 Build 405

Linux/64
Release 11.0.1 FP2|October 20, 2020 Build 452 *
Release 11.0 November 25, 2019 Build 451 *
Release 10.0.1 November 29, 2018 Build 450 *
Release 9.0.1 October 14, 2013 Build 405 *

*) notes.ini setting required: LS64BITCCALLOUTPointerSupport=1

The required notes.ini settings for some of the 64-bit servers is a fix that has been applied because of 32/64 bit incompatibility. This setting is necessary for the relevant server to perform a user activity scan on a database. When an argument is passed by reference, the C function receives a 32-bit pointer to the value area. API calls from LotusScript on a 64-bit Domino server will fail (crash) when an 64-bit pointer is expected.

For example: when a variable is a pointer that is passed by reference, this works perfect on a Domino 32-bit server when this variable is defined in LotusScript as a Long. However, the same situation on a 64-bit Domino server – where the variable is defined in LotusScript as a (64-bit) Double – makes the server crash.

In order to be able to return the pointer in double, you need to set environment variable LS64BITCCALLOUTPointerSupport=1

You can set this variable by directly editing the server’s notes.ini, or by issuing a console command on the server:

set config LS64BITCCALLOUTPointerSupport=1

A server restart is not necessary after this notes.ini change. It will take effect immediately.

After adding the Notes.ini setting, the Agent Manager task has to be restarted on the Domino server.

During our extensive testing, we learned which servers this setting should apply to. If a user activity scan is initiated on such a server – and this setting is not found in the server’s notes.ini – this is reported in the scan log on the Task document. The database scan will continue its run, but the user activity scan task is skipped.

Furthermore, if a scheduled User Activity scan task is run from a server that may very well run without problems, but was not tested by us, we don’t allow the agent to run the User Activity scan anyway. Other scan tasks are allowed though. The Task document log will tell you that the User Activity scan is not possible on this server.

This also applies to non-tested 32 bit servers. As mentioned above, we tested the newest versions of all relevant Domino server major releases from R6 up to R12. Only on these Domino servers, the server-based agent is allowed to do a User Activity scan. If one of your servers is not allowed to do the User Activity scan, you can run a User Activity scan from the client (background agent).  You can also contact Lialis and pass us the Domino server release, so we can test the User Activity scan on this specific Domino version, and adjust the script.

If for whatever reason the agent is interrupted during the scan (for example by an unexpected error, or an automatic Domino server reboot or crash), the Notes database that the agent was processing at the time of the interruption is skipped when the NDDM scan agent starts again. Just in case the scan of this database has caused a server crash, omitting this database in the successive run prevents the server from crashing again. The skipped database is logged:

Domino server performance impact

All scanning tasks are carried out by the Domino server on its own local Notes databases by a NDDM Lotus Script agent. See image below:

Domino server-based scan of the Notes database user activity. The impact on the Domino server is zero because the scan is very fast. For example scanning user activity for 1.000 Notes databases will take less than 10 minutes (depending on server speed).

Domino Server-based scan of the Notes database content. The impact on the Domino server performance is average; not high and not low. The agent will scan each Notes document in all Notes databases on the Domino server running the scan. The scans are very fast because the Domino servers scan their local Notes files. However, a single Domino server scan can take up to a few days depending on the amount of Notes files on the Domino server.

NDDM has been designed in such manner that multiple Domino servers will run their own local scans in a parallel operation, resulting in enormous scan speeds. So, scanning 1 Domino server will take the same amount of time (a day or 2) as scanning 10 or even 100 Domino servers.

The Domino server agent manager must allow long agent run times of at least 6 hours (12 hours is even better). If the Domino agent manager cuts off the NDDM scan agent there is no problem. The NDDM scan agent will pick up the Notes database scan in the new run at the exact moment it was interrupted in the previous run.

A Domino administrator can adjust the NDDM scan agent, simply by adjusting the Agent schedule. For example to set the NDDM agent to only run at night time.

 

With NDDM we have scanned many Notes databases, from small to extremely large. The database in the example below holds 115,087 Notes documents and the NSF file size is 51 GB. The Notes content scan of this particular NSF file took 30 minutes.

01/06/2021 05:57:09 PM GMT – Start Database Analysis
01/06/2021 05:57:09 PM GMT – Start scanning Design Elements
01/06/2021 05:59:25 PM GMT – Finished scanning Design Elements
01/06/2021 05:59:26 PM GMT – Start scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished Database Analysis

Here is an example of a small Notes database of 400 MB holding 876,430 Notes documents. The scan took 15 minutes.

01/06/2021 01:23:47 PM GMT – Start Database Analysis
01/06/2021 01:23:47 PM GMT – Start scanning Design Elements
01/06/2021 01:24:46 PM GMT – Finished scanning Design Elements
01/06/2021 01:24:46 PM GMT – Start scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished Database Analysis

The scan speed depends mostly on the amount of Notes documents and to a lesser extend on the Notes database file size.

The total Notes database content scan speed also depends heavily on the Notes database setup and the OS server performance. We have managed to scan 280 GB in one hour. In this one hour, 4,823,000 Notes documents where scanned by one particular Domino server. If 100 Domino servers carry out the scan, the total scanned GBs and Notes documents can by multiplied by 100.

Setup and Configuration

When you read this technical manual, we assume you have received the NDDM Notes database file from Lialis. It is not possible to download an evaluation version.

Open the local NDDM Notes database on your Notes client and make a Notes database copy (using your Notes client) to the main IBM Domino (HUB) server that replicates with all other Domino servers. Make sure to include the documents in your copy. Do not copy the ACL.

Open the new NDDM Notes database copy on the first Domino server and continue.

ACL – First adjust the ACL. Make sure all Domino servers have manager access with delete permissions and the Admin ACL role. The same goes for the signer ID and the Administrators of NDDM. You might also want to add viewers with read permissions, or even set the default access to Reader so that any Notes user in your organization can view the content.

Signing – NDDM will run unrestricted scheduled agents on all Domino servers to be scanned. This means you have to sign NDDM with a Notes ID that is allowed to run unrestricted agents on each Domino server. In most environments, the ID of the Notes administrator has these privileges. You can also use a certain Domino server ID file to sign NDDM. If the ID does not have the required permissions, no worries: NDDM will report this so that you can adjust it later on in the scan phase.

Configuration

NDDM holds multiple Settings views listed in the left navigator where the NDDM configuration must be set prior to scanning.

Excluded Files and Folders – First exclude all Domino system files and folders from the database import. You may also want to exclude the mail folders. We already made a pre-selection of system files that we are aware of. We do this because it’s a waste of resources to scan large Domino system databases like the log.nsf. It is up to you to add additional folders and files to the exclude list prior to carry out any scanning. Because Linux servers are case-sensitive, the file/folder names entered here are case-sensitive as well.

You may use the button “Add additional files/folders” to paste files and folders to quickly setup the exclusions.

It is not a problem when some system databases were accidentally imported. You can delete these system database documents from the NDDM, and you can then improve the exclusions list after the Domino server import has been completed.

Users and servers to exclude from the user activity scan – Here you must add users and servers to exclude from the Notes database user activity scan. This step is mandatory, because filtering away all Domino servers, signer ID and names of administrators and developers will result in a much more accurate list of activity of real users. Generally, it’s not interesting to know what servers and admins are doing in a Notes database. So add as many names now. Later in this manual you will be guided how to improve this list.

Try to add as many known Domino servers and ‘unreal’ users to the exclude list. See the sample below:

You may use the button “Add additional servers/users” to paste Server and user names from Domino groups to quickly setup the exclusions.

Search keywords – The strings listed in this view will be used for the design string scan. In the setup phase we leave it for now.

NDDM Servers –  The roll-out part of NDDM allows you to create replicas of the NDDM Notes database to all (or a selection of) Domino servers in the Domino domain that you are planning to scan with NDDM. Your Notes client will create the new replicas in the Notes client background. Depending on the network bandwidth and the amount of Domino servers you are actually going to scan, this may take some time.

You may also use the Domino Adminp process to create NDDM replica’s.

You need to make sure that all Domino servers you are going to scan are setup with a so-called ‘pull push replication’ to the main or HUB Domino server. Normally this is the case in Domino environments. Check this anyway just to be sure.

If you have Domino cluster servers, you only need to install NDDM on the main Domino cluster server.

When all NDDM replicas are created with NDDM or via Adminp, please press the button “Check NDDM Replica” to make sure the NDDM file is present on each Domino server.

You may assign Keys to each group of Domino server, for example to indicate global regions where the Domino servers reside in.

Send Errors to – If scan errors occur on a database, a button will appear on the Database info document that allows you to send the error message to the specified address.

Targets – Lialis NDDM has a set of Target calculation rules that will assist you to calculate the Targets automatically when the scan is completed. Please refer to the section Target analysis on this page for details.

Pipeliner

NDDM can be integrated with the Pipeliner, a tool to archive Notes databases to SharePoint Online. You will find a lot of information on this topic in this guide.

Status Management

Here you can define the list of status values to be assigned on Notes databases to manage the life cycle. The status value can be used when carrying out communications with the NDDM questionnaire functionality, or during the archiving of the Notes database to SharePoint Online.

Mailings

NDDM offers a questionnaire tool to send out communications to your end users and collect their responses. Here you can manage the email templates and questions to be send.

User Mapping

This part of NDDM is used in the Notes database archival to SharePoint Online process.

The Connection view contains the SharePoint Online Tenant and SPO credentials. You can also specify the local temporary folder where data is saved before it is uploaded to the SPO site.

The Users view contains the mappings of Notes user names with AD names, so NDDM can add the users to the SharePoint site permissions.

NDDM runs 4 scheduled agents, of which the agent ‘ProcessMailingQueue’ need adjustments in the Notes Designer Client prior to running.

You don’t need to make changes to the first 3 scheduled agents.

Agent DBDISCOVERYANALYSISONSERVER is the agent that does all of the import and analysis work. This agent needs to be able to perform restricted operations with full administration rights. This will prevent a lot of database access errors. No changes needed.

Agent EXPORTONSERVER creates a CSV export of all databases that are included in the NDDM tool. To enable the agent to save the CSV file on a drive, this agent should be allowed to perform restricted operations. No changes needed.

Agent ProcessMailingQueue sends out email messages to database owners. To make sure the message is only sent once from one server, and to provide a single reply address, this agent must run on one specified server. Please adjust the Domino server where the agent should run on, for instance the HUB Domino server.

Please also adjust the run on behalf of the reply user:

This is also described in section ‘Sending batch mailings and questionnaires to end users for information collection’.

The NDDM Notes views are set to NOT refresh automatically. This has been done to speed up the NDDM scan process. Please press F9 in most of the NDDM views when you are working in NDDM!

At this stage all selected Domino servers have a replica of the Lialis NDDM Notes database. You do not need to open all NDDM replicas. NDDM has been designed in such a way that you can initiate all scan work in 1 NDDM Notes database on your main central HUB server. With task documents, each Domino server is instructed to carry out its own scan work. You only need to wait for the results to be replicated back to the central NDDM Notes database.

You might want to check the Domino server agent manager task settings in the Domino server documents. Scanning may take a few hours or days depending on the size. It is recommended to increase the agent manager run time to 12 hours for example. If the scan takes more time, then there is no problem: NDDM will continue the scan at the point where it was interrupted in the previous scan.

You also might want to create new Domino server replication connections documents between the HUB server and all the Domino servers to be scanned, where only the NDDM Notes database is replicated (pull push) in the shortest time interval of 5 minutes for example. This will speed up the work considerably.

Follow the order of steps as described below carefully.

Scanning – Import Notes databases. In this step, the Domino server adds a Database Info document for each database found. To make sure that no .nsf file is omitted, only information that can be retrieved regardless of database access is recorded (replica id, server, filepath and -name). This task will only take a few seconds to complete.

Select one, multiple or all Domino servers. Use the Key to filter away Domino servers.

The scheduled tasks can be seen in this view:

Now wait until the Domino servers have completed the import task. Each database on the server now has a Database Info document in NDDM.

A Domino server will run this Notes database import task in a few seconds. The only delay is waiting on the agent manager and the scheduled replication.

Note: if there are multiple new/in progress task documents for the same server, the scan agent will always pick the newest Task document to run on, which is the first displayed in this (unsorted) view. So make sure that you only create a second task document AFTER the server completed its first task.

Ideally, use the ‘Scheduled On Domino’ option for all scan tasks to perform. Domino servers agents process data much faster than client-based agents. However, in certain cases you might want to run a task manually on the client. You can do so by using the ‘Manual On Client’ option in the navigator .

You can always re-run this step. The agent will simply add new Notes databases when they are found.

If you delete all NDDM content you can run this task also to start from scratch.

Make sure you improve the NDDM exclusion list and delete newly excluded files from NDDM simply by deleting the Notes database records in NDDM.

If Notes databases are deleted from the Domino server, the import task will inform you about this with help of this column in the Database Info view. A red cross will be displayed when a Notes database is no longer on the Domino server. You may then delete these records from NDDM.

When the Notes database import – conducted in the previous step – is ready, it is time to carry out the Notes database user activity scan. In this scan, the excluded Domino servers and Admin user names are skipped, thus not reported and not used in the analysis phase of the scan.

You want to know the real end users activity. Domino server and Administrators user activity is irrelevant

This manual will assist you to carry out the scan multiple times until only real end users remain. The separation of real end users from servers or system users is crucial for determining if a Notes database is still (actively) used in your organization.

As mentioned before, the user activity scan should preferably be performed by the scheduled server agent.

You may carry out the user activity scan on all imported Notes databases or on a selection of Notes databases.

If you scan a selection of Notes databases, select “Analyze User activity” in the pop-up screen. When you perform a scan on all server databases, this option is selected by default.

 

A good approach is to start with 1 Domino server holding the majority of the Notes databases, so that we can extend the list “Users and servers to exclude from the user activity scan” in an effective and efficient way.

Besides collecting all Notes database user activity details per year, NDDM will also collect all user activity details per user. The ‘User Activity after Date’ box can be used to list all active users and their activity after the specified date. By default, the date is set one year back into the past. You can adjust this date if you like to.

NDDM will also collect the top x users in this time period. These 5 (for example) most active users can be used to communicate with, using the NDDM email tool (explained later on in this guide). You may change the top x number users if you like.

When the date and/or top x users fields are blank (or 0), these settings are omitted during the scan. All activity is then included in the activity per user and the top x active users.

 

Press the button “Initiate Task(s)”. The scheduled scan tasks can be seen in this view:

When the scan is completed open view ‘Database Info’:

Scroll to the right and sort the column ‘Last Read By’.  Zip through the content and identify the first non real end user name (for example a Domino server, Admin, agent signer or Developer name).

Simply open the Notes database scan document and press this green marked link. A pop-up will appear where you can select the non end user name(s) to add to the exclude list.

The user name(s) will be added to the exclude list of NDDM.