NDDM Technical Manual
IBM Notes Domino Application Scan and Analysis tool.
Optimized for Domino Server decommission projects.
Lialis will always support your Domino Server Application Scan in short daily remote calls to ensure a perfect and swift scan.
Domino server performance impact
The NDDM Notes database scan is done in 2 different main parts.
Notes client based scan of the Notes database user activity. This scan is carried out in the Admin Notes client background. Scanning one Notes database takes around 1 second. The impact on the Domino server is zero. So for example scanning 100.000 Notes databases will take a day (depended on network speed).
Domino Server based Notes database content scan. This scan is carried out by the Domino server on its own local Notes databases by a NDDM Lotus Script agent. See image below:
The impact on the Domino server performance is average; not high and not low. The agent will scan each Notes document in all Notes databases on the Domino server running the scan. The scans are very fast because the Domino servers scans their local Notes files. However, a single Domino server scan can take up to a few days depending on the amount of Notes files on the Domino server.
NDDM has been designed in such manner that multiple Domino servers will run its own local scans in a parallel operation resulting in enormous scan speeds. So scanning 1 Domino server will take the same amount of time (a day or 2) as scanning 10 or even 100 Domino servers.
The Domino server agent manager must allow long agent run times of at least 6 hours (12 hours is even better). If the Domino agent manager cuts off the NDDM scan agent there is no problem. The NDDM scan agent will pick up the Notes database scan in the new run at the exact moment it was interrupted in the previous run.
Clients can adjust the NDDM scan agent simply by adjusting the Agent schedule. For example to set the NDDM agent to only run at night time.
With NDDM we have scanned many Notes databases, from small to extremely large Notes databases. Below example database holds 115.087 Notes documents and the NSF file size is 51 GB. The Notes content scan of this particular NSF file took 30 minutes.
01/06/2021 05:57:09 PM GMT – Start Database Analysis
01/06/2021 05:57:09 PM GMT – Start scanning Design Elements
01/06/2021 05:59:25 PM GMT – Finished scanning Design Elements
01/06/2021 05:59:26 PM GMT – Start scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 06:27:51 PM GMT – Finished Database Analysis
Another example is a small Notes database of 400 MB holding 876.430 Notes documents. The scan took 15 minutes.
01/06/2021 01:23:47 PM GMT – Start Database Analysis
01/06/2021 01:23:47 PM GMT – Start scanning Design Elements
01/06/2021 01:24:46 PM GMT – Finished scanning Design Elements
01/06/2021 01:24:46 PM GMT – Start scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished scanning content (incl. scan links)
01/06/2021 01:38:26 PM GMT – Finished Database Analysis
The scan speed depends mostly on the amount of Notes documents and to a lesser extend on the Notes database file size.
The total Notes database content scan speed also depends heavily on the Notes database setup and the OS server performance. We have managed to scan 280 GB in one hour. In this one hour 4.823.000 Notes documents where scanned by one particular Domino server. If 100 Domino servers carry out the scan the total scanned GBs and Notes documents can by multiplied by 100.
Setup and Configuration
When you read this technical manual we assume you have received the NDDM Notes database file from Lialis. Its not possible to download an evaluation version.
Open the NDDM Notes database with your Notes client and make a Notes database copy (with your Notes client) to the main IBM Domino (HUB) server that replicates with all other Domino servers. Make sure you copy documents. Do not copy the ACL.
Open the new NDDM Notes database copy on the first Domino server and continue.
ACL – On NDDM on the main Domino server first adjust the ACL. Make sure all Domino servers have manager access with delete permissions. Also add all administrators and viewers if applicable. You might want to add people to view the content with read permissions.
Signing – NDDM will run scheduled agents on all Domino servers to be scanned. This means you have to sign NDDM with a Notes ID that is allowed to run agents on the Domino server. This ID must also have read access to all Notes databases. In most environments the ID of the Notes administrator has these privileges. You can also use a certain Domino server ID file to sign NDDM. But no worries, NDDM will report if the ID does not have enough permissions and it has means to adjust this later on in the scanning phase.
License – add the license key which you have received from Lialis.
Configuration – The configuration form has 3 parts which will be described in the following sections.
Go to the view ‘Configuration & Roll-out’ and press the button ‘New’ to create a new configuration.
Directories and databases to exclude from the scan – First exclude all Domino system files and folders from the scan. You may also want to exclude the mail folders. We have already preselected all files we know. It is up to you to add folders and files to the exclude list. We use this list because its a waste of resources to scan large Domino system databases like the log.nsf file.
Users and servers to exclude from the user activity scan – In the new configuration form you may add users and servers to exclude from the Notes database user activity scan. This step is important because it will result in a much more accurate user activity scan by filtering away all Domino servers, signer ID and names of administrators and developers. Its simply not so interesting to know what this group of users does with the Notes databases. So please add as many names now. Later in this manual you will be guided how to improve this list.
Please try to add as many known Domino servers and ‘unreal’ users to the exclude list. See the sample below:
Configuration of the Roll-Out – The roll-out part of the NDDM configuration will allow you to quickly create replicas of the NDDM Notes database to all (or a selection of) Domino servers in the Domino domain that you are planning to scan with NDDM. If you have multiple domains you can create manual Notes replicas, or, you can add the Domino server names to the list in this configuration. Your Notes client will create the new replicas. This may take a short while depending on the amount of Domino servers you are actually going to scan.
You may also use the Domino Adminp process to create NDDM replaca’s.
Now press the ‘Rescan servers’ button to find all running Domino servers.
It will show that NDDM is placed on the first Domino server and you will be able to create new replicas to all Domino servers which are running.
You have to ensure that all Domino servers you are going to scan are setup with a so-called ‘pull push replication’ to the main or HUB Domino server. Normally this is the case in Domino environments. Please check this anyway just to be sure.
If you have Domino cluster servers you only need to install NDDM on the main Domino cluster server.
At this stage all selected Domino servers have a replica of the Lialis NDDM Notes database. You do not need to open all NDDM replicas. NDDM has been designed in such a way that you can initiate all scan work in 1 NDDM Notes database on your main central HUB server. This will instruct the Domino server to carry out the scan work. You only need to wait for the results to be replicated back to the central NDDM Notes database.
You might want to check the Domino server agent manager task settings in the Domino server documents. Scanning might take a few hours or days depending on the size. Its advisable to increase the agent manager run time to 12 hours for example. If the scan takes more time, then no problem, NDDM will continue the scan where it was interrupted by stopping the task by the Domino server.
You also might want to create new Domino server replication connections documents between the HUB server and all the Domino servers to be scanned where only the NDDM Notes database is replicated (pull push) in the shortest time interval of 5 minutes for example. This will speed up the work considerably.
Please follow below order of steps carefully.
Scanning – Import Notes databases. You can now instruct all Domino servers to import all Notes databases. In this step we only add the NSF file names. You may now instruct all Domino servers to carry out this task.
You may select one, multiple or all Domino servers.
In the background a Notes task document will be replicated to all NDDM replica databases. Each Domino server NDDM agent will see this task document and the Domino server agent manager will carry out the work accordingly. Notes documents will be created in the remote NDDM replica which will then replicate back to the central NDDM Notes databases where you will be looking at the results.
The scheduled tasks can be seen in this view:
Please wait till all Domino servers have carried out the import and all tasks are completed before you continue.
A Domino server will run this Notes database import task in a few seconds. The only delay is waiting on the agent manager and the scheduled replication.
All scan tasks should be carried out by the Domino servers preferably because this will go very fast.
You may also run them manually just in case by using the left menu options; for example.
You may always rerun this step. The agent will simply add new Notes databases when they are found.
If you delete all NDDM content you can run this task also to start from scratch.
Scanning – User activity. When the Notes database import conducted in the previous step is ready its time to carry out the user activity scan. In this scan the previously excluded Domino servers and Admin user names are excluded from this scan. When the user activity scan is carried out there is a very big chance you will find more Domino servers from the past and also Admins who left the company. This manual will assist you to carry out the scan multiple times until only real end users remain. This is crucial because its not so interesting that a Domino server or a Notes developer is using the Notes database for your migration project.
You want to know the real end users activity. Domino server and Administrators user activity is irrelevant
The user activity scan is carried out by the Notes client of the Domino administrator. The scan runs in the Notes client background.
Please make sure you enable Full Access Administration in the Domino Admin client before you carry out manual scans. This will give better scan results.
In the view “Database Info” you will find all imported Notes databases
The best approach is to start to select all Notes databases on the Domino Hub server for example instead of selecting all Notes databases.
Press the menu “Manual On Client” to display the task screen.
Select the first two scan options. And all excluded names are listed as well.
When the Notes client scan is completed open this view please:
Scroll to the right and sort the column ‘Last Read By’. Zip through the content and identify Domino server names, Admin names, agent signer names and Developer names in order to find all names that are not the real end users. Do the same with the column ‘Last Write By’.
Now add the newly found non end user names to the configuration exclude list. Simply open the Notes database scan document and press this green marked link. A pop-up will appear where you can select the names to add to the exclude list by making them color blue.
The user name will be added to the exclude list.
Carry out this exclude task for all non end users names found in the user activity scan.
You may always rescan all or selected Notes databases on the first Domino server in the same manner.
When the first Domino server user activity scan does not report non end user names you are set to scan the other Notes databases in NDDM. There is a big chance more non end user names will surface. Simply exclude these and rescan the user activity. Continue this process until the Database info view in the columns ‘Last Read By’ and ‘Last Write By’ only show real end users or na (‘not applicable’).
A few important remarks on the user activity scan:
- “na” means that the Notes database holds user activity details. But no real end users are left after filtering out the excluded names. Hence it can be concluded that the Notes database is not used by end users. Only by the excluded servers and admins for example.
- “User Activity not recorded” means; the Notes database does not record your activity.
Scanning Notes database user activity with the Notes client and exclude all non end users will take a day max.
A Notes database user activity scan result example:
It is crucial the user activity scan does not show non end user names. You may improve the results later in the scan proces
Scanning – Notes database content.
Normally the Notes database content scan is done after the user activity scan. This is because the user activity scan phase is carried out multiple times until only the real end users are left in the report.
The Notes database content is executed by the Domino servers in parallel because this task will process a lot of Notes data.
Instruct the Domino servers to carry out the scan and sit back for the results to appear in the central NDDM Notes database.
You want to discover the dates when the Notes database content was last modified
You may select one, multiple or all Domino servers.
Do not enable Deep Scan Doclinks yet. We will do this later on.
Please wait until the view “Scheduled Scan Tasks” shows that all servers are ready with the content scan.
A few important remarks on the Notes database content scan:
- The exclude list is not used in the content scan because it is very interesting who has last modified the content; a Domino server, a signer ID or a real person. The exclude list is only used in the user activity scan.
A Domino server will run this Notes database content scan task in a few hours or a few days depending on the size. You must wait for the scheduled replication. The Domino servers will run this task in parallel, not sequentially.
A sample Notes database content scan result:
Below part of the scan is very important to determine what to do with the Notes database in the step ‘target analyses’ which is explained further below.
At a later moment you may rescan all – or selected Notes databases to update the scan details using the options in the left menu. No worries, you cannot damage the tool or scan results in any way by doing this.
Manual scans can be used at the end of the scan process if there are a few Notes databases that have scanning problems during the Domino server based scan. This way you will find ‘not scanned Notes databases’:
Please make sure you enable Full Access Administration in the Domino Admin client before you carry out manual scans.
Now select all Notes databases where the forms used shows empty and carry out a Notes client content scan.
Scanning – Notes database content scanning issues.
The Domino server will run the scanning agent under a certain Notes ID used to sign NDDM. Its possible that this ID does not have access to the complete Notes database content. Reasons can be that an ACL Role is used in reader fields on the Notes document.
When not all content can be scanned the Notes database is highlighted in red in the view. For example:
The form shows the number of documents that could not be scanned (marked with a green square in the image below).
The fix could be simple. Make sure the scanning signer ID has all the roles in the Notes database ACL. Or the cause might be different and it needs a Domino Administrator to step in. He/She can figure out what;s wrong and implement a fix resulting in a complete scan of this particular Notes database. If there is no doubt about the scan target then a complete scan for this particular Notes database is not needed.
Below Notes view will display all Notes databases where there are protected Notes documents found in the scan.
A good way to fix these is to carry out a manual content scan of these Notes databases by enabling the Full Access Administration option in the Notes admin client. This way you will have access to many protected documents and the scan will be better for these Notes databases. Please check below images for the steps to take to rescan read protected documents.
Scanning – Logging of scan errors
On the bottom of the NDDM Notes document form there is a scan log listed where scan errors, if any, are listed.
Now the scan is ready. You have collected a huge amount of details on all Notes databases which now have to be analyzed.
In this phase NDDM will try to classify all Notes databases into one of these 3 main categories:
- Replace – Notes database is actively in use
- Archive – Notes database is only read
- Delete – Notes database is not used at all
Lialis NDDM has a set of Target calculation rules that will assist you to calculate the Targets automatically.
For each Notes database the content scan and the user activity scan are reviewed by above listed target definitions. NDDM will calculate in which group (Replace, Archive or Delete) the particular Notes database fits.
The Targets setup we have supplied in your NDDM version may differ from the examples we give in this guide. The Targets setup we supply are based on the latest knowledge we have gained in our projects.
Lialis will always guide clients in the target calculation phase because of its complexity and importance
First you have to decide the target cut-off date; for example 1 or 2 years in the past.
In below Diagram we have simplified the cut-off date approach we use in NDDM.
This way you can adjust the cut-off date easily on all targets
The problem with Notes databases is that there are always databases that do not fit nicely in one of the 3 categories (Replace, Archive or Delete). This is the main reason why there are more than 3 target calculation documents provided in NDDM by Lialis.
Lets look at the Replace target definition document in more detail.
Target number is used later on in the process when we worry about Notes database replicas on multiple Domino servers.
Target short can be Replace, Archive or Delete. You may rename these if you prefer.
Target description has a very detailed naming of this particular target.
Availability its possible to enable 3 options:
The blue marked criteria are applied to the Notes database content scan results.
The purple marked criteria are applied to the Notes database user activity scan results.
- No Documents – when the Notes database has 0 documents
- No Activity – when the user activity is “na” (means there are no real users found after applying the exclude list on the user activity)
- Not scanned – when the user activity is not enabled in the Notes database
In the following fields you can set if this target is applicable. For example if the last document creation date is after a certain date.
Last doc created, Condition: Created after Date: 01-01-2020
Last doc modified, Condition: Modified after, Date: 01-01-2020
Last read date, Condition: Read after, Date: 01-01-2020
Last write date, Condition: Written after, Date: 01-01-2020
Reads, Condition: Larger, Number: 10
Writes, Condition: Larger, Number: 10
All criteria must apply to the particular Notes database scan results before the target is applied.
To carry out the target calculation please select all documents in the view and press the action button. The calculation will be done in a few minutes.
With the targets supplied we try to achieve that all Notes databases are targeted,. However, this might not be always the case.
If a target cannot be calculated, below columns are empty in the main view Database Info (though this will not happen a lot).
On the Notes document you can follow the target calculation decisions made by the system. This will help you understand the target calculation and it will help you finetune the target calculation.
Lialis support is always included to assist you in the target calculation and fine-tuning
If a Notes database is replicated across multiple Domino servers NDDM will calculate which Notes database replica is used the most and it will assign this particular Notes database replica the HUB role. The calculation mechanism is not very difficult. In the initial target calculation each Notes database got a target number assigned. The HUB assignment calculation phase will make a collection of all Notes databases with the same replica ID. The Notes database with the highest target number will get the HUB role. The highest target number is equal to the highest usage by people. See below image as an example. The hub server icon indicates this particular Notes database is the HUB Notes db. If a Notes database is not replicating it will always get this HUB role.
When all Notes databases with the role HUB are migrated, or closed, or other wise dealt with, you can be sure that all Notes databases in the whole environment are migrated and the all Domino servers can be decommissioned.
NDDM supplies below 3 views to display only the Delete, archive or replace HUB databases
It is advisable to use these views to carry out batch ACL close operations or to send out emails with the NDDM survey tool.
A good way to study and review the targets is to export the NDDM content to Excel as illustrated below.
In Excel its easy to review the target calculations made by NDDM by having a close look at the columns that have the target definitions. Simply make the deletions in the Excel columns in the same manner as defined in the Target settings document and you will understand how NDDM does the target calculations for you.
NDDM offers a way to find certain Notes database design strings. In this example we search for the design strings “@SendTo” and “.Send” to find out which Notes databases are sending out emails with for example workflow links to Notes databases. Listing these Notes databases is important if you plan to do a Notes email migration first since the database workflow will likely no longer be functioning.
Knowing these details is only interesting for the replace hub Notes databases. Hence first go to the view Replace hubs.
Select all Notes databases in this view and carry out below scan instructions to find the design strings “@SendTo” and “.Send”.
The scan will be carried out in the Notes client background and will take a few minutes per Notes database.
An example of the found strings:
In order to decide which Notes databases are sending emails, with for example Notes document links, below information can be used to make the selections.
The Notes database holds memo forms with people in the SendTo field:
The memo form has been used recently:
The Notes database holds design elements capable to send out workflow emails.
Below view displays a selection of the Notes databases where the design strings are found. In this example databases in the target ‘Replace HUB’ and holding Notes memo documents where users are found in the SendTo or CopyTo fields.
NDDM offers a deep Notes document link scan. Normally this scan is only carried out for the Notes databases with the target Replace HUB.
Go to below displayed view, select all Notes databases and press the highlighted menu option.
The selected Notes databases will now be scanned on Notes document links by the Domino servers.
When the scan is completed:
A result of the Deep Notes document link scan is displayed in the image below
With NDDM its easy to export all content to Excel as illustrated below. Simply select all documents and press the Export button.
In Excel its easy to review the target calculations made by NDDM by having a close look at the columns used the target definitions.
The Excel export provides more information which is not displayed in the NDDM Notes views. For example a listing of the design forms the Notes database has. Sorting on the forms column will give a great insight in the Notes database that have a similar design. This is because if the design forms look similar, the Notes database designs can be grouped into one group, even if the template names are different.
NDDM supplies great Notes database ACL adjustment functions that can be applied bidirectionally to multiple Notes databases in one go. With this approach its easy to close the access for regular users for all Notes databases that have been targeted to 1400 Delete. If needed this can be undone with one mouseclick.
First please configure the ACL lists first.
The “Added ACL entries” function allows you to list the ACL entries to be added when the ACL is adjusted by NDDM. For example, always add the servers and admin groups just in case.
The “Unchanged ACL entries” is important because NDDM will not change these in the ACL when you are going to batch change Notes db ACL’s in NDDM. Please add the standard ACL Manager groups and server names and admins to this list. These entries will not be adjusted when a Notes database is closed to ensure proper replication between Domino servers and Admin access later on.
Change the access with NDDM on the main Notes database replica and let the replication do its work to the other replicas.
In below example the Admin is going to put the selected Notes databases with the target 1500 Delete to read only.
Its a good practice to set Notes databases with the target “1500 Delete” to read only and wait a few weeks for users to complain that they are not able to modify the content in Notes. Then you know that the Delete classification should be adjusted.
Below view gives details on ACL closure status.
The NDDM Notes form displays details on all ACL changes made with NDDM.
Changing back ACL changes made with NDDM is easy using the appropriate action menu:
Reverting access is logged as well.
When the targets are applied nicely its time to reach out to the recent users of the Notes database to get confirmation of the calculated target.
NDDM has a way to send out emails in batch based on predefined templates and specific data from your scan which can be used in your project.
To send a mailing click in the menu on ‘Mailings’.
We have supplied a mailing example:
In ‘Send to’ we have selected the option ‘Unique users after’. This means the email will be send to the following users:
The “Unique users after” always contains less users compared with the Unique users list. So its very handy to communicate with the recent users.
TIP: You might want to shorten the “Unique users after” date if the list of users is too long for certain Notes databases.
Above email can for example be sent to all Notes databases with the target Archive. Select the Notes databases you want to use for your mailing:
Then press this menu action to choose the email template.
Email documents are listed in a ‘Send Queue’ which can be accessed this way:
NDDM has an agent that needs some configuration if you are going to use the NDDM mailing features.
Please open NDDM in the Notes designer and open this agent please:
Please set the Domino server which will run this agent to send out the emails.
Please decide on the user who will be the email sender. Use this Notes ID to save the agent and set this user name here
NDDM is mostly used in environments where the email is already migrated to Outlook for example. So getting this NDDM mailing feature to work smoothly in hybrid email domains might become challenging. But no worries Lialis will assist you in this set-up.
Another function of NDDM is a ACL role scanner which reports the roles the person pressing the NDDM Scan ACL Roles button is missing for certain Notes databases.
This feature is handy when the Notes Admin needs to rescan certain Notes databases because the Domino server based scan was showing an incomplete scan due to Notes document readers limitations.
Go to the view “Database Roles”; select all documents in this view; or only select the Notes database you are interested in. Then press the button “Get Role List”
The agent will find all Roles a Notes database has and it will report the roles the Notes admin is missing.
When the agent is ready sort the column “Missing roles for Current User”
Select the Notes databases where the Admin is missing ACL roles and press the button Set Manager in the left navigator.
This action will add the Admin to the Notes database ACLs and will assign all roles to this user.
When you then rescan these Notes databases adjusted in previous step, the column “Missing roles for Current User” will show no roles. This means the Admin has all roles assigned.
The next step could be to rescan a certain Notes database where the Domino server scan resulted in a red marked Notes database. Scanning with the Notes client and with all roles assigned might result in a complete scan result.
In most projects we carry out for our clients, the NDDM generated Excel file is used as a project master file in which the status of each Notes database is tracked carefully by the project team. In some use cases its handy to import the Excel master file status into the NDDM field named ‘Target’. For this purpose we supply a way to import these targets into NDDM.
All you need is to create an Excel file with 4 columns. The Domino server name in Column A, the file path of the Notes database in Column B, the title in Column C and the target you wish to import in column D. For example:
Then import the Target values in NDDM as illustrated below:
The imported targets will then show up in the Status column in the NDDM view “Database Status”.
For the Notes databases with category ‘Replace’ it might be handy to quickly find out if there are ODBC connections designed for example.
NDDM allows you to import all Notes database design text into NDDM for full text searching.
Select the Notes databases in NDDM of your interest and press the button “Export Design to XML”.
This process is not very fast. So it is advised not to select too many Notes databases for the first batch.
Select a temporary location to store the exported XML files so they can be attached to NDDM.
You can read XML files with Notepad ++ for example:
You may follow the progress in the Notes client task bar.
Finaly, the design elements are attached to the NDDM Notes document at the bottom of the form
Please enable the NDDM full text index as illustrated below.
When searching for the word “Lotus” we get the following results.
The XML files where this word is found are highlighted.
This scan will report the users and their access level to the Notes database. If groups are found in the Notes database ACL the members of these groups will be collected from the Domino Directory. Initiate the scan select the Notes databases you want to scan and press the menu option “07. Scan ACL”
A result example is displayed in the image below:
Doing a proper Domino scan is challenging. Lialis will support the scan from begin to end to ensure a perfect result.
Please contact us here if you are interested in NDDM commercials or a demonstration.
Thank you for your interest.