NDDM Technical Manual – Scanning

The NDDM Notes views are set to NOT refresh automatically. This has been done to speed up the NDDM scan process. Please press F9 in most of the NDDM views when you are working in NDDM!

At this stage all selected Domino servers have a replica of the Lialis NDDM Notes database. You do not need to open all NDDM replicas. NDDM has been designed in such a way that you can initiate all scan work in 1 NDDM Notes database on your main central HUB server. With task documents, each Domino server is instructed to carry out its own scan work. You only need to wait for the results to be replicated back to the central NDDM Notes database.

You might want to check the Domino server agent manager task settings in the Domino server documents. Scanning may take a few hours or days depending on the size. It is recommended to increase the agent manager run time to 12 hours for example. If the scan takes more time, then there is no problem: NDDM will continue the scan at the point where it was interrupted in the previous scan.

You also might want to create new Domino server replication connections documents between the HUB server and all the Domino servers to be scanned, where only the NDDM Notes database is replicated (pull push) in the shortest time interval of 5 minutes for example. This will speed up the work considerably.

Follow the order of steps as described below carefully.

Scanning – Import Notes databases. In this step, the Domino server adds a Database Info document for each database found. To make sure that no .nsf file is omitted, only information that can be retrieved regardless of database access is recorded (replica id, server, filepath and -name). This task will only take a few seconds to complete.

Select one, multiple or all Domino servers. Use the Key to filter away Domino servers.

The scheduled tasks can be seen in this view:

Now wait until the Domino servers have completed the import task. Each database on the server now has a Database Info document in NDDM.

A Domino server will run this Notes database import task in a few seconds. The only delay is waiting on the agent manager and the scheduled replication.

Note: if there are multiple new/in progress task documents for the same server, the scan agent will always pick the newest Task document to run on, which is the first displayed in this (unsorted) view. So make sure that you only create a second task document AFTER the server completed its first task.

Ideally, use the ‘Scheduled On Domino’ option for all scan tasks to perform. Domino servers agents process data much faster than client-based agents. However, in certain cases you might want to run a task manually on the client. You can do so by using the ‘Manual On Client’ option in the navigator .

You can always re-run this step. The agent will simply add new Notes databases when they are found.

If you delete all NDDM content you can run this task also to start from scratch.

Make sure you improve the NDDM exclusion list and delete newly excluded files from NDDM simply by deleting the Notes database records in NDDM.

If Notes databases are deleted from the Domino server, the import task will inform you about this with help of this column in the Database Info view. A red cross will be displayed when a Notes database is no longer on the Domino server. You may then delete these records from NDDM.

When the Notes database import – conducted in the previous step – is ready, it is time to carry out the Notes database user activity scan. In this scan, the excluded Domino servers and Admin user names are skipped, thus not reported and not used in the analysis phase of the scan.

You want to know the real end users activity. Domino server and Administrators user activity is irrelevant

This manual will assist you to carry out the scan multiple times until only real end users remain. The separation of real end users from servers or system users is crucial for determining if a Notes database is still (actively) used in your organization.

As mentioned before, the user activity scan should preferably be performed by the scheduled server agent.

You may carry out the user activity scan on all imported Notes databases or on a selection of Notes databases.

If you scan a selection of Notes databases, select “Analyze User activity” in the pop-up screen. When you perform a scan on all server databases, this option is selected by default.

A good approach is to start with 1 Domino server holding the majority of the Notes databases, so that we can extend the list “Users and servers to exclude from the user activity scan” in an effective and efficient way.

Besides collecting all Notes database user activity details per year, NDDM will also collect all user activity details per user. The ‘User Activity after Date’ box can be used to list all active users and their activity after the specified date. By default, the date is set one year back into the past. You can adjust this date if you like to.

NDDM will also collect the top x users in this time period. These 5 (for example) most active users can be used to communicate with, using the NDDM email tool (explained later on in this guide). You may change the top x number users if you like.

When the date and/or top x users fields are blank (or 0), these settings are omitted during the scan. All activity is then included in the activity per user and the top x active users.

Press the button “Initiate Task(s)”. The scheduled scan tasks can be seen in this view:

When the scan is completed open view ‘Database Info’:

Scroll to the right and sort the column ‘Last Read By’.  Zip through the content and identify the first non real end user name (for example a Domino server, Admin, agent signer or Developer name).

Simply open the Notes database scan document and press this green marked link. A pop-up will appear where you can select the non end user name(s) to add to the exclude list.

The user name(s) will be added to the exclude list of NDDM.

Carry out this exclude task for all non end user names found in the user activity scan in the column ‘Last Read By’.

Do the same with the column ‘Last Write By’. Go through the content and identify Domino server names, Admin names, agent signer names and Developer names in order to find all names that are not the real end users.

At this stage you have excluded all non end users for the first Domino server. Now you must do a new server-based user activity scan for the same Domino server. There is a big chance that new non end users will pop up in the columns ‘Last Read By’ and ‘Last Write By’. Exclude these names now and redo the user activity scan a few times until the 2 columns ‘Last Read/Write By’ only report real end users.

When the first Domino server only reports end users and all non end users are excluded, you can start scanning the other Domino servers for user activity. It is likely that you will find a few more non end users which you must exclude. When you exclude more names, you should do a re-scan of the user activity on the Domino servers where non end users where found. After a few new user activity scan operations, only real end users should be listed in the columns ‘Last Read/Write By’. Once again:

You want to know the real end users activity. Domino server and Administrators user activity is irrelevant

One remark on the user activity scan: “na” (not available) can indicate that – although user activity recording is enabled – there are no user activity entries, or that all user activity is designated to users that are excluded. In the latter case, it can be concluded that the Notes database is not used by real end users.

These are the possible scenarios regarding User Activity scanning:

Scanning Notes database user activity with the Notes client and excluding all non end users will take up to one day.

It is crucial the user activity scan does not show non end user names. You may improve the results later in the scan process

Normally the Notes database content scan is done after the user activity scan. The Notes database content is executed by the Domino servers in parallel, because this task will process a lot of Notes data.

Instruct the Domino servers to carry out the scan and sit back for the results to appear in the central NDDM Notes database.

You want to discover the dates when the Notes database content was last modified

You may select one, multiple or all Domino servers.

Do not enable Deep Scan Doclinks yet. We will do this later on.

The option “Analyse Content with restricted access” will identify the number of Notes documents that have readers fields with no ACL roles listed in them. These documents cannot be read and will not be migrated to SharePoint in its current state.

The option “Scan Readers and Authors” will give a detailed report on how these fields are used in the Notes content.

Wait until the view “Scheduled Scan Tasks\All” shows that the content scan is completed on all servers.

A Domino server will run a Notes database content scan task in a few hours or a few days, depending on the database size and number of documents.  All Domino servers will run this task in parallel, not sequentially. You must wait for the scheduled replication to retrieve data from all parallel scans.

A sample Notes database content scan result:

An important remark on the Last Doc Created/Modified fields: the Users/Servers exclude list is NOT applied here, because it is important to know if it was a Domino server, a signer ID, an Administrator or a real person who did last create or modify the content. The exclude list is only used in the user activity scan.

The Last Doc Created/Modified (By) data is very important to determine what to do with the Notes database in the step ‘target analysis’, which is explained further below.

At a later moment you may re-scan all or selected Notes databases to update the scan details, using the options in the left menu.

For the few Notes databases that encounter problems during the Domino server-based scan, you can apply a manual scan or redo a scheduled scan. You can find these Notes databases by – for instance – checking the ‘# Forms Used’ column:

Before you manually scan these Notes databases, make sure to enable Full Access Administration in the Domino Admin client.

Now select these Notes databases and click on “Manual Analysis On Selected Databases”

The Domino server-based scanning agent runs under a same Notes ID that was used to sign the NDDM. Its possible that this ID does not have access to the complete content of some Notes databases. This sometimes happens when – for instance – an ACL Role is used in readers fields on the Notes document.

When the agent was unable to scan all content, the Notes database is highlighted in red in the view. For example:

The form shows the number of documents that could not be scanned, because they were read-protected:

To fix this, make sure the scanning signer ID has all roles in the Notes database ACL enabled. If this does not fix the issue, the problem may be caused by something else. A Domino administrator needs to step in to implement a fix resulting in a complete scan of this particular Notes database. However, it is already clear that this database is not going to be archived or replaced (see chapter ‘Target Analysis’ below), a complete scan for this database is not needed.

Under ‘Access\Read Protected’, all Notes databases are listed that contain 1 or more protected Notes documents.

To get round the protected Notes documents issue, run a manual content scan of these databases with the Full Access Administration option in the Notes admin client enabled. By doing this, you can access most protected documents, resulting in a much better scan result. The image below displays the steps to be taken with regards to scanning read-protected documents.

NDDM has 2 views to report Notes databases where the user activity or content scan failed or did not complete.

Make sure these 2 views are empty, then all Notes databases are scanned properly.

The scan log is displayed on the ‘Scan Log’ tab of the Database Info document.

If an error or a situation of special attention occurs during a scan, the issue is listed at the bottom of this log:

For the User Activity scan, additional warnings and info are logged as well:

• Warning: User Activity recording not enabled for this database
• Info: User Activity recording is enabled for this database, but no activity was found

In the Errors views we have created multiple views where the scan errors are grouped.

In case of scheduled tasks, additional logging is displayed on the task document itself. This logging will also list errors that can not directly be traced back to a specific database.

The Scheduled Scan Tasks\All view displays the number of errors found during the scheduled scan. These do not include the Warning and Info messages mentioned above.