Lotus Notes Application Design Analysis with AI

AI is booming these days, and I wanted to research if it’s possible to set up a smart AI system that will provide a decent answer to the following question:

What are the complex aspects of the Notes application “abcde” and what’s the reason?

This is a very complex question because of the “why” part. It would take a Notes developer a lot of time to answer it (assuming the Notes app contains loads of code).

Lotus Notes Design Export

All the AI work is happening in the cloud. At the moment, it’s not possible to connect an on-premises IBM Domino server to any cloud automation platform and have it check all Notes application design elements.

The AI system we are going to create needs Lotus Notes database design input. We have extended our Domino application scanner (NDDM) with a function to export all the HCL / IBM Notes application design elements as text. We can only use text as input for the AI system, by the way. Please check some Notes app design exports in the images below.

Each Notes application design element is exported to a single Notes document. In this example, the CSW DEV HCL Notes application has 3 agents, 8 forms, and so on. NDDM creates a highly detailed export of all the Lotus Notes functions on a Notes form. For example, it exports the Notes fields on the form, types of fields, all buttons, and all formulas and LotusScript code used in various places on the form. At the time of writing this blog, I must admit that the design exports we create can be improved, but for now, they are sufficient.

I have used make.com to create automation scenarios that build the AI system to process the IBM Notes application design data (text) in order to answer my questions.

Workflow Scenario 1: From Notes Design Data to Pinecone

The goal of this scenario is to take all Notes application design data (forms, views, agents, script libraries, and so on) from NDDM and store it in a cloud database for further processing in scenarios 2 and 3. Pinecone will be explained later on.

I use make.com because it’s easy to use, and I can’t program. The same result can be achieved with Python or other similar platforms.

The complete make.com scenario is displayed in the image below.

I will explain the important parts of this flow in the following sections.

SharePoint

The Notes design data stored in NDDM is uploaded to a SharePoint list. We do this just to get the Notes design data into the cloud. There’s nothing fancy going on in the SharePoint part, just a simple Notes field to SharePoint field mapping.

Text Parser

This is used to convert HTML to text.

OpenAI Completion Prompt

Here it becomes interesting. We use an OpenAI prompt to build a vector (explained below). The following prompt is sent to OpenAI:

We create a prompt to be sent to OpenAI. This prompt contains the task we want OpenAI to perform (e.g., “I am building a vector database…”), some Notes database metadata (Notes database title, file name, etc.), and the raw Notes design code (form NDDM). All Notes database information is taken from the SharePoint list.

Small Text Alterations

These tool modules replace newlines with a space, replace ” with a space, and replace the \ with a space. This operation is needed to prepare the input for the next important step: creating vectors.

OpenAI Get Vectors

Please refer to this video about vectors >> https://www.youtube.com/watch?v=ZGa8cQDel80

We ask OpenAI to create a vector based on the output of the OpenAI completion prompt.

Pinecone

Each vector generated by OpenAI’s “get vectors” is stored in the Pinecone vector database, along with Notes app metadata and the raw (original) Notes design code generated by NDDM.

An example of a Pinecone vector is displayed below:

It has an ID, all 1536 vectors are stored under values, and all the Notes app metadata is attached to the vector.

Why Vectors?

Currently, the OpenAI GPT-4 model has a limit of 120,000 tokens per prompt (approximately 200-300 pages of text). The design of the CSW dev Notes application contains more code than can fit in one prompt, which is why we need to split the prompts into multiple parts. Keeping the OpenAI prompts smaller increases the accuracy of the results. And I want to store many Notes application designs in a database.

The reason we use a vector database like Pinecone will become clear when we look at the following make.com scenarios.

Workflow Scenario 2: Finding Vectors Close to My Query

The goal of this scenario is to find a certain number of vectors in the Pinecone database that contain data close to my query: “What are the complex aspects of the Notes app CSW DEV and what’s the reason?”. The word close is crucial in this setup.

The complete make.com scenario is displayed in the image below.

Clear the data in some Google Sheets

OpenAI API Call

This operation will convert my query into a vector.

Pinecone Query Vectors

This operation will find (10) vectors in Pinecone that are close to my query (which we also converted to a vector). The number 10 can be freely chosen.

OpenAI Create a Completion Prompt

Now, it becomes complicated. Let’s break it down step by step:

1. Our query about the Notes database is converted into a vector.
2. Pinecone is asked to return 10 vectors that are close to the vector from step 1. Pinecone also returns the raw Notes source code (form, agent, script library, and so on) we stored as metadata in the Pinecone database.
3. We now have a collection of 10 Notes design elements with metadata and the raw Notes design code export from NDDM created earlier.
4. We send the raw Notes design code to an OpenAI prompt and ask OpenAI to tell us how important the raw design code is (Yes 84% or No 30%) in relation to the query and the context.

 

Collect the Data in Google Sheets

The first sheet gives us the Yes and No outputs.

In the second sheet, we only store the Yes values, which we will use for Prompt 3.

Workflow Scenario 3: Getting the query answered.

The make.com scenario used here is displayed in the image below.

Collect the Yes Results and Sort Them

This part will collect all the results of Notes design code that matter (valued with Yes) and sort them by the highest result percentage first.

The text parser will remove HTML and return only the text.

OpenAI Create a Completion

This is the final operation, and OpenAI will answer the initial query we had:

What are the complex aspects of the Notes app abcde and what is the reason?

The prompt we send to OpenAI is explained below:

The Final Answer

OpenAI will return one answer to the query based on all the given input (10 vector results with Notes design (meta) data).

The “CSW DEV” Notes application contains intricate and complex functionalities for several reasons. Below is an analysis of the identified complex aspects and the underlying reasons contributing to that complexity:

—

### *1. Data Import and Transformation Workflow*
– **Description**: The application imports data from other Notes databases and performs numerous transformations to present the data in a “highly convenient” format for the users.
– **Complexity Factors**:
– **Dependency on External Databases**: The application relies on other Notes databases for its data, inherently introducing challenges in maintaining data consistency, synchronization, and error handling.
– **Transformation Logic**: The data is “altered in many ways,” which involves complex processing logic such as determining relationships, reformatting, and optimizing for user presentation.
– **Risk of Data Integrity Issues**: If any steps in the import or transformation process fail (e.g., mismatched field mappings or unhandled edge cases), it might compromise the data integrity.
– **Example Code Reference**:
– The BaseData class in *slBackground* initializes source and destination documents, processes relationships, and calculates document types (e.g., initializeDestinationRKDocument and calculateType methods).
– The use of nested conditions, multiple document sources, and complex data relationships adds to the inherent complexity.

—

### *2. Multi-Document Linked Navigation*
– **Description**: The application implements a navigable flow between different types of documents such as Service Requests, Activities, Knowledge documents, and Steps. This involves tracking parent-child relationships and detours in workflows.
– **Complexity Factors**:
– **Handling Relationships**: Relationships among documents (e.g., parent-child, detours, and next steps) require retrieving and processing multiple linked documents dynamically at runtime.
– **Environment Variable Management**: Navigation is managed using environment variables (e.g., ENV_CSWID, ENV_COUNT). This creates additional complexity as it must ensure proper state tracking and cleanup.
– **Dynamic Navigation Flow**: Functions like processPreviousAndNextLinks, findDetour, and AC_ClickAction handle creating and processing navigation links. This involves querying document views, matching flow conditions, and dynamically determining the next or previous document.
– **Example Code Reference**:
– Functions such as processPreviousAndNextLinks and findDetour in *slForeground* demonstrate handling of navigation flows.
– Code like ST_ClickNextActivityFirstStep and AC_ClickAction facilitate transitions and must handle exceptions like missing documents (MsgBox "Document not found").

—

### *3. Support for Multiple Document Types*
– **Description**: The application supports various forms (ServiceRequest, Step, Activity, Knowledge, etc.), each having distinct processing logic.
– **Complexity Factors**:
– **Document-Specific Logic**: The application needs to account for the differences between form types (e.g., Service Request vs. Activity) for workflows, navigation, and transformation processes. This differentiation is evident in logic like calculateType (which maps document types to constant identifiers like TYPE_SR, TYPE_ST, etc.).
– **Shared Dependencies**: The same shared resources (e.g., views or environment variables) may be used by multiple document types, requiring robust handling to avoid conflicts.
– **Example Code Reference**:
– The DestinationForm property in *slBackground* maps document types to specific forms (FORM_SUMMARY, FORM_ACTIVITY, etc.).
– getFlow dynamically determines the type of flow based on the document type (TYPE_SR, TYPE_RK, etc.).

—

### *4. Error Handling and Logging*
– **Description**: Custom errors and logging mechanisms are implemented to handle issues during operations like document initialization, navigation, and workflows.
– **Complexity Factors**:
– **Custom Error Management**: Error handling logic (e.g., On Error statements in multiple methods like initializeSourceDocument) contributes to code overhead because it must account for all possible exceptions.
– **Logging Dependencies**: Logging is centralized in the slLogging script library, with frequent calls to log events or issues (WriteLog method).
– **Example Code Reference**:
– Error-handling blocks in methods like getDocument, initializeDestinationRKDocument, and findDetour ensure partial failure doesn’t crash the whole process.
– Logging is an integral part of the workflow, e.g., logging invalid lines in SourceDocUNID.

—

### *5. Advanced Data Manipulation*
– **Description**: Complex algorithms are used for manipulating data before presenting it back to the user or routing it through workflows.
– **Complexity Factors**:
– **View-Based Data Queries**: Extensive reliance on Notes views such as DocsByKey, ServiceRequests, UnprocessedActivities, and others, makes maintaining data accuracy challenging.
– **Field-Based Logic**: Many fields (e.g., FIELD_FLOW, FIELD_UNPROCESSED, etc.) are dynamically modified based on certain trigger conditions, which can introduce bugs if not carefully managed.
– **Example Code Reference**:
– The testList function in *slForeground* processes list structures to match keys, requiring nested loops and array handling.

—

### *6. Cross-Platform Compatibility*
– **Description**: The application includes platform-specific code for Windows, Linux, and other environments.
– **Complexity Factors**:
– **Platform Constants**: Constants like PLATFORM_W32 and PLATFORM_TUX support environment-specific features (e.g., external DLL calls for Windows vs. SO libraries for Linux).
– *External API Calls**: Functions like NSFDbOpen, NSFDbClose, and NSFNoteDelete in *slBackground** involve external interactions with the Notes database API.
– **Example Code Reference**:
– Cross-platform logic is evident in the Declare Function statements that differentiate between Windows (nnotes.dll) and Linux (libnotes.so) for database operations.

—

### *Root Causes of Complexity*
The primary reasons for the complexity in “CSW DEV” stem from:
1. **Feature Richness**:
– A wide array of workflows, from data import to multi-document navigation, adds functional depth but makes development and maintenance challenging.
2. **Dynamic Nature of the Application**:
– The application’s dependency on runtime document state, environment variables, and view-based handling introduces additional unpredictability.
3. **Tightly Coupled Components**:
– Shared views, libraries, and navigation links increase interdependencies within the codebase, reducing modularity.
4. **Backward Compatibility Requirements**:
– As a “backup” for CSW, the application must handle many edge cases and failures gracefully, which adds substantial overhead.

—

Let me know if any specific aspect requires deeper elaboration or analysis!

Concluding

I was impressed with the output. Even the Notes developer of the CSW DEV app was stunned by the result because he knows the code he built and could easily compare the OpenAI output with his knowledge of the Lotus Notes application.

At the moment, we have 3 Make.com scenarios. We can combine these 3 into one scenario, and it would be nice for the user if we added a lovely ChatGPT-like prompt.

We will improve the NDDM design output, and the Make.com scenarios can be improved here and there.

This setup can be used to ask questions about hundreds of Notes applications at once. We can even decide to add Notes logs and configuration documents to the Pinecone vectors to improve the outcome.

I am sure this vector approach has many more use cases. It would be possible to add all Notes database document content into vectors and ask questions like:

Does Notes application XYZ store personal information? Please provide 10 Notes document ID examples.

Please reach out to me if you see any use cases for your business.