Bring the same test to build and buy

Before I would choose an AI diligence tool, I'd ask where the documents travel and how a reviewer can challenge the result. The same questions belong in a custom-build proposal.

- Sasha Deneux

The Take: Your deployment choice belongs in the pilot

NIST's January 2020 Privacy Framework includes mapping data processing as part of understanding privacy risk. That is a useful starting point for a CRE team comparing an AI diligence product with a custom build.

Where does an uploaded lease go? Who processes its text? Which systems retain the original file, the extracted fields, or the prompt? What can the team delete, and what evidence confirms deletion? Ask those questions about the configuration you would actually use.

Owning part of the software doesn't settle them. A custom workflow can still call an external model, use an OCR service, or leave sensitive content in a log. A packaged product can offer controls worth evaluating. The architecture and the agreement need to support the answer.

I would put data handling beside accuracy and reviewer effort in the same pilot. A technically impressive extraction is a poor fit if the firm cannot use it with the documents it actually receives. Equally, an acceptable deployment arrangement doesn't establish that the model understood the rent clause.

The buying decision becomes clearer when both options must pass the same practical checks. Ask for the source reference, inspect a deliberately conflicting document, follow a correction through the export, and trace a file through processing and deletion.

Record what the demonstration established and what remains a contract question or an untested claim. That gives the team a basis for deciding which option fits its work and what must be resolved before live documents enter the process.

The Teardown: Run one evaluation for build and buy

Choose a permitted, redacted practice packet that represents the work your team wants to automate. Include an operating statement, a rent roll, a relevant lease or amendment, and a short market note. Define the accepted answers and known conflicts before either option sees the packet.

Draw the data path. List upload, OCR, model processing, storage, exports, logs, and deletion. Ask which provider handles each stage, what access is required, and what the contract says about retention and use. Record unanswered questions instead of accepting a broad label such as private or enterprise.

Test material facts and contradictions. Check the reporting period, occupied area, rent, concessions, and any assumptions the workflow introduces. Have it identify a conflict between documents and show the evidence. A result with no empty fields can still contain an incorrect value.

Test the reviewer experience. Open the source from a material result. Correct one field and rerun the output. Check whether the correction survives and whether affected calculations and narrative change consistently. Record the effort needed to reach an accepted result, including the work outside the vendor's interface.

Check the market evidence separately. For each market claim, record geography, property type, observation period, publication date, and whether it is an observation or forecast. A current article may discuss older data. A national office trend is not a rent assumption for the particular asset being underwritten.

Compare the full operating commitment. Include integration, review, model or vendor charges, maintenance, and the person responsible when an input format changes. Define what happens when the workflow cannot answer. A usable escalation path is part of the deliverable.

Our build-versus-buy diligence guide provides the comparison framework. Keep an evidence register behind the market section so each claim can be checked against its original scope and date.

Signal

Reference desk: sources for a reusable build-versus-buy review.

• January 2020: map the processing. NIST's Privacy Framework includes inventorying data processing and the parties involved. Ask each bidder to show the path from upload to deletion, including logs and exports. Keep unknown steps visible in the evaluation.
• February 2022: ask the supplier about maintenance. NIST's Secure Software Development Framework gives purchasers and developers a shared basis for discussing secure development. Ask who handles updates, vulnerabilities, and incident response for the proposed workflow, including its third-party components.
• January 2023: evaluate the intended use. NIST's AI Risk Management Framework emphasizes context and documented measures. Give both options the same permitted practice packet and acceptance criteria, then compare correction effort and unresolved questions alongside output quality.

From NextAutomation

We scope and build underwriting workflows around the documents and review standards a firm can actually use. Book a conversation with a redacted practice packet and your team's acceptance criteria. The underwriting model-builder pack can help organize the model handoff.

Make the evaluation reusable. The next model or vendor will need to pass it too.

Next: carrying the acquisition assumptions into the first operating review.

- NextAutomation Team