FileMaker AI in the Real World

Introduction to AI in FileMaker

Building an automated invoice processing app to scan, and code invoice line items using OCR and AI in FileMaker

For the last few years, I’ve enjoyed the opportunity to work as the primary developer on the custom FileMaker app for Blueprint Capital REIT, a real estate lending and development firm based in Seattle, WA. Although their app uses and integrates with a variety of tools and services, my head is usually living exclusively in FileMaker-land, extending the data model, building new layouts, and scripting workflows. So when the opportunity comes to integrate with something new outside of FileMaker, I get excited. And maybe a little nervous. And much more so (both excitement and nerves) for the prospect of adding some Artificial Intelligence to their app!

I, like many software developers, can often hold a (healthy?) skepticism of AI.

Toyota adds more operators to its “driverless” vehicle after it crashes into visually impaired athlete at Paralympic Games. https://t.co/nVFpWMIDbm

— hypervisible dot pdf (@hypervisible) August 31, 2021

How Tesla’s autopilot system interprets traffic lights on the back of a moving truck. I guess this scenario was probably not part of the system’s training data. A good illustration of how it will likely be impossible to reach full driving autonomy just by recording “more data”. https://t.co/eZEhxhpXBC

— Max Little (@MaxALittle) June 3, 2021

I used GPT-3 to generate new Disney villains.

I, too, rebel against the mandatory sasquatch dress code.https://t.co/n6nvZn50Jz pic.twitter.com/WDh73g3fv6

— Janelle Shane (@JanelleCShane) April 30, 2021

But as Blueprint’s accounting team took on more customers, they faced some challenges that improving the data-entry workflow alone couldn’t resolve. They estimated it cost $8 to process a single invoice; multiply that by an average of 1,200 (and growing) per month, and invoice processing quickly becomes an expensive task. Perhaps even more costly were the long hours and late nights the team put in each month. To make a significant impact, we would need to automate more of the process with the help of AI.

The Backstory

Blueprint began as a residential construction lender for its network of builders, but their business model quickly grew to include all facets of the construction project lifecycle: from evaluating and sourcing new projects, to budgeting and financing construction loans, to completing plans and permits, to marketing the properties for sale. As a real estate investment trust, Blueprint also began their own construction projects, building beautiful apartment homes in the booming Seattle area.

To support managing these projects, SeedCode built an accounting module in their FileMaker app to set and update budgets, receive and categorize invoices, and export costs and payables to their general ledger application. It wasn’t long before Blueprint realized that, thanks to their small-but-talented accounting team and the right software, they could offer these same accounting services to their customers (builders). Fast-forward a few years, and Blueprint now provides bookkeeping for many of their builders’ companies and projects.

Before AI

Coding invoices and receipts is a critical function of this accounting module. Documents get uploaded, scanned, or emailed into the system. Then a user reviews and inputs the pertinent info, including vendor, date, invoice number, and amounts. Users also assign one or more cost codes (budget categories) to the amounts to properly track actual expenses against the budget. This manual process worked fine, particularly when the number of bookkeeping projects was low. But when the amount of invoices processed monthly stretched over 1,000, coding invoices became an onerous and time-consuming task, especially for the month-end crunchtime when most invoices are received.

We mitigated the pain by refining and streamlining the invoice entry process, reducing clicks, consolidating tabs, and providing shorthand inputs for dates and vendors. But those changes, while helpful, could only make a marginal impact on their invoice input time.

The “FileMaker AI” Request

Enter, the request for automation using AI and FileMaker. Blueprint’s FileMaker system already automates a wide range of processes (creating invoices from emails and scans, sending project notifications and reports, preparing doc packages to vendors containing missing/needed forms, etc), but management asked about the possibility of using OCR (optical character recognition) and AI tools to extract data and even predict cost codes from the PDF documents automatically.

I confess: when they first suggested adding OCR functionality, my FileMaker-focused brain first went to some OCR FileMaker plugins I had read about years ago. But on the FileMaker platform, we’re not restricted to a set of tools designed just for our ecosystem; a worldwide web of services and APIs are available to us through the integration capabilities FileMaker provides. We were given the greenlight to explore options with the aim of reducing the workload around processing invoices.

Exploring AI Vendors

To get my bearings on what was available, I started researching OCR vendors, finding several generic options that extract text from a variety of documents and images, as well as services tailored specifically for processing invoices and receipts. To narrow in on the best contenders, I settled on a few goals:

1. The service needed to offer API endpoints for uploading docs and retrieving results. Other upload methods (e.g., emailing invoices directly to the service) could be useful, but Blueprint already has a system for importing the docs; integrating an API means we can add OCR/AI capability without changing up the user’s existing workflows.
2. It needed to extract “fields” specific to invoices. Generic OCR services can return document text, but we couldn’t use a text blob for our purposes; we need to know specific values, like vendor, invoice number, date, tax, total amount, etc.
3. More than that, it needed to handle a wide range of invoice formats. To extract specific fields, some OCR vendors need you to “annotate” individual invoice templates, defining where specific fields can be found. But Blueprint has worked with over 1,500 vendors and counting; we needed a vendor that can process “unstructured” docs.
4. Ideally, it would offer the ability to predict cost codes based on Blueprint’s set of cost codes. Even if we could find one that offered this, we knew we’d want to compare code predictions with other AI services; we didn’t want to have to limit our search for the best OCR based on its ability to also predict codes.

It’s easy to think that points 2-3 are just “OCR” problems and that “AI” doesn’t really come into play until number 4. But for a service to be able to interpret an invoice and tell us the invoice date — distinct from the due date — is an incredibly powerful layer of AI at work. Now, some services claimed they could do this, but required a large up-front investment just to get started; I focused on ones that provided free trials or low startup costs, including Google (Invoice OCR Beta), Klippa, Typless, Nanonets, and Veryfi.

After some testing, Veryfi clearly rose to the top. Not only was it faster than other services I tested (often taking only 3 seconds to process an invoice, about twice as fast as others), it also attempted to predict cost codes right out of the box, with the promise of the ability to train the model to get better cost code predictions over time. But even more than that, its OCR AI did almost all of the heavy lifting for me.

In particular, Veryfi’s json results use normalized field names as the keys for the corresponding extracted values. For example:

{
"InvoiceNumber" : "20210048",
"InvoiceDate” : "8/2/2021",
…
}

Other vendors provide an array of field objects, where the labels and values are separate properties within each object:

{
"label": "Invoice Number",
"ocr_text": “20210048",
},
{
"label": "Invoice Date",
"ocr_text": "8/2/2021",
}

The latter may have more flexibility for different document types, but it also would require a lot more work on the FileMaker side to extract the few values that are meaningful to us. Instead, with minimal coding in FileMaker, Veryfi allowed us to accurately extract most invoice values that previously needed to be input by hand.

Similary, Veryfi returned invoice line item results in a logical array of line item objects, containing keys like “description” and “amount” for each item. Some other vendors return an array of individual cells, providing the row, column, and value of each cell. Again, this design may be flexible, but would have required a lot more processing on the FileMaker side to build meaningful invoice line data.

When we demo’d extracting invoice data within Blueprint’s app, the CEO’s first reaction was “This is literally magic.”

The Right Combination of  AI Vendors

As easy as it was to pull meaningful, specific invoice fields from Veryfi, the raw OCR results alone couldn’t link related records to invoices for us. So, in the case of vendors, we augmented the OCR results with logic in FileMaker, “training” the database to assign the correct vendor to invoices. To do so, when a user manually selects the vendor on an invoice, either because no vendor was assigned or because the wrong vendor was assigned by Verify, we create a record in a new “OCREntities” table, storing the vendor’s primary key along with the vendor name and address returned by the OCR. Then, when the same vendor name and address are returned on future invoices, the system can find that matching vendor in the OCREntities table to grab the vendor’s primary key. That way, the system can properly link the vendor to their invoices, since a vendor’s name and address stay consistent regardless of the builder or project billed.

Extracting the vendor and other invoice values was a huge and immediate leap forward in terms of invoice processing burden; after deployment, the team finished their first month-end accounting process, typically a week long, one and a half days sooner than normal. But we still wanted to get to the next level and predict the cost codes on invoice line items. Although Veryfi included this capability, after more extensive testing, the results weren’t dependable for our purposes. Even after training the model with thousands of invoices, Veryfi only hit on the correct cost code less than 25% of the time. But we didn’t need to regret our choice in OCR vendor; we had envisioned comparing code predictions with other AI services from the beginning anyway. Our idea was that, once we have the invoice results from Veryfi, we should be able to send that data in a separate API request to another AI vendor specifically to predict codes.

We decided to dig into a well-documented AI service called Aito, in part because it provided examples of the very cost-coding problem we were hoping to solve — and also because it was easy to get started. I’m not a data scientist, and Aito didn’t require me to be one. I simply needed to upload data (invoice line data including description, amount, vendor, and cost code). From there, to predict a cost code, I could send an invoice’s descriptions, amounts, and vendor via an API call, and Aito would return predictions and confidence percentages (the likelihood a prediction is correct). There are no models to retrain or maintain, other than by uploading new data as invoices get processed.

When we send codes to Aito, we use FileMaker’s Insert from URL script step with a request payload like this:

"predict" : “CostCode”,
   "from" : 
   {
      "from" : "InvoiceLinesOrText_csv",
      "where" : { "CostCodeListID" : “E0B060A4-8BA7" }
   },
   "where" : 
   {
      "Amount" : 8573.4,
      "Description" : “Z-MAX DECK POST CONNECTOR“,
      "VendorID" : “08754845-9F8B-0847-B95A"
  }

And we end up with a response that looks like this, where $p is the confidence percentage of the result:

"hits" : 
   [
      {
         "$p" : 0.993500630562667,
         "feature" : "4005 - Framing Materials",
         "field" : "CostCode"
      },
      {
         "$p" : 0.000738495964824645,
         "feature" : "1204 - Structural Engineering”,
         "field" : "CostCode"
      }
   ]

Not only did Aito return a higher percentage of correct cost code results, Aito allowed us to predict project assignment for each invoice in a similar way! Though many invoices include a project name, different vendors often record different name variations for the same project. Linking projects in FileMaker just based on the raw OCR data, like the method we used for vendor assignment, would be very difficult; AI with Aito makes it possible.

The Quest for Better

It probably goes without saying, but Aito doesn’t always get it right. In fact, our models’ predictions are highly confident only about 50% of the time. We omit low confidence predictions altogether and flag mid-tier confidence predictions to minimize inaccuracies. Flagged invoices appear with a warning to check the prediction. Invoices cannot be approved until a user clears the warning by clicking a button to signal that the prediction was correct or by manually changing the selection to the correct project or code.

Although continual training of the models with more data should improve predictions over time, we expect those improvements to be marginal, as we had seeded the original models with years of existing invoice data. Still, the opportunities to try to improve the predictions seem endless. We could review/massage the historical data that our models depend on. Or go deeper into Aito’s tools and options to squeeze the best results out of our data. Or set up models with other AI vendors to compare with Aito.

For now, I’m proud that Blueprint’s software was able to make a real and felt improvement on their invoice processing burden — to a degree that wouldn’t be possible without the help of some AI.