Domain Group builds 'listing assistant' to help staff find property metrics

Domain Group has around one-third of its staff using an AI assistant to locate property listing metrics and other data with natural language, reducing reliance on dashboards as a data presentation layer.

Head of business intelligence Suyash Masurkar told a Snowflake World Tour event in Sydney that the property marketplace operator had shifted “from dashboards to dialogues” over the course of the so-far two-year project.

At the heart of this shift is Cortex Analyst, a managed service “that provides a conversational interface to interact with structured data in Snowflake”, according to the vendor’s documentation.

It is connected to one of Domain’s key statistical datasets, containing about a decade’s worth of data that can answer about “80-to-90 percent of all queries,” Masurkar said.

Previously, when staff wanted to access a property-related metric, they would be pointed to multiple dashboards to locate their answer.

“At one point in time, we [had] 7500 dashboards, about seven dashboards per person [in the organisation],” Masurkar said.

“If you wanted to search for a metric, chances are you’d get bombarded with seven-to-10 dashboards, but not everyone needs a dashboard for answering all of their questions. Usually, people are just after the metric.

“Also, what happened is people started losing trust in the data because the same metric was used in different [ways] across different dashboards by different teams.”

Around 2022, the group kicked off a project to set up a single portal, “where any user in the business can come in, ask a question and self-serve their data.”

The result is the “listing assistant”, essentially a Cortex Analyst-powered chatbot, two versions of which have so far been produced.

“At a high level, Cortex Analyst takes in a user question, and it spits out an SQL query,” senior BI developer Ahmad Faizi said.

“As developers, we can execute this SQL query, get the output, and then share it with the user as a response to whatever questions they've asked.

“It seems pretty straightforward, but the devil's really in the details, especially when it comes to human language and the nuances that surround that.”

Version one, Faizi said, was a “bit of a letdown” from a usability perspective.

“Every time I used Cortex Analyst, the listing assistant, it performed perfectly, [but when] we gave it to the end users, for some reason they could just not get the same level of satisfaction out of it that we could,” Faizi said.

“One of the problems we had was that we tested it within the data team who were very intimate with the underlying table, with the database language, etc., so we could speak a language that the Cortex Analyst would really understand. 

“We were very precise in what we would ask, whereas end users would come to us with all sorts of questions.”

Faizi said an end user might ask, for example, ‘What market share does x [real estate] agency hold?’

“They want to know the market share of that agency. However, in our business, we have 10 different ways of measuring market share. The end users were aware of maybe one or two or three of these,” he said.

“It's not really their fault, but at the same time, this really annoyed our listing assistant, because it didn't want to assume and it didn't want to hallucinate [a response].”

Faced with ambiguous prompts, and inconsistent use of similar language across different internal departments, the team tried to “get the organisation aligned on language”, and to give guidance to data consumers on how to refine their prompts.

“It’s a really difficult task to do this, so what we did was we passed this burden over to the chatbot itself,” Faizi said.

“We took the easy path, because the chatbot asks the user whether they're happy with the response or not, so it knows when to step in, when to interfere with the workflow and say that ‘Hey, have you tried to frame your question according to these principles of prompt engineering? And to help you get there, here is a searchable data dictionary as well’. 

“The searchable data dictionary was one of the solutions that we came up with to handle that different language the company has.”

Faizi said the effect of this “was almost immediate”.

“Users would type similar, ambiguous questions: ‘What's the market share for an agency?’ and the Cortex Analyst would say, ‘You're not being specific enough’. 

“The user would not be very happy with it, but this time they have the tools at their disposal to help them get to the end goal. 

“So now they're very specific. They want to know market share by listing volume. That actually helps quite a bit because you can break that down into SQL logic.”

Faizi said that version two of the listing assistant was also successful because the team iterated quickly based on user feedback collected via a Slack channel.

“We did what we do as developers - we sifted through this feedback, and prioritised and worked on the most impactful feedback,” he said.

“Most importantly, we closed that loop by getting back to the test users, to help them understand how their feedback and testing is actually impacting the application.

“After the letdown of version one, we still were able to build momentum, build engagement, and build trust with our end users.”

In addition to growing usage, the next steps are to open the listing assistant to more than the one data table it currently has access to.

Future improvements will also tackle context retention and misinterpretation.

“There's limited context retention: when a conversation gets too lengthy, it's possible for Cortex Analyst to lose track of previous prompts and context and details,” Faizi said.

“[In addition], if there's a prompt that is incorrect or false, sometimes what this does is it establishes incorrect contents for subsequent prompts. This is a misinterpretation problem. It’s still an issue, but we're working on it.”

Other future work may see the listing assistant surfaced through Slack where users will be able to interact with it; it is currently a web app built using Streamlit in Snowflake.