Figure Eight & Eagleview

Using Machine Learning and the power of the crowd to accurately assess property conditions at speed and scale.

Figure Eight really won me over with how user friendly it is as well as how stable it is… With a remote team working nights in some cases, the stability of the system is a major asset.

– Allison Lechner,
Machine Learning Project & Quality Manager, Eagleview

The Company

EagleView is a property-data analytics company that provides insights from aerial imagery to customers in the insurance, construction, government, and energy industries. Founded in 2008, and merging with Pictometry International Corp. in 2013, EagleView has extensive image capture experience and leverages machine learning, computer vision, and property data analytics to help clients improve the lives of their customers and constituents.

The Problem

While EagleView is well-known for its high-resolution aerial imagery captured with multiple image sources, the company’s area of expertise is found in its ability to use machine learning and computer vision to extract rich property data, enabling customers to make informed decisions without the need to visit the site. Its engineers and data scientists are tasked with making models based on the imagery, performing quality assurance, and delivering insights to customers.

Quality Assessment (QA) engineers review the images that EagleView captures based on a specific category, as determined by the proprietary machine learning models (e.g. roof architecture, roof material, roof condition, property tree coverage). EagleView Data Scientists then use the results of the QA assessment to improve the machine learning models where needed. Since ultimately, the model results serve industries like insurance underwriting, where accuracy is critical, the models are constantly evaluated and updated. EagleView wanted to make this feedback process more efficient by serving data to QA engineers in an accessible format and visual platform. The current method they used to gather feedback from over 60,000 images daily was proving to be time consuming and difficult.

Allison Lechner, Machine Learning Project and Quality Manager at EagleView, recalled,

“We were using an in-house tool originally designed for another task. However, it used too many resources, didn’t provide clear user performance metrics and job tracking, and required our team to maintain multiple logins, among other issues.”

EagleView’s machine learning team needed a more efficient way to perform QA at scale.

Figure Eight & Eagleview

Humans-in-the-loop for intelligent conversational data

The Solution

EagleView turned to Figure Eight to help scale its QA processes. Figure Eight provides the EagleView team with a platform purpose-built for creating high-quality machine learning training data.

For example, the EagleView machine learning model is fed an image of a home to assess for roof condition. The model then provides a confidence score associated with the roof being “good,” “fair,” “poor,” “damaged,” etc. Once the algorithm returns the results, a human QA engineer assesses the same image and the results are compared between the model and engineer’s assessments. Differences are then used to automatically guide model retraining.

Human assessments can be wrong, however, and cannot always be used for model improvement and retraining.

“One of the best parts about using Figure Eight is the platform’s ability to provide golden answers.”

Golden answers are the set of answers that the machine learning team knows and trusts are accurate. They serve as the ground truth for assessing and monitoring the quality of the human QA engineer’s feedback.

Golden answers allow the team to perform regular audits, take random samples, and generate reports to easily check how humans are doing. Previously, this process required an individual to regularly spot check human analysts, perform constant check-ins or worse, perform the manual assessment multiple times. With Figure Eight, EagleView can now track QA engineers’ progress at a team level to see how accurate and consistent their answers are and which tasks are proving confusing or difficult. According to Lechner,

“EagleView can easily see if there are points of confusion in the analysis process and check the results the analysts provide for consistency.”

Figure Eight is helping EagleView put the best technology and most comprehensive information in the hands of every customer.