What changed
Property Finder, a digital real estate application, found that 96.8 percent of their evaluation labels were inherited from a single model’s predictions rather than verified by human experts. This created a systematic bias that made alternative models appear to underperform.
After rebuilding unbiased ground truth and applying evolutionary prompt optimization on Amazon Bedrock, they achieved higher accuracy at 86 percent lower cost.
What this means in practice
Property Finder processes tens of thousands of property images daily across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image, including watermark detection, room classification, and photo quality.
The team hand-annotated 356 UAE images and a separate set of Egypt images to create a trusted evaluation dataset. The first 256 UAE images formed the development set for tuning, while the remaining 100 images were used for final model comparison.
The team used Micro F1 as the primary metric, alongside macro F1, per-label precision and recall, and per-sample intersection over union (IoU) to measure model performance.
Limitations and availability
The evaluation dataset included 356 images, which is considered sufficient to detect moderate differences between models. However, rare classes within individual attributes may have fewer examples, making smaller differences on those attributes harder to detect.
The methodology described is applicable to vision models with similar evaluation needs, particularly in domains requiring detailed image classification.
The prompt optimization loop used by Property Finder and the AWS Generative AI Innovation Center team is available for similar workloads on Amazon Bedrock.
