HomeLens AI combines supervised sentiment models with a rule-based aspect dictionary and a provider-level scoring layer. Every step can be inspected.
One- and two-star reviews are the risk class, four- and five-star the comparison class; three-star reviews are held out.
A fine-tuned DistilBERT model is the primary scorer. A TF–IDF logistic regression is the fallback and powers this website’s demo.
Thresholds keep negative-class recall at or above 90%.
A curated phrase dictionary over eight aspects finds the complaint language in each sentence and picks the evidence sentence.
High-risk reviews with no match go to manual review.
Reviews roll up to providers with at least 20 reviews.
A score blends mean risk, recent risk and high-severity rate as percentiles, then maps to a relative tier.
The split holds out whole businesses, so test scores are not inflated by a provider appearing in both training and test data.
| Model | Threshold | Accuracy | Macro-F1 | Neg. precision | Neg. recall | ROC-AUC |
|---|---|---|---|---|---|---|
| Fine-tuned DistilBERT | 0.950 | 0.976 | 0.973 | 0.972 | 0.955 | 0.996 |
| TF–IDF Logistic Regression | 0.620 | 0.957 | 0.951 | 0.950 | 0.917 | 0.991 |
| TextBlob | 0.405 | 0.793 | 0.785 | 0.630 | 0.909 | 0.911 |
Test set: 3,331 reviews from 213 held-out businesses (1,100 negative, 2,231 positive). TextBlob is the lexicon baseline. Source: artifacts/model_comparison.csv.
Filtered from 23,685 candidates. 1,054 providers and 12,384 reviewers in Tucson, AZ, Nov 2006–Jan 2022. Median length 83 words.
Train: 10,411 reviews, 630 businesses
Validation: 3,691 reviews, 211 businesses
Test: 3,331 reviews, 213 businesses
Phone numbers, emails, links, dollar amounts and star phrases are replaced with tokens so a model cannot lean on a rating written into the text.
The risk score is a weighted blend of percentiles, which makes it a ranking inside this corpus.
100 × (0.50 · mean-risk percentile + 0.30 · recent-risk percentile + 0.20 · high-severity percentile)
Recent means 2021 onward. The trend compares recent against the 2020 mean and only appears with at least three reviews in each period.
StableScore below 50 · 126 providers
WatchScore 50–75 · 75 providers
ElevatedScore 75–90 · 47 providers
High concernScore 90 and above · 6 providers
Paste a review and read the evidence, the matched phrases and the term weights.
