The sales team asked me which accounts they should focus on for expansion. Customer success asked me which accounts were at risk of churning. Product asked me which users to interview about new features.
I didn't have answers. We had usage data on 15,000 users, but no systematic way to identify who was highly engaged, moderately engaged, or disengaged.
I could pull individual metrics (last login, feature usage, support tickets), but I couldn't synthesize them into a single view of engagement health.
I needed an engagement scoring model.
Three months later, I had a system that assigned every user an engagement score from 0-100 and predicted with 84% accuracy who would:
- Churn (score <30)
- Stay flat (score 30-70)
- Expand (score >70)
Here's how I built it and what it revealed.
The Problem: Too Many Metrics, No Synthesis
Before building a scoring model, I tracked engagement with disconnected metrics:
Usage metrics:
- Days since last login
- Logins per week
- Time in product
- Features used
Activity metrics:
- Projects created
- Reports generated
- Data processed
- Exports/shares
Adoption metrics:
- Core features adopted
- Advanced features used
- Integrations connected
Team metrics:
- Team size
- Active team members
- Collaboration actions
Support metrics:
- Tickets opened
- Time to resolution
- Satisfaction scores
Each metric told me something, but no single metric told me enough.
Example: A user who logged in daily but only used basic features (high frequency, low depth). Were they engaged or at risk?
Example: A user who logged in monthly but ran complex analyses each time (low frequency, high depth). Were they satisfied power users or losing interest?
I needed a way to combine these signals into a single health score.
Step 1: Defining What "Engagement" Meant
Before building a scoring model, I had to define success. What outcomes did I want the score to predict?
Three outcomes:
Churn Risk
Users likely to cancel within 90 days
Retention Stability
Users likely to renew but not expand
Expansion Opportunity
Users likely to upgrade, add seats, or increase usage
I pulled 12 months of historical data:
- 523 users who churned
- 11,847 users who renewed without expanding
- 1,630 users who expanded (upgraded or added seats)
For each group, I looked at their engagement patterns 90 days before the outcome.
This would tell me: What behaviors in Month 1-3 predicted the outcome in Month 4?
Step 2: Identifying Predictive Behaviors
I tested 37 different metrics to see which correlated with each outcome.
Methodology:
- For each metric, calculate average value for churned vs. retained vs. expanded users (90 days before outcome)
- Test correlation strength (how well does metric predict outcome?)
- Rank metrics by predictive power
Results:
Highest Predictors of Churn (scored negative in model)
1. Declining usage trajectory
- Churned users: 73% showed declining usage in final 90 days
- Retained users: 21% showed declining usage
- Correlation: 0.71
2. Time since last core feature usage
- Churned users: Average 32 days since using core feature
- Retained users: Average 4 days
- Correlation: 0.68
3. Support ticket volume (unresolved)
- Churned users: 2.4 open tickets average
- Retained users: 0.3 open tickets
- Correlation: 0.54
4. Error rate
- Churned users: 8.2 errors per session average
- Retained users: 1.1 errors per session
- Correlation: 0.49
5. Abandoned workflows
- Churned users: 67% had workflows started but never completed
- Retained users: 18% had abandoned workflows
- Correlation: 0.47
Highest Predictors of Expansion (scored positive in model)
1. Power feature adoption
- Expanded users: 89% used advanced features regularly
- Non-expanded users: 31% used advanced features
- Correlation: 0.79
2. Team growth
- Expanded users: Average 3.2 new team members added in 90 days
- Non-expanded users: Average 0.4 new members
- Correlation: 0.76
3. Integration expansion
- Expanded users: Average 2.1 new integrations added in 90 days
- Non-expanded users: Average 0.2 new integrations
- Correlation: 0.72
4. Usage at tier limits
- Expanded users: 81% hitting usage caps
- Non-expanded users: 12% hitting usage caps
- Correlation: 0.71
5. Cross-feature usage
- Expanded users: Used average 8.3 features regularly
- Non-expanded users: Used average 3.1 features
- Correlation: 0.68
Neutral Behaviors (didn't predict outcome strongly)
Surprisingly, some metrics I expected to matter didn't:
Login frequency: Weak predictor (0.23 correlation)
- Some power users logged in weekly but did deep work
- Some churning users logged in daily but accomplished nothing
Time in product: Weak predictor (0.19 correlation)
- Not all time is valuable time
- Users stuck troubleshooting spent lots of time but churned
Support ticket volume (total): Weak predictor (0.15 correlation)
- High-engagement users sometimes had lots of tickets (complex use cases)
- What mattered was unresolved tickets, not total tickets
These insights shaped my model: I excluded weak predictors and focused on the 15 metrics with >0.45 correlation.
Step 3: Building the Scoring Formula
I designed a 0-100 point system with three categories:
Usage Health (40 points max)
Frequency (15 points):
- Daily usage: 15 points
- 3-5x per week: 12 points
- 1-2x per week: 8 points
- Less than weekly: 3 points
Recency (10 points):
- Used today: 10 points
- Used in past 3 days: 7 points
- Used in past 7 days: 4 points
- Used in past 30 days: 2 points
- No activity 30+ days: 0 points
Trajectory (15 points):
- Usage increasing over 90 days: 15 points
- Usage stable: 10 points
- Usage declining <20%: 5 points
- Usage declining >20%: 0 points (churn signal)
Feature Adoption (30 points max)
Core feature usage (10 points):
- Using 80%+ of core features: 10 points
- Using 50-80%: 7 points
- Using 25-50%: 4 points
- Using <25%: 1 point
Power feature adoption (10 points):
- Regular power feature user: 10 points
- Occasional power feature use: 6 points
- Tried once: 3 points
- Never tried: 0 points
Depth of usage (10 points):
- Advanced workflows/customization: 10 points
- Intermediate usage: 6 points
- Basic usage only: 2 points
Expansion Signals (30 points max)
Team growth (10 points):
- Added team members recently: 10 points
- Team stable and active: 6 points
- Team shrinking: 2 points
- Single user only: 0 points
Hitting limits (10 points):
- At 90%+ of plan limits: 10 points
- At 60-90% of limits: 6 points
- At 30-60% of limits: 3 points
- Below 30% of limits: 0 points
Feature expansion (10 points):
- Added new integrations recently: 10 points
- Trying features outside original use case: 7 points
- Consistent with original use case: 4 points
- Narrowing use case: 0 points
Deductions for Risk Signals
Subtract points for:
- Unresolved support tickets: -5 points per ticket
- High error rate: -10 points
- Abandoned workflows: -5 points
- Declining engagement trajectory: -10 points
- Team members deactivating: -5 points per deactivation
Final score: Max 100, min can go negative for very high-risk accounts
Step 4: Scoring Results and Segmentation
I ran the model on all 15,000 users and plotted the distribution:
Score distribution:
- 0-29 (High Risk): 1,847 users (12%)
- 30-49 (At Risk): 3,210 users (21%)
- 50-69 (Stable): 6,423 users (43%)
- 70-89 (Healthy): 2,820 users (19%)
- 90-100 (Champions): 700 users (5%)
Validation: Did Scores Predict Outcomes?
I compared scores to actual outcomes 90 days later:
Users scoring 0-29:
- 71% churned
- 24% stayed but didn't expand
- 5% expanded
Users scoring 30-49:
- 38% churned
- 54% stayed
- 8% expanded
Users scoring 50-69:
- 11% churned
- 79% stayed
- 10% expanded
Users scoring 70-89:
- 3% churned
- 51% stayed
- 46% expanded
Users scoring 90-100:
- 0% churned
- 12% stayed
- 88% expanded
Overall prediction accuracy: 84%
The model wasn't perfect, but it was highly predictive. Scores strongly correlated with outcomes.
Step 5: Making Scores Actionable
A score is useless unless it drives action. I created playbooks for each segment:
Segment 1: High Risk (Score 0-29)
What it means: These users are likely to churn within 90 days
CS Action:
- Emergency intervention: Schedule call to understand issues
- Identify and resolve blockers
- Consider offering discount/concession if at risk due to price
- Last resort: Document churn reason for product insights
Product Action:
- Identify common drop-off patterns
- Prioritize fixing common issues
- Study why engagement declined
Segment 2: At Risk (Score 30-49)
What it means: Users not getting enough value to ensure renewal
CS Action:
- Reach out proactively: "How can we help you get more value?"
- Share best practices and use cases relevant to them
- Offer training or onboarding refresh
- Monitor for movement to High Risk or Stable
Product Action:
- Increase engagement through in-app prompts
- Highlight underutilized features
- Reduce friction in their workflows
Segment 3: Stable (Score 50-69)
What it means: Users getting value but no expansion signs
CS Action:
- Quarterly check-ins (not monthly)
- Share new feature announcements
- Passively monitor for growth signals
Sales Action:
- Monitor for signals they might need higher plan
- No proactive outreach unless signals appear
Segment 4: Healthy (Score 70-89)
What it means: High engagement, strong expansion candidates
Sales Action:
- Proactive expansion outreach
- Share use cases that require higher tier
- Offer trials of premium features
- Collect referrals and case studies
CS Action:
- Nurture relationship
- Early access to beta features
- Invite to customer advisory board
Segment 5: Champions (Score 90-100)
What it means: Power users, expansion likely, low churn risk
Sales Action:
- Expansion conversations ongoing
- Enterprise tier upgrades
- Multi-year contract discussions
Marketing Action:
- Recruit for case studies
- Ask for referrals
- Invite to speak at events
- Feature in customer spotlight
What the Scoring Model Revealed
Discovery 1: Engagement Drops Happen Gradually
I expected churn to be sudden. It wasn't.
Average engagement trajectory for churned users:
- Month -6: Score 62 (Stable)
- Month -5: Score 58 (Stable declining)
- Month -4: Score 51 (Stable to At Risk)
- Month -3: Score 42 (At Risk)
- Month -2: Score 31 (At Risk to High Risk)
- Month -1: Score 19 (High Risk)
- Month 0: Churned
Users don't suddenly disengage. They gradually drift away over 4-6 months.
Implication: We had a 3-4 month window to intervene before churn became inevitable.
Discovery 2: Power Feature Adoption Was The Strongest Expansion Predictor
Users who adopted advanced features:
- Expanded at 11x the rate of basic users
- Had 91% retention vs. 58% for basic users
- Increased usage 3.4x over 12 months
This changed our product strategy:
- We prioritized helping users discover and adopt power features
- Built progressive onboarding that introduced advanced capabilities after users mastered basics
- Created in-app prompts when users' behavior indicated readiness for advanced features
Discovery 3: Team Size Growth Was An Untapped Expansion Signal
Users who added 2+ team members in a quarter:
- Upgraded plans 67% of the time within next quarter
- Rarely churned (4% churn rate)
- Had highest NPS (72 average)
But we weren't tracking team growth proactively.
New process:
- Weekly report of accounts adding team members
- Sales reaches out: "I see you added 3 team members. Our Team plan might be a better fit..."
- Conversion on these outreaches: 43% (vs. 8% on cold expansion outreach)
Discovery 4: Single-User Accounts Were High-Risk
Users who never invited team members:
- 42% churn rate (vs. 11% for multi-user accounts)
- 3% expansion rate (vs. 38% for multi-user)
- Lower engagement scores overall
Implication: Getting users to invite teammates became a key activation metric.
New onboarding goal: Get every user to invite at least 1 team member within first 30 days.
Building Your Engagement Scoring Model
Step 1: Define Your Outcomes
What do you want to predict?
- Churn risk
- Expansion opportunity
- Product satisfaction
- Advocacy potential
Step 2: Gather Historical Data
Pull 12-24 months of data on users who:
- Churned
- Renewed without expanding
- Expanded
For each group, identify their behaviors 90 days before the outcome.
Step 3: Test Metrics For Predictive Power
Test 20-40 metrics. Look for:
- Strong correlation with outcomes (>0.45)
- Measurable from usage data
- Actionable (you can influence it)
Common strong predictors:
- Usage frequency and recency
- Feature adoption breadth and depth
- Team growth
- Hitting plan limits
- Support ticket patterns
- Usage trajectory (increasing vs. declining)
Common weak predictors:
- Raw time in product
- Total logins
- Account age
Step 4: Build Weighted Scoring Formula
Assign points based on:
- Predictive strength (stronger predictors = more points)
- Category balance (usage, adoption, expansion signals)
- Simplicity (don't overcomplicate)
Recommended structure:
- 40 points: Usage health (frequency, recency, trajectory)
- 30 points: Feature adoption (breadth, depth, power features)
- 30 points: Growth signals (team, usage expansion, limit approaching)
- Deductions: Risk signals (errors, unresolved tickets, declining usage)
Step 5: Validate and Refine
Run model on historical data. Check:
- Does high score predict retention/expansion?
- Does low score predict churn?
- Are there false positives/negatives?
Refine weighting based on results.
Step 6: Make It Actionable
Create playbooks:
- Score 0-29: Emergency intervention
- Score 30-49: Proactive value realization
- Score 50-69: Passive monitoring
- Score 70-89: Expansion nurturing
- Score 90-100: VIP treatment
Automate workflows:
- Weekly reports to CS (at-risk accounts)
- Weekly reports to Sales (expansion opportunities)
- Alerts when scores drop significantly
- Dashboards for executives
The Uncomfortable Truth About Engagement Scoring
Most companies track engagement through disconnected metrics and rely on gut feel to identify at-risk accounts.
The result: They discover churn risk too late and miss expansion opportunities.
An engagement scoring model forces you to:
- Define what healthy engagement looks like
- Identify leading indicators of outcomes
- Systematize interventions instead of relying on intuition
But building a scoring model is work:
- Requires data infrastructure
- Needs ongoing maintenance
- Must be validated and refined
Most teams skip it because it's not urgent.
The best teams build it because it's important:
- Predict churn 3-4 months in advance (time to intervene)
- Identify expansion opportunities systematically (not randomly)
- Focus CS resources on highest-risk and highest-opportunity accounts
- Measure impact of engagement initiatives
We went from guessing which accounts needed attention to knowing with 84% accuracy.
That clarity transformed how CS and Sales operated.
Stop tracking disconnected metrics. Build a scoring model that predicts outcomes.
Because you can't optimize what you can't measure. And measuring 37 metrics separately is overwhelming.
Synthesize them into one score, validate it predicts outcomes, and use it to drive action.
That's how you systematically reduce churn and increase expansion.