Key Takeaways
- Companies are rapidly adopting AI but largely lack the infrastructure to measure its actual return on investment.
- The current challenge in measuring AI's impact parallels the 'attribution problem' that early internet advertising faced.
- Many employees are using AI tools without management's knowledge, primarily driven by a desire for increased efficiency.
- Effective AI measurement requires moving beyond traditional surveys to integrate passive behavioral and aggregate productivity data.
- Employee anxiety about AI stems more from tool overload and lack of training than from fear of job loss.
Deep Dive
- Russ Fradin notes 85% of companies feel they have 18 months to become AI leaders, drawing parallels to early internet and ad tech growth.
- The 'attribution problem' in early ad tech, where measuring ad effectiveness was difficult, mirrors today's challenge of proving AI value.
- Establishing measurement infrastructure was crucial for ad tech's trillion-dollar boom, as seen with companies like ComScore.
- Fradin's new company, Laradin, aims to build AI measurement and governance tools to track usage and ROI.
- Rapid AI adoption allows widespread use of unmonitored tools, with many unknown to management.
- Driving employee AI adoption requires making users feel safe from penalty or appearing unskilled, especially given data privacy concerns.
- Enterprise AI tool adoption is currently lower than expected, with companies struggling to measure actual usage and ROI.
- Measuring AI productivity combines traditional surveys with proprietary behavioral data to overcome biased answers and unknown usage, similar to ComScore's past methods.
- Measuring individual AI productivity is difficult; companies need aggregate data for overall output and cost-effectiveness.
- Goodhart's Law warns that a measure, once targeted, ceases to be accurate, potentially corrupting metrics like email count.
- Surveys for AI tools like Harvey are flawed, as users may self-report positive experiences, requiring passive behavioral data for true insights.
- Traditional metrics like monetary spending on tools such as Cursor may not reflect true productivity or value in developer-centric companies.
- Objective, third-party measurement is needed for AI tools to provide reliable data to CFOs and CIOs, similar to internet advertising's evolution.
- The challenge is to define and measure the true output and value of AI adoption without optimizing for the measurement itself.
- Larridin establishes baseline productivity metrics, adapting them to departments and focusing on interdepartmental responsiveness, not just employee usage.
- Interviews with 350 IT heads reveal $700 billion is spent on enterprise AI, but 70% of leaders believe much is wasted due to absent measurement systems.
- Companies, unlike in ad tech with 20 years of infrastructure, lack systems to measure AI success, contributing to perceived waste.
- Employees are anxious about AI, not primarily due to job loss fears, but from being overwhelmed by numerous new tools without training or clear guidelines.
- Employee adoption of AI tools is hindered by concerns over productivity, job security, and regulatory compliance.
- Larridin's product acts as a secure wrapper around AI models, guiding employees to use them effectively.
- This helps employees avoid mistakes or policy violations, encouraging internal expertise and effective leveraging of tools like Cursor and Harvey.
- Historical technological shifts, like the agricultural revolution, led to job shifts but overall societal improvement.
- Skepticism exists regarding large-scale AI-driven job loss; capitalist incentives suggest companies will use AI to scale and innovate.
- New roles are expected to emerge, potentially in podcasting, skilled trades, data center engineering, and even space exploration.
- Concerns that AI may disproportionately affect highly educated, white-collar workers are met with the argument that these individuals are more adaptable.
- Employee anxiety about AI necessitates providing tools for better usage and understanding its productivity impact.
- The 'product marketing problem' for AI tools questions whether they truly enable productivity or face employee hesitation.
- Selling broad AI capabilities is difficult; a focus on specific, valuable use cases is more effective, similar to ComScore's early strategy.
- Demand exists for specific market share data in industries like finance and pharmaceuticals.