There are three kinds of lies: lies, damned lies, and statistics. Mark Twain
I was told one time that there are firms that want to innovate, and there are firms that want to say they innovate. Meaning, of course, that some firms just want to get on the bandwagon and appear to follow trends for appearance’s sake. These firms either don’t know how to innovate or have no interest to begin with.
I thought about this idea when I was recently presenting with Dr. Maura Grossman. Dr. Grossman is a professor of Computer Science at the University of Waterloo in Canada, a consultant, and a lawyer. She has been a court-appointed special master on eDiscovery matters. She has represented Fortune 100 companies and financial institutions in dozens of civil actions. She invented one of the most widely used protocols for technology-assisted review. Oh, and in her spare time, she is an affiliate faculty member at the Vector Institute of Artificial Intelligence in Ontario, Canada.
Anecdotal Evidence Suggests Limited Use of Gen AI
And she is one of the smartest people I know. She and I were presenting on Generative AI to a group of high-end trial lawyers. At one point, Dr Grossman mentioned that she had had several recent conversations with lawyers about the use of Gen AI. Based on these conversations, Dr. Grossman believed Gen AI was being used sparingly, if at all, by lawyers. But many of the lawyers she talked to insisted they were using Gen AI tools frequently.
The disconnect, said Dr. Grossman, was a misunderstanding of the tools, how they work, and what they could do. And the lawyers may also be influenced by the “fear of missing out” syndrome. They have heard all the publicity about AI and don’t want to admit they aren’t on the bandwagon.
Many lawyers have little familiarity with what Gen AI could do, how it works, and how they could be affected by it
I have written about similar findings based on my conversations with lawyers. Many of the lawyers I talk to have not adopted Gen AI tools. As Dr. Grossman concluded, I too have found many lawyers have little familiarity with what Gen AI could do, how it works, and how they could be affected by it.
Gen AI Surveys
On the other hand, numerous surveys done primarily by vendors would seem to suggest the opposite. These surveys would suggest that lawyers are embracing Gen AI and using it frequently.
One survey, for example, suggests that some 73% of lawyers responding say they plan to use Gen AI tools in the next year.
According to a UK survey, more than a quarter of legal professionals say they already use generative AI. The same study reported that the proportion of AI users has more than doubled in six months.
Another vendor survey of 1,200 legal professionals found that nearly two-thirds (62%) of law firms have already made changes to their day-to-day operations as a result of using Gen AI. These changes include the launching of AI-powered products for internal use. They also include such things as running AI-related training for staff and hiring AI experts. The firms say they have even developed policies for the future use of the technology. These steps all suggest a sophisticated familiarity with and use of Gen AI tools.
These surveys seem fundamentally inconsistent with the anecdotal findings of Dr. Grossman and myself.
Inherent Limitations of Surveys
The problem of course with these surveys is that they are conducted solely in writing. There is no ability to ask follow-up questions to determine if those who say they are using the tool. There is no way to tell if they understand what they are saying and whether they really are doing what they say. A better way to find out the true state of adoption is to ask questions.
Most of the time, when I and Dr. Grossman probe about usage, we find that the true use and adoption rates are much lower. Granted, our experiences are based on much fewer numbers than the mass surveys. However, the consistency of our findings and the anecdotal evidence reported by others makes me think our conclusions are closer to the truth.
Surveys are better than nothing. But they are far from perfect.
Bias
“Figures often beguile me, particularly when I have the arranging of them myself”. Mark Twain
I know lots of the folks at the vendors who do these surveys. By and large, they are good, hard-working, and honest people. However, the way you write the questions and interpret the results can make a difference in your conclusions. It only stands to reason that your unconscious bias toward your employer or what you want to find can impact your findings. Since there is no way to question the response, you are free to make assumptions about what information comes back. You may unconsciously count something one way when it could be counted another way by someone else.
Surveys are better than nothing. But they are far from perfect.
A Reality Check
The inconsistency between what the surveys show and reality was also evident in another recent presentation in which I was involved. I was part of a panel whose goal was to demonstrate –not just talk about—how Gen AI could be used in litigation.
The audience was an elite group of trial lawyers frequently on the cutting edge of innovation in the litigation space. The kind of lawyers who have more to do than they can get done and are under constant time and client pressure. The kind of people you would assume are in that big group of savvy Gen AI users.
They were stunned when we showed the group all the things that could be done with GenAI tools to streamline what litigators do
But they were stunned when we showed the group all the things that could be done with GenAI tools to streamline what litigators do. Mouths flew open. As panelists, we could tell that we had opened up a new world for many of those in the audience. Gauging from their reaction and comments to us after the session, this was not a group that was using Gen AI on an hourly, daily, or even weekly basis. I suspect most of them had not used Gen AI at all or even planned to do so before our session. Clearly, these lawyers showed the same disconnect Dr. Grossman mentioned. They suffered from the same misunderstanding of Gen AI tools, what they do, and what they could do that Dr. Grossman and I have discovered.
The Conclusion?
The contrast between vendor survey results and the anecdotal evidence of Dr. Grossman and others reveals a gap in our understanding of the true legal use of Gen AI. Surveys are a start but they often fail to capture the nuanced reality of day-to-day practice. Care should be exercised in relying solely on surveys to understand what’s really happening.
