This checklist assumes that the reader has a basic understanding of key concepts related to Generative AI (GAI) including Large Language Models (LLM), Machine Learning, neural networks, deep learning and retrieval augmented generation [Editor’s note: . Many vendors and legal tech related organizations including the American Association of Law Libraries offer webinars and resources to their members or customers.
In response to discussions with colleagues and my own discussions with vendors I have outlined below the key issues to be considered in planning for the acquisition of GAI products related to legal research. Every organization is different and my hope is that this list can provide an outline which can be adapted to the specific issues and needs across law firm, government and academic law libraries.
Outline of Generative AI Acquisition Issues
Read full story on Legal TechHub
Getting a “seat at the table.” Is there a different stakeholder mix for GAI?
Due to the complex mix of technical, ethical, training and client issues, many firms have established committees to assess the introduction of Generative AI products, often including but not limited to involvement by the following stakeholder groups:
- Library/KM,
- IT,
- Professional Development,
- IT Security,
- Counsel’s Office General,
- Innovation,
- Practice Group Leaders or Innovation Committee,
- Client Intake.
[Editor’s note: LTH Survey results from Jan 2024 show that in most AmLaw 100 firms the Innovation department leads the firm’s generative AI taskforce or committee, with KM and IT vying for second place]
If this is a research tool, a leader on the library or research team must have a seat at the table.
Why are you looking at GAI Products? Your assessment issues will change depending on why you are starting to look at GAI products.
- Just exploring / experimenting
- Looking to address specific workflow issues
- Responding to client demand for more efficiency
- Firm has a commitment to transform how lawyers work and is willing to get rid of the billable hour
Dealkillers – There are several issues that stand out as potential dealkillers. These should be addressed before proceeding with a vendor or product
- Can the content be trusted?
- Security Compliance
- GC Ethical Concern’s, Client Concerns, Court Rules restricting GAI
Content
- What legal content was the product trained on? Trusted sources? Is it continuously trained as the law changes?
- What are your processes for updating the training data? How often?
- Are there gaps in content? E.g. dates, types of content e.g. caselaw, statutes, regulations.
- How are content gaps addressed in training?
‘Security
- Can the company pass the firm’s security compliance standards? SOC2, ISO2701 certifications.
- If deployed in the cloud – who hosts ? Amazon Web Services. Microsoft?
- Where are the servers located?
- Does the solution support single sign on and multi-factor authentication?
- Can they guarantee that your data wont be used to train the LLM?
- Can they guarantee that uploaded documents will be purged immediately after ingestion and analysis?
GC Ethical Concern’s and Client Concerns. What Risks are unique to GAI?
- Fear of hallucinated answers common in Open Source platforms such as GPT 3.
- GAI is a black box – so is natural language research!
- Ingesting documents is not safe – but Brief checkers and Deal analyzers have been ingesting documents for years.
- Results may not be complete and reliable – Did anyone have a policy warning lawyers not to use Google Scholar where results may also be released? This is similar.
- The limitations of GAI can be overcome with proper “research Hygiene.”
Company
- How long has the company been in the market?
- If a private company, how are they financed?
- How many subscribers/beta testers do they have in the legal market? Any of our peer firms?
- What type of licenses does the company offer? Enterprise only? Practice group? Limited number of licences which are priced by tiers?
- Product trials – free or paid? What type of restrictions in terms of people or duration?
- Can we get testimonials or talk to other subscribers?
- Do you have a customer user group? Or Advisory board?
- What do your colleagues say?
Use Cases
- What use cases does the product support?
- Common Use Cases for AI legal research tools that go beyond basic research capabilities include: drafting, summarision, translation, contract/document review, data extraction.
- Which uses cases do you need? Can you select only the use cases you want from the provider, or do you have to buy into all of them?
Technical Issues
- Deal Killer: Are research results based on “trusted content” owned by the vendor or open web sources?
- What is the underlying AI technology? Claude, OpenAI GPT 3, GPT4. LamDa, DALLe.
- Do they rely on multiple LLM’s depending on the use case?
- Is the architecture model agnostic, in other words is it possible to swap one model out for another as the landscape evolves?
Research Results and Issues
- How are you controlling for hallucinations?
- Have clients ever experienced a hallucinated result? How was this addressed?
- What type of testing do you do to make sure results remain reliable?
- What is your system for notifying customers if there are bugs?
- How does the platform respond if it doesn’t know the answer?
- Does the result include a “confidence score”?
- Does the answer to a research query include links to the source document?
- Does the answer include a full citation to a sources including citation signals?
- What is the speed of response? Do lawyers have an “email the results” option due to speed?
- Contract /Document Review: are there limitations on the size of the size of the document or data set that can be uploaded? Can specific clauses be analyzed?
- Can product extract data and create a table of clauses?
Contract Issues
- Ask to see the license, provide it to procurement team/internal contract attorney and review it yourself. Highlight any concerns.
- Smaller vendors may be more flexible on contract terms.
- Does company offer “all or nothing” pricing?
- Are you required to purchase unrelated products?
- Liability Who is liable for false answers?
- Are there any usage limitations e,g, caps of the number of questions or results?
- Liability if violation client confidentiality
- Enterprise vs. practice group vs. individual licenses
- Length of term. One year or multiple years?
- Product downtime
- Notifications of system issues
- Training support
- Password management vs IP authentication
- Cancellation/Autorenewal
- Usage statistics
- Notice of Violation
- Opportunity to Cure
- Copyright
- Ongoing training
Trials – Depending Firm Culture or Ethical/Client Concerns
- Tight control or loose control over who uses GAI at the firm or how it is used?
- Restrict Trials to professional researchers?
- Limit to attorneys who have a strong interest in evaluating GAI
- Require Training on the ethical use of GAI
- Require training on how GAI and LLMs work
- Restrict use of GAI product to non-client work
- Require work to be verified in a traditional legal research resource
Testing
During your testing of a product, look for:
- Research capabilities
- Quality of results
- Speed of results
- User Experience
- Links to primary Sources
- Citation flags
- Editorial enhancements
- Input -Carefully crafted Prompts
- Input – Natural Language question like an associate assignment
- How many “turns” in conversational Search
- Uploading documents
Training, Deployment, Ongoing Help
- How does the vendor support training?
- Does the product require special prompt training?
- Are there training guides, videos embedded in the product?
- Is there a 24 X 7 helpline?
Keep Learning
- Talk to colleagues
- Attend programs at AALL ILTA, Legal Tech
- Set up news alerts to monitor developments.
