Digital White Papers

I L T A W H I T E P A P E R & S U R V E Y R E S U L T S | L I T I G A T I O N & P R A C T I C E S U P P O R T 15 I L T A ' S 2 0 2 3 L I T I G A T I O N & P R A C T I C E S U P P O R T S U R V E Y R E S U L T S Limitations Understanding what the LLM can and cannot do is key as this will dictate how you use it. The following are some limitations to consider: • Formulaic responses – How sophisticated is the model? Will it only generate a formulaic response? If so, what data works best with this? For, it may be bad to use it for case strategy considerations. • Oversimplification – Will it oversimplify the content? Do you need more particulars, or will the output suffice? Perhaps it depends on the use case and if so, exactly what are the use cases? • Memory – Does the LLM have limited memory? If you are summarizing a document, what length of the document can it correctly summarize, and what if any issues will you run into with longer documents? • Data Types and Dates – Will it work on all data types such as Excel spreadsheets and can it undertake mathematical functions? If it can summarize records, then how does it deal with dates? Considerations What do you need to consider at the outset before using a tool that incorporates LLM features? • Accuracy, Reliability, and Authenticity – How factually accurate is the tool? If using a similar data set, will it produce the same results, or will they differ? Is there an explanation for the difference? Has accuracy and reliability been established by independent testing? • Explainable – Can the work of the LLM be explained? For an understanding of explainability check out NIST's, Four Principles of Explainable Artificial Intelligence at https://www.nist.gov/ publications/four-principles-explainable-artificial-intelligence. • Data Cleansing – Those who have worked with Technology Assisted Review tools know that garbage in equals garbage out. Do you need to cleanse the data set before using the LLM tool with it as you may otherwise receive skewed results? • Cost - Is the use of the LLM tool appropriate for the problem or is there a more economical or proven method to solve it? The use of certain LLMs is expensive given the cost of tokenization so you may want to obtain a cost estimate at the outset. • Environmental Impact – Most LLMs currently need a lot of computing power which translates to energy usage which in turn leads to environmental considerations. I do not recall a time when the legal profession was more interested in a particular type of technology as is the case with LLMs. While LLMs certainly hold a lot of promise, several issues need to be examined to satisfy ethical and other legal obligations. The best course of action would be to become educated in LLMs and ask lots of questions. Be curious, take precautions, and don't make assumptions. There is no magic bullet. ILTA

• Cover