RedditPP Mascot

RedditPP

r/trueantiAI • … • u/squidwardtufte

They put me in charge of our project’s legal review

...because the legal team was swamped? Unavailable? I was tasked with running content through the "legal agent" (it's just Copilot, as far as I can tell) and getting it to report on any issues that the team needed to resolve. I am absolutely not a lawyer.

Sure, it made me nervous to have this responsibility, but I went ahead and plugged these docs into the chatbot. For each it would return 10-12 issues to look into, ranked by severity as I'd asked it to, spit out an excel doc, all nicely formatted. Great, my job is done?

Knowing that I have no legal background, and being nervous about missing something, I tried feeding the docs back into another instance of the agent. Repeatedly. The "non-determinative algorithm" would always give me 10-12 issues, ranked by severity, in an excel doc... but never exactly the same issues for the same document. Basically just about the same length of text, no matter what I gave it, which really seems to me like it's just generating some amount based on how many tokens it's supposed to use.

Anyway, I think companies (this is a very big company I am talking about) are willing to roll the dice on legal issues to save on legal review.

— 0