The tasks LLM's are good at, and the tasks they're sh**t at: a personal review

The tasks LLM's are good at, and the tasks they're sh**t at: a personal review
Estimated reading time: 10 minutes

My views on how LLM’s might impact our economies and personal lives have shifted a lot in recent years. This is an explanation of that shift. LLM’s can be good at some tasks, and awful at others. Being able to understand the distinction between the two modes helped make my understanding complete. In this essay I’ll focus on opportunities of LLM’s for personal use, and how they can help employees grow.

The Rise of Personal LLMs: A Shift in Human-Machine Collaboration

I find myself at a sterile lecture hall inside UW-Seattle. The white LED’s cast diffuse shadows as my classmates hunch over their laptops. The lesson is on engineering leadership. We’re discussing on distinctions of different kinds of engineering leaders. I remain unsatisfied. I have my own little theory, that’s been shaped by my professional experience. I don’t know where to start formulating it. So many frameworks and competing ideas! How would I even begin structuring my thoughts!?

I decide to use Gemini, Google’s LLM, to help me. I need to evaluate what the whole genAI thing is about anyway. It’s an unexpected partner in the trip. I am skeptical, but I feed it my jumbled collection of connections and questions. What follows is impressive. Gemini doesn’t just offer facts or explain existing models. It counter-questions my thoughts, reflects my ideas back in a clearer, more structured form. It prods for points I haven’t considered. It reveals blind spots in my ‘framework’, pushes me to explore nuances I’d considered unimportant. It has acted as a catalyst. It has forced me to put down my thoughts precisely, identify important points. It tells me I need ‘a compelling narrative’ to market it.

I am impressed.

The process isn’t seamless. There are moments of frustration. Gemini is limited in understanding the nuances, and the complexities of specific arguments. But it helped me think better! It forced me to refine my ideas, rephrase them for clarity, and sharpen the core material of my framework. My essay changed in structure, depth, conviction on each iteration.1.

Not. Too. Bad.

The final paper didn’t look like my initial, fragmented thoughts. It presented a cogent, well-defined theory. It engaged with established frameworks and offered a distinct perspective. Gemini played an important role in its development. It was a thought-provoking companion, a decent questioner.

Fair enough, I thought. If nothing else, LLMs are excellent tools for knowledge exploration. What’s the harm in using a tool that helps express ideas with clarity and confidence?

What else might they be good for? I kept playing around with Gemini for the following weeks and months.

Personal LLMs: Bridging the Knowledge Gap

There are different ways to learn new things and automate tasks in one’s computer. Programming skills allow people to automate tasks, but you need to understand the tools to be good at them. Searching the internet is much easier, but is not interactive. It’s challenging to communicate your intent. Online courses and textbooks come with high investment costs and limited accessibility.

Personal LLMs are a bridge between these disparate learning and skill paradigms. LLMs let individuals with basic computer literacy to automate tedious tasks without programming knowledge. They are much simple web searches om understanding user context and intent, and provide interactive, responsive feedback. LLMs provide a cost-effective “try-before-you-buy” experience for online courses, allowing individuals to sample the benefits of guided learning before committing to larger investments. LLMs act as intelligent assistants. They simplify technical tasks and guide folks towards detailed learning resources.

Stepping Out of the Shadows: Comparing LLMs to Existing Solutions

Let’s now discuss the strengths and weaknesses of LLM’s compared to existing methods in detail.

Automation of repetitive computational tasks

  • Traditional methods: Scripting languages like Python need programming knowledge and time investment. Online tutorials are available, but they lack personalized support.
  • LLMs: Gemini, GPT, and Microsoft’s Copilot can turn simple human instructions into scripts. That democratizes programming for those who are less technical. However, relying solely on LLMs does lead to not understanding underlying process, and potential hard-to-debug bugs.
  • Human experts: You can always ask your friend, child, or coworker. Hiring somebody is an option too. But they come at a higher cost and may not be readily available.

Verdict: LLMs provide an accessible entry point to automation, but combining them with human expertise can lead to optimal results.

Information Retrieval and Navigation

  • Traditional methods: Search engines like Google search through massive troves of data. But they need precise queries and don’t filter according to your exact intent. Specialized databases provide specific information, but require prior knowledge of their structure.
  • LLMs: Can understand user context and intent. They offer personalized search results and summarize complex information. However, their factual accuracy can be uneven, and navigating large datasets might still require human intervention.
  • Human experts: Offer nuanced understanding of information and can guide users through complex topics. Their availability and cost can be limiting factors.

Verdict: LLMs are really good at retrieving information based on user context. Combining them with human expertise assures accuracy and clearer understanding. Without that, it’s hard to fully trust the LLM’s.

Guidance and Coaching

  • Traditional methods: Online courses and books offer structured learning, but can’t give personalized feedback and adapt to the users. Human mentors provide individualized guidance, but are expensive and limited in availability.
  • LLMs: Can act as personal coaches, adapting their guidance based on user progress and offering feedback. However, they lack the emotional intelligence and experience of human mentors. They might also be wrong at times.

Verdict: LLMs provide a scalable and accessible form of guidance. They can be unreliable, and can’t really provide emotional and intellectual support.They should ideally combined with human mentors for a rounded learning experience.

From early grammar checkers like Dr.Gram to the open-source spirit of NLTK, we see a consistent theme: AI empowering individuals and supporting innovation. LLMs could be the next stage of this narrative, they might change not just writing, but how we interact with information, and learn new skills.

However, responsible development demands acknowledging and addressing potential threats. These extend beyond simple comparisons of strengths and weaknesses, including broader societal concerns like:

  • Bias and Fairness: LLMs trained on biased data can perpetuate and amplify existing societal inequalities. Imagine an LLM used for hiring decisions unintentionally favoring resumes with specific keywords reflecting unconscious biases. Mitigating this requires diverse training data, responsible development practices, and ongoing monitoring.

  • Misinformation and Disinformation: LLM-generated content, with its fluency, could speed up the spread of fake news and propaganda. Imagine an LLM creating articles indistinguishable from legitimate news sources, manipulating public opinion. Combating this requires fact-checking mechanisms, user education on identifying AI-generated content, and promoting algorithmic transparency.

  • Job Displacement: Automation through LLMs could disrupt certain industries, leading to job losses. Imagine an LLM automating tasks currently performed by customer service representatives or data entry clerks. Mitigating this requires proactive workforce reskilling initiatives, social safety nets, and exploring the potential for LLMs to create new job opportunities.

  • Privacy and Security: LLMs trained on personal data raise privacy concerns. Imagine an LLM used in customer service unknowingly revealing sensitive information about user interactions. Protecting privacy requires strong data anonymization techniques, clear user consent procedures, and good security measures.

Addressing Potential Threats and Charting a Course for Responsible Development

While unlikely, considering worst-case scenarios helps guide responsible development. Imagine an LLM used for malicious purposes, manipulating public opinion or even creating deepfakes indistinguishable from reality. Amelioration involves establishing ethical guidelines for LLM development.International cooperation is important, and so is promoting public awareness of these potential risks.

Here are potential solutions to the threats from above:

Bias and Fairness:

  1. Diverse Training Data: LLMs must be trained on datasets that reflect the richness human language and experiences. This requires actively accounting for data bias by incorporating underrepresented perspectives.

  2. Algorithmic Auditing: Regular audits should be conducted to identify and mitigate potential biases within LLMs. Transparency in model development and evaluation is important if users are to trust the system.

  3. Human Oversight: LLMs should be used in combination with human oversight. This is specially important for sensitive tasks like like hiring or loan approvals. Human judgment can help mitigate bias and ensure fair outcomes.

Misinformation and Disinformation:

  1. Fact-Checking Mechanisms: Robust fact-checking systems should be implemented to verify the accuracy of LLM-generated content. Collaboration between LLM developers and fact-checking organizations is important.

  2. User Education: Educational campaigns can teach users the critical thinking skills required to evaluate the credibility of information encountered online.

  3. Algorithmic Transparency: Understanding how LLMs generate text is critical for identifying potential biases and misinformation. Algorithmic transparency will allow users to make informed judgments about the information they consume.

Privacy and Security:

  1. Data Anonymization: Techniques like data anonymization can be employed to protect user privacy while still enabling LLMs to learn from vast datasets.

  2. User Consent: Clear and informed user consent should be obtained before collecting and using personal data for LLM training.

  3. Robust Security Measures: Strong security measures must be implemented to safeguard user data from unauthorized access or breaches. This includes regular security audits and updates.

  1. Here is the final essay that came off the collaboration. 

Sirish
Shirish Pokharel, Innovation Engineer, Mentor

This is where all my quirky comments will go.