Computer Vision vs. LLMs: Choosing the Right AI Tool

The Wrong Way to Start an AI Project

The single most common mistake I see AI projects make: choosing a technology — "we're going to use GPT" or "we're going to use computer vision" — before defining the problem precisely enough to make a coherent choice.

Both computer vision and large language models are powerful. Both are overused in contexts where they're the wrong tool. Here's the framework I use to choose between them.

What Computer Vision Does Well

Computer vision excels when your input is image or video data and your output is spatial — detecting objects, tracking movement, reading text from images, estimating poses, or classifying scenes. The key property of these tasks is that they require understanding the visual structure of data: where things are, how they relate to each other in space, and how they change over time.

At Sunbots, we use computer vision for three distinct applications:

• Currency detection in SmartON: Identifying Indian rupee denominations from a camera feed. The task is inherently visual — the information lives in the color, texture, and pattern of the note, not in any text.
• Scene understanding for SmartON: Describing the physical environment to visually impaired users — identifying obstacles, doors, and spatial relationships.
• Retail theft detection: Pose estimation and action recognition from CCTV feeds to identify suspicious behavior patterns. This requires understanding body position and movement in space — fundamentally a visual problem.

Where computer vision struggles: when the meaning of what you're seeing depends heavily on context that isn't visible in the image. "Is this a fraudulent document?" isn't primarily a visual question — it's a semantic one that requires understanding the content.

What LLMs Do Well

Large language models excel when your input is text (or can be reduced to text) and your task requires understanding meaning, context, or generating coherent natural language. Document summarization, question answering, code generation, classification of text into categories, and conversation are natural LLM problems.

The key property: LLMs have internalized patterns from an enormous amount of human language, which means they perform well on tasks that humans solve using linguistic and cultural knowledge. Legal document analysis, multilingual translation, and understanding conversational intent are strong fits.

Where LLMs struggle: tasks requiring precise spatial reasoning, real-time performance at under 100ms, or very high accuracy in narrow, well-defined domains where fine-tuned smaller models consistently outperform larger general-purpose ones.

The Gray Zone: Multimodal Problems

Some of the most interesting problems require both. SmartON's MIRA assistant is a good example: a user holds up a chemistry worksheet with a graph on it and asks "explain this graph to me." Computer vision reads and interprets the visual graph structure; an LLM generates the natural language explanation.

The routing logic — deciding which component handles which part of the request — is itself an interesting design problem. MIRA uses intent classification (an LLM task) to decide whether the user's request is better served by vision (scene understanding, currency detection, document OCR) or language (document search, explanation, translation).

Building multimodal systems is more complex than either standalone, but the user experience gains are substantial when the routing is accurate. If you're in the gray zone, start by clearly defining the routing logic before writing any model code.

A Simple Decision Guide

Use this as a starting point, not a rigid rule:

• Input is image/video → start with computer vision. If the information is inherently visual, don't fight it by converting to text.
• Input is text → start with an LLM. For text classification, generation, summarization, or conversation, LLMs are almost always the right starting point.
• Input is audio → transcribe first, then LLM. Whisper-class models for speech-to-text, then LLM for understanding and generation.
• Task requires sub-100ms latency → consider smaller fine-tuned models. Large LLMs and general-purpose vision models are often too slow for real-time applications. Domain-specific fine-tuned models are faster and often more accurate.
• Task is well-defined with clear categories → consider classical ML first. If the problem has clear boundaries and sufficient labeled data, gradient boosting or a fine-tuned small model will be faster to train, cheaper to run, and easier to audit than a large foundation model.