Leaderboard

No matter how advanced today's Large Language Models (LLMs) are, there are many limitations due to which LLMs are not ideal to solve all kind of problems. Let's review some of the problem categories with examples which LLMs cannot solve: 1. Problems Requiring Reasoning & Inference: LLMs depend on matching the pattern and lack the deep understanding like human brain. So, in many problems where in-depth understanding and logical reasoning are required, LLMs fail to solve the problem. Example: Apple did one study, where it demonstrated failure of LLMs in standard mathematical problem where in by changing names and numbers led to drop in performance up to 9.2% and when irrelevant details were included, there was drastic drop to 65.7%. 2. Problems Requiring Maintaining Context in Long Conversations: Many a times, we need to give a long context to LLMs for getting a response considering all the constrains and entire context. But they most of the time loose the context in a long and extended conversations. Example: In ChatGPT (If you are extensively using it, then you know!) when we give long context, many a times it forgets earlier context then we again need to reiterate to consider that and reshare the response. 3. Problems With Ambiguity and Incomplete Information: If there is incomplete or ambiguous inputs are given then LLMs may generate incorrect or nonsensical responses. Example: If you are typing open ended question like "What to do if my car is not starting?", the LLM response might suggest checking the battery, but it may also suggest unrelated actions like checking the oil due to no situational understanding. Here, Situational Understanding becomes important as it eliminates many probable root causes in the problem. 4. Problems Requiring UpToDate Knowledge: LLMs are generally good at solving problems using historical data & context till which model is trained, but there is no real time updating capabilities. This means LLMs might give some outdated information. Example: LLM trained before 2024 will not be aware of latest events, leading to inaccurate responses. 5. Biased or False Information: LLMs has large backend data upon which it is developed. If biased data is used in the backend, then LLMs will generate biased or false information which is dangerous. Example: This is recently in news that the countries developing LLMs feed the back-end data as per country's stand on controversial topics (e.g. Land, Wars etc.). Two countries may have different LLMs giving different response to same question as who won the historical war between A & B?