90%? Chopie, kup se spirytus!

Jednego dnia wyjaśniasz nieco dalszym kolegom po fachu, co robisz w okolicach AI - pilnujesz, żeby sobie ludzie nim paluszków nie poobrywali.

Drugiego dnia przychodzi w pracy mail tak straszno-śmieszny, że wart uwiecznienia. Z obrazkiem. 
(Przyjmijmy, że aplikacja z dostępem do LLM-ów, przy której pracujesz, nazywa się Tatanka).

Hi (...) and Tatanka Dev Team,

We trying to understand condition and investigating options to increase accuracy and reliability of generated  output for a number of use cases.

Primary use case is;

User can querying  sets data and the output is highly accurate (~90+%) and highly repeatable.  

Use cases  applicable to;

1. Financial calculations and projections
2. Product portfolio risk assessment (+ status changes)
3. Supply chain risk assessment (+ status changes)

1. Where are we today on our system's accuracy and reliability?

• Are we greater than 50% accuracy using current method of vectorRAG? 
• See attached example vectorRAG vs GraphRAG
• What are the conditions (system and usages requirements) that ensures higher accuracy with greater reliability?

2. What are our plans for increasing accuracy of Tatanka?

• What is our current benchmark by % of accuracy achievable, and what is our next target of accuracy %?
• What are the conditions (requirements, behaviors) that ensures higher accuracy with greater reliability?
• What will it take for our system to get 99.9% accuracy with high degree reliability?

3. How are we factoring knowledge graphs, GraphRAG, and other methods/models towards increasing both accuracy and reliability?

4. If in the meantime, interested in examples and guidance;

• to articulate what the accuracy limits that currently exist
• to articulate what conditions contribute to maximizing accuracy for the current capabilities of Tatanka (to help inform best practices).