Reviewing Feedback for LLM-Based Questions
Intended audience: POWER USERS
AO Easy Answers: 4.3
Overview
The Feedback Review screen is available when Easy Answers is working in AI Enhanced (LLM) mode. It allows power users to validate any feedback provided on Easy Answers questions and related results and decide if and where improvements in the ontology shall be made, including creation of words, synonyms, and/or prompt instructions.
The entry point to the Review Feedback screen is from the Easy Answers Review Feedback icon in the Header area.
Click the Review Feedback icon to open the full-screen dialog. The dialog is generally split into two main sections:
Left-side panel - provides a list of unique questions with status information - see details below.
Right-side panel - provides details about each question (when clicking a question in the left-side panel, including Question Feedback, Comments, Similar Questions, and Words that have been suggested through Annotations - see details below.
Review Dialogs
Left-side panel - My Issues vs All Issues
There are two tabs in the left-side panel: My Issues and All Issues. Each tab includes a list of unique questions and various status indicators:
My Issues - this is the default tab when Review dialog is opened.
Question - list of unique questions. The list of questions are pre-filtered based on Status = Failed, Assignee = current user, Enrichment = Pending, and In Progress.
Last Accessed By - username of the person who most recently asked the question.
Last Accessed On - timestamp for most recent time the question was asked.
Enrichment Status - the status of the current enrichment process. Options include: Not Required, Pending, In Progress, In Review, Completed.
Error - to indicate the Success or Failed status of the most recent question, using Yes (for Error), and No (for No Error).
Feedback - indicators to show how often thumbs up/down have been used for the question.
Asked - count to show how many times the question has been asked by any user.
All Issues
Question - list of unique questions. The list of questions is pre-filtered based on Status = Failed, Assignee = current user, Enrichment = Pending, and In Progress.
Last Accessed On - timestamp for most recent time the question was asked.
Assignee - username for the user assigned to deal with the feedback.
Enrichment Status - the status of the current enrichment process. Options include: Not Required, Pending, In Progress, In Review, Completed.
Resulted in Error - to indicate the Success or Failed status of the most recent question, using Yes (for Error), and No (for No Error).
Feedback - indicators to show how often thumbs up/down have been used for the question.
Asked - count to show how many times the question has been asked by any user.
The following Search and Filter options are available for easy access to the questions to be reviewed:
Question - search to find a question containing the search term(s).
Hide Linked Questions - ON/OFF Toggle. On is default. When enabled, it hides any question which has been added as a Similar Question for another Question, thereby only showing questions which are a parent for other Similar Questions and Questions which are not yet linked.
Error - Filter on most recent Query Execution Status, using Yes (for Error), and No (for No Error).
Assignee - filter on the user’s name assigned to deal with the feedback. Only shown for All Issues tab. Default is: Unassigned.
Enrichment - filter on the Enrichment status. The default are Pending, In Progress, and In Review.
My Issues | All Issues |
|---|---|
 |  |
Right-side panel
The top section has the following buttons.
View Ontology - click to go to the Ontology View in a new tab to make any updates due to the Feedback provided.
View in Easy Answers - click to execute this question in Easy Answers and show the results in a new tab.
Retry in Easy Answers - click to clear the Feedback, execute the question in Easy Answers from LLM, and show the results in a new tab.
Refresh - click to refresh the details of the selected question.
Close - click to close the Review Feedback full-screen popup.
The header section consists of the following.
User Question - the question that was asked.
Resulted in Error - the most recent result of executing the question, eg. Yes (for Error), and No (for No Error).
Most Recent MSO Query - the most recent MSO query generated for the question.
Assignee - select a user from dropdown to be assigned to validate and update ontology due to the feedback.
This action results in a Desktop/Bell icon Notification being sent to the user being assigned to do the enrichment for the current question. Use the link in the Notification message to open the Easy Answers Feedback Review popup on the question assigned to the current user.
Enrichment Status - once the enrichment task is started the user can choose the next relevant status from the dropdown, including Not Required, Pending, In Progress, In Review, Completed. Completed status can only be selected if any Similar Questions added are first resolved and set to Completed status.
This action results in a Desktop/Bell icon Notification being sent to all users who have provided thumbs-down Feedback to the question.
Reopen Question - use the button to reset Enrichment Status to Pending. The button only shows when Enrichment Status is set to Completed.
The following tabs are available on the page:
Question Feedback
This tab includes statistics on the type of Feedback as well as a repeater section on all Feedback related to the selected question, including:
Last Accessed On - timestamp for each Feedback provided.
Feedback By - the username of the user providing the Feedback.
Type - the type of Feedback, such as Thumbs Up / Down, and whether or not the Feedback was the non-technical User Feedback, or the more technical LLM Feedback.
Feedback - the actual Feedback provided from both preselected options or the free-text Feedback. The MSO Query is also provided.
Filters - include the selection of question type and the user who asked the question.
Comments
This tab includes the Comments functionality related to the selected question allowing the user to add comments which will be sent as Notification to other users. The Comments tab label shows a red dot if new Comments have been added.
Similar Questions
This tab includes a repeater section to provide Links to other questions.
+ Add button - opens dialog to allow the user to select a Similar Question.
Questions - questions that a user wants to enrich in the same way as the current parent question.
Has Words - to indicate if a Similar Question has Words suggestions (Yes or No).
Enrichment Status - similar Questions will receive the same Enrichment Status as the main question.
Resulted in Error - the status of the Query Execution Status when it was last used, using Yes (for Error), and No (for No Error).
Options menu:
View in Easy Answers - opens a popup dialog with the details of the question. This dialog includes the content of the right-side panel for the question being selected.
Retry in Easy Answers - clears the Feedback, executes the question in Easy Answers, and shows the results in a new tab.
Show Details - opens up the Similar Question to allow the user to further review.
Reopen Question - reopens the Enrichment Status for the Similar Question. When selected, the Enrichment Status will be reset to Pending. This option is only enabled when Enrichment Status is set to Completed.
Remove - deletes the Similar Question entry.
When the Enrichment Status changes for current question, the following will happen for both current and Similar Questions:
set the Enrichment Status for all Similar Questions to be the same as the current (parent) question.
Send an update notification to all users who provided Feedback on current or any of the Similar Questions. See the example in the Mockups section. The link in the Notification message opens a new tab with Easy Answers with the question being executed.
Notifications will only be sent when Enrichment Status has changed to either “Not Required” or “Completed”. Other types of Enrichment Status will not trigger a Notification.
add a Comment in all impacted Similar Questions: The Enrichment Status for this question has been changed to: [enrichment_status] by: [reviewer_assignee] caused by linked question reference: [question_reference]
Words
This tab includes a table with the columns below. The Words label shows a red dot as decorator if any entries on the Tab have “Pending” as the Status.
Word - a Word requested to be added as a synonym during Easy Answers NLQ.
Type - the Type of request made for the Word, eg. MSO Synonym, MSO Property Synonym, Value-based Synonym, Rule-based Synonym.
Requested by - username of the person requesting the Word.
Status - the Status of the Word being actioned, including Pending, Approved, Rejected.
Action
Options are to either Approve or Reject the Word request.
Info icon - opens a popup dialog showing all details of the Word/Synonym that has been suggested, including MSO, Property, Rule or Value-based Synonym configurations.
