This post follows on from our previous post: Use Power Automate to link your Power Virtual Agent to Azure OpenAI and enterprise documents. As such, prerequisites are:
A working MS Teams Chatbot (Power Virtual Agent) connected to Azure OpenAI ChatGPT and Azure Cognitive Search via Power Automate.
Optional/helpful:
Comfortable with Power Automate and regular expressions within Power Automate processes.
We want to mimic, as closely as possible, the references that are displayed in Azure OpenAI's chat playground. Within the Teams interface, the goal is to modify the response received as follows, so that the [doc_n] references provide value to users. The references list displays a list of the names of documents, hyperlinked with their location on the Azure Storage Account (which will work if access is permitted, or if public view is enabled), and then a suffix that refers to the relative position in that document.
Currently for PDFs, the URLs do not link directly to a location within the document; however still serve some value to help locate relative positions within a PDF.
The n in a [doc_n] reference refers to the index of the relevant citation pulled by Azure OpenAI (e.g. it might only utilise one reference from a possible list of 10); hence, [doc2] refers to the second item in the array, but may be the only reference used in a response (as below).
original:
goal:
Logic overview
1. Initially, we were pulling just the 'response body' from Azure OpenAI. Now, we will also be parsing the 'citations' component. Thinking back to the initial Parse_JSON step, this citations component is located at:
body('Parse_overall_response')['choices'][0]['messages'][0]['content']2. This response contains several slots, and we refine the output with a 'Select' action. We use 'filepath', 'url', and 'chunk_id'. We store this in an array variable called 'refined_citations_array'.
3. We will end up hyperlinking the filepath (filename) with the corresponding URL, which doesn’t reference locations within a document, but still links to the file in the storage account. We then use the chunk_id to signify the relative 'part' of the document (as is done in the playground). For example, a chunk_id of 0 (as they are indexed from 0), represents 'Part 1'.
4. We match the positional index of items in the citations array to '[doc1], [doc2], … [doc_n]' references in the main response. For example, [doc2] refers to the second item in the citations array (and within the citations array, the chunk_id reference could be any number that refers to a relative position in the document; e.g. 'chunk_id: 28'. The larger the document, the more chunks there are.
5. We compare the index of items in the citations array to the 'doc ref' indexes, and identify overlap, signifying that a citation was used in the main response body.
6. Then another 'Condition' step comes into play: If there is no overlap, it is implied that no docs are referenced, and only the core response is relayed to PVA.
7. If there is overlap, it is implied that a reference is made, and we use the array of overlapping indexes as references to isolate the relevant citation data.
8. When there is overlap, the citation data is processed to hyperlink the filepaths, add a '- Part_n’ suffix (based on the chunk_id), in a way that is markdown-friendly for PVA. Note that HTTP is not supported, and only certain markdown methods can be used in PVA (in Teams).
Steps
1. Firstly, we need multiple new variables that are used throughout the flow: array variables, string variables, a counter variable that is used for incrementation, and some text variables used for markdown syntax. Since the flow is much longer, we have also removed the 'OAI_response' variable for efficiency purposes, instead focussing on referring directly to outputs within expressions.
The following variables should be added at the top level (via an Initialize Variable action) after the existing HTTP step (that you should have already), with names and types as follows. The names are important as they are referred to directly in expressions later on.
refined_citations_array ; array
citation_index ; array
filtered_array ; array
doc_ref_indexes ; array
overlapping_indexes ; array
newline ; string
Counter ; integer
citation_string ; string
Your top level of the flow should now look something like the image to the right.
2. Then, we have the original Condition step, used to detect errors from the HTTP Post request. If status does not equal 200, then no connection has been established and we reply with a response directly to the PVA by creating a 'Return value(s) to Power Virtual Agents' action with a text response:
3. If the status equals 200, we then proceed with the initial Parse_JSON action (as before), but now with the following schema. Again, if you have issues with this schema, please test the flow and generate a schema from the output of the HTTP response.
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"model": {
"type": "string"
},
"created": {
"type": "integer"
},
"object": {
"type": "string"
},
"choices": {
"type": "array",
"items": {
"type": "object",
"properties": {
"index": {
"type": "integer"
},
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"index": {
"type": "integer"
},
"role": {
"type": "string"
},
"content": {
"type": "string"
},
"end_turn": {
"type": "boolean"
},
"url": {
"type": "string"
}
},
"required": [
"index",
"role",
"content",
"end_turn"
]
}
}
},
"required": [
"index",
"messages"
]
}
}
}
}
4. Now on the left branch, we process the core content response to extract the [doc_n] references, modify them, and append to the 'doc_ref_indexes' array variable. This is done with an 'Apply to each' loop, containing 2 actions: Compose and Append to array variable.
![Initial 'apply to each' loop to process the response body and detect [doc_n] references](https://static.wixstatic.com/media/75b708_db21598990914bc6a8e7b26d110b1e10~mv2.png/v1/fill/w_980,h_811,al_c,q_90,usm_0.66_1.00_0.01,enc_auto/75b708_db21598990914bc6a8e7b26d110b1e10~mv2.png)
4.a. In the 'select output from previous steps' box, enter the following expression.
skip(split(body('Parse_overall_response')['choices'][0]['messages'][1]['content'], '[doc'), 1) This processes the core response, splitting when '[doc' is mentioned and skipping the first portion.
4.b. In the first 'Compose' step called 'isolate numbers, …' use the expression:
sub(int(first(split(item(), ']'))),1)This isolates the numbers from the prior step, converts them from string to int, and subtracts 1 (to match the corresponding item in the citations array, since they index from 0).
4.c. In the second action in the loop, the 'Append to array variable' action, we select the 'doc_ref_indexes' array variable, and set the value as the output from the prior compose step. Each [doc_n] reference is thus processed, isolated for its number, and appended to the array. This works when one or more references are in the response body.
5. In parallel to the above, we have the following to process the citation information itself, which is in a separate item in the overall response array.
6. First, we parse the citations response (which is a string of JSON within the [messages][0] slot from the initial 'parse_JSON' action):
For content, use the expression below. Note the change in message index compared to when we pull the response body.
body('Parse_overall_response')['choices'][0]['messages'][0]['content']For schema, use the following.
{
"type": "object",
"properties": {
"citations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"content": {
"type": "string"
},
"id": {},
"title": {
"type": "string"
},
"filepath": {
"type": "string"
},
"url": {
"type": "string"
},
"metadata": {
"type": "object",
"properties": {
"chunking": {
"type": "string"
}
}
},
"chunk_id": {
"type": "string"
}
},
"required": [
"content",
"id",
"title",
"filepath",
"url",
"metadata",
"chunk_id"
]
}
},
"intent": {
"type": "string"
}
}
}7. We then have a Select action called Select URLs, which isolates the slots from the above schema. This will be used going forward.
Set the dynamic content per the image below (for the 'From' and 'Value' sections) and then type the following keys as well, which correspond to their values.
In the value for the 'chunk_id' key, use the following expression, which converts the chunk_id strings to integers.
int(item()['chunk_id'])You should have something similar to the snippet below. Note, the dynamic content is selected from the output of the prior 'Parse_JSON' action (which we named 'Parse_citations').
8. We then use a 'Set variable' action to send the output from this Select_URLs action to the 'refined_citations_array' array variable:
9. After this, we create an array of the indexes of all the references, setting this to the 'citations_index' array variable, using the following expression in the Value slot:
range(0, length(body('Parse_citations')?['citations']))
10. The image below shows an overview of the steps above. The left branch shows the processing of the citations component and the right branch occurs in parallel, extracting the [doc_n] references from the main response body.
11. Now we can compare the 'doc_ref_indexes' array to the 'citations_index' array, and identify the overlap of indexes. We have written an expression so that if the length of the overlap is 0 (implying none of the citation indexes are used as references in the response), it responds with only the main content. If the length of this overlap != 0, then there are citations to be referenced, and further processing is triggered. The expression in the condition is:
length(intersection(variables('citations_index'), variables('doc_ref_indexes')))
12. In the 'yes' section (implying no overlap), we create a 'Return value(s) to Power Virtual Agents' action and use the following expression to respond with just the main content:
body('Parse_overall_response')['choices'][0]['messages'][1]['content']
The above 2 steps should look like this:
13. As mentioned, if the length is not equal to zero, then further processing is triggered. The overview of this is below (which are 2 'Apply to each' loops, followed by the final 'Return value(s) to Power Virtual Agents' action).
14. In the first loop, we take the overlapping index values and use them as indexes to reference the 'refined_citations_array' and pipe the relevant indexes to the 'filtered_array' variable:
Under 'Select output from previous steps', use the expression:
intersection(variables('citations_index'), variables('doc_ref_indexes'))Under 'Name', Select the 'filtered_array' variable.
Under 'Value' use the expression:
variables('refined_citations_array')[items('Use overlap index to filter citations array')]15. Then, we have a subsequent 'Apply to each' loop that modifies the filtered array:
15.a. The output from previous steps is dynamically set to the 'filtered_array' variable.
15.b. We then have an 'append to string variable' action with the following.
Name: select the ‘citation_string’ var.
For Value, use the following expression.
concat(variables('newline'), add(variables('Counter'), 1), '. ', '[', item()['filepath'], ' - Part ', add(item()['chunk_id'], 1), '](', item()['url'], ')', variables('newline'))This incorporates the 'newline' variable (used to allow newlines with markdown that works in PVA), the 'counter' variable, and the url, filepath and chunk slots. It forms the citation string that will be used in the final response.
We then have an 'increment variable' action, to increase the 'Counter' variable we made. This helps with incrementing the ' - Part_n' suffix in the citation string.
Select the Counter variable for the dropdown.
Type '1' in the Value section.
16. Outside of these loops, we have the final step of providing the response to the PVA (being the combined response of content + citations), using the following expression:
concat(body('Parse_overall_response')['choices'][0]['messages'][1]['content'], variables('newline'), variables('newline'), '**Doc References:**', variables('newline'), variables('citation_string'))
Concluding
You should now have an MS Teams Chatbot that can interact with your enterprise documents, and reference which documents that information was pulled from! If the hyperlinks do not seem to work, it may be due to security configurations in your Azure Storage Account, or user access permissions.
Keep an eye out for updates and optimisations to this Power Automate flow.
Comments