Build an Agentic RAG with Azure AI Search & Microsoft Foundry—Step-by-Step Guide for Beginners

Introduction:

Unlock the power of Agentic Retrieval-Augmented Generation (RAG) using Azure AI Search and Microsoft Foundry. This step-by-step guide will help you build an intelligent AI agent that can read documents, understand data, and provide accurate answers.

In this tutorial, you will learn how to:

• Create a Microsoft Foundry project
• Deploy an AI model
• Build an AI agent
• Upload documents to Azure Storage
• Configure Azure AI Search
• Connect everything to enable a fully functional RAG system

This guide is perfect for beginners, developers, and cloud engineers who want to build production-ready AI solutions.

1. Create a Microsoft Foundry Resource

Before building your AI chatbot, you need a Microsoft Foundry project. This project acts as the central workspace where all AI components are created and managed.

Steps:

  • Click Create Project
  • Select your:
    • Subscription
    • Resource Group
    • Region (Recommended: East US or Sweden Central)
  • Enter Project Name: my-ai-chatbot
  • Click Create

Step 1:

Screenshot – Create a Microsoft Foundry Resource: Click Create

Step 2:

Screenshot – Create a Microsoft Foundry Resource: Click Create (2)

Step 3:

Screenshot – Create a Microsoft Foundry Resource: Click Create (3)

Step 4:

Screenshot – Create a Microsoft Foundry Resource: Click Create (4)

Output:

Screenshot – Create a Microsoft Foundry Resource: Click Create (5)

Your project is now ready. Think of it as a container that holds all your AI resources

2. Create an Azure AI Agent in Microsoft Foundry

The AI agent is the brain of your application. It uses a deployed model to understand questions and retrieve answers from your knowledge base

Create the Agent

  • Go to Build ? Create Agent
  • Enter an agent name
  • Select the deployed model

Note: By default, Foundry may deploy:

  • gpt-4.1-mini
  • text-embedding-3-small

These help with reasoning and vector search.

Step 1:

Screenshot – Create the Agent: These help with reasoning and vector search.

Step 2:

Screenshot – Create the Agent: These help with reasoning and vector search. (2)

Step 3:

Screenshot – Create the Agent: These help with reasoning and vector search. (3)

Output:

Screenshot – Create the Agent: These help with reasoning and vector search. (4)

By default, Foundry may deploy

Step 5:

Prompt Configuration:

Create a strong system prompt to ensure the AI agent only retrieves answers from your Azure AI Search knowledge base rather than generating unrelated responses.

Example Prompt:

You are an AI assistant whose responses must be grounded only in the information found in the company’s AI search knowledge base, specifically the uploaded documents. Do not use general knowledge, the wider internet, or your own assumptions—only cite, summarize, or synthesize relevant content from the provided document data. If you cannot find an answer in the document data, clearly state that the knowledge base does not contain the requested information

Screenshot – Example Prompt: You are an AI assistant whose responses must be grounded only in the…

Step 6:

Screenshot – Example Prompt: You are an AI assistant whose responses must be grounded only in the… (2)

Example user prompt:

Screenshot – Example user prompt

3. Create an Azure Storage Account:

Azure Storage is used to store your documents such as PDFs, Word files, and text files. These documents will later be indexed and converted into searchable knowledge.

Step 1:

Screenshot – Create an Azure Storage Account: Azure Storage is used to store your documents such as PDFs, Word files, and…

Step 2:

Screenshot – Create an Azure Storage Account: Azure Storage is used to store your documents such as PDFs, Word files, and… (2)

Step 3:

Screenshot – Create an Azure Storage Account: Azure Storage is used to store your documents such as PDFs, Word files, and… (3)

Step 4:

Screenshot – Create an Azure Storage Account: Azure Storage is used to store your documents such as PDFs, Word files, and… (4)

Step 5:

Screenshot – Create an Azure Storage Account: Azure Storage is used to store your documents such as PDFs, Word files, and… (5)

Step 6:

Storage Account Ready ? Now you can upload files.

Screenshot – Create an Azure Storage Account: Storage Account Ready ? Now you can upload files.

Step 7:

Screenshot – Create an Azure Storage Account: Storage Account Ready ? Now you can upload files. (2)

4. Create Azure AI Search Resource

Azure AI Search enables fast and intelligent document retrieval. It converts your files into indexed data so the AI agent can quickly find relevant information

Steps:

  • Create Azure AI Search
  • Choose pricing tier (Basic is enough for testing)
  • Waiting for deployment

Step 1:

Screenshot – Create Azure AI Search Resource: Waiting for deployment

Step 2:

Screenshot – Create Azure AI Search Resource: Waiting for deployment (2)

Step 3:

Screenshot – Create Azure AI Search Resource: Waiting for deployment (3)

The search service was created successfully.

5. Import Data and Set Up RAG

Now it’s time to connect your storage to Azure AI Search and prepare your Retrieval-Augmented Generation (RAG) pipeline

Process Overview:

  1. Connect Storage Account
  2. Select the document container
  3. Enable vector search
  4. Create an index
  5. Configure embeddings
  6. Run the indexer

Step 1:

Screenshot – Import Data and Set Up RAG: Run the indexer

Step 2:

Screenshot – Import Data and Set Up RAG: Run the indexer (2)

Step 3:

Screenshot – Import Data and Set Up RAG: Run the indexer (3)

Step 4:

Screenshot – Import Data and Set Up RAG: Run the indexer (4)

Step 5:

Screenshot – Import Data and Set Up RAG: Run the indexer (5)

Step 6:

Screenshot – Import Data and Set Up RAG: Run the indexer (6)

Step 7:

Screenshot – Import Data and Set Up RAG: Run the indexer (7)

Step 8:

Screenshot – Import Data and Set Up RAG: Run the indexer (8)

Output:

Screenshot – Import Data and Set Up RAG: Run the indexer (9)

Your knowledge base is now ready.

6. Complete the RAG Setup in Microsoft Foundry

Finally, connect Azure AI Search to your AI agent so it can retrieve answers directly from your indexed documents.

Process Overview:

Open your agent.

  Add Knowledge Source

  Select Azure AI Search

  Test with sample queries

Step: 1

Screenshot – Process Overview: Step: 1

Step 2:

Screenshot – Process Overview: Step: 1 (2)

Step 3:

Screenshot – Process Overview: Step: 1 (3)

Step 4:

Screenshot – Process Overview: Step: 1 (4)

Step 7:

Screenshot – Process Overview: Step: 1 (5)

Step 8:

Screenshot – Process Overview: Step: 1 (6)

Outputs:

Screenshot – Outputs
Screenshot – Outputs (2)
Screenshot – Outputs (3)

Get new Microsoft 365 and Azure guides from Azure365Pro first in Google.

Premnath
Premnath
I have been working as Cloud engineer in System Administration field in Microsoft Azure, Microsoft Office 365 / Exchange Servers / Endpoint Manager / Azure Active Directory (IAM) / PowerShell

Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here