Ingestion & Indexing

Check whether the documents that users uploaded are ready to be used in AI search. The search data created here is used for RAG, in which the AI finds related documents and answers from them.

Before you begin

• Required permission: Workspace administrator
• The AI runtime and model that run embedding must be ready.
• To use vector search, complete the connection and the embedding model settings in Knowledge Storage.

Open Ingestion & Indexing

1. Open the administrator page.
2. Select Knowledge Services in the left menu.
3. Select Ingestion & Indexing.

Knowledge Indexing Overview

Check the current indexing status at a glance.

You can leave Auto refresh on, or update the status manually with the Refresh button.

While the vector DB status is being checked, or when the check fails, Vectors in database shows -. If the server briefly returns a snapshot from an earlier point in time for the same space, the page keeps the more recent value that it last confirmed and shows a stale snapshot warning. When you switch between the team space and a personal space, the values of the previous space are cleared immediately, and unknown values are shown as - until the query for the new space finishes.

Before you start

Before you start indexing, check the required settings below. Select each button to go to the corresponding setting.

Here you only choose the models to use. A vision model and an embedding model can be used for indexing only after you download them in AI model settings.

The AI vector database item checks not only whether the setting exists but also the actual connection status. If the vector DB cannot be reached, Connection error · check settings is shown in the warning color. If the dimension or the distance metric of the collection does not match, Collection mismatch · review needed is shown. If the collection has not been created in Milvus yet, Collection not found · create it first is shown. For a connection error or a collection mismatch, hover over the card to see the detailed error. No separate error detail is shown when a Milvus collection is missing, so select the card to go to the vector DB settings and create the collection by following Connect Milvus. A Qdrant collection that does not exist yet is not an error, so Ready · collection is created on first indexing is shown. For detailed actions, see the vector DB operations guide.

Collection & Processing Rules

Set when search data is created and which folders' documents are used for AI search.

Enable knowledge indexing

Turns the entire knowledge indexing feature on or off.

Indexing Mode Settings

Included roots

Specify the folder paths to index. If left empty, all paths are indexed.

Example:

/@team/docs
/@client/projects

AI Embedding Model Settings

Select the AI model that converts documents into search data.

A provider that has no installed model is shown as No models available.

PDF/image include patterns

Enter the extensions or path rules of the PDF and image files from which text is read.

Example:

*.pdf, *.png, image/**/*

PDF/image exclude patterns

Enter the extensions or path rules of the PDF and image files that are not processed. If left empty, all files included above are processed.

Example:

thumbs.db, *.gif, cache/**

Progress by storage space

Check how much content in each team or personal storage space is waiting to be delivered to the AI.

You can search by space (team/client) and user, and sort by the number of pending refreshes.

AI learning data overview

Check how much AI search data has been created for each model. This is not a feature that trains a new AI model with the source documents.

You can sort by count to compare the amount of search data for each model.