'/%3e%3cpath%20id='cross_2'%20data-name='cross%202'%20class='cls-2'%20d='M20.612,30.43h0a5.361,5.361,0,1,0-10.723,0H0A33.709,33.709,0,0,1,30.569,0V9.75a5.361,5.361,0,1,1,0,10.723V30.43Z'%20transform='translate(67.193%200.007)%20rotate(90)'/%3e%3cpath%20id='cross_3'%20data-name='cross%203'%20class='cls-3'%20d='M20.612,30.43h0a5.361,5.361,0,1,0-10.723,0H0A33.709,33.709,0,0,1,30.569,0V9.75a5.361,5.361,0,1,1,0,10.723V30.43Z'%20transform='translate(67.186%2067.041)%20rotate(180)'/%3e%3cpath%20id='cross_4'%20data-name='cross%204'%20class='cls-4'%20d='M20.6,30.5h0a5.361,5.361,0,1,0-10.723,0H0A33.71,33.71,0,0,1,30.56,0V9.818a5.361,5.361,0,1,1,0,10.723V30.5Z'%20transform='translate(0%2067.041)%20rotate(-90)'/%3e%3c/svg%3e)
Preprocessing Training Data
Convert and clean raw data (XML, HTML, JSON, TXT) to fit the training purpose (PT/SFT), and connect it directly to the Training tab.
Specifying the Source Folder
Specify the path of the source folder containing archive files (.zip/.tar.gz) or raw files (XML, HTML, JSON, TXT, etc.). The default path is /@team/.AppData/fine_tuning/raw.
Selecting the Target Training Stage
Select the purpose for which the data will be converted.
| Stage | Description |
|---|---|
| PT — Pre-training raw corpus | Uses the entire set of converted MD files as a domain corpus for pre-training |
| SFT — Instruction dataset (Alpaca JSONL) | Converts data into an instruction dataset in instruction/input/output format |
| DPO | Not supported yet |
In PT mode, no separate output file is required. Once conversion is complete, .md files are created inside the source folder, and the source folder path is connected directly to the Training tab.
When you select SFT, Final output file path (JSONL) appears. If you leave it empty, the file is saved as dataset.jsonl in the source folder.
Raw Formats Contained in the Source Folder
For reference, this shows which formats of raw files are mixed in the source folder. You can choose among KIPRIS XML (Korean patents), HTML (web documents), JSON (structured data), and TXT/MD (plain text); this selection only affects the pipeline description below. The actual server-side processing batch-converts every file in the folder.
Running the Pipeline
PT runs in two stages and SFT in three. Each stage can be run individually or all at once in a batch.
- Automatic archive extraction: Automatically unpacks .zip/.tar.gz files in the source folder and lists the files.
- Document → Markdown conversion: Batch-converts document files (XML, HTML, JSON, etc.) in the source folder to Markdown. The converted .md files are created in the source folder.
- Alpaca JSONL generation (for SFT): Shown only when SFT is selected. Converts .md/.txt files into Alpaca JSONL with the instruction/input/output structure.
Under Run Full Pipeline in Batch, select Run in Batch to run all stages in sequence. When it finishes, select Connect to Training Tab to pass the dataset to the Training tab.