Fine-Tuning AI Models

Fine-Tuning AI Models with Bagel

In this tutorial, you'll discover how to leverage Bagel's APIs to fine-tune AI models.

Step 1: Preparing Your RAW Asset

Before we can fine-tune a model, you need to have your Asset ready within Bagel. If you haven't already, use this code to create your Asset:

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)

const api_key = "insert api key"
const payload = {
    dataset_type: "input dataset type", // e.g "RAW", "VECTOR", "MODEL"
    title: "Input title", // e.g "Apple"
    category: "inpute category", // e.g "Cat1", "Cat2"
    details: "input details", // e.g "testing"
    tags: ["Optional input"], // e.g ["AI", "DEMO", "TEST"]  or [ ]
    user_id: "input user id" // e.g "345281182308180743453"
}

// Calling the create asset method
const createAsset = async () => {
  // get response of created asset
  const res = await client.create_asset(payload, apiKey)
  console.log(res)
}

createAsset()

Create a MODEL

// Define the payload for creating a dataset. Just replace the payload above with this one
const payload = {
    dataset_type: "MODEL",
    title: "Python testing Create MODEL",
    category: "Cat1",
    details: "Testing",
    tags: ["AI", "DEMO", "TEST"],
    user_id: "101481188466180740994"
}

Upload file to RAW Asset

Navigate to your console and select my datasets on the left side of your page. Select the RAW asset you created and navigate to the files tab. Add a .txt file of your choice. You can as well run this code to upload file from your computer:

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)

const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"
const file_path = "input file path" // e.g "product_catalog.csv" or "path/dir/file.csv"

// Calling the file upload method
const fileUpload = async () => {
    const res = await client.add_file(asset_id, file_path, api_key)
    console.log(res)
}

fileUpload()

Step 2: Initiating the Fine-Tuning Process

With our Assets in place, we can now leverage Bagel's fine-tuning capabilities to train a custom model. The POST /api/v1/asset endpoint allows us to initiate a fine-tuning job for a pre-trained model using our Asset:

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: 'rest',
  bagel_server_host: 'api.bageldb.ai',
})

const client = new Client(settings)


const apiKey = 'input your API key', // e.g 'm34282rbS6g86xSqPhi27fjfZqEc2ErkloJ'

const payload = {
  dataset_type: 'MODEL',
  title: 'model1',
  category: 'CAT1',
  details: 'test',
  tags: [],
  user_id: '9a198051-7fae-4e06-81aa-dcfc988d4d0f',
  fine_tune_payload: {
    asset_id: 'bc0f4e4e-d468-4c07-8db1-1fc66873c682', // Move asset_id here
    model_name: 'model1', // Same as the title
    base_model: 'b0cb6e74-5f4a-4af1-ac5e-872ef3962b1c',
    file_name: 'test.txt',
    user_id: '9a198051-7fae-4e06-81aa-dcfc988d4d0f',
  },
}

// Calling the fine tune method
const fineTune = async () => {
  const asset = await client.fine_tune(payload, apiKey)
  console.log(asset)
}

fineTune()

Once the fine-tuning job is started, Bagel will return a asset_idof the fine-tuned model. We can use this to retrieve a job id which will allow us monitor the progress of the fine-tuning process.

Step 3: Monitoring the Fine-Tuning Job - Getting Job by Asset

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: 'rest',
  bagel_server_host: 'api.bageldb.ai',
})

const client = new Client(settings)

const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"

// Calling the get job by asset id method
const getJobByAsset = async () => {
    const job = await client.get_job_by_asset(asset_id, apiKey)
    console.log(job)
}

getJobByAsset()

In this example, we're sending a GET request to the /api/v1/asset/{asset_id} endpoint. The response contains information about the job id.

Step 4: Monitoring the Fine-Tuning Job - Get Job

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: 'rest',
  bagel_server_host: 'api.bageldb.ai',
})

const client = new Client(settings)

job_id = "input job_id" # Replace with the actual job ID. e.g "8566657552882860032" 
api_key = "input API key" # Replace with your actual API key. e.g "67YUtJPByf8qnUihA388378GYsZ766HG"

// Calling the get job method using job id
const getJob = async () => {
  const result = await client.get_job(job_id, apiKey)
  console.log(result)
}

getJob()

In this example, we're sending a GET request to the /api/v1/jobs/{job_id} endpoint, providing the job_id as a path parameter. The response contains information about the job, including its status (e.g., "running", "completed", "failed") and progress (e.g., a percentage value indicating how much of the job has been completed).

You can periodically check the job status and progress until the fine-tuning process is complete.

List Jobs

You can also list all jobs using this code:

import {Settings, Client} from 'bagelml'

server_settings = Settings(
    bagel_api_impl = "rest",
    bagel_server_host = "api.bageldb.ai",
)

const client = new Client(server_settings)


// Replace "your_api_key_here" with the provided API key
const api_key = "input API key" # Replace with your actual API key. e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const user_id = "input your user id" # e.g "231243124466184567909"

// Call the list jobs method
const listJob = async () => {
    const res = await client.list_jobs(user_id, api_key)
    console.log(res)
}

listJob()

Step 5: Accessing the Fine-Tuned Model

Once the fine-tuning job is complete, you can access the fine-tuned model and its associated files using the GET /api/v1/jobs/asset/{asset_id}/files/{file_name}

import { Settings, Client } from 'bagelml'

// Settings config
const settings = new Settings({
  bagel_api_impl: "rest",
  bagel_server_host: "api.bageldb.ai",
})

// Create Bagel client
const client = new Client(settings)


const api_key = "input API key" // e.g "67YUtJPByf8qnUihA388378GYsZ766HG"
const asset_id = "input dataset id" // e.g "eR57yd73b-7gt4-4a71-b0fd-afKJUIY6578Ge"
const file_name = "input file name" // e.g train.txt"

// Calling the download file method
consgt fileDownload = async () => {
    const res = await client.download_model_file(asset_id, file_name, api_key)
    console.log(res)
}

fileDownload()

In this example, we're using the GET /api/v1/jobs/asset/{asset_id}/files/{file_name} endpoint to download a specific file. You can replace file_name with the appropriate file name for your use case.