Single-Turn Training Quickstart ↗

    git clone https://github.com/eval-protocol/quickstart-gsm8k.git
    cd quickstart-gsm8k
    pip install -r requirements.txt
    ```

Create the `gsm8k_artifacts/` folder structure and copy files:

```bash
    mkdir -p gsm8k_artifacts/{tests/pytest/gsm8k,development}
    cp evaluation.py gsm8k_artifacts/tests/pytest/gsm8k/test_pytest_math_example.py
    cp gsm8k_sample.jsonl gsm8k_artifacts/development/gsm8k_sample.jsonl
    ```

The repository includes:

* **Evaluator** (`evaluation.py`): Defines how to evaluate math answers
* **Dataset** (`gsm8k_sample.jsonl`): Contains example math problems to test on
  <span class="tab-end"></span>

  <span class="tab-start" data-tab-title="Download files manually"></span>
Install the latest `eval-protocol` SDK, `pytest`, and `requests`:

```bash
    python -m pip install --upgrade pip
    python -m pip install pytest requests git+https://github.com/eval-protocol/python-sdk.git
    ```

Download the evaluator and dataset files:

Run this Python script to download two files from the Eval Protocol repository into a folder on your machine called `gsm8k_artifacts/`.

* **Test script** (`test_pytest_math_example.py`): Defines how to evaluate math answers
* **Sample dataset** (`gsm8k_sample.jsonl`): Contains example math problems to test on

```python
    from pathlib import Path
    import requests

    ARTIFACT_ROOT = Path("gsm8k_artifacts")
    TEST_PATH = ARTIFACT_ROOT / "tests" / "pytest" / "gsm8k" / "test_pytest_math_example.py"
    DATASET_PATH = ARTIFACT_ROOT / "development" / "gsm8k_sample.jsonl"

    files_to_download = {
        TEST_PATH: "https://raw.githubusercontent.com/eval-protocol/python-sdk/main/tests/pytest/gsm8k/test_pytest_math_example.py",
        DATASET_PATH: "https://raw.githubusercontent.com/eval-protocol/python-sdk/main/development/gsm8k_sample.jsonl",
    }

    for local_path, url in files_to_download.items():
        local_path.parent.mkdir(parents=True, exist_ok=True)
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        local_path.write_bytes(response.content)
        print(f"Saved {url} -> {local_path}")
    ```

Expected output:

    ep logs
    ```

This will start a local server, navigate to `http://localhost:8000`. Keep this terminal running.
  <span class="step-end"></span>

  <span class="step-marker" data-step-title="Run the test script"></span>
In a **new terminal**, call the test script to run the evaluator on your dataset of sample math problems.

```bash
    cd gsm8k_artifacts
    ep local-test
    ```

This command discovers and runs your `@evaluation_test` with pytest.

As the test runs, you'll see evaluation scores appear in the browser, with detailed logs for each problem the model attempts. `pytest` will also register your evaluator and dataset with Fireworks automatically, so you can use them in the next step for RFT.
  <span class="step-end"></span>
<span class="steps-end"></span>


  <img src="https://mintcdn.com/fireworksai/XAK4ji8XrlzPoITj/images/fine-tuning/gsm8k-local-eval.jpeg?fit=max&auto=format&n=XAK4ji8XrlzPoITj&q=85&s=e922357a54237c828ae8204fd9c0e5b0" alt="GSM8K evaluation UI showing model scores and trajectories" width="1372" height="932" data-path="images/fine-tuning/gsm8k-local-eval.jpeg" />


## 3. Start training

First, set your Fireworks API key so the Fireworks CLI can authenticate you:

```bash
export FIREWORKS_API_KEY="<your-fireworks-key>"

cd ..
eval-protocol create rft 
    --base-model accounts/fireworks/models/qwen3-0p6b