Asosiy qismga o'tish

Baholash: MOTA, IDF1 va HOTA

16-bob 16 kod bloki 9175 so'z

Bu bobning kodi: kod/16_baholash.py — kitob papkasidan ishga tushiring: python kod/16_baholash.py

Uchta son yetmaydi

9–14-boblarda tracker larni uchta son bilan solishtirdik: ID lar soni, ID almashishi va topilgan GT box lar ulushi. Ular tushunarli edi, lekin uchta kamchiligi bor.

  1. Ular bir-biriga qarshi tortadi. 10-bobda ishonch chegarasini 0.4 ga ko'targanimizda ID almashishi 101 dan 73 ga tushdi, topilgan GT esa 71.8% dan 69.7% ga kamaydi. Bu yaxshilanishmi yoki yomonlashish?
  2. Ortiqcha box lar sanalmadi. Tracker bo'sh devorga yoki mashinaga box chizsa ham, uchala son o'zgarmaydi.
  3. Hammasi bitta videoda. Xulosalarning deyarli hammasi MOT17-09 ning 525 kadridan chiqdi — qo'zg'almas kamera, bitta ko'cha, 26 odam.

Tracking da bu muammolar uchun uchta standart metrika bor: MOTA (Bernardin va Stiefelhagen, 2008), IDF1 (Ristani va boshqalar, 2016) va HOTA (Luiten va boshqalar, 2021). MOTChallenge reytingi va yangi tracker maqolalari shu uchtasini beradi. Bu bobda ularni noldan yozamiz, MOTChallenge ning rasmiy baholovchisi TrackEval bilan bir xil natija berishini tekshiramiz va shu paytgacha qurgan 11 ta tracker ni MOT17 ning 7 ta videosida qayta baholaymiz. Asosiy savol: 9–12-boblardagi "yaxshilanishlar" dan qaysilari haqiqiy edi?

Yetti video

MOT17 train qismida 7 ta video bor (9-bob). Hozircha box larni faqat ikkitasi uchun saqlagan edik (9- va 11-boblar). Qolgan beshtasi uchun ham xuddi shunday qilamiz: YOLO11n, 1280 o'lcham, ishonch ≥ 0.1, bir marta.

kod/16_baholash.pyyangi fayl · 1-qator: from kitob_yordam import fig, show
python52 qator
import configparser
import contextlib
import inspect
import io
from pathlib import Path

import cv2
import numpy as np
import pandas as pd
import torch
import torchvision
import matplotlib.pyplot as plt
from scipy.optimize import linear_sum_assignment
from ultralytics import YOLO
from ultralytics.engine.results import Boxes
from ultralytics.trackers import BOTSORT, BYTETracker
from ultralytics.trackers.byte_tracker import STrack
from ultralytics.trackers.utils.kalman_filter import KalmanFilterXYWH
from ultralytics.utils import LOGGER, YAML, IterableSimpleNamespace
from ultralytics.utils.checks import check_yaml

LOGGER.setLevel("ERROR")
MOT = Path("datasets/MOT17/train")
VIDEOLAR = ["MOT17-02", "MOT17-04", "MOT17-05", "MOT17-09", "MOT17-10", "MOT17-11", "MOT17-13"]
KAMERA = {"MOT17-05": "harakatda", "MOT17-10": "harakatda", "MOT17-11": "harakatda", "MOT17-13": "harakatda"}

def seqinfo(video):
    ini = configparser.ConfigParser()
    ini.read(MOT / f"{video}-FRCNN" / "seqinfo.ini")
    s = ini["Sequence"]
    return int(s["seqLength"]), int(s["frameRate"]), (int(s["imHeight"]), int(s["imWidth"]))

model = YOLO("weights/yolo11n.pt")
for video in VIDEOLAR:
    fayl = Path(f"out/mot17/{video}_yolo11n_1280.txt")
    if fayl.exists():                                                # 09 va 10 oldingi boblarda saqlangan
        continue
    qatorlar = []
    for k, yol in enumerate(sorted((MOT / f"{video}-FRCNN" / "img1").iterdir()), 1):
        b = model.predict(yol, classes=[0], conf=0.1, imgsz=1280, device=0, verbose=False)[0].boxes
        for x in np.hstack([b.xyxy.cpu().numpy(), b.conf.cpu().numpy()[:, None]]):
            qatorlar.append(f"{k},-1,{x[0]:.1f},{x[1]:.1f},{x[2] - x[0]:.1f},{x[3] - x[1]:.1f},{x[4]:.4f}")
    fayl.write_text("\n".join(qatorlar) + "\n")

qatorlar = []
for video in VIDEOLAR:
    n, fps, (h, w) = seqinfo(video)
    gt = np.loadtxt(MOT / f"{video}-FRCNN" / "gt" / "gt.txt", delimiter=",")
    d = np.loadtxt(f"out/mot17/{video}_yolo11n_1280.txt", delimiter=",")
    qatorlar.append({"video": video, "kadrlar": n, "fps": fps, "o'lcham": f"{w}x{h}", "kamera": KAMERA.get(video, "qo'zg'almas"),
                     "GT piyoda box": int((gt[:, 7] == 1).sum()), "YOLO11n box": len(d)})
show(pd.DataFrame(qatorlar))
videokadrlarfpso'lchamkameraGT piyoda boxYOLO11n box
MOT17-02600301920x1080qo'zg'almas18 58121 571
MOT17-041 050301920x1080qo'zg'almas47 55751 116
MOT17-0583714640x480harakatda6 91710 580
MOT17-09525301920x1080qo'zg'almas5 3258 301
MOT17-10654301920x1080harakatda12 83916 775
MOT17-11900301920x1080harakatda9 43612 945
MOT17-13750251920x1080harakatda11 64214 985

Videolar har xil: to'rttasida kamera harakatda (qo'lda yoki mashinada), MOT17-05 esa kichik (640x480) va sekund iga atigi 14 kadr. MOT17-04 ning o'zi GT box larning 40% dan ko'prog'ini beradi — 7 videoni birlashtirgan natijada u eng katta og'irlikka ega bo'ladi.

Kim hisobga olinadi: MOT17 qoidalari

gt.txt da faqat piyodalar emas, 12 xil sinf bor (9-bob). Baholashdan oldin qaysi box "javob" ekanini aniqlash kerak.

kod/16_baholash.pydavomi
python10 qator
SINF = {1: "piyoda", 2: "transportdagi odam", 3: "mashina", 4: "velosiped", 5: "mototsikl", 6: "motorsiz transport",
        7: "turgan odam", 8: "chalg'ituvchi", 9: "to'suvchi", 10: "yerdagi to'suvchi", 11: "to'liq to'suvchi", 12: "aks (oyna)"}
DISTRAKTOR = [2, 7, 8, 12]                       # transportdagi odam, turgan odam, chalg'ituvchi, aks
jadval, belgi = {}, []
for video in VIDEOLAR:
    gt = np.loadtxt(MOT / f"{video}-FRCNN" / "gt" / "gt.txt", delimiter=",")
    jadval[video] = {SINF[s]: int((gt[:, 7] == s).sum()) for s in SINF}
    belgi.append(np.array_equal(gt[:, 6] == 1, gt[:, 7] == 1))
show(pd.DataFrame(jadval))
print("6-ustun (hisobga olinadi) = 1 aynan piyodalarda, 7 videoning hammasida:", all(belgi))
Chiqish
6-ustun (hisobga olinadi) = 1 aynan piyodalarda, 7 videoning hammasida: True
indexMOT17-02MOT17-04MOT17-05MOT17-09MOT17-10MOT17-11MOT17-13
piyoda18 58147 5576 9175 32512 8399 43611 642
transportdagi odam1 54903150000
mashina01 05019602504 918
velosiped1 55911 550315000103
mototsikl01 05000000
motorsiz transport00110000
turgan odam5 2714 79805141 376820
chalg'ituvchi1 2000241 5754703224
to'suvchi1 84302351 05005960
yerdagi to'suvchi023 100002 74002 733
to'liq to'suvchi018 9000000680
aks (oyna)0001 9470181122

MOTChallenge qoidasi uch qadamdan iborat:

  1. GT — faqat piyodalar. MOT17 train da 6-ustundagi "hisobga olinadi" belgisi aynan piyoda sinfida 1, boshqa hamma sinfda 0 — buni hozir tekshirdik.
  2. Distraktorlar kechiriladi. Tracker box i IoU ≥ 0.5 bilan transportdagi odam, turgan odam, chalg'ituvchi yoki oynadagi aks ga tushsa, u box baholashdan butunlay chiqariladi — na to'g'ri, na xato. Detector u yerda haqiqatan odamni ko'rgan; uni kuzatish kerakmi-yo'qmi — munozarali, shuning uchun jazolanmaydi. 14-bobda zonadagi ortiqcha sanashlarning sababi aynan shunday box lar edi: MOT17-09 da 1 947 ta aks va 514 ta turgan odam box i bor.
  3. Qolgan hamma narsa — soxta (FP). Mashina, velosiped yoki to'suvchi ustidagi box, bo'sh joydagi box.

11 ta tracker, 7 ta video

Endi 9–12-boblardagi hamma tracker ni 7 ta videoda ishga tushiramiz. 10-bobdagi mini-SORT ni qayta ko'chiramiz, faqat bitta farq bilan: o'zimiz yozgan Kalman o'rniga ultralytics ning KalmanFilterXYWH i — 10-bobda ular 1e-12 aniqlikda bir xil ekanini ko'rgan edik.

kod/16_baholash.pydavomi
python56 qator
def iou_matritsa(a, b):
    return torchvision.ops.box_iou(torch.tensor(np.asarray(a)[:, :4], dtype=torch.float64),
                                   torch.tensor(np.asarray(b)[:, :4], dtype=torch.float64)).numpy()

def xyxy_to_xywh(b):
    return np.array([(b[0] + b[2]) / 2, (b[1] + b[3]) / 2, b[2] - b[0], b[3] - b[1]])

def xywh_to_xyxy(x):
    return np.array([x[0] - x[2] / 2, x[1] - x[3] / 2, x[0] + x[2] / 2, x[1] + x[3] / 2])

kf = KalmanFilterXYWH()

class Trek:
    def __init__(self, tid, box):
        self.id, self.box, self.tasdiq, self.yoqolgan = tid, box, 1, 0
        self.m, self.P = kf.initiate(xyxy_to_xywh(box))

def mini_sort(det, kalman=True, venger=True, max_yoqolish=30, min_tasdiq=3, min_ishonch=0.25, min_iou=0.3):
    """10-bobdagi mini-SORT."""
    treklar, natija, oxirgi_id = [], [], 0
    for k in sorted(det):
        D = det[k][det[k][:, 4] >= min_ishonch]
        if kalman:
            for t in treklar:
                t.m, t.P = kf.predict(t.m, t.P)
        taxmin = [xywh_to_xyxy(t.m[:4]) if kalman else t.box for t in treklar]
        juftlar = []
        if treklar and len(D):
            M = iou_matritsa(taxmin, D)
            if venger:
                r, c = linear_sum_assignment(1 - M)
                juftlar = [(i, j) for i, j in zip(r, c) if M[i, j] >= min_iou]
            else:
                band_t, band_d = set(), set()
                for i, j in sorted(zip(*np.nonzero(M >= min_iou)), key=lambda juft: -M[juft]):
                    if i not in band_t and j not in band_d:
                        juftlar.append((i, j))
                        band_t.add(i)
                        band_d.add(j)
        for i, j in juftlar:
            t = treklar[i]
            t.m, t.P = kf.update(t.m, t.P, xyxy_to_xywh(D[j]))
            t.box, t.tasdiq, t.yoqolgan = D[j, :4], t.tasdiq + 1, 0
        ulangan_t, ulangan_d = {i for i, _ in juftlar}, {j for _, j in juftlar}
        for i, t in enumerate(treklar):
            if i not in ulangan_t:
                t.yoqolgan += 1
        treklar = [t for t in treklar if t.yoqolgan <= max_yoqolish]
        for j in range(len(D)):
            if j not in ulangan_d:
                oxirgi_id += 1
                treklar.append(Trek(oxirgi_id, D[j, :4]))
        for t in treklar:
            if t.yoqolgan == 0 and t.tasdiq >= min_tasdiq:
                natija.append((k, t.id, *(xywh_to_xyxy(t.m[:4]) if kalman else t.box)))
    return np.array(natija)

Ultralytics tracker lari 11-bobdagi kabi tayyor box lar bilan yuritiladi. Endi video ning haqiqiy fps i beriladi (MOT17-05 da 14, MOT17-13 da 25): ByteTrack yo'qolgan trek ni necha kadr kutishini shunga qarab hisoblaydi. Har bir tracker natijasi MOTChallenge formatidagi faylga yoziladi — kadr, id, x, y, w, h, ishonch, -1, -1, -1 — rasmiy baholovchi aynan shu formatni o'qiydi.

kod/16_baholash.pydavomi
python61 qator
def tracker_sozlama(yaml_nomi, **ozgartirish):
    s = YAML.load(check_yaml(yaml_nomi))
    s.update(ozgartirish)
    return IterableSimpleNamespace(**s)

def ultralytics_tracker(det, sozlama, video):
    n, fps, olcham = seqinfo(video)
    klass = BOTSORT if sozlama.tracker_type == "botsort" else BYTETracker
    klass.reset_id()
    tracker = klass(args=sozlama, frame_rate=fps)
    yollar = sorted((MOT / f"{video}-FRCNN" / "img1").iterdir())
    natija = []
    for k in sorted(det):
        kadr = cv2.imread(str(yollar[k - 1])) if sozlama.tracker_type == "botsort" else None   # GMC va ReID kadrni ko'radi
        treklar = tracker.update(Boxes(np.c_[det[k], np.zeros(len(det[k]))], olcham), kadr)
        natija += [(k, int(t[4]), *t[:4]) for t in treklar]
    return np.array(natija)

def yukla(video):
    d = np.loadtxt(f"out/mot17/{video}_yolo11n_1280.txt", delimiter=",")
    return {k: np.c_[d[d[:, 0] == k][:, 2:4], d[d[:, 0] == k][:, 2:4] + d[d[:, 0] == k][:, 4:6], d[d[:, 0] == k][:, 6]]
            for k in range(1, seqinfo(video)[0] + 1)}                    # kadr -> (n, 5): x1 y1 x2 y2 ishonch

BAHO = Path("out/mot17/baho")

def oqi(fayl):
    a = np.loadtxt(fayl, delimiter=",", ndmin=2)
    return np.c_[a[:, :2], a[:, 2:4], a[:, 2:4] + a[:, 4:6]]           # (kadr, id, x1, y1, x2, y2)

def natija_fayli(papka, video, yarat):
    """Tracker natijasi MOTChallenge formatida. Fayl bo'lsa qayta hisoblamaydi."""
    fayl = BAHO / papka / "data" / f"{video}-FRCNN.txt"
    if not fayl.exists():
        fayl.parent.mkdir(parents=True, exist_ok=True)
        fayl.write_text("".join(f"{int(q[0])},{int(q[1])},{q[2]:.2f},{q[3]:.2f},{q[4] - q[2]:.2f},{q[5] - q[3]:.2f},1,-1,-1,-1\n"
                                for q in yarat()))
    return oqi(fayl)

TRACKERLAR = {"9-bob: oddiy IoU tracker": dict(kalman=False, venger=False, max_yoqolish=0, min_tasdiq=1),
              "+ Venger ulash": dict(kalman=False, venger=True, max_yoqolish=0, min_tasdiq=1),
              "+ Kalman bashorati": dict(kalman=True, venger=True, max_yoqolish=0, min_tasdiq=1),
              "+ xotira: 30 kadr": dict(kalman=True, venger=True, max_yoqolish=30, min_tasdiq=1),
              "+ tasdiq: 3 kadr": dict(kalman=True, venger=True, max_yoqolish=30, min_tasdiq=3),
              "+ ishonch 0.4 (mini-SORT)": dict(kalman=True, venger=True, max_yoqolish=30, min_tasdiq=3, min_ishonch=0.4),
              "ByteTrack": tracker_sozlama("bytetrack.yaml"),
              "ByteTrack, ikkinchi bosqichsiz": tracker_sozlama("bytetrack.yaml", track_low_thresh=0.25),
              "BoT-SORT, GMC siz": tracker_sozlama("botsort.yaml", gmc_method="none"),
              "BoT-SORT, GMC bilan": tracker_sozlama("botsort.yaml"),
              "BoT-SORT, GMC + ReID": tracker_sozlama("botsort.yaml", with_reid=True, model="weights/yolo26n-cls.pt")}
PAPKA = {nom: f"t{i:02d}" for i, nom in enumerate(TRACKERLAR)}

def ishga_tushir(sozlama, det, video):
    return mini_sort(det, **sozlama) if isinstance(sozlama, dict) else ultralytics_tracker(det, sozlama, video)

treklar = {}
for video in VIDEOLAR:
    det = yukla(video)
    for nom, s in TRACKERLAR.items():
        treklar[nom, video] = natija_fayli(PAPKA[nom], video, lambda: ishga_tushir(s, det, video))
print("MOT17-09, mini-SORT bosqichlaridagi ID lar (10-bob jadvali bilan solishtiring):",
      [len(np.unique(treklar[nom, "MOT17-09"][:, 1])) for nom in list(TRACKERLAR)[:6]])
Chiqish
MOT17-09, mini-SORT bosqichlaridagi ID lar (10-bob jadvali bilan solishtiring): [483, 480, 483, 196, 135, 90]

ID lar soni 10-bobdagi jadval bilan bir xil — ko'chirilgan kod o'sha natijani beradi. Birinchi ishga tushirish o'n daqiqalab vaqt oladi: BoT-SORT har kadrni diskdan o'qiydi, ReID esa har bir box uchun tasniflash modelini ishlatadi. Keyingi safar fayllar tayyor bo'ladi.

Har kadrda moslash

Uchala metrika ham bitta tayyorgarlikdan boshlanadi: har kadrda GT box lar va tracker box lari orasida IoU matritsasi. Yuqoridagi uch qoidani shu yerda qo'llaymiz: distraktorga Venger bilan (IoU ≥ 0.5) juftlangan tracker box larini chiqaramiz va faqat piyoda GT ni qoldiramiz. Keyingi hisoblar matritsa indekslari bilan ishlashi uchun ID larni 0, 1, 2, ... ga o'tkazamiz.

kod/16_baholash.pydavomi
python23 qator
def tayyorla(video, trek):
    """Qaytaradi: har kadr uchun (GT ID lar, tracker ID lar, IoU matritsa). ID lar 0 dan ketma-ket."""
    g = np.loadtxt(MOT / f"{video}-FRCNN" / "gt" / "gt.txt", delimiter=",")
    gt = np.c_[g[:, :2], g[:, 2:4], g[:, 2:4] + g[:, 4:6], g[:, 7]]     # kadr, id, x1, y1, x2, y2, sinf
    kadrlar = []
    for k in range(1, seqinfo(video)[0] + 1):
        G, T = gt[gt[:, 0] == k], trek[trek[:, 0] == k]
        S = iou_matritsa(G[:, 2:6], T[:, 2:6]) if len(G) and len(T) else np.zeros((len(G), len(T)))
        if len(G) and len(T):
            r, c = linear_sum_assignment(-np.where(S >= 0.5, S, 0))
            juft = S[r, c] >= 0.5
            olib = c[juft][np.isin(G[r[juft], 6], DISTRAKTOR)]           # distraktorga tushgan tracker box lari
            T, S = np.delete(T, olib, 0), np.delete(S, olib, 1)
        piyoda = G[:, 6] == 1
        kadrlar.append((G[piyoda, 1].astype(int), T[:, 1].astype(int), S[piyoda]))
    gi = np.unique(np.concatenate([k[0] for k in kadrlar]))
    ti = np.unique(np.concatenate([k[1] for k in kadrlar]))
    return [(np.searchsorted(gi, g), np.searchsorted(ti, t), S) for g, t, S in kadrlar]

tayyor = {v: tayyorla(v, treklar["ByteTrack", v]) for v in VIDEOLAR}
show(pd.DataFrame([{"video": v, "ByteTrack box lari": len(treklar["ByteTrack", v]),
                    "distraktorga tushib chiqarildi": len(treklar["ByteTrack", v]) - sum(len(t) for _, t, _ in tayyor[v])}
                   for v in VIDEOLAR]))
videoByteTrack box laridistraktorga tushib chiqarildi
MOT17-0210 8211 196
MOT17-0437 6071 387
MOT17-055 4217
MOT17-095 324818
MOT17-109 117634
MOT17-117 832112
MOT17-136 31176

MOT17-09 da ByteTrack box larining taxminan 15% i distraktorga — oynadagi aksga, turgan yoki chalg'ituvchi odamga — tushgan. Bu qoidasiz ular soxta box bo'lib sanalardi va tracker "detector to'g'ri ko'rgan narsa" uchun jazolanardi.

MOTA: xatolarni sanash

MOTA (Multiple Object Tracking Accuracy) — CLEAR-MOT metrikalaridan biri. Har kadrda GT va tracker box larini moslaymiz, keyin uch xil xatoni sanaymiz: topilmagan GT (FN), ortiqcha box (FP) va ID almashishi (IDSW). Ularning yig'indisi GT box lar soniga bo'linadi:

Moslashda bitta nozik joy bor. Oldingi kadrda GT odam A tracker ID 7 bilan juftlangan bo'lsa va hozir ham ular IoU ≥ 0.5 bilan ustma-ust tursa, bu juftlik saqlanadi — hatto boshqa box ning IoU si biroz kattaroq bo'lsa ham. Aks holda ikki odam yaqinlashganda tracker aybsiz bo'lsa ham ID almashishi sanalardi. TrackEval buni narxga katta bonus (1000) qo'shish bilan qiladi — Venger algoritmi avval eski juftliklarni saqlaydi, qolganini IoU bo'yicha tanlaydi. 9–12-boblardagi id_tahlili ham shu qoidaga amal qilgan, lekin ochko'z usulda edi.

ID almashishi: GT odamga oxirgi marta boshqa tracker ID si berilgan bo'lsa. "Oxirgi marta" — oldingi kadrda emas: odam 20 kadr yashirinib, boshqa ID bilan qaytsa, bu ham almashish. MOTP esa moslangan juftliklarning o'rtacha IoU si — box larning aniqligi.

kod/16_baholash.pydavomi
python28 qator
def clear_mot(kadrlar):
    tp = fn = fp = idsw = 0
    iou_yigindi = 0.0
    oxirgi, oldingi_kadr = {}, {}                  # GT -> oxirgi ulangan tracker ID; GT -> oldingi kadrdagi tracker ID
    for g, t, S in kadrlar:
        if not len(g) or not len(t):
            fn, fp = fn + len(g), fp + len(t)
            continue
        eski = np.array([oldingi_kadr.get(a, -1) for a in g])[:, None] == t[None]
        ball = np.where(S >= 0.5, S + 1000 * eski, 0)
        r, c = linear_sum_assignment(-ball)
        r, c = r[ball[r, c] > 0], c[ball[r, c] > 0]
        for a, b in zip(g[r], t[c]):
            idsw += a in oxirgi and oxirgi[a] != b
            oxirgi[a] = b
        oldingi_kadr = dict(zip(g[r], t[c]))
        tp, fn, fp = tp + len(r), fn + len(g) - len(r), fp + len(t) - len(r)
        iou_yigindi += S[r, c].sum()
    return {"TP": tp, "FN": fn, "FP": fp, "IDSW": idsw, "IoU": iou_yigindi}

def jami(sanoqlar):
    return {k: sum(s[k] for s in sanoqlar) for k in sanoqlar[0]}

for nom, s in (("MOT17-09", clear_mot(tayyor["MOT17-09"])), ("7 video", jami([clear_mot(tayyor[v]) for v in VIDEOLAR]))):
    gt_soni = s["TP"] + s["FN"]
    print(f"ByteTrack, {nom}: GT {gt_soni} | FN {s['FN']} ({s['FN'] / gt_soni:.1%}) | FP {s['FP']} ({s['FP'] / gt_soni:.1%}) | "
          f"IDSW {s['IDSW']} ({s['IDSW'] / gt_soni:.2%}) | MOTA {1 - (s['FN'] + s['FP'] + s['IDSW']) / gt_soni:.1%} | "
          f"MOTP {s['IoU'] / s['TP']:.3f}")
Chiqish
ByteTrack, MOT17-09: GT 5325 | FN 1455 (27.3%) | FP 636 (11.9%) | IDSW 64 (1.20%) | MOTA 59.5% | MOTP 0.798
ByteTrack, 7 video: GT 112297 | FN 48713 (43.4%) | FP 14619 (13.0%) | IDSW 901 (0.80%) | MOTA 42.8% | MOTP 0.789

MOTA ning eng katta zaifligi shu yerda ko'rinadi: 7 videoda ID almashishi MOTA dan atigi 0.8 ball oladi, topilmagan GT esa 43 ball. MOTA — amalda detection metrikasi: tracker ID larni qanchalik yomon aralashtirsa ham, bu sonni deyarli sezmaydi. Bundan tashqari, ID almashishi bir marta sanaladi — odam 1 soniyadan keyin to'g'ri ID ga qaytsa ham, butun video davomida noto'g'ri ID bilan yursa ham.

IDF1: kim kim bilan qoldi

IDF1 savolni boshqacha qo'yadi: "Butun video davomida har bir haqiqiy odamga bitta tracker ID si mos qo'yilsa, kadrlarning qancha qismida to'g'ri ID turibdi?" Buning uchun kadrlar bo'yicha emas, butun trek lar bo'yicha bitta moslash qilinadi:

  1. Har bir (GT odam, tracker ID) juftligi uchun ular IoU ≥ 0.5 bilan ustma-ust turgan kadrlar sonini sanaymiz.
  2. Venger algoritmi bilan GT odamlar va tracker ID lari orasida birga-bir moslash topamiz — shu kadrlar yig'indisi eng katta bo'ladigan qilib. Bu yig'indi — IDTP.
  3. Qolgan hamma GT box — IDFN, qolgan hamma tracker box i — IDFP.

Bu yerda N_GT va N_tr — GT va tracker box larining umumiy soni. Tracker bir odamni ikki ID ga bo'lsa, ulardan faqat bittasi moslashadi, ikkinchisining kadrlari xato bo'ladi. Shuning uchun IDF1 uzun, uzilmagan trek larni mukofotlaydi.

kod/16_baholash.pydavomi
python17 qator
def sonlar(kadrlar):
    """Har bir GT va tracker ID necha kadrda bor."""
    return (np.bincount(np.concatenate([k[0] for k in kadrlar])).astype(float),
            np.bincount(np.concatenate([k[1] for k in kadrlar])).astype(float))

def idf1_sanoq(kadrlar):
    n_g, n_t = sonlar(kadrlar)
    M = np.zeros((len(n_g), len(n_t)))                  # M[g, t]: birga turgan kadrlar soni
    for g, t, S in kadrlar:
        a, b = np.nonzero(S >= 0.5)
        M[g[a], t[b]] += 1
    r, c = linear_sum_assignment(-M)
    return {"IDTP": M[r, c].sum(), "GT": n_g.sum(), "TR": n_t.sum()}

for nom, s in (("MOT17-09", idf1_sanoq(tayyor["MOT17-09"])), ("7 video", jami([idf1_sanoq(tayyor[v]) for v in VIDEOLAR]))):
    print(f"ByteTrack, {nom}: IDTP {s['IDTP']:.0f} | GT box {s['GT']:.0f} | tracker box {s['TR']:.0f} | "
          f"IDF1 {2 * s['IDTP'] / (s['GT'] + s['TR']):.1%}")
Chiqish
ByteTrack, MOT17-09: IDTP 2831 | GT box 5325 | tracker box 4506 | IDF1 57.6%
ByteTrack, 7 video: IDTP 46601 | GT box 112297 | tracker box 78203 | IDF1 48.9%

HOTA: aniqlash va bog'lash — alohida

MOTA detection ga, IDF1 esa bog'lashga (association) og'ib ketadi. HOTA (Higher Order Tracking Accuracy) ikkalasini alohida o'lchaydi va keyin teng og'irlik bilan birlashtiradi:

  • DetA — aniqlash sifati: moslangan box lar ulushi. MOTA dan farqi — ID almashishi yo'q, faqat TP, FN, FP.
  • AssA — bog'lash sifati. Har bir moslangan juftlik c uchun (GT odam g, tracker ID t) so'raymiz: g ning butun hayoti va t ning butun hayoti qanchalik ustma-ust? TPA — g va t birga moslangan kadrlar, FNA — g boshqa ID bilan moslangan yoki topilmagan kadrlar, FPA — t boshqa odamga yoki hech kimga tushgan kadrlar.

Uchinchi farq — IoU chegarasi. MOTA va IDF1 bitta chegara (0.5) bilan ishlaydi. HOTA 19 ta chegarada (α = 0.05, 0.10, ..., 0.95) hisoblanadi va o'rtachasi olinadi: box lari aniqroq tracker yuqori α larda ham moslashadi va ko'proq ball oladi.

Moslash ham boshqacha: har kadrda Venger narxi IoU emas, IoU × "juftlikning butun video bo'yicha o'xshashligi". Kadrda bir xil IoU li ikki nomzod bo'lsa, butun video davomida shu odam bilan ko'proq birga yurgan ID tanlanadi.

kod/16_baholash.pydavomi
python36 qator
ALFA = np.arange(0.05, 0.99, 0.05)                      # 19 ta chegara

def hota_sanoq(kadrlar):
    n_g, n_t = sonlar(kadrlar)
    P = np.zeros((len(n_g), len(n_t)))                  # 1) juftlikning butun video bo'yicha o'xshashligi
    for g, t, S in kadrlar:
        maxraj = S.sum(0)[None] + S.sum(1)[:, None] - S
        P[np.ix_(g, t)] += np.divide(S, maxraj, out=np.zeros_like(S), where=maxraj > 0)
    umumiy = P / (n_g[:, None] + n_t[None] - P)
    TP, FN, FP = np.zeros(len(ALFA)), np.zeros(len(ALFA)), np.zeros(len(ALFA))
    C = np.zeros((len(ALFA), len(n_g), len(n_t)))       # 2) har alfa uchun: juftlik necha kadrda moslandi (TPA)
    for g, t, S in kadrlar:
        if not len(g) or not len(t):
            FN, FP = FN + len(g), FP + len(t)
            continue
        r, c = linear_sum_assignment(-umumiy[np.ix_(g, t)] * S)
        for i, alfa in enumerate(ALFA):
            m = S[r, c] >= alfa
            TP[i], FN[i], FP[i] = TP[i] + m.sum(), FN[i] + len(g) - m.sum(), FP[i] + len(t) - m.sum()
            C[i, g[r[m]], t[c[m]]] += 1
    A = C / np.maximum(1, n_g[:, None] + n_t[None] - C)  # TPA / (TPA + FNA + FPA)
    return {"H_TP": TP, "H_FN": FN, "H_FP": FP, "AssA_TP": (C * A).sum((1, 2))}

def sanoq(video, trek):
    k = tayyorla(video, trek)
    return {**clear_mot(k), **idf1_sanoq(k), **hota_sanoq(k)}

def yakuniy(s):
    """Sanoqlardan metrikalar (%). Bir nechta video: avval sanoqlar qo'shiladi (jami), keyin shu funksiya."""
    DetA = s["H_TP"] / np.maximum(1, s["H_TP"] + s["H_FN"] + s["H_FP"])
    AssA = s["AssA_TP"] / np.maximum(1, s["H_TP"])
    return {"HOTA": np.sqrt(DetA * AssA).mean() * 100, "DetA": DetA.mean() * 100, "AssA": AssA.mean() * 100,
            "MOTA": (1 - (s["FN"] + s["FP"] + s["IDSW"]) / (s["TP"] + s["FN"])) * 100, "MOTP": s["IoU"] / s["TP"] * 100,
            "IDF1": 2 * s["IDTP"] / (s["GT"] + s["TR"]) * 100, "IDSW": s["IDSW"], "FP": s["FP"], "FN": s["FN"]}

show(pd.DataFrame([{"ByteTrack": "MOT17-09", **yakuniy(sanoq("MOT17-09", treklar["ByteTrack", "MOT17-09"]))}]).round(2))
ByteTrackHOTADetAAssAMOTAMOTPIDF1IDSWFPFN
MOT17-0947.1153.8641.559.5379.7657.59646361 455

Bir nechta videoni birlashtirishda metrikalarning o'rtachasi olinmaydi: avval sanoqlar (TP, FN, FP, IDSW, IDTP, ...) qo'shiladi, keyin metrika hisoblanadi. Shunda katta video kattaroq og'irlikka ega bo'ladi. AssA uchun esa har bir video ning AssA si uning TP soni bilan tortiladi — AssA_TP aynan shu ko'paytmani saqlaydi.

Rasmiy baholovchi bilan tekshiruv

O'zimiz yozgan metrikalar rasmiy natija bilan bir xilmi? TrackEval — HOTA mualliflarining kodi, MOTChallenge sayti ham shuni ishlatadi. O'rnatamiz — PyPI dagi trackeval paketi boshqa odamning fork idan yig'ilgan va matplotlib, opencv ni yangilab yuboradi, shuning uchun asl repodan, aniq commit bilan, qo'shimcha paketlarsiz (--no-deps: unga faqat numpy va scipy kerak, ular bor):

Terminal
bash1 qator
pip install --no-deps "trackeval @ git+https://github.com/JonathonLuiten/TrackEval.git@12c8791b303e0a0b50f753af204249e622d0281a"

TrackEval ning asl kodi hali np.float va np.int ni ishlatadi — ular numpy 1.24 da olib tashlangan. Import dan oldin ikkalasini qaytarib qo'yamiz, ular shunchaki Python ning float va int i edi.

kod/16_baholash.pydavomi
python25 qator
np.float, np.int = float, int                          # TrackEval eski numpy nomlarini ishlatadi
import trackeval

bizniki = {(nom, v): sanoq(v, treklar[nom, v]) for nom in TRACKERLAR for v in VIDEOLAR}

with contextlib.redirect_stdout(io.StringIO()):         # TrackEval ning jarayon xabarlari kerak emas
    baholovchi = trackeval.Evaluator({"USE_PARALLEL": False, "PRINT_RESULTS": False, "PRINT_CONFIG": False, "TIME_PROGRESS": False,
                                      "OUTPUT_SUMMARY": False, "OUTPUT_DETAILED": False, "PLOT_CURVES": False})
    dataset = trackeval.datasets.MotChallenge2DBox({"GT_FOLDER": str(MOT), "TRACKERS_FOLDER": str(BAHO), "BENCHMARK": "MOT17",
                                                    "SKIP_SPLIT_FOL": True, "TRACKERS_TO_EVAL": list(PAPKA.values()),
                                                    "SEQ_INFO": {f"{v}-FRCNN": None for v in VIDEOLAR}, "PRINT_CONFIG": False})
    rasmiy, _ = baholovchi.evaluate([dataset], [trackeval.metrics.HOTA(), trackeval.metrics.CLEAR(), trackeval.metrics.Identity()])

farq = {}
for nom, papka in PAPKA.items():
    for v in VIDEOLAR + ["7 video"]:
        r = rasmiy["MotChallenge2DBox"][papka]["COMBINED_SEQ" if v == "7 video" else f"{v}-FRCNN"]["pedestrian"]
        s = yakuniy(jami([bizniki[nom, x] for x in VIDEOLAR]) if v == "7 video" else bizniki[nom, v])
        te = {"HOTA": r["HOTA"]["HOTA"].mean() * 100, "DetA": r["HOTA"]["DetA"].mean() * 100, "AssA": r["HOTA"]["AssA"].mean() * 100,
              "MOTA": r["CLEAR"]["MOTA"] * 100, "MOTP": r["CLEAR"]["MOTP"] * 100, "IDF1": r["Identity"]["IDF1"] * 100,
              "IDSW": r["CLEAR"]["IDSW"], "FP": r["CLEAR"]["CLR_FP"], "FN": r["CLEAR"]["CLR_FN"]}
        for m in te:
            farq[m] = max(farq.get(m, 0), abs(te[m] - s[m]))
print(f"{len(PAPKA)} tracker x {len(VIDEOLAR) + 1} (har video + 7 video birga) ta natija solishtirildi")
print("bizniki va TrackEval orasidagi eng katta farq:", {m: float(f"{x:.1e}") for m, x in farq.items()})
Chiqish
11 tracker x 8 (har video + 7 video birga) ta natija solishtirildi
bizniki va TrackEval orasidagi eng katta farq: {'HOTA': 0.0, 'DetA': 0.0, 'AssA': 7.1e-15, 'MOTA': 1.4e-14, 'MOTP': 2.8e-14, 'IDF1': 0.0, 'IDSW': 0.0, 'FP': 0.0, 'FN': 0.0}

Farq nol yoki 1e-14 atrofida — bu kompyuterning kasr sonlarni yaxlitlashi, mantiq farqi emas. Endi o'z kodimizga to'liq ishonishimiz mumkin: u rasmiy baholovchi bilan bir xil sonni beradi va har bir qadami ko'z oldimizda.

Qaysi yaxshilanish haqiqiy edi

kod/16_baholash.pydavomi
python2 qator
jadval = pd.DataFrame([{"tracker": nom, **yakuniy(jami([bizniki[nom, v] for v in VIDEOLAR]))} for nom in TRACKERLAR])
show(jadval.drop(columns="MOTP").round(2))
trackerHOTADetAAssAMOTAIDF1IDSWFPFN
9-bob: oddiy IoU tracker36.3542.4432.0638.9737.893 41117 01848 107
+ Venger ulash36.4342.4432.1939.0238.083 34817 02048 109
+ Kalman bashorati36.8842.532.9239.438.893 10716 92848 020
+ xotira: 30 kadr40.242.4638.940.345.282 10916 91648 013
+ tasdiq: 3 kadr40.3642.3439.2542.1345.881 60013 69949 687
+ ishonch 0.4 (mini-SORT)40.0739.1941.3542.4746.968086 39657 399
ByteTrack41.8742.4942.0442.848.9290114 61948 713
ByteTrack, ikkinchi bosqichsiz41.7241.5942.5243.2349.3699111 16551 600
BoT-SORT, GMC siz42.1642.8842.1943.1149.1486414 73248 288
BoT-SORT, GMC bilan43.3943.5243.9743.6551.2572015 68946 869
BoT-SORT, GMC + ReID43.7743.3344.9741.1851.461 04119 75045 263

Jadvalni 9–12-boblar tartibida o'qiymiz (7 video birga):

  • Venger ulash (10-bob) — deyarli hech narsa: HOTA 36.35 dan 36.43 ga. 10-bobda ham MOT17-09 da farq yo'q edi; endi 7 videoda tasdiqlandi.
  • Kalman bashorati — kichik, lekin haqiqiy: AssA 32.19 dan 32.92 ga, ID almashishi 3 348 dan 3 107 ga.
  • Xotira (30 kadr) — eng katta bitta qadam: HOTA +3.3, IDF1 38.89 dan 45.28 ga, AssA 32.92 dan 38.90 ga. 10-bobdagi eski sonlarda bu qadam ID almashishini deyarli o'zgartirmagan edi (113 dan 111 ga) — faqat ID lar soni kamaygan edi. Eski o'lchov uning foydasini ko'rsatmagan.
  • Tasdiq (3 kadr) — soxta box larni 16 916 dan 13 699 ga kamaytirdi, MOTA +1.8. HOTA deyarli o'zgarmadi.
  • Ishonch 0.4 — metrikalar kelisha olmaydi. MOTA va IDF1 biroz o'sdi (ortiqcha box lar ikki barobardan ko'proq kamaydi), HOTA esa pasaydi: DetA 42.34 dan 39.19 ga tushdi, chunki haqiqiy odamlarning ham 7.7 mingga yaqin box i yo'qotildi (FN). Bobning boshidagi savolga javob: bu nimani o'lchashingizga bog'liq. Bobning oxirida bunga qaytamiz.
  • ByteTrack — mini-SORT ning ikkala variantidan ham uchala metrikada yaxshi (11-bobdagi xulosa tasdiqlandi). Ikkinchi bosqich esa deyarli neytral. 11-bobda u recall va ID barqarorligini oshiradi degan edik: 7 videoda DetA 41.59 dan 42.49 ga o'sdi va ID almashishi 991 dan 901 ga kamaydi — bu to'g'ri. Lekin qo'shilgan box larning bir qismi soxta, shuning uchun MOTA va IDF1 0.4 ga pasaydi.
  • GMC — HOTA +1.2, IDF1 +2.1, ID almashishi 864 dan 720 ga. Buni har video bo'yicha ko'rish kerak (keyingi jadval).
  • ReID (12-bob) — aralash: AssA 43.97 dan 44.97 ga o'sdi, HOTA +0.4, lekin MOTA 43.65 dan 41.18 ga tushdi — soxta box lar 4 mingdan ko'proq ko'paydi, ID almashishi ham 720 dan 1 041 ga oshdi.

Eng muhim kuzatish: ustunlarga qarang. DetA ishonch chegarasini ko'targan qatordan tashqari hamma tracker da 41.6 dan 43.5 gacha, AssA esa 32 dan 45 gacha o'zgaradi. Tracker aniqlashni deyarli o'zgartirmaydi — box larni detector beradi. Tracker ning ishi bog'lash, va HOTA ning AssA qismi aynan shuni ko'rsatadi.

Har bir video alohida:

kod/16_baholash.pydavomi
python2 qator
tanlangan = ["9-bob: oddiy IoU tracker", "+ ishonch 0.4 (mini-SORT)", "ByteTrack", "BoT-SORT, GMC siz", "BoT-SORT, GMC bilan", "BoT-SORT, GMC + ReID"]
show(pd.DataFrame([{"HOTA": nom, **{v: yakuniy(bizniki[nom, v])["HOTA"] for v in VIDEOLAR}} for nom in tanlangan]).round(1))
HOTAMOT17-02MOT17-04MOT17-05MOT17-09MOT17-10MOT17-11MOT17-13
9-bob: oddiy IoU tracker26.340.232.242.529.447.427.7
+ ishonch 0.4 (mini-SORT)30.24434.545.432.950.433.3
ByteTrack31.445.936.447.136.449.837.1
BoT-SORT, GMC siz32.346.538.245.535.149.836.9
BoT-SORT, GMC bilan32.746.638.7454249.741.5
BoT-SORT, GMC + ReID31.847.938.747.940.548.841.8
  • GMC ning foydasi deyarli butunlay ikki videodan keladi: MOT17-10 (35.1 dan 42.0 ga) va MOT17-13 (36.9 dan 41.5 ga) — ikkalasida kamera tez harakatlanadi (11-bobda MOT17-10 da kadriga o'nlab piksel siljishni ko'rgan edik). Qo'zg'almas kamerali 02, 04, 09 da farq ±0.5 ichida. Xulosa: GMC ni kamera harakatlanadigan joyda yoqing; qo'zg'almas kamerada u faqat vaqt oladi.
  • ReID — 7 videodan uchtasida yaxshi (04, 09, 13), uchtasida yomon (02, 10, 11), bittasida bir xil. Jami +0.4 HOTA ni "yaxshilanish" deb bo'lmaydi — bu videolarga qarab u yoki bu tomonga og'adigan farq. 12-bobdagi xulosa o'z kuchida: tayyor tasniflash modeli bilan ReID bepul yutuq emas.
  • Bitta videodagi farq aldashi mumkin. Faqat MOT17-09 ga qarasak, ReID HOTA ni 45.0 dan 47.9 ga oshiradi — "ReID ni yoqish kerak" degan xulosa chiqardik. MOT17-10 da esa u 42.0 dan 40.5 ga tushiradi.
  • ByteTrack va GMC siz BoT-SORT — 11-bobdagi "deyarli teng" xulosasi 7 videoda ham to'g'ri: har biri uchta videoda yutadi, MOT17-11 da teng.

HOTA ichidan

HOTA ni bitta son sifatida ko'rsatish qulay, lekin uning ichidagi qismlar ko'proq narsa aytadi. Chapda — ByteTrack ning har α dagi DetA, AssA va HOTA si; o'ngda — 11 ta tracker ning DetA va AssA si.

kod/16_baholash.pydavomi
python18 qator
_, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 4.8))
s = jami([bizniki["ByteTrack", v] for v in VIDEOLAR])
det_a = s["H_TP"] / (s["H_TP"] + s["H_FN"] + s["H_FP"])
ass_a = s["AssA_TP"] / s["H_TP"]
for y, nom in ((det_a, "DetA"), (ass_a, "AssA"), (np.sqrt(det_a * ass_a), "HOTA")):
    ax1.plot(ALFA, y * 100, marker="o", ms=3, label=nom)
ax1.axvline(0.5, color="gray", lw=0.8, ls="--")
ax1.set_xlabel("IoU chegarasi α")
ax1.set_ylabel("%")
ax1.set_title("ByteTrack, 7 video", fontsize=10)
ax1.legend(fontsize=8)
for i, (nom, q) in enumerate(jadval.set_index("tracker").iterrows(), 1):
    ax2.scatter(q["DetA"], q["AssA"], color="C0" if i <= 6 else "C1")
    ax2.annotate(str(i), (q["DetA"], q["AssA"]), xytext=(5, 2) if i % 2 == 0 else (-12, 2), textcoords="offset points", fontsize=9)
ax2.set_xlabel("DetA, %")
ax2.set_ylabel("AssA, %")
ax2.set_title("1–6: mini-SORT bosqichlari, 7–11: ultralytics (jadval tartibida)", fontsize=10)
fig("hota_ichidan.png", "Chapda: IoU chegarasi oshgan sari aniqlash va bog'lash sifati tushadi; HOTA — 19 nuqtaning o'rtachasi. O'ngda: tracker lar asosan AssA bo'yicha farq qiladi.")

Chapda: IoU chegarasi oshgan sari aniqlash va bog'lash sifati tushadi; HOTA — 19 nuqtaning o'rtachasi. O'ngda: tracker lar asosan AssA bo'yicha farq qiladi.

Chapda: IoU chegarasi oshgan sari aniqlash va bog'lash sifati tushadi; HOTA — 19 nuqtaning o'rtachasi. O'ngda: tracker lar asosan AssA bo'yicha farq qiladi.

Chapdagi egri chiziq: DetA α = 0.5 gacha sekin, keyin keskin tushadi — α = 0.95 da box GT bilan deyarli piksel-piksel mos kelishi kerak, bunga detector juda kam holatda erishadi. Shuning uchun faqat α = 0.5 dagi son box aniqligini ko'rmaydi, HOTA esa uni ham hisobga oladi. O'ngdagi rasm yuqoridagi xulosani ko'rsatadi: nuqtalar asosan vertikal, AssA bo'yicha tarqalgan, DetA bo'yicha esa bir-biriga yaqin. 6-nuqta (ishonch 0.4) eng chapda — u ko'p haqiqiy odamni tashlab yuborgan; 8-nuqta (ikkinchi bosqichsiz ByteTrack) ham biroz chapda — u past ishonchli box larni ishlatmaydi.

Detector mi, tracker mi?

Tracker lar orasidagi eng katta farq 7 HOTA ball (36.35 dan 43.77 gacha). Detector ni almashtirsak-chi? Chegaraviy tajriba: tracker ga YOLO11n box lari o'rniga GT box larining o'zini beramiz — xatosiz "ideal detector".

kod/16_baholash.pydavomi
python14 qator
ideal = {"ByteTrack": [], "oddiy IoU tracker": []}
for video in VIDEOLAR:
    g = np.loadtxt(MOT / f"{video}-FRCNN" / "gt" / "gt.txt", delimiter=",")
    p = g[g[:, 7] == 1]
    det = {k: np.c_[p[p[:, 0] == k][:, 2:4], p[p[:, 0] == k][:, 2:4] + p[p[:, 0] == k][:, 4:6], np.ones((p[:, 0] == k).sum())]
           for k in range(1, seqinfo(video)[0] + 1)}
    ideal["ByteTrack"].append(sanoq(video, natija_fayli("ideal_bytetrack", video,
                                                        lambda: ultralytics_tracker(det, tracker_sozlama("bytetrack.yaml"), video))))
    ideal["oddiy IoU tracker"].append(sanoq(video, natija_fayli("ideal_oddiy", video,
                                                                lambda: mini_sort(det, **TRACKERLAR["9-bob: oddiy IoU tracker"]))))
show(pd.DataFrame([{"tracker": "ByteTrack", "detector": "YOLO11n", **yakuniy(jami([bizniki["ByteTrack", v] for v in VIDEOLAR]))},
                   {"tracker": "ByteTrack", "detector": "ideal (GT box)", **yakuniy(jami(ideal["ByteTrack"]))},
                   {"tracker": "oddiy IoU tracker", "detector": "YOLO11n", **yakuniy(jami([bizniki["9-bob: oddiy IoU tracker", v] for v in VIDEOLAR]))},
                   {"tracker": "oddiy IoU tracker", "detector": "ideal (GT box)", **yakuniy(jami(ideal["oddiy IoU tracker"]))}]).round(2))
trackerdetectorHOTADetAAssAMOTAMOTPIDF1IDSWFPFN
ByteTrackYOLO11n41.8742.4942.0442.878.8748.9290114 61948 713
ByteTrackideal (GT box)93.7494.9692.6199.3895.4796.721457547
oddiy IoU trackerYOLO11n36.3542.4432.0638.9779.3637.893 41117 01848 107
oddiy IoU trackerideal (GT box)96.7399.5394.029999.7795.991 11355

Ideal detector bilan ikkala tracker ham HOTA da 90 dan oshadi — YOLO11n bilan 36–42 edi. Detector xatolari tracker lar orasidagi farqdan bir necha barobar katta. Tracking dagi murakkablik — Kalman, xotira, ikki bosqich, ReID — asosan detector ning xatolarini yamash uchun kerak.

Kutilmagan natija: ideal box larda oddiy IoU tracker HOTA bo'yicha ByteTrack dan yuqori, garchi ID ni ko'proq almashtirsa ham. Sababi MOTP ustunida: oddiy tracker GT box ni o'zgarishsiz qaytaradi (MOTP 99.77), ByteTrack esa Kalman bilan silliqlangan box ni (95.47). Yuqori α larda silliqlangan box lar moslashmay qoladi. Ikkinchi sabab ultralytics kodida:

kod/16_baholash.pydavomi
python2 qator
print("\n".join(q.strip() for q in inspect.getsource(STrack.activate).splitlines() if "frame_id == 1" in q or "self.is_activated" in q))
print("\n".join(q.strip() for q in inspect.getsource(BYTETracker.update).splitlines() if "if x.is_activated" in q))
Chiqish
if frame_id == 1:
self.is_activated = True
return np.asarray([x.result for x in self.tracked_stracks if x.is_activated], dtype=np.float32)

Yangi trek faqat 1-kadrda darhol faollashadi, keyingi kadrlarda esa ikkinchi marta ko'ringanda. Faol bo'lmagan trek natijaga chiqmaydi — shuning uchun 1-kadrdan keyin paydo bo'lgan har bir odamning birinchi kadri natijada yo'q. Jadvaldagi FN ning katta qismi shu. Real detector bilan bu to'g'ri qaror (bitta kadrlik soxta box lar trek ochmaydi), ideal box larda esa ortiqcha ehtiyotkorlik.

Ishonch chegarasi: uch metrika, uch javob

Bobning boshidagi savolga qaytamiz: mini-SORT uchun eng yaxshi ishonch chegarasi qaysi? 0.1 dan 0.7 gacha sinaymiz, 7 videoda.

kod/16_baholash.pydavomi
python17 qator
qatorlar = []
for chegara in (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7):
    s = dict(TRACKERLAR["+ ishonch 0.4 (mini-SORT)"], min_ishonch=chegara)
    sanoqlar = [sanoq(v, natija_fayli(f"ishonch_{chegara}", v, lambda: mini_sort(yukla(v), **s))) for v in VIDEOLAR]
    qatorlar.append({"ishonch chegarasi": chegara, **yakuniy(jami(sanoqlar))})
chegaralar = pd.DataFrame(qatorlar)
show(chegaralar.drop(columns="MOTP").round(2))

_, ax = plt.subplots(figsize=(8, 4.5))
for m in ("HOTA", "MOTA", "IDF1"):
    ax.plot(chegaralar["ishonch chegarasi"], chegaralar[m], marker="o", label=m)
    eng = chegaralar.loc[chegaralar[m].idxmax()]
    ax.scatter(eng["ishonch chegarasi"], eng[m], s=120, facecolors="none", edgecolors="black")
ax.set_xlabel("detector ishonch chegarasi")
ax.set_ylabel("%")
ax.legend()
fig("ishonch_chegarasi.png", "mini-SORT, 7 video. Qora doira — har metrikaning eng yuqori nuqtasi: uchala metrika uch xil chegarani tanlaydi.")
ishonch chegarasiHOTADetAAssAMOTAIDF1IDSWFPFN
0.135.4639.3633.1520.6637.593 74545 17740 177
0.239.4242.5137.4339.544.332 05418 90146 990
0.339.9341.4938.9943.1146.011 24510 34852 293
0.440.0739.1941.3542.4746.968086 39657 399
0.539.5836.0543.7239.8447.185364 02662 998
0.638.6332.0246.8235.745.623922 43669 377
0.734.3426.3744.8729.2139.032461 48677 758

mini-SORT, 7 video. Qora doira — har metrikaning eng yuqori nuqtasi: uchala metrika uch xil chegarani tanlaydi.

mini-SORT, 7 video. Qora doira — har metrikaning eng yuqori nuqtasi: uchala metrika uch xil chegarani tanlaydi.

Uchala metrika uch xil javob beradi: MOTA uchun eng yaxshisi 0.3, HOTA uchun 0.4, IDF1 uchun 0.5. Past chegarada soxta box lar MOTA ni yeydi (0.1 da FP 45 mingdan oshadi). Yuqori chegarada detector ko'p haqiqiy odamni yo'qotadi — DetA tushadi, lekin qolgan ishonchli box lar orasida bog'lash osonlashadi, shuning uchun AssA 0.6 gacha o'sadi. IDF1 ning cho'qqisi o'ngroqda: uning maxrajida tracker box lari soni ham bor, shuning uchun kamroq, lekin ishonchliroq va uzilmagan trek lar unga foydaliroq.

Demak, "eng yaxshi tracker" yoki "eng yaxshi sozlama" metrikasiz savol emas. Avval vazifaga mos metrikani tanlang, keyin sozlang.

Mijozga qaysi raqam

Metrika tanlash — biznes savoli. 14-bobdagi vazifalar bo'yicha:

Vazifa Nima muhim Qaysi metrika
Kirish-chiqishni sanash har odam bir marta sanalsin, soxta box kam bo'lsin DetA, MOTA; chiziqda hodisalarni GT bilan juftlash (14-bob)
Band bo'lganlik, issiqlik xaritasi har kadrda odam bor-yo'qligi DetA, MOTA (ID muhim emas)
Kutish vaqti, xaridor yo'li bir odam — bitta uzun trek IDF1, AssA
"Kaskasiz" signal soxta signal kam, lekin odamni o'tkazib yubormaslik detection precision va recall, trek darajasida tasdiq
Umumiy solishtirish, reyting hammasi birga HOTA

Amaliy qoida: mijozga MOT17 dagi son emas, uning o'z kamerasidagi son kerak. Uning kamerasidan 3–5 daqiqalik video oling, uni belgilang (masalan, CVAT dasturida — u MOT formatiga eksport qiladi) va shu bobdagi kodni ishga tushiring. Bir necha soatlik belgilash ishi keyin oylab bahslashishdan saqlaydi: "tizim 90% aniq" emas, "sizning kirish eshigingizda IDF1 = ..., ma'nosi shu".

Bizning YOLO11n bilan MOT17 train dagi HOTA ~42–44 bilan maqolalardagi sonlarni solishtirmang: masalan, ByteTrack maqolasi MOT17 test da HOTA 63.1 beradi, lekin u yerda detector — MOT17 va boshqa piyoda datasetlarida o'qitilgan YOLOX-X. Yuqorida ko'rdik: farqning asosiy qismi detector da.

Asosiy fikr

  • ID lar soni va ID almashishi yolg'iz aldaydi: ular ortiqcha box larni ko'rmaydi va bir-biriga qarshi tortadi. MOTA, IDF1 va HOTA — tracking ning standart o'lchovlari, ularni noldan yozdik va rasmiy TrackEval bilan 1e-14 aniqlikda bir xil ekanini tekshirdik.
  • MOTA — amalda detection metrikasi (ByteTrack da ID almashishi MOTA dan 1 balldan kam oladi). IDF1 — butun trek lar bo'yicha moslash, ID saqlanishini o'lchaydi. HOTA aniqlash (DetA) va bog'lashni (AssA) alohida o'lchaydi va 19 ta IoU chegarasida o'rtachasini oladi.
  • 7 videoda: eng katta haqiqiy yaxshilanish — trek xotirasi; GMC faqat harakatdagi kamerada foyda beradi; ReID va ikkinchi bosqich — neytral yoki aralash. Bitta videoda chiqarilgan xulosa boshqa videolarda teskari bo'lishi mumkin.
  • Tracker lar asosan AssA bo'yicha farq qiladi, DetA ni detector belgilaydi. Ideal detector bilan hatto eng oddiy tracker ham HOTA > 90 oladi: sifatni oshirishning birinchi yo'li — detector.

Mashqlar

  1. HOTA o'rniga faqat α = 0.5 dagi qiymatni (ALFA[9]) hisoblang. Ideal detector tajribasida ByteTrack va oddiy IoU tracker orasidagi tartib o'zgaradimi? Nega?
  2. ByteTrack ning track_buffer parametrini 30 dan 60 va 90 ga o'zgartiring. 7 videoda IDF1 va AssA qanday o'zgaradi? MOTA-chi?
  3. CLEAR-MOT ning yana ikki metrikasini qo'shing: MT (trekining ≥ 80% kadrida topilgan GT odamlar) va ML (≤ 20%). Natijani TrackEval ning r["CLEAR"]["MT"] va r["CLEAR"]["ML"] bilan solishtiring.
  4. MOT17 box larini YOLO11n o'rniga kattaroq model (masalan, yolo11s.pt) bilan qayta hisoblang va ByteTrack ni baholang. Detector ni almashtirish HOTA ni tracker ni almashtirishdan ko'proq oshiradimi?
  5. 15-bobdagi poldagi tracker ni shu bobning funksiyalari bilan baholang. tayyorla da IoU o'rniga masofadan o'xshashlik oling: S = max(0, 1 − d / 100 sm) — shunda S ≥ 0.5 "50 sm ichida" degani. clear_mot, idf1_sanoq va hota_sanoq o'zgarishsiz ishlaydi. Bitta kamera va uch kamera, tezlik bilan va tezliksiz — qaysi biri HOTA va IDF1 bo'yicha yaxshi? 15-bobdagi "ko'proq topish, ko'proq almashish" savoliga javob bering.

Bu bobni bepul o'qidingiz.

Kitob o'rgatadi, lekin sizning kodingiz nega ishlamaganini ayta olmaydi. Kursda uy vazifangiz tekshiriladi.

Telegramda yozish Narxlar