RESTRICTED BOLTZMANN MACHINES FOR IMAGE TAG SUGGESTION

RESTRICTED BOLTZMANN MACHINES FOR IMAGE TAGSUGGESTION Michal Hradiš, Jiří Král Vysoké učení technické v Brně, Fakulta informačních technologií v Brně Božetěchova 2, 612 66 Brno zemcik,ihradis,herout@fit.vutbr.cz 5.3.2012

Image tag suggestion • Upload images to web repository • System suggests tags for the images • User picks from suggested tags Záhlaví (99.99.9999)

forest outdoor person horse BOW singing Záhlaví (99.99.9999)

RBM – RestrictedBoltzmanMachine v1= P(horse) h h3 h2 h1 v3 v2 v1 v P(horse) P(dancing) P(person) Záhlaví (99.99.9999)

P(h1) = sig(w1 v + bh1) h3 h2 h1 v3 v2 v1 Záhlaví (99.99.9999)

P(h) = sig(w v + bh) h3 h2 h1 v3 v2 v1 Záhlaví (99.99.9999)

P(v1) = sig(w’1 h + bv1) h3 h2 h1 v3 v2 v1 Záhlaví (99.99.9999)

P(v) = sig(w’ h + bv) h3 h2 h1 v3 v2 v1 Záhlaví (99.99.9999)

Gibbs sampling • Algorithm • Start with random values of v • Compute P(h) • Take sample from P(h) • Compute P(v) • (Take sample from P(v)) • Continue … • Fix known variables in v h v Záhlaví (99.99.9999)

CRBM h c v P(v) = sig(w’ h + bv), bv=Ac+ba P(h) = sig(w v + bh), bh=Bc + bb Záhlaví (99.99.9999)

Semanticdetectors - TRECVID data • 340 classes, 400 hours of video, 18M shot-level annotation • outdoors, waterscape, mountains, door, singing, people, cars, dancing, politician, woman, children, face close-up, football • 340 classes x 8 bins per histogram = 2720 features Michal Hradis et al.:Semantic Class Detectors in Video Genre Recognition

Záhlaví (99.99.9999)

RESTRICTED BOLTZMANN MACHINES FOR IMAGE TAG SUGGESTION