SOLUTION MANUAL FOR PATTERN RECOGNITION 4TH EDITION

S OLUTION M ANUAL FOR
M M




P ATTERN R ECOGNITION AND M ACHINE
M M M




L EARNING

EDITED BY
M




ZHENGQI GAO m




themStatemKeymLab.mofmASICmandmSystemm
SchoolmofmMicroelectronics
Fudanm University
NOV.2017

, 1


0.1 Introduction

Problemm1.1mSolution

WemletmthemderivativemofmerrormfunctionmEmwithmrespectmtomvectormwme
qualsmtom0,∂w
m(i.e.m
∂Em=m0),mandmthismwillmbemthemsolutionmofmwm=m{w }mwhich
i

mminimizesmerrormfunction mE.mTomsolvemthismproblem,mwemwillmcalculatemthe

mderivativemofmEmwithmrespectmtomevery mwim,mandmletmthemmequal mtom0minste

ad.mBasedmonm(1.1)mandm(1.2)mwemcanmobtainm:
=>
∂E ∑
N
{my(xn ,mw)m−mt }xinm =m0
m
=
∂wi n=1 n
=>
∑
N
m
N
∑m
y(xn,mw)xnim =m nxi
n=1 n=1
m tn
=>
N
∑N
m ∑m
M
∑m
(m w j mnxmj mm)xn im =m n
n=1 j=0 n=1
ximtn
=>
N
∑ m ∑m
N m
∑m
w j x ( mj+i)m =m x
M
im
tn n n
n=1mj=0 n=1
=> Mm m N N
∑m ∑m ∑m
x(mnj+i)wjm =m xim tn
n
j=0mn= n=1
Ifmwem denotem Aimjm=m 1 ∑
xin+mjm andmTim=m N xnimtn,m them equationm abovem can
∑N n=1 n=1

bemwrittenmexactlymasm(1.222),mThereforemthemproblemmismsol
ved.
Problemm1.2mSolution

ThismproblemmismsimilarmtomProb.1.1,mandmthemonlymdifferencemismthemlas
tmtermmonmthemrightmsidemofm(1.4),mthempenaltymterm.mSomwemwillmdomthemsa
memthingmasminmProb.1.1m:

=>
∂E ∑
N
{my(xn ,mw)m−mt }xinm +mλwim=m0
m
=
∂wi n=1 n
=>
∑ m ∑m
Mm m N N
∑m
xn(mj+i)wjm+mλwim=m n
j=0 mn=1 n=1
xim tn
=> M N N
∑m ∑m ∑m
{m x(mnj+i)m+mδ ji λ }w j m=m nximtn
j=0m n= n=1
1

, 2


where
{
0 j ̸=mi
δ ji
1 jm=mi
Problemm1.3mSolution

ThismproblemmcanmbemsolvedmbymBayes’mtheorem.mThemprobabilitymofmsele
ctingmanmapplemP(a)m:
3 1 3
P(a)m =m P(a|r)P(r)+P(a|b)P(b)+P(a|mg)P(g)m =m ×0.2+m ×0.2+m ×0.6m =m0.34
10m 2m 10m
Basedm onm Bayes’m theorem,m them probabilitym ofm anm selectedm orangem co
mingmfrommthemgreenmboxmP(g|o)m:

P(o| mg)P(g)
P(g m| o)=
P(o)
WemcalculatemthemprobabilitymofmselectingmanmorangemP(o)mfirstm:
4 1 3
P(o)m =mP(o|r)P(r)+mP(o|b)P(b)+mP(o|mg)P(g)m =m ×0.2+m ×0.2+m ×0.6m =m0.36
10m 2m 10m
Thereforem wem canm getm :
P(o|mg)P(g)m m3 ×m0.6m
P(g|o)m=m =m =m0.5
10 m
m

P(o) 0.36
Problemm1.4mSolution

Thismproblemmneedsmknowledgemaboutmcalculus,mespeciallymaboutmChain
mrule.m WemcalculatemthemderivativemofmP y ( my)mwithmrespectmtomy,maccordingm
tom(1.27)m:


dp y ( my) d(p x (g( my))| mg ‘ ( my) dp x (g( my)) ‘ d|mg‘(my)|
|)
= = dy (∗)
dy dy |mgm (my)|m+mpx(g(m dy
y))

Themfirstmtermminmthemabovemequationmcanmbemfurthermsimplified:
dp x (g( my)) ‘ dp x (g( my))m dg( my) ‘
|mgm(my)|m= |mgm(my)| (∗∗)
dy dg( my dy m m
)
Ifmxˆm ismthemmaximummofmdensitymovermx,mwemcanmobtainm:

dp x (x)
¯ xˆm =m0
dx
Therefore,mwhenmym=myˆ,ms.t.xˆm=mg(myˆ),mthemfirstmtermmonmthemrightmsidemo
fm(∗∗)mwillmbem0,mleadingmthemfirstmtermminm(∗)mequalsmtom0,mhowevermbec
ausemofmthemexistencem ofm them secondm termm inm (∗),m them derivativem maym no
tm equalm tom 0.m But

, 3


whenm linearm transformationm ism applied,m them secondm termm inm (∗)m willm va
nish,m(e.g.m xm=may m+mb).mAmsimplemexamplemcanmbemshownmbym:

px(x)m =m 2x, xm∈m[0,m1] => xˆm=m1
Andm givenm that
: xm=msin(my)
Therefore,m p y ( my)m=m2msin(y)m|cos(my)|,m ym∈2m[0,m πm],m whichm canm bem simplifiedm :
π π
p y ( my)m =msin(2y), ym∈m[0,m ] => yˆm =m
2m 4
However,mitmismquitemobviousm:

xˆ ̸=msin(ŷ)

Problemm1.5mSolution

Thism problemm takesm advantagem ofm them propertym ofm expectation:

var[mfm]m m =m m E[(mfm(x)m−mE[mfm(x)])2]
=m m E[mfm(x)m2−m2mfm(x)E[mfm(x)]m+mE[mfm(x)]
2
m ]


=m m E[mfm(x)m2]m−m2E[mfm(x)]2m+mE[mfm(x)]
2

=>mvar[mfm]m =m E[mfm2(x)m ]m−mE[mfm(x)]2

Problemm1.6mSolution

Basedm onm (1.41),m wem onlym needm tom provem whenm xm andm ym ism independent,
Ex,y[xy]m=mE[x]E[my].m Becausem xm andm ym ism independent,m wem havem :

p(x,m y)m=m px(x)m p y ( my)

Therefore:
∫∫ m ∫∫ m
xyp(x,my)mdxdy m m =m m xyp x (x)p y ( my)mdxdy
∫m ∫m
=m m ( xpx(x)mdx)( yp y ( my)mdy)

=>mEx,y[xy]m m =m m E[x]E[my]

Problemm1.7mSolution

ThismproblemmshouldmtakemadvantagemofmIntegrationmbymsubstitution.
∫+∞m∫+∞m
m 1m m m 1m m
I2 =m exp(−m x2m−m y2)mdxdy
−∞ −∞ 2σ 2 2σ 2
∫2πm∫+∞ m 1m

= m 2 )rdr mdθ
exp(−m 2m r
0 0 2σ