From e189e095f09a48ee06e87bdb4ee38c747687c138 Mon Sep 17 00:00:00 2001 From: ablaom <30517088+ablaom@users.noreply.github.com> Date: Tue, 29 Sep 2026 00:30:07 +0000 Subject: [PATCH] update notebook markdown --- .../01_data_representation/notebook.md | 10 +- .../MLJTutorial/02_models/learning_curve.png | Bin 15350 -> 15822 bytes .../MLJTutorial/02_models/notebook.md | 313 +++++----- .../MLJTutorial/03_pipelines/notebook.md | 100 ++-- .../MLJTutorial/04_tuning/gamma_sampler.png | Bin 9095 -> 9276 bytes .../MLJTutorial/04_tuning/notebook.md | 58 +- .../MLJTutorial/05_composition/notebook.md | 131 ++--- .../99_solution_to_exercises/exercise_7c.png | Bin 22694 -> 22708 bytes .../99_solution_to_exercises/notebook.md | 152 ++--- .../notebooks/UsingMLJ/01_basics/notebook.md | 120 ++-- .../UsingMLJ/02_model_composition/notebook.md | 57 +- .../03_model_tuning/learning_curve.svg | 64 +-- .../UsingMLJ/03_model_tuning/notebook.md | 98 ++-- .../03_model_tuning/tuned_model_grid.svg | 534 +++++++++--------- .../03_model_tuning/tuned_model_random.svg | 492 ++++++++-------- docs/src/notebooks/lightning_tour/notebook.md | 49 +- learning_curve.svg | 64 +-- tuned_model_grid.svg | 534 +++++++++--------- tuned_model_random.svg | 492 ++++++++-------- 19 files changed, 1645 insertions(+), 1623 deletions(-) diff --git a/docs/src/notebooks/MLJTutorial/01_data_representation/notebook.md b/docs/src/notebooks/MLJTutorial/01_data_representation/notebook.md index ca841c7..63dd2b0 100644 --- a/docs/src/notebooks/MLJTutorial/01_data_representation/notebook.md +++ b/docs/src/notebooks/MLJTutorial/01_data_representation/notebook.md @@ -290,7 +290,7 @@ csv_file = Downloads.download(url) ```` ```` -"/tmp/jl_nAeUTc/horse.csv" +"/tmp/jl_ruz0kP/horse.csv" ```` Entering these lines of code downloads the data to a temporary file at the location @@ -531,8 +531,8 @@ A = rand(2, 3) ```` 2×3 Matrix{Float64}: - 0.84235 0.324562 0.731728 - 0.233998 0.746378 0.485095 + 0.113155 0.33251 0.397839 + 0.171581 0.90293 0.436722 ```` ````@julia @@ -550,8 +550,8 @@ Asparse = sparse(A) ```` 2×3 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: - 0.84235 0.324562 0.731728 - 0.233998 0.746378 0.485095 + 0.113155 0.33251 0.397839 + 0.171581 0.90293 0.436722 ```` ````@julia diff --git a/docs/src/notebooks/MLJTutorial/02_models/learning_curve.png b/docs/src/notebooks/MLJTutorial/02_models/learning_curve.png index 85b533a7f3fe985421ff057146b061debb8deb09..7234cf4b5a2f95adb72495a9da41c977bf2d3725 100755 GIT binary patch literal 15822 zcmZvDcRZJE6!&d}>`jq9%Pb)~BP%<5g~*nSY_j*v&di?Ko2(=|GLpAibDi^DcbKZOEFShvYy?5@#O>|Y+$#gpsc)En@0lm(bIJ&PNk5kRp z+I`-e`h2ctqkQ)T_#?x0ux?9B_ec^ksy$5e-)G_2d~+1z%}p_u->;RqO?&cqcJX2N zNO$VJUr#v(7~tID7|AXu@Vyyri3&kTiF*VPgpDcG0YOyd2m`oyZ&&xSvJdS6%f_3Kv| zEF$#9KYuclk`7nR`(kKSl$7i)wfK}GY!=WELP^OS)k$U~ELLv*RfvM3qKmV$o~w(C zi=u+Uk5=2ue-|ewCmtRixrX>5BqSv73JNAqZ@!k0AVd%8jJ@D}GbhS~M zgZq}-4vI}|s>2-c@S;9`{FpSSU1_v)axTDKQ&n|C%^+%W}K{&a+rO-)VX1xlWKcge`eaBy%01O$pHun=Tdxx{<^ z`S9B&_dOQLPvhenii#T(#Tr;SD=RClt*upNeF`;K_wP&`whgB?p|r5Ei6B&0QzH`1 zmBzX_UXDrTGL*zx6r>>|JJ{)?ZE0=}4hiwSyf{irON%c;N02v((n2Z|U%$=^(eSdc z6qpP1@bHLVea%lykP5BBs+Z<{JBnJaynpB}k{J-tE&OBe=(&zNiJsWnEeA9N!G6Ra z!{s@OCFQD)`ne*X;FOp;rSqEK4_P`w+wYBWxoo-=U#fPN`#e274kzYF#oCo7Mh|$` z0+?1P6f^J`4&17_un;;!W@ctgd}^Jhm-%Hl>mTZDrWy^ndX8)+{wKsw?2PHTCrLc4_B_hK5AXFv{Yc1)gGg-%ru`=^xW5!Az=XO-_Lzv9t=r^9*~p z{+coN7f%%#XMNycLlEY|Cp%Gu*nGY5lKo2SgMAH_7 z+N_2kpC8@8zz+|gul;hZVfCF_a#2=rLe45rHX%Z%`}60|qd#*4BO|oDradhknm@$M zqaEVII;`hoG4z{$klr$F^8R_qjI`(mEwqa{?Gf!2jGSne3HjyRiZZwBSZ}3O8y7#-Z z$iiGtPfwYCgL3WM_wORs6Gh?S*!v%QyomG9GgvwK9;_J+wQxiq#KP~S~jHOkM#6BTlo~%ad9}CD-(f#ONf++Xl&WI zN~-NnyNsCY$E~{ev3DkDxJ}cP9M;89vy*wWq;O8Vb_5Tue&dV9ucg`)96F;f{|<0>;9_HY?5~X-Zp{Yx`~Ugg#Mp%D zVfITNhvYs7?;Ed6p<6HFP8kW4JP^b=s=d8ECMKq)#%Ul9I$B6bNW*5hmThvZ(XDwb zdKcmg6h%?!qeboQ|KM=Oz844ZO`o=*swz$oudQJ(FUYbuB=P+O8j?L89v<#?&r`NpvdAmL(vt%fb`5VYdH&*QnfkDZ~} z`{#(|(~pBWv~1)Bbrz3_e_3hl>-wGY++0&TyC}kM9ljmG!NEB>IUFVN+>>e3TaGKW zHcu?6UGDm=I6X5YM3CCOwJ*@dM@M;th0hx1eP)|oY?PGnB#Ux$m*4@abEp`>lYjry zteyACLs5}IbmUX!^l^ce5CcQ5@8!RH&xZ}_tl<{PB-yyQDvUdPP+v=glYPb+QzzKl za@?joRldO_h)8_w`FNBm>gjXxGr2p2AcFh_Nh*&Cs*IYvyu5m`5{>s}B<(JZ-fD`xiDPqbS~PF3g1Ge|dWwV+s^* zZ>}S;cvb5fT3VAGzLyP|)|QsBI-=nw?`e{Vm8F02ZX1WZ+CV|7>G|8w4!6=XGk>f; zH8C-^$OtS|M+$yVN2$gu(rr`TuEcNdLP5T;@VCQ~UKAA-k+3S)QnRzMrSO_v zJ;8S`}MMO0(7)|@&<+IcM^|7%rxIqK1ql=?? zz^X=aGx>=HxCXsVk%zc;g;$@~s9wY{zULbxeF7Lk$!zF!;yup~EPMU^C+d`&1 zET4B_Ry_VXFRr8v2NhuxuEOgrC@A>-`*+eDl$k^O$?BA`FBx`_I{V`t%NL{T5)6aX zM}mr-p0I1=NoG%ts!U}72$-5ONUn7TVE*~@2j&e{4lMFp;aAEY*KU)J@zp2`=~z$DP&7myuwX4gKT|JJj$1FuR_t8K7E9@S z@Y{rp4^~9_0i`^lNftG2*LqlYBpC5)`;o3Q}V*I`f zQ;yavY=?Z53?EtJHC^@`O1mEx7WPgyX7z=jnwpvc7X`iZ`j`CXW+4?76?UznzZTOA z7j;$FeZR2hZx+y_ddc`rr0f+dlwC z9sK>P)8^r#F|P07TT@+Kppx}xagm6I#^>Vn;Ll_wrNPudO9@o)1&Rn zv+a(KqaU77v(3XTGp{$HU{t1Kj6D(&p%BmisBQX_2ZghXd4@w4zH8y1CB-XhqkZ>KEGiXcHJy0EuC$7 z<*>n(MovH@5*i)-1DdL(wv^QJ@2{nRz{@L%u+Y&Tz7PzrJ#Q(+XucV0!Q?ppdAdz1 zu!Q(T5m~rw^EklnUI=UQU)*tHnt8}4$MoRX%V&ek9ZIjRu1?=LHaS_q{(EC%qo9h) z$oaV!EVYx9ldtb3|IzwobJ3XCE$b~oqmo9kC%!!z74GR--PMkhA z>hQUM@dz^xpiKw?ZN2aKje8abO)r10j~8O)c%K~x1qNDLSD`Vj2fdLJ5^`7$5;_p0x?Nka+1m4{u$fl)6M(-PLh*()$o9b$5)&y5sHmvK#Kf&u6P59O073`*`_h%p?d%K;T|+~5&}X;2$Yi{nPYR|bU3ZjD|Dms; zV~SQ*R48j`XiRGQWlq%}KySZAO3L(2SwR5+hqZSSU1;JRZC92SBFU1HlC6HzmI`WjWxq*zIt;y$E0W9#+kY7FM_OvPRFsr%~A*uEyRfbTU%OU zsEYTf<8czH2sMnA&cdgAW{87GRD{`iQ{B9&*I;UGt*5J-nv`T=ZT)$9Cr{VZp~jnS zr!0EzQXmo?6RDN8-SC{LHn)rB!L6|vN`pK4_U)TOV~LSckOb?+^=-<}0|-?aX2t^I z*Rat=he_QB_3!p;_|{l@g2YSP<7oi5Knx0WC-VVTX-<7CB0|q8Ej?T5C%LYPruVqM z3+?gy82iy`E_8&hu$g?GRr&3wfq{TwlJ)7IG9u;XzMq1XUmzd!olREy0nCI)M2L!r zprWEO8BXS?4skJAxgHxNj9OA&QqGW-hMlAxG2!v9YHRy z_N%@9pV1ttjo-h!y1EJ;r+%59c3b`!1$cFTf1lR(TrG_mH88e~ydH+&%fxo!%!xp_ zO9|nfHj{1)nwaFYG;4G7uP+|N@C#~HIJ--huZ6@u?_oC;6@eo3K>X;hb*Is=uk@-Nw0zHO8DWolsx)WKU!Nx-vTJw_Q^*4@8RL$ zrZ4CY4+I4{^y@iTS!LNsC%=B}?d=6sqTHzMTVJ27)=z(d8>KlB@4^y`oxeL#Dqu-) zGaQcnEYU1|6iKf6y&t z?N-mf=8X9GyUuH){Fu#=1qB7355QYkO6MP@vR=-{w&z(-NN`!uC1%35&Wx&`$-BFsz*732t`{a<0Tk%D zlVf9X3U=ATu5QFN#A`EQ``bR~_;U*g8+WAn?8a@*J9k>V&s+g9!{};jyMW!{XyEZ~ zoPmMi`}gnUz$b_AqrUc|!@|c*;yI>8iF#z13pi#rgqWK0@$?3)I?R*L&Df)RLMhx* z#>V1&d{2b({W7~4N1Nzgp1N2|@YQ`qqkWD-gzrb}!P#z==5w-Z9+EXSH5D>h@pfG8 zVRAtK@vqJ5+5IG3i$2w^SwatWMD=!M?<9N_dv$gdCGLA29=HS3x-aDORyB^EjPJXe$R3?#VSS_pGD}Bzok2>8XI4>Y+s>}?K~~KSX$U|zbdEMr;7F- zGF{goN2T*+I&gCT`I!vhj;w4S*&QuD0f7#8ly06D*R7X;kDQ&~ig`B&enz%U8z?$^ zt7HUBn6Q!EqPEFb9v-JK(o>R?SqZW9Xy@x}k`od%glPdKzZG%cRS~}cBy|@Elu|ms z_rX_fy9$6DLq7+S|0kz%29fs9Bmq(bx+cJq0`r}J!TQ6D>x6y%tjn9d9Kw2_vv(nN^f_k;|{1i(Mq9y1p7#J9w z6nTY(>8YvA&^#YKvbM6~)UO|$n`@)`ipCBq7KJQ{Pt}{So8(VX(bSMw3e)A5_wQc- zg_`?hAE*kbrM|x261G?ZUFerDUnX}9L3DZYgcMkv`9RW}H>Bd?9l$sg6$ki_`UeLi z{}j>EJ{;fv0o(?OdA6q_Dhe-p8IWPFp;*=>P!)k~DZ;?a%uFa~Z5$+?X9#2D zb>{-|DQaq|rKo5vfI!T0MpHwBiq|v{FmO|o`vzvrt!RXaE9|&Gx;Kg2Wz6^C=?i2< z(o@_YP%KOd=;Y-!E%#t}fs8Kd_V=l`_XXe`D8R62nCU+OQ{--ekmoXzDGHwu6BDE2 zw}r|1=Z652XB$J&`QK%uJbtN|eLbiPu)ra$4|I#&6EDZ-pql>gxo~S?EPwy&^|4T# zi6=H*%(xiZ=wLmJ9+kggI$rKud?7^L_(j)C<=qJ$q#DE%r|dVmX{e9yZj{6Y=baUm+ok+|0^sOH9D47JU+lj>1OLx?lVvk>xVs(r=yJ zX^oE?A}$ToG*+mi2~e$(H|`B*2(vcbP?O+ZEDHc(hRd<1yFwM{_{)cVv?z#k@bqv( zc6K%tmZ+fMm52eBSGh{(Cz4TSO39k7zb|mk2Rv(+trpWB_NcS2DWWVG)-@PNd3Xr@ zCjkzzQ@W!zKTMBs^(3O5URI1>H&aQy5DN?ulNLngSozx<>}C&pDU4eWNFzm1=J<;U zl1LxXcy|T%sl9u=lVK!HOQt)8?q^C&HP6q_|D_CVpC-5c4Uf}#>DkpuF$9+0`HUu_ zN+i4doivA%g(eS^7mS*ShykeoJPmSH524$Z4HUe6e>%L`fAk=RAec30w;jXC)NA*0 z2l_cc4rEU{K=0*POtw*=k(ou9yd zqL&JW)-EI}dI8&Azuwk;ccp*2$_%<|$zy;DIC`U0S#=zL_S<8fjJe zcAMuB%V!UNmw4NXqwJkp~qebja%Hev`!wV%vNPvuhP;|W0;Yj z;ut$SJFjTR+X{=gP`ek0-tc+-xi|<56F8e^vr7C*+0`6O zW3IDC+!FFxTMQ;kAt>B?_cU_9BBOX*&C_vlJy~Y`-}RpPsvm##ZobK)iPq)^ z7DcnmMx*1h^enIr&_O_^wp@jVT66F>;NOza0v6|LqFchQf`Wo_r6LHE zWY(1sNwI%*jK?%JcN4`@BW~@}KQRg7&lcoKN}O-<{r|9=Nmk4`*6gvRbZ5P5RJY`G zY;Nn{YyH2n6KnmpF!TuM44f_L{B|#$oZ$T5m-}M!)hUrL-Hf9u-C;5pJ2}6KoKAQM z$W?LS*?>f#U130EmnG23vHOv{3^nHuW&ck3^@Pv^f5@x zSUETpT!5TvTkAny>;CyN|H&<+T4KpELU^Z+>JNqD#V%!Y*Xy^$!phblBU}NGJ9i=? zBB*$t!os5wN&^1_V*~6p^)fO8K@_MAgp+15C=KjiOb5v68p8cth7#r-f zbmVl}P^`d0;4_aON0ygwy_)xC6>?i0yaBx%L}aT(Og|31iQYevjL(@>Q7DQ@P{8t- zzGFo`$7Rk_{Hh0Xx* z%7*8UppgNvBqt{awXTq@YER*`Qu1zpxi7N9>^-w=T|(9`=sg6bv^XJTdsq0tV|nL}F{7=BPYQ05$*oLd$rh};0> zJ&p&F=b|BFVhjfHj~N3G-AS0Yos=;Mp`g6K&em5ypILl0FuiGga%D=bt?nL zz?iF@9h?v5T{-vhQUo`kYn>!;)WCrQmAeek{&=BE`q>pV1+L+=I%vfn*0)@Dfgwk{ zMD?jWF<@qqqNq;F?e5#~By(M`j#9nd>3B_rRZ!8;fET)hCYi*D`g$QjMXb1@;&5-z z=CS~2RjvR(kz@>w=v@H;Z8i5Tieb1Y==bQ?uD-~~$*)XUhb`JZM+4~QzdC*l+9NA( z+sIdQVrV_FMHeJcb$CH-+uYhBl7xu_&5Kslqqe_5eRN$d*)J$>is=wC* zod4qPy|+39egi)RP|)Ji(pQ)Bw8+XVmf7Qo2rlw2EXmPsrn5=rZXUrO>%|N|5C=ol zpMO2^NfL()2;!731zKH-%bK&wGn&tYp9vreX4Pbw`XTBA*(mrPUu=e6L&yZwUVQNy z$K^*l<&=KIjE0#_BN=pZ4RpAg57O;fMmagSyHZoh4b6d36Ihr z&*LM=P4@OaM}G`;6H)z$$5*mvT#tW*3^qvRz$q~HmC)#zvxJt##VxF#<~WQ>LCh>R zzWuk3s_{9ijyK*uA!N|!p&$cioc;#sGL5G7;o$ zUMK2lL8#e$tJ8}O3O95xNLm@IoYn^4KG=hn5B2$$&nh11r@8r0uvoJ|*Lc<+51P$Y zv;%1D`uh4Ut~ms;luDGwIa~Ckxr^zwKt?m_4Me}Xnz(NVz-Us?OP!>P&6*KVec*uS z>xEhPH}0;0NC}IVpjC{p#q>r80;r#|4>T&AqU*1GejT^1DNl5*4hF!JClipF(c^v@ zY%&d+E9}tSy**4^%7yLitfVBCLjZS<(P|Ptr^>nDaJ290^ZfTiJI3kR*@XnrP;4vT_A%L6cofM3r1_l@e&S1 zZpLh@KsIJ!Vfj)-gqVYIjW^ltV^fD~k&4`M?|KYQ-lZ%czi-lb5wub(kC|V261^^< z410Gy=@_Cv`^={fZkyM?t&5zjraO3fdqbiLHaKh;K+Tt@8i*iP6M#aF$xSgLvxqdhw zMR6!l`yeAk|Ar@^imgqv?{WjA9TCsN$dO#>UjZew~|}>+F08?k)%-MHLlEdX9*ntF+K!#Obn6m9t|&nIcLxUHySK zG@sbmSg=m*zt#4C`BGg~b>&iAIWI#)N^)`^c#fxNkeRcI8>| zhed6d>o*7q3(q$?zWZ5s?7p`O#-;gS3NQHnJgex)gB=I$)>IQ&!`zCXSnns#4Dn(a zsxnxCGD4@ayO-a;Hg=T;1F4MeqzgeZ>N=yac)L-#)d8q;U6YHOJ_aeS7-p7Jtock@ z+D)3u@o_!`$umt5=KV~H(6yZ%eeo z%UC%u>|mDYQe5+(DRa5*SFOgZ=1f<0KcW2zTSwHrzkiH2&bZjwPtNy7gwF>7wOU;% z@bL(;{n-Fbm`tLbLS+t%5pVI9F!O<)7vw@*EFQm-LTk;%@oU}V?U}@5=~`YBzu;XH zQd5A=8x$mCVNp<~fP(Co0h*4)=3nCf<;ByLgID34y-kIf*H5Jv5#?5nKcx#N_CLmu zNOgSSJSgk7f8FzL^7^rLwkq^X)tAi%{6euh9udQ3fv?#V@`dksLj-srTWs21#k4ES z`L|pDu)?SJYrk>G`(pVitQ?850(P20F}HmcFd!$~*2PC9@-9B(Wg@_4R|}b;KQn3l z=lcX^6|b$+)LibB*jd;Fg_;v(MI2K5wR5kYvOy)>a~_-`SGdbdD~pIa>B3tHD8B{3T}MGp800+KkDWi9$xvDf?VoPC$R zW=WhhH)V&Evwdll<^E8hGGq8!Ao~ny)XpriI-W8AO7mE0&8o4y|5AwdV&gQdN#=Hu%u~Dv|udr;ioDnX_M2O?5>_F+DdZ#4$e5@%KIQW4C`QfV=)OEAnR8a1?_k z7CMfSGuqXaDaW#tuLs6NyrU|jX`Ict>#EP<-IcZAdPliY@HB?jq`MDx3)%Z=eQgN{ z2~(9*h6?|FiW6_SI+wQab;$DfeWB0E?Nd>l3A;Zcsn!YjLlu&`*kT3qW^Tc`OD7>ElNq@U}CB-DS)6Fc>|k*H)Y@Psk7yz zC~=v)p%)C1MLp}QZIaXfdD(E^yzMh`b-izHhgC$rwsX-#qU1<48b5b>!cl zc{AQ!2W^A&2&nkVNP6npX9)`sOd62c{3*Gr#V)KvKxGx#MqJH(Q`Lo}#NI-Bd8Ra7 zP3hjO+=XxSMiHGI*Oz6J$+S<(I5j`b4)s!UN_rH7;8`9mEwO!0p5BYyiOx|zPUwzo zj!iiln8CAhen~mg#^vCcB($h!utKZY7tMlk5U1`xg+2E5*I**U;Y*?jLhL^goG*J+ zFqgI(D%n>rom!i3Xx`6JM@iR@AP%3Ani)*cs}Flr?=^UJT1H_pi^8Cp9o_(%A!qqs zkOFpOX}z5{COIM61BsC1lasopr{E$KDrbBi8G$W$HMhl%7tpS2q9T^Yc_crKsnWd? zPc5)`=W{en4DLv4RF0jsUZ?UIVcoP9emf%5nOwXk@kQ1&Lg_tLzV>_V@aX}W!HbuZ zvxHQ06qh|WwhQNGD&>~sVnX(RP>yh^&qZcVR;yS~7uKzP!#NDGU6seX^DMTUyuL{M zpUJO(Z-vc2xRwCyS_E)}l9^rO*su_e|vDFxZjMQvlImvQ2xG6cg$ZdnmU(!FKG$|3y2>NEg zCrv<=d!)CcskCOrS=zUe%1s}a9}aYB!Mm&LW)m#YXjueg@pxyDXW0| za1^&!8O9}z%{P=f9`SMr5lL7(^%{{U4pASc%ioYy@Zc(Wqwd<3HDTni)OPt#nEsin z6{G(5P=%YS^6=fyh`)No7*-knSK+2^u#7d+msL}1CUxf6dTe!8UQObi2XYAk6LNXV zrg{c>!w=hoQ@#pECdFYATX~CnpFRQe{|)2~AQbm+Z$x~FtD>X?-Uv85>bT?#Q-Cp6 zI3z>yC}F^1Txvv(_Z`+*17tqSo=yG8Pv~OGoWr75LF2H;eXxpsalu&d*iV6K<}*v) zNO$Dd*s7a%pY%SSP!+f1_<(++Wa*>zC%L*`RAal38Rl)w&GUnU9}R$etOee)eq|n; z^)N(;`l7vUM&u#c4){P?I?z(v&`=ae%YXLH=qX0FccoEV*LyC$dmXNiTz~hxr)2q@ zfX(A!7Z-wYGo5;5MkMquecJC0&*ssFs-@hG*J~#C?aMy@-isBBvHK;Qb4~8Hj67OQ zA7elMt}eq)d1w)RvtxK8gXD=$~gal5_#c4ETF4Z*+VvEdDLQ~dmI_o1+gTznvG z&s!V=_4djxA9GPY_4;Lak;OQ7X*Zu&26)Odhz?Q?y&|8lVnA`d`RNbm2@2ERWPVhX zl8j6*C;~DqfD&Lj+f|t8x`MU}ItH-l+qdK2O3)yqEsiy(8qo7 z`<>^^d@~$t7-2n-nBDpuh3}dm|_!Ds)@j98FD^U1D7i4Z%r5MI3Wc_8G@X+{G{Wf93xEK7KUqk7j8%;qL zQln6L=fo>eRY_?GgnrIuh%D+G8n&4WejZG{Cw{U*a-9qfS)+l}89q~ctL#~?dPoR~ zvQW6_6RCc=XA4hNUcU}NT3dz*sBLBKZ`qVDbZ@?i=^7DCfv?n5_Kn*}tcGL`&+Xg4 z=eE&iI78qi2yTPM*pw8u|9%8CPrt!F(S=Nc0e*slWWRzjjUYkjJ9zM-fqaE*CN8a* z2&7alLRNvfGs%O$C=alM6swyD92ZCeL*&SL=zEis@Z0MO70WGvjGLRAFDNdlsHo)Q z1L5LAHT3oGc|9$mfFjWjunn@|_OtavBO?wF zrvJO9n8QB%P7zoZ1PLJ(2hmv2X);7TwI<0>=n8Rhae>~pG{>XG(te&i_1Rep3Jbd~ z`RU7-yW--;lRqIy6;1tMX=4LtrhN2`-ESA_KHa2Vp2z|#N?iO8`n`R9+p`VZyFUlu z3AB;~g>8d919u-E9}jANz0rdjkQE>lkK6q?Q)C8AI5;jVYv1tjAIRE+igK5O5@Gk; zS%RFekc=7_{N*uM0U)6C@Tvl2kVR;?d3hbbmS}>?3{SNBHrYN_(7FgO506f(+YS_m z{^Z6(EMQZ3H~HA$9^bzo{rdF)0*8u89J*Dey$2gp9l&D_r`!>VRCxUYwG0g}##O}W1If9KB2A|MU7W!OAywN3UN^M=f!b$CUSV-@7*jt~mr@E( zPFdMw=U5XEfYZT=fj}m|fWYjxZ}!vgb07xsQ-g?z2*jDYw{NuGfqPWb(D2-Q>g9C~ z!o2h>Jw1Jv&ndg4B}h~tv?N$NBA?R2!#}5E0vCpc#b-Ui7D$*pfQO4aI6Qn6Mx7`{ zBhKN;V0Ab+KXHPr{#bon+!%N<=K2K4%b&1y;dvTxY1i7=+MX}<;>;wWd&HXd2|Ogk z?GCNK5p3maFNt`hPm0rF(Wez$_m4FZ&3?}A>ID@|K|v8w4}vP_DE}R-cY3<9xmk~n zFw_-B&cbu)R5n*HOiauyabKfaZ|E5d3r}Yz(fkZx5v!`J!=s{Pui|bj$8slfZ!F9=PMn z!2jVjtiOL>+`Cr;S~}fTva||<(O}ZdHM=Ov%F2@dlz71Q*sCxv4-VU~>EZ4U>db#( zQ=l3n)wXVKZW~Y=Ynm{@;}a6}T|paeYi*UDg;1~D`~P_n@BeKU*ZFdlQ^DA%P@oWP8kk zV&9$vQOL>8{?ONlNzbI0@c~{(VrG_c24N1HzSec?8}#y*g4EP~nophpteM`Ofm($< z#TY0ZHV?TSkO>>!f@%l_S-kdUZ55`^^yp$Lc143zkTBv;4*Hf?%M&y55&(g zKA@i7kVYC~hZH!3{Ma=LNXf{c;vjECMDKjKsXL|%H4I7W>gu~{L5_}&Dw!g2hqQRA zMpxH$emLuZl>;l^K<)nT{6Vk@j6Cy8s`(#3esDgA6sEJYvvkxNyoAV=4B3_MNi`5c z5uw=z4g8sIJ?sM+8S~Y_RELXZ8X6i%ra&|b7Z+#rR>C_vdP1KasRi!cv(Z(#3syNu5#VShuqfTUZ%ta}jOIZOJ|YIxw_x;@ ze0K0~>@y8#3*?8(M-JD=Wl85j(Z`}UG@RpqZrtt_Mp#l>DuvZ{^0p=ju+oNMJd;%# zx!sij1fgPp79g)z#G?=Rjr(W@6W`z3cv}8XB`5zP|q@5w%{v+=Xj; zqNxd*q&6s<{&WDmL7z*1`?f46M@?7P9VFx{=^GuLE)btqZ1C3}JjieA{V*&bF$vLL zb$$Jfjg48DlcS@m>gq2+TLZxx0!`unhLCK`Pp32Nh^Q1Bsr^iK@8l#oy>xtDP*5*W zr+w)0haHS6uzBmA^%JXUAH>p2LFXMmT?RBs+~x>2nUtg?jkwSE)6FKQUvLH<+-}G3 zpFdlJ1e3cA?z>UFt?tGh$R&dm`!dNKtT`~dArOY;PlwetJY4Zp1KGz)*+XZe%25qn z7JNv^DGMb6&OSc1;Cst4hCp~02{4iL#qSVexrt;p3Xk(aAJf)8+xk(6tP!4bUwfl8 z(6YM}IslO?;h88x>kAdhUoP7kh%*+AAu7Dl3>{I`2jZfz_+)s5=n_^E^H(i0?-GrB XdTxn=<>(SxFd{FbEL|b_)c=0~&E1py literal 15350 zcmaibbyU?`^yimOrI8M$K|;DyQb3eY5d={L0THAdBqgOoQbGX%L8MDsIwYjKq`PPH zezVrh{58Dw)>}{S{c`U)XYc)~6Leog=^8E-E`lJ}RFoASAPCAT{L6)n3V-sx_Ei)7 z2lKg_k^*vh^-o$&P85PLASw#7+D@O=CmbJ<4xeCc@xL@)sW}YOMlt`uhAKqJrS%y_ zP>4%fu-W1B@E5-CQ)U7!!7Rg%p7XF_DsU-lk)jjDurbh*qGxjRmh7IISRdT1Ol#Af z?0J`Z+S%RRS&M#bEAnoUCd8Nm9YJLNPS0|~$B?m>aYIAAw0({+5#%fS_B8}~fHO;t zAeqEX!U&?w9B7Ro5vqh!@Bw2OYEcjw4%z?S5V?FLWi#6}QRP@A?VoMV9mkpIft|13m2D^R7q!l{`@}Myl)WAOJMe`$kpFsu!jWyEG^B-%6e&PIy5-w zc6ok?E?e!6AJBNbl9PIepMQOQz3ya{Q&3RQSQ{3tSe5PDrcyYa`~Dxjm6a9!vNvRd z0)|CKvTkl?Kga2~8jg5@F2qWX-4i64$-z|kb;(!OT zGe-|OhUezy{`m3Zh^Mr)l#`QlO6uQe$tx8V6;_Q;(>2=K+Gp!!Q>0{M;`EI(Z?S!h zzCV(?O@XZZ86OWZjxWzc=>5A)qh1gkbEO>V-oI~VVKG-X|5n66E}^x)zP`DcnTg5w zFp^2-_U&)nmxP)z5j5Pj^|BJru#w2{>=nlgHxw2BIZ4ORK-T?`&`>osHA?@Pmo_RY z?c$q;9GsluA|mKmSdQ=5r6mP+I9hJ@Nw@~lq2qX!Pm5egJ!Ys8q3O<6Ph$H(?8qod zE$vYq97^va(apx zX`D~-<=?HpIG$odrjubM4%WxYUjN~+xzYcAZMd+byOqeGCe)L zI?qcfIkVO<+Ba|B6ciM2ky{KG7<~NrF(oC1l9E#TZ0mh?wrQbs_X7+AdNuSj70!=l ziDXP7$az>|V&Z(OqpPdyloSeQ^pR=Fs9vGrLf!o3jKk5E@r7hddlV}Hz0_IXXC)ya zAw@++`r{y>*N?{8p1+{UQpn)tSAL6l{UcFzGodgwH$OhO=>DuuC?_T+))+vrnMYth z)HJIYWirc-+A4Q%T1p&|X=&G%w@)K?V2uCd9$abI5LiWyAcWirAFO>N%D5e*0zWyt zuhhZ7xsF7iFkAe%KMW#-0{Rcd38K$HDQtug_+w z`)dDR^`w;k_+j_%-P$g}x?x?QwQuhOrBAaNL~$!g>6x`Y<#{LwcBl~bZtupaZlcg> zXq>;lKPSdKP}#K5;PBrm>gw9gMu;D2O04y!RD$ z@7{6WgIDY2KO&-``IMYYOHIw`qxnzn;IvkJY*{u$u)*gtbM>!_6a|7*!b`^c`-_bl zd;$ZpzjFol)y+~f{uqtI7HE-Z1BN=lb9s!Ddo9THswK0LvuFY z9z`qe03|e4<*4a|vG=)~q}FP(DMQ=k9~p;-z}G?gOEd&I@ed2@j$oAk^y$-)E7S=Q zIeCkxxs}!JYR%2?6+gxw*4TVJ2`O)5Dn%k+kVvLN+e(K1m6elYJzj1Dm5_X>|M+oG zzQ4z?^RM@ed)Cr&NulcHs-=Ci1(*nf)$iZGFU~hTc?>G;&W^TcbJL}N);eS3pG$=t z%~MI9W4ha-%N`6odx3aO7L}Dr+pk7DSu78yH~;Eek;w=(HJ!?+*ivReWOA`XHlbBH zcE{VF?5!jwQW?iGJ+KuKqg9SA<9!#!*noocD;T+Lj#uo?HV0d8ri$6;VzbDRGGh=M zP^pj$@OHeB=YbRa+U?PjiiHJdBQKmTxm?Akq$+miq|RlD$2-@rPPmv|js0VNeb@EB zg1+}3KYpyC(Y-Q|b8KxR@+nE!y7&>-yLa!jV%ey2x$IuzQ|__uTd~wMptTUAG6?eG z=M@yB+UG~hD9LY#Yg{&v@3koI!$ds?! zi;@Z_;+NlRCmFZKJXlMrM3jk0NTO~jh21A*-jH)yTv+%MptnXf1^otU2Rr?4FnL8q zMOTt=_$7H#*U<3rjRnh}899xL!>y5xJP(<|bSv9-^l-g8>uubA`QpqL$l+JElvP(( zXJs+VEia{d>gwta4GfSd>C9&4c@eh|Ha<6hNqpPZ3o1&P%{zR3i|)L$qobp>v}H~A|fJq zaY0_5JT~W7FepdVJCmcMfl=`C!7>n0w#q74GWhYVJ zNRY7HOJn0;X1>c*8kA7V5i1h=$UfY)n2?wTVX#4WNTHGTaBGibez(}U;wBx}Wtxi9OQn6Z>8Uf-nWvE9Vvz(D548i%-2aHz#@`N!)sufaM;||V^5oGYGZT|r^SMmHN#qVv@t^F2D<9)rHAqs_ zxzmy!so=0MDd!y~PfeIndn*I`dwZy;s0t5~(6|3IcVk9AEPU9m6ETe-K@3Hkt`}#= z=H}+99w%R8W69;%7#PwrGZh{`KJL#_vlPA!N1XrjGLmO6+Kd?R~6pU(vp#{nSoAiTt{$AzVN=yXBFYwR~=Yn zzvypwJfBHR2<5HB2bsUjOf07eZ_eY!h{W364}g6Q46Q$UV_{)oWwp?w5fc^lZl#~u z#AFUmcF7|ytr*rImc+R(QrwfMN+P+xUuAvJkS2K}x5B3tcf`fT+1YUz_qJx_H8pz< z4jg=YE8Z3rVU2M0~@9*zF zt946=j3lO}PD)5%&)s1feky(zjyyLdt~sGoZ!{xCyxMR=Hld;?2et7Vn_63y6cm=1 zm+>j@G&MBLzzn~cf8X{Re|<4a#ACsGTdCSQc~_lyj~{hTUq>fKc&r`J8labBlNuZ! zvo~+Zwd{#C141uo)K935HkQPRJJN~ZlhJy+Pf1BhogS>0l$AkCEUc`otgDlj3(Q{m z^-C@xmG|I1A;ptTjia?W{>bGDhY>Mh)PQbS+$|hS_rk{|n62AdpB0;1TQ861!fkDB zvD3@T%UfC$>FMd=eA6mYJYO+xD#bO`me?m218n?@wP@eU((-I;rt$2|&C1GZ@_igO zqquk#weTAXDyk2;66Vh{eJ3c?-u%v%H2E@z(qtrQ=8J~-eyM78!1gIBDOsy-ZE4A_ z>(bkoRi^po$ckysPeS~)@Kaz+L2o!JLP#FpSh?w!_?&XCJ^uN|QJ$Sa($&JWu?1SE z^J@Ki;}?lG_;JjcM=0|hX$a-*io$7kG-@*`rxZ}_2m;dV85}%o; z#~2R^vU9Kc?;(WV+sRQ;51&3gJGf|SYC1V_fre|*8U0;9Ju`@SCoG+AP)-RKNwbE& ze2rS@lbHSL`Z+W$x4mVp@Zqk`PJBGP-Gc+e5>s{kG4YKlNsqH5C?WWx6kcOg>*rjQ zEShN(wN0j)ZrU_t*!y$4pwo`~m{PX~hqcZ@W|S_tDgmsa_mJi_fu4+zYCrLcF5a#5StZ(`Pm& zs<6}Jc%CgTEwQq&>`Y~_t)A?!q07c6B-nfQ!DB?l#JC?9qci?^x6}x`sLaD{@j^S$yi)ZC;8+v*Qclm^^C(q`i zG#_#4%F8#_x}TbMe>y&}$0VRT+go`^Mn*QN#>|`rXqKIW!$HcwafT3`F%%Dx%xyKD zjHS+RoLE(4Cw!|$g01%OVN`K(@&88uhZGhD`cp7T>FDUN(+#Q|zJC3>Um!1k_ik3{ zzUQuxps?`n4TCv$Ztm*@1mWS~hx_}tIXHIq_W?pI^r&>avbLtDqB?;wuww|bRZ_Bs zyS3;M7kj8RJz4bEaPpi!dE&O^bOkRII3mU~W!_#~L?TN;!NF>UJg?upxtMP!nRI|Q ztE8k9x4$wlFc2CN;zY{=OC%p~y~cTWacsNm`FA)~w{PEeH3;&3XqrA1KxC8Exuo)= zPU>ezTyON|F&FT~iKatX&o|@c+udaz=lcNlhu?6Exu4iU(}VK0uvn{cJ@ohW-Fh;W zqmkS{IEasn3pDVm%SW+lV&YS5t^Uaqcig!5No4z09r21_3%ctWP6c1EgOu9&e6VJ* z*Cb%u9`u&7loZ1yQ)@WEfEErXOIrxV2Y0oE7 z6DuoPQquX=)k=@^6GnM|3To=O*jSBZG5g=q?Cv}J%ItTE;fVaYmr&8Zm;P$}Tps%t z^RgNWVwbb_t>ewxig7`Up%*V+Kwl+6@0;6(g#fY#{DR_Og9&N@eh-j{>QaVD7^Mp{ zGkl&GXK$qLx>y_3k_0H~)AiO#%y=xxc9sMjo66FgSz7wmS|{wGc;?KB)S3@uyB=;B z)VS=shv{i+3kV63M^9igMl!)JpucpyY+~tkahzSH*qMGd&bO^?Lvq!Js>cHCONi0a zzkWUbH_FebgXZ*9;QJ4oc?y&9UDVQjlsJEUETnp&arP7d?A^O}hlYmaaxu3)iac5L zox))y`CTE;qZPi!2qkBI{Sx3~8%9()Ghh%8({4iY;d;n-E;H6595P&~TK>Z4ow`s^HshqnAP z4njyX)e*;Y<0WmHE9Qq~OG2A!FYyZw!3o^U?)Ue-rqYo;>&^WfCncqTqBUgGB#|tI z|D(>Nj|BI9O2J<0>oRpi25!=x7ZNvb{u&?u)MW}23W^^Hc0h;UAIv6|q|h)dLTZVv zUe3W)S37YxVPv@wy~WYkI6W?IID!9_grsEM#mQQa^yR1Ql`OTm?j&K&=zVX;L`!e! z_%@dZ;SxOeMT-z>DRNDWvNyazf<2-NoA}j)fh`gflTk-24=yibzi5dp*{(%#7V>eKZ+QM&4RU zNy)>_sb>!#4md9XmsL|$J;F;z!SQDN(TP(>BzGYmF3ysYYvye#f)r~SO&ul*TDZBn zSz20VYb3w6vYHqij0UP4A1`9w&!nlTsTGSo8TA=6mQHcQt<7Fs^SmSJsFRAW7hNV6 zC&U^HD*$@vpFBN!F}p9{zFlL4gZ{&~(O)M|H>gNr%NWL{ZF2yn5r5ROudV4Np;ILm zlJIAO0NCg5LPy-#*jQd(9>8${LBYGaWI%|>V7pwL zJ^>og)`nfIU&$7^-1hc)BJ+l`_iq;rgBhePP58+aloZx=n&s_JK+|sBx|JS-e$sh| zbzB)IG%Ux1R@Fc8SJ7MK#c%3ugNF}GT=rM#dbW3W-yg-2r#_l)Q60%ta4F@iZ6bPO zipn59dE^Z9=H&&zbkkR_%G{23K=BC+3j_4$>x&L;)bnhPUfv&{NUtL8hMV2@Xq%=B z*6}Z-i9R;)X>h%}zQckLPzzZC)QOFaWl{kx1q%zSwY7C~vgUNM#x;YB$V~!|?4O3n zHXx ze*MluM@B{l39AM=CT8xJFCo;z8qv%?wWXan;Rm-`4AQwt<&{K`2lu8;#Dj0}6ny*^mN9^zg|Tld<_L5jIGSQqlNMN8@}Zo0C4>4k6L&w&rkN@ zof?{&JX~Dbu-Y)X4-Q&vDBM`^pwWyOD?CCq7`k_P$JeOAkqj{(Xbho%2AH0np23b0 z7@vMKmQ>Zl;XR$JA5hEOj-Vq@FhT|FrZ+A6_kP$UcC<#r2L_W zZ|}@x{25adc296JaMQ;J6<{7+4+v@Atd1`*z0+@}_{QK4OUH^)Dxa*$`=Bw7Kf}jp2wU;2*YXN`+WdXnK)OYju)il>YV!VQ3}4Z&({#4A zQHa`R#>NgoiCS7&J*)G8WQki_ zdKwznJqgqY)`q_2ef_$9cz8p_O2#LIzxf?YB-Kva0*(Em=sg{5uOpm)r-oGy0Oj+O zlE!+|sZY5uOO-TVGxk#(R>dTxVSE}pI7wmQLEBA3-4@2g>)z z3$)HRC@$?fO?O*fr??q=JUro~j3&I$OUc4=gR@r#Og1iW-Kbiz7 zJcxufGAios<7*MGhjK@bjetGc*I|M#knZ|aL zSvH=>A#75^66cM^u8^C+Wj3BML40$oyx+6@_j8jIqs`wY`$)`@G*Koe>{jXMrmL>_ zjjS3YGfWIBxn;C1L#R{W$)IQ)wA*Wb68ZjS0%&7pZ*MQaK@N6yd(pJdpXbJ`>m(ii zJ+iRK17Hf&IMt0i0v!))`eoNWGgNzbuCF5ocqo$YT};vZdcVutbwX;ybQ@(ESwlT`T0-lJkG~jxeO{{I0TZo7X;y+<&_ZYN*pCI3-_o&_I_>v<*5l_m7-Py5KQ|kZ;G$Aq3;T>)Ku0pQ; zTdgCV-^xU=d})-?=O0wKUJ-MaALdeO(sH#C65RPUURuRn)ylBL)Z8V=< z<{k>XR~IL1KRGULX?6AYn1j7NbhvK?1siTM8X94Kei)k#-Cz(v4*_eVDcJAx!&`0t z^g^bXRS(Tt$6r+b41K(=T748^Z;xN`o9xhu6JQ2~T*s%aIh6TNFI*mycbZ{)$xJxh zJJ1E4^0KT3)CE~WSf!JblQT4!Ogp=~snVWw)YPsp0^!R}fo#8S-c(Xtobmbd=Zp*| zoaB+yeJJ#* z^4A|!A0|SFv9kHvXj-CD9NSpo!wUB_o*3YD*B=f|f{{B^4B(7tk(mt=acqdaO z>N=eS;Pq1nw<-|qlEuqTyGbIM7;Bjm?#O*itP$;dLdooxzFEASj?)xGoa+ezbB6FKU78H ziaAG8@Sy75jEx&I*(<+x3J9RNxfxJNvg36s0c8V&nxqn7d@FNvQ3(kv)6+#jjVSp| zw>I?Oj21Vyx3dcf3<6_qZG8pE@$#rudIQ!BCaGhX1`5I`&`qjI7vLVLJe6k*yp#=R zfkzR^46mp61Hm02dNL57^>kpc>;Owv7BkQ=@kfAw9&OK2Q&A~~Qk9wyuuNV6nhl-` zwmsUK5q-C4IrdW>MzomN->$CpxmrSoj(i;$&hr@pe-Cfkn==__wvJ7PNR8Hv+cBtu}g{5cXqIAiMcj}0*2#RATFC;E9} z{ZWdPyUV?MAxwqx1Shw7Xt`d}Y!BSpC>|4#7)Cp}?VZfp(9qBtPEQYNZvNvEJ3BkE zx<7MmB#3X99EKXLQdpg;yGyc^#3!RGx!fy+h+-;9$m(}@w<31Uxq@r>Elq?FMcvuX z9$$2goc4&--iGNbGtY((LU`MIyT)qlr*-Y|VzTw<=qS!NIz-A#!utZ7L1;YZoJY$u z%pb)s%@dvo;4mW}&1Go`#zJClLLqHA|QF=zs6|< zYEvj()6*(~q|O{axrsttbT4$`BkKbY*tN8@ZYhR5`eF95wA9|j#N_qs)$U|*umet~ z8_+~WMYVKs>QA?Znj_`gzw0e$C@xmX`lcy5J;ZX+Zk5P4v0aNFf5MUzYrZTaX4$EZ z_ijM$&ENPWa?FWknH?NAGbs@fiq>rq*t#t`m*(fotEyU?ntXkI1JgvD%;@&ytf;9x zWBT>#ozQUExjD>D>HhE~2zJhi?R^Pjw|v2+!>e?yQx?tH%p{B|-VaxI^w*1_vqR7D zwcb3T78SJJ-iyu4e}DMA>#i};NeJ-dq5rJK!^PDtcseQyy#|^sI8GqiUoC}Uf3%XG zV$?^KACCBbgf2t%@f^n8iSZ>YLVt)W%w}VAH`|`e-|g%rr9M2Ib?14aHN?rXMnQ>b#EYlP?a(2IqTu(f;AFg7kWG5IxBy zG-Jlto#74YoGVO~$;nBul%WGqio8vOUR_af42%=}&{t+=Y`=+7BG|^N10A?exP!?Z z8Pn=WMw~M;Gwq;1fXmU^+zfbfd2KDg&#x;*a;%|2HhLe0!CaXpnMU?{rA&pHEo-7U z(x8nX)Tvil0KAtKvdE;8<9h>@$dLbjNFNI#$`}S1b}WG~r?1^_#r&eS9zn{mkhTu0 zm(z;T6v0gvmi}u+Z9;@3;M}{I*Gg{W1aztTRK?3kfPKGQ1GHnubnzL1@dP~sCfNFW zDH7h@=tSc5{WuG~=vWz;+t=!q__Th7VMOr1dC)`6*x;>PW9>FPJPh)*C@ZVdmp=$Y zC@Ovt8Fy2rLS+oGBYT=O)EQ_p*9Zuftn_qs#Y9AW{rzG37Z}z&(Jn<{5Uh1*Nf!M6 zh14k)D=-Y`liP7|QWCZG)zpDS#KFMKTo zjVANk7Zhf}ktT&zMeNaWZ#sL`)lW~d-v`qS@?8}fZIY$8_-i9u^quPew)$~BFYlHy zddnf`VQ8qRdwY9MW55ov)1fI34-P^l2N%d7nO$PSH%t5?YF&O1^X6s)LA7EnUO(KpqM+^k27 zpK{t78Qze{d z=jIy6r0kX>>gqfnJm?3%0`?FT+q*h+$a!Ebsu-EFn@2E*8CY2aosmJ>Ji76=B z#x76xEuaq*>0rrR?s{H7Vz9C_ymK!V)rH~h^fW+|p#lSa1B2vrb$yA%^n}s4FhacQg28`|X)hk~xb8}xSC@5H2 zeQW!GmX@l&5i*wBz(F`Yfqvi)eA3s?55#&9(kH+#0fFH5?RDU3fTlGyG+J!RKcjlF zovmsf8D<_x&u~*9(st9=pzlHHK#BuMIxg;Ia+%1TJGMLX?Xt46KwRg0=rRbwUCLU{P`L@2)N~4`amZ@{V4^bpqC$iL{O+ zX2&!XLP_uKZ)s`wK?!@w8rShHFYlqYHmR)+;w20QUNGi#3g2Mmy~<1+IU0sRbvG;w zjB_xSz$ZA9q`AHn%v1ZNu#f;7n^46Ho_%b};Bm{>aw$Xh#+PZ4)xCGz7$qbJ)~Kho z;QCYFxUsakxw*Rf+Ss`7?_Zb8i}TghRWM~9fiYjKfFR+?R6oJ?nbvs9HKVv>Q&-*< z`lEq-WNwjOn3qPG-TRxhmDM`*Dyxwq`2LKXoOwk>Z-P|p)Dd3_Uw7SB9YOJGoT9N zZk`~BFX8EC%h7^X)0)G)LrS7s=o@b=!BcPHlUh(lmQ>THKR*K-V99AV3mw|mQd-%VO;J^K)^GB{-(lya7bH%*!P^yz!4C9rMu(0$iJ zk2aQ
    (XL})Y)v6z^cXv7`jFy>0dSb>hd5+uJuJZReF06ns-s=;W4RV~);td#1vyx#Ix# zjsy1Pe)3~^PMQd*f0gfho{IMUgznG@5vYHeaQ~zh4-fCShSR~p0mK(;;StZ{g&Uv^ zZP3@|u;l`%-?9clfKag%X-kRWK1z=3{KnBv8-L}E+f32Vfl2_@O10I?moNQ|4eLC* zOiPZjNN?N#9ZwvrlG8OPdL|Z@Mc|CpabQWp;W0=#_Wj6;R1cJC9QAAO47DA^E}G)M zHzZ~MKcr=Cd3kVnc=zYp+8T~eFOU|udT7??XJ<~FL&8Er*;!d&hF6xC+f{^tP6`mW zl3qkPk_m!y)=3M27rURt4AMoKBeZmN$sI-0HgU3cn}9*^^GB5>fBa}{Z9TOx9GIL8 zO6LG9yJo7?#K=f|V&Vw^LrA^Jc!KM4^onJo7e(EAlqq`X@a%Xu073vBXHM`ANY`;c zEsrRZhH?ah0~o4)ormP5k*n(|K-Ca>X+w$_q&Ky~+}K2YYdt*3z+;vh1Ka!l?9~{9 zqK6~&8n#Wyic0zlY0WPzRMAL4>HUM zU`I20BZJ)ITj1RHXR3@;v9YsX8G}#D3H`TG7|6&+A2{!K#@uPQ`rK}ps@UY@Yihy>!_55KA6SzSadO`Fs?Mb zJHAzyIF3QL=F9#OQcM9|!@I#f|4LX6yRiuP9_;3g8*sS9!zuV~a&oFM-9<&DhQg(q ziXOdABrE9JI5uJ+L%-kvO?3r;e{oboN4s|x!c!~UOmh&CJ%3KUpt(6rR#tl zVl6CO26GC$rk%qdFnfT1L3peB_#Q4E9=HT|wIF!JW4XtmA-yD(@8=qSN6u)e9X!&_ zrKMLUCM2Y!FJ8ZX^6Z%>G!S^7`{@BYCubc1q~hW$26&T;4B65WnD2WcJR2;&SNdJf zc^WM+zJ90-4vvGv0hFj=wUf>@z-}47nuh>huSzhg(>~sm-w&@s{|=MIFWvW zHb?}@Eqp#!b<4wBW?pJF8?)kke0_jj!I%RZvAbIXq8eACQV%*F67k@2r=@JEH|F+i z0h${PFu*eFsp~j6d_qDOGXeCSU0uM-A3l5-PA3T&5~tlo_4spIWaVYnhpwN^!=}A; zC-PVyNMW!8nzXmIU9GO{0v{;nUcwy+TxDi5QM^O-($-&797)aI(BV%J>m`lSMHSby zefREFSK^)EKlC`M|CuSLkP`wq#l^+tiN3zDpdcKDLKwp9e@lTHfq{T9FkMM9Zyl{HPp-LS zi2fW%VlY;eTVdRC&C1J5`uOp?!}=5MI9K2ZpFYumi3(-maX8_akxzgySjz8z&&Qd6 zR+x|`f~!cPO7w{98bUcdGV&!OqkqmAUv7-Z#J1P^5ud1>zA)<09dnjgOipDcX1 zu1B%?o&ECI?p`M?ZcPcPS+u?`BRP)2VoFG!5m5EogG-SGrhKd z%KPA<;W!3L1E}-xfn4)dy zhxk}=@4ulf`;qYdK?9Aze=M6;)ZAz&_t9Y#m95!cam-gUSswj_-@4j=d~!uYqA=mJ z1&k6_dBMu;QQC_NG!v)p9Cip%E!3FO4f9plchY09m4)dKbpmm@z8W8e2@at!R^0v>^bYh?s0}QSIvmZ+(#W%fng-hM1pUKWv~ z8USBknNP@Dw`!K(Y71(i{pt|EE~SLYjhI@Bk10%E6RB>a){=Tz~gewrn}Iua8ad*a)e68+L2|8(*d ze3Lff`0Op~)mnXhl@{o+%WAu}MRom-mK@9P_@|n*jRW7VmOT;w@65ecA)Z-&i?jEM zzrQng`e?g+z&IFldN9*PBdGhUmNM|%gdycgGmT4&y~G&bqZ_B=i+^YE9xS6dk3T!Z#@bIQ?Ctf|%^c9{Nch|sG=qs6-b%PL(0aBP z1UiBm^SFFZxfc5#6P`pr)#1Qqk_K71Jr1%wJpQBT2k#CEo|_Zbe$TzinAMHxCfk+; zEJ%RbX!pSb%cAXed-rGK7=7=oy_W`4{d9}?xa>%vo`m3+(JF z<2ndKI>tF{sP|s^9?*1&J9~6f(wQXq>JG1@UXEd1YeVm zpG3Ycc3A80>*Hf&Y=G$cVpmd?!+Pc08J}`aR2jLI)m0Ex!GaI2BJ5Qw9y%%b@`YDG zKmj`;DGBgea1{;0GzfW5fZq&NS&8}T6v1;Mjl2~8eFE3sH=sD>K==;4rk2)Lc)*rc zCQB`(UI9e`efB4yFZleqaq5|_uEWV5+*J{G-gyBq9GPp%UZ}78d5>;(`qW4^B+1GCf`PDaQM7R>3lEuvOu< z2}DN07!|b~;hpRTOq^R=djW}ddFxzVP4)|wagiijft@Utru5emIk;#CgZT9;0Z%FW8{JCN{-R()ZtHcJ@0Gv2T|0rUY{`|?p z#8lrC$n`PtBP*wFJ{vc8E$j~@@gVgNh|uiC3vFFph?tX6P$);ffIOj#OJzbrLOCaf z3_cv%xZC==5cmV%pPbzE`Sap;OB%ktu$ysQkBKQMolg&}fD0H#qm42MZo7kZ1n&EV z+OucRpi}tX2eAm!Wt9FnczA1PA_T}egcQ`CJfXVGij0f|waNMv&OPWa5Nv^Mdh4ki ztE_%iw9EG@PoZklrvCr8xArC>ALTaRMr5Oha>7n=cui*0+0~Vrk`idS#<-u?1@@Ew zu6Rx3JYjgav$F$q8ukJ68dcM?yhP)80JvaZkq5^B%?G^Kza1Sh!>N4n#gmOS=>}_m}zdsTQgQz`R(@OPx1O6*m-Gp-0 zO~d{DV(%95FDKw}a3CWO$v0;~pr5Y+$3PH*w6Lru6fLLTi2~-taUBaapg3EBoajqv zEqf=D$$FxaIpdNhjUGT(u~*}ah=>S21@AOSC{MY|fKT!A-q+JBgEo3BUJL!)<7g%T z3IqrQ1T;WoVTVM04oD|o1Qiz4{o-w;Kwg{-i%ArDs1&vTTf4Ezz%8v zII0uS4`+5ZHlQ~tWfTa!>}w0zGM?Ck{$e}R1i_HxjP^tA{B!YLVY z&}Hx$qf=74ryd#_h8d51yrmRhH|Gk!_RJAd`GNuhPNB>H#W^a#x89!mmYe&$WE7%p zFDbHXYa#F_b@S$Ia_ZW`0x%AcJ%72X4E%s&edC#1gX`Da+)9V_zqRM8DAVt2tr_q@ zYlQ48{5UuWdFDh$_5)>T_M{{wZMyQ(4z{25Xu3C!%l~4Gt_Q2ErDiTVY~1fHX5=Rx8XugqyM!k$IXrAEX0XA z|;166EVSE8s-mvMB+J;~xXpmh?8Ic=ro=B^HYeRv#0 zBjE&1$qIBO=vp2g9#FVok-|iWUvqWq6Ru`B*YP9EXtTABxH40ql8h9;3@$AdlnZ21 z;pZ;_(;U{C-!oid6zpvnf6fMf-eX>?blfyt8i4Cx^{%su8XDY8O!KgF$y1?0LEBqf z^}P74HqbO69KC(e6VGRIyt9xT{s}H?K;W3df990.326, Iris-versicolor=>0.331, Iris-virginica=>0.343) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.533, Iris-versicolor=>0.291, Iris-virginica=>0.177) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.524, Iris-versicolor=>0.293, Iris-virginica=>0.183) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.38, Iris-versicolor=>0.293, Iris-virginica=>0.327) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.449, Iris-versicolor=>0.271, Iris-virginica=>0.281) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.454, Iris-versicolor=>0.27, Iris-virginica=>0.276) ```` We'll have more to say on the form of this prediction shortly. @@ -701,7 +701,7 @@ report(mach) ```` ```` -(training_losses = Float32[1.2405895, 1.5638204, 1.4046175, 1.2446647, 1.125359, 1.0727392, 1.0180935, 1.0194263, 0.9750622, 1.0149823, 0.9778134, 0.94241244, 0.9262781],) +(training_losses = Float32[1.7850709, 2.023422, 1.5918993, 1.4905419, 1.2054352, 1.2269526, 1.1768, 1.1895036, 1.128155, 1.0898077, 1.0582713, 1.0594574, 1.056259],) ```` You save a machine like this: @@ -720,9 +720,9 @@ yhat[1:3] ```` 3-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, String, UInt32, Float32}: - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.314, Iris-versicolor=>0.33, Iris-virginica=>0.356) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.531, Iris-versicolor=>0.291, Iris-virginica=>0.178) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.322, Iris-versicolor=>0.332, Iris-virginica=>0.346) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.379, Iris-versicolor=>0.304, Iris-virginica=>0.317) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.454, Iris-versicolor=>0.269, Iris-virginica=>0.277) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.377, Iris-versicolor=>0.292, Iris-virginica=>0.331) ```` Machines remember the last set of hyperparameters used during fit, @@ -737,10 +737,10 @@ fit!(mach, rows=train, verbosity=2); ```` [ Info: Updating machine(NeuralNetworkClassifier(builder = Short(n_hidden = 0, …), …), …) (with warm restart if possible). -[ Info: Loss is 0.926 -[ Info: Loss is 0.9323 -[ Info: Loss is 0.8731 -[ Info: Loss is 0.8909 +[ Info: Loss is 1.067 +[ Info: Loss is 1.038 +[ Info: Loss is 1.034 +[ Info: Loss is 1.038 ```` @@ -771,10 +771,10 @@ fit!(mach, rows=train, verbosity=2); ```` [ Info: Updating machine(NeuralNetworkClassifier(builder = Short(n_hidden = 0, …), …), …) (with warm restart if possible). -[ Info: Loss is 0.8501 -[ Info: Loss is 0.8156 -[ Info: Loss is 0.7756 -[ Info: Loss is 0.7181 +[ Info: Loss is 0.9739 +[ Info: Loss is 0.9423 +[ Info: Loss is 0.9133 +[ Info: Loss is 0.8122 ```` @@ -789,26 +789,26 @@ fit!(mach, rows=train, verbosity=2); ```` [ Info: Updating machine(NeuralNetworkClassifier(builder = Short(n_hidden = 0, …), …), …) (with warm restart if possible). [ Info: MLJFlux: converting input data to Float32 -[ Info: Loss is 1.152 -[ Info: Loss is 0.9842 -[ Info: Loss is 0.948 -[ Info: Loss is 0.8713 -[ Info: Loss is 0.8544 -[ Info: Loss is 0.8761 -[ Info: Loss is 0.7668 -[ Info: Loss is 0.7384 -[ Info: Loss is 0.6662 -[ Info: Loss is 0.6725 -[ Info: Loss is 0.7011 -[ Info: Loss is 0.6599 -[ Info: Loss is 0.6516 -[ Info: Loss is 0.6764 -[ Info: Loss is 0.6492 -[ Info: Loss is 0.6193 -[ Info: Loss is 0.6555 -[ Info: Loss is 0.6376 -[ Info: Loss is 0.6776 -[ Info: Loss is 0.7155 +[ Info: Loss is 1.285 +[ Info: Loss is 0.9493 +[ Info: Loss is 0.9647 +[ Info: Loss is 0.8598 +[ Info: Loss is 0.8699 +[ Info: Loss is 0.859 +[ Info: Loss is 0.843 +[ Info: Loss is 0.8556 +[ Info: Loss is 0.8016 +[ Info: Loss is 0.7762 +[ Info: Loss is 0.7672 +[ Info: Loss is 0.6868 +[ Info: Loss is 0.7414 +[ Info: Loss is 0.7573 +[ Info: Loss is 0.7062 +[ Info: Loss is 0.7683 +[ Info: Loss is 0.725 +[ Info: Loss is 0.8516 +[ Info: Loss is 0.7587 +[ Info: Loss is 0.7132 ```` @@ -827,7 +827,7 @@ yhat[1] ```` ```` -UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.155, Iris-versicolor=>0.582, Iris-virginica=>0.262) +UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.156, Iris-versicolor=>0.519, Iris-virginica=>0.326) ```` What's going on here? @@ -858,7 +858,7 @@ pdf(yhat[1], "Iris-virginica") ```` ```` -0.26227638f0 +0.32571402f0 ```` To get the most likely observation, we do @@ -879,10 +879,10 @@ broadcast(pdf, yhat[1:4], "Iris-versicolor") ```` 4-element Vector{Float32}: - 0.58242285 - 0.03553577 - 0.03338368 - 0.035628445 + 0.5187301 + 0.0025167037 + 0.00241053 + 0.0027305766 ```` ````@julia @@ -921,10 +921,10 @@ pdf(yhat, L)[1:4, :] ```` 4×3 Matrix{Float32}: - 0.155301 0.582423 0.262276 - 0.963971 0.0355358 0.000493214 - 0.966181 0.0333837 0.000435089 - 0.963876 0.0356284 0.000495223 + 0.155556 0.51873 0.325714 + 0.997162 0.0025167 0.000321372 + 0.997287 0.00241053 0.000302146 + 0.996915 0.00273058 0.000354911 ```` However, in a typical MLJ workflow, this is not as useful as you might imagine. In @@ -936,7 +936,7 @@ log_loss(yhat, y[test]) ```` ```` -0.3571473942077973 +0.41664216811287624 ```` To apply a deterministic measure, we first need to obtain point-estimates: @@ -946,7 +946,7 @@ misclassification_rate(mode.(yhat), y[test]) ```` ```` -0.044444444444444446 +0.08888888888888889 ```` For more on metrics provided by MLJ, see the [StatisticalMeasures.jl @@ -970,20 +970,21 @@ evaluate!( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: NeuralNetworkClassifier-481 +Tag: NeuralNetworkClassifier-433 Extract: ┌─────────────────────────┬──────────────┬─────────────┐ │ measure │ operation │ measurement │ ├─────────────────────────┼──────────────┼─────────────┤ -│ LogLoss( │ predict │ 0.357 │ +│ LogLoss( │ predict │ 0.417 │ │ tol = 2.22045e-16) │ │ │ -│ MisclassificationRate() │ predict_mode │ 0.0444 │ -│ BrierScore() │ predict │ -0.187 │ +│ MisclassificationRate() │ predict_mode │ 0.0889 │ +│ BrierScore() │ predict │ -0.236 │ └─────────────────────────┴──────────────┴─────────────┘ - +Apply `describe` to this result for a named tuple summary. ```` Or applying cross-validation instead: @@ -999,27 +1000,28 @@ evaluate!( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: NeuralNetworkClassifier-855 +Tag: NeuralNetworkClassifier-136 Extract: -┌───┬─────────────────────────┬──────────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼─────────────────────────┼──────────────┼─────────────┤ -│ A │ LogLoss( │ predict │ 0.319 │ -│ │ tol = 2.22045e-16) │ │ │ -│ B │ MisclassificationRate() │ predict_mode │ 0.04 │ -│ C │ BrierScore() │ predict │ -0.168 │ -└───┴─────────────────────────┴──────────────┴─────────────┘ -┌───┬───────────────────────────────────────────────────────┬─────────┐ -│ │ per_fold │ 1.96*SE │ -├───┼───────────────────────────────────────────────────────┼─────────┤ -│ A │ [0.363, 0.271, 0.265, 0.305, 0.32, 0.388] │ 0.0432 │ -│ B │ [0.04, 0.04, 0.0, 0.08, 0.04, 0.04] │ 0.0222 │ -│ C │ Float32[-0.205, -0.135, -0.139, -0.17, -0.15, -0.206] │ 0.0279 │ -└───┴───────────────────────────────────────────────────────┴─────────┘ - +┌───┬─────────────────────────┬──────────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼─────────────────────────┼──────────────┼─────────────┼─────────┤ +│ A │ LogLoss( │ predict │ 0.276 │ 0.035 │ +│ │ tol = 2.22045e-16) │ │ │ │ +│ B │ MisclassificationRate() │ predict_mode │ 0.04 │ 0.022 │ +│ C │ BrierScore() │ predict │ -0.143 │ 0.026 │ +└───┴─────────────────────────┴──────────────┴─────────────┴─────────┘ +┌───┬──────────────────────────────────────────────────────────┐ +│ │ per_fold │ +├───┼──────────────────────────────────────────────────────────┤ +│ A │ [0.312, 0.256, 0.219, 0.263, 0.277, 0.33] │ +│ B │ [0.04, 0.04, 0.0, 0.04, 0.08, 0.04] │ +│ C │ Float32[-0.181, -0.127, -0.0987, -0.137, -0.142, -0.172] │ +└───┴──────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Or, Monte Carlo cross-validation (cross-validation with repeated @@ -1037,28 +1039,29 @@ e = evaluate!( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: NeuralNetworkClassifier-990 +Tag: NeuralNetworkClassifier-762 Extract: -┌───┬─────────────────────────┬──────────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼─────────────────────────┼──────────────┼─────────────┤ -│ A │ LogLoss( │ predict │ 0.316 │ -│ │ tol = 2.22045e-16) │ │ │ -│ B │ MisclassificationRate() │ predict_mode │ 0.0422 │ -│ C │ BrierScore() │ predict │ -0.172 │ -└───┴─────────────────────────┴──────────────┴─────────────┘ +┌───┬─────────────────────────┬──────────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼─────────────────────────┼──────────────┼─────────────┼─────────┤ +│ A │ LogLoss( │ predict │ 0.318 │ 0.035 │ +│ │ tol = 2.22045e-16) │ │ │ │ +│ B │ MisclassificationRate() │ predict_mode │ 0.042 │ 0.023 │ +│ C │ BrierScore() │ predict │ -0.171 │ 0.023 │ +└───┴─────────────────────────┴──────────────┴─────────────┴─────────┘ ┌───┬─────────────────────────────────────────────────────────────────────────── │ │ per_fold ⋯ ├───┼─────────────────────────────────────────────────────────────────────────── -│ A │ [0.363, 0.328, 0.281, 0.313, 0.355, 0.22, 0.29, 0.403, 0.309, 0.386, 0.4 ⋯ -│ B │ [0.0, 0.0, 0.04, 0.0, 0.08, 0.0, 0.04, 0.12, 0.0, 0.04, 0.12, 0.08, 0.08 ⋯ -│ C │ Float32[-0.21, -0.181, -0.144, -0.152, -0.208, -0.103, -0.164, -0.257, - ⋯ +│ A │ [0.263, 0.309, 0.294, 0.313, 0.317, 0.216, 0.388, 0.297, 0.427, 0.275, 0 ⋯ +│ B │ [0.0, 0.04, 0.04, 0.0, 0.04, 0.04, 0.2, 0.0, 0.0, 0.04, 0.04, 0.08, 0.08 ⋯ +│ C │ Float32[-0.133, -0.164, -0.153, -0.156, -0.175, -0.117, -0.246, -0.158, ⋯ └───┴─────────────────────────────────────────────────────────────────────────── - 2 columns omitted - + 1 column omitted +Apply `describe` to this result for a named tuple summary. ```` We finally note that you can restrict the rows of observations from @@ -1082,51 +1085,51 @@ predict(mach, rows=test) # and predict missing targets ```` 45-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, String, UInt32, Float32}: - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.212, Iris-versicolor=>0.565, Iris-virginica=>0.223) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.989, Iris-versicolor=>0.0114, Iris-virginica=>4.52e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.988, Iris-versicolor=>0.0125, Iris-virginica=>5.2e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.986, Iris-versicolor=>0.0142, Iris-virginica=>6.58e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.236, Iris-versicolor=>0.564, Iris-virginica=>0.2) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00401, Iris-versicolor=>0.257, Iris-virginica=>0.739) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.99, Iris-versicolor=>0.00983, Iris-virginica=>3.51e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.989, Iris-versicolor=>0.0115, Iris-virginica=>4.51e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0575, Iris-versicolor=>0.445, Iris-virginica=>0.497) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.3, Iris-versicolor=>0.612, Iris-virginica=>0.0877) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00131, Iris-versicolor=>0.194, Iris-virginica=>0.805) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00405, Iris-versicolor=>0.259, Iris-virginica=>0.737) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00183, Iris-versicolor=>0.21, Iris-virginica=>0.788) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.15, Iris-versicolor=>0.514, Iris-virginica=>0.336) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.988, Iris-versicolor=>0.0124, Iris-virginica=>5.19e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.333, Iris-versicolor=>0.615, Iris-virginica=>0.052) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00342, Iris-versicolor=>0.243, Iris-virginica=>0.754) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00366, Iris-versicolor=>0.249, Iris-virginica=>0.748) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0363, Iris-versicolor=>0.416, Iris-virginica=>0.548) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.209, Iris-versicolor=>0.555, Iris-virginica=>0.236) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.392, Iris-versicolor=>0.584, Iris-virginica=>0.024) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.171, Iris-versicolor=>0.546, Iris-virginica=>0.282) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.225, Iris-versicolor=>0.589, Iris-virginica=>0.186) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00249, Iris-versicolor=>0.225, Iris-virginica=>0.772) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.983, Iris-versicolor=>0.017, Iris-virginica=>8.92e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00176, Iris-versicolor=>0.208, Iris-virginica=>0.79) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.262, Iris-versicolor=>0.597, Iris-virginica=>0.14) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0433, Iris-versicolor=>0.405, Iris-virginica=>0.552) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00694, Iris-versicolor=>0.287, Iris-virginica=>0.706) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00519, Iris-versicolor=>0.269, Iris-virginica=>0.726) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.161, Iris-versicolor=>0.568, Iris-virginica=>0.271) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.984, Iris-versicolor=>0.0164, Iris-virginica=>8.65e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.176, Iris-versicolor=>0.569, Iris-virginica=>0.255) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.236, Iris-versicolor=>0.593, Iris-virginica=>0.17) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00228, Iris-versicolor=>0.222, Iris-virginica=>0.776) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.04, Iris-versicolor=>0.419, Iris-virginica=>0.541) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00363, Iris-versicolor=>0.249, Iris-virginica=>0.748) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.302, Iris-versicolor=>0.606, Iris-virginica=>0.0924) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.988, Iris-versicolor=>0.0124, Iris-virginica=>5.31e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.987, Iris-versicolor=>0.013, Iris-virginica=>5.6e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00524, Iris-versicolor=>0.271, Iris-virginica=>0.724) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.985, Iris-versicolor=>0.0155, Iris-virginica=>7.67e-8) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0264, Iris-versicolor=>0.39, Iris-virginica=>0.584) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.245, Iris-versicolor=>0.582, Iris-virginica=>0.173) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0246, Iris-versicolor=>0.38, Iris-virginica=>0.595) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0903, Iris-versicolor=>0.528, Iris-virginica=>0.382) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.979, Iris-versicolor=>0.0153, Iris-virginica=>0.00547) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.978, Iris-versicolor=>0.0163, Iris-virginica=>0.00586) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.975, Iris-versicolor=>0.0185, Iris-virginica=>0.00674) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.134, Iris-versicolor=>0.524, Iris-virginica=>0.342) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>8.56e-5, Iris-versicolor=>0.228, Iris-virginica=>0.772) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.983, Iris-versicolor=>0.0127, Iris-virginica=>0.00446) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.98, Iris-versicolor=>0.0149, Iris-virginica=>0.00533) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00347, Iris-versicolor=>0.395, Iris-virginica=>0.602) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.135, Iris-versicolor=>0.521, Iris-virginica=>0.343) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>2.86e-5, Iris-versicolor=>0.19, Iris-virginica=>0.81) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>9.2e-5, Iris-versicolor=>0.231, Iris-virginica=>0.769) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>3.58e-5, Iris-versicolor=>0.198, Iris-virginica=>0.802) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0527, Iris-versicolor=>0.52, Iris-virginica=>0.427) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.978, Iris-versicolor=>0.0164, Iris-virginica=>0.0059) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.144, Iris-versicolor=>0.518, Iris-virginica=>0.338) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>8.25e-5, Iris-versicolor=>0.227, Iris-virginica=>0.773) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.000144, Iris-versicolor=>0.248, Iris-virginica=>0.752) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00414, Iris-versicolor=>0.404, Iris-virginica=>0.592) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0957, Iris-versicolor=>0.528, Iris-virginica=>0.376) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.215, Iris-versicolor=>0.494, Iris-virginica=>0.291) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.045, Iris-versicolor=>0.515, Iris-virginica=>0.44) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0742, Iris-versicolor=>0.526, Iris-virginica=>0.4) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>5.45e-5, Iris-versicolor=>0.212, Iris-virginica=>0.788) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.964, Iris-versicolor=>0.0261, Iris-virginica=>0.0098) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>4.02e-5, Iris-versicolor=>0.202, Iris-virginica=>0.798) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.131, Iris-versicolor=>0.523, Iris-virginica=>0.346) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00176, Iris-versicolor=>0.361, Iris-virginica=>0.637) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.000152, Iris-versicolor=>0.25, Iris-virginica=>0.749) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.000158, Iris-versicolor=>0.252, Iris-virginica=>0.748) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0393, Iris-versicolor=>0.51, Iris-virginica=>0.451) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.969, Iris-versicolor=>0.023, Iris-virginica=>0.00853) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.0425, Iris-versicolor=>0.513, Iris-virginica=>0.445) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.097, Iris-versicolor=>0.527, Iris-virginica=>0.376) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>4.55e-5, Iris-versicolor=>0.206, Iris-virginica=>0.794) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00276, Iris-versicolor=>0.383, Iris-virginica=>0.614) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>9.93e-5, Iris-versicolor=>0.234, Iris-virginica=>0.766) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.138, Iris-versicolor=>0.521, Iris-virginica=>0.341) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.977, Iris-versicolor=>0.0165, Iris-virginica=>0.00596) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.977, Iris-versicolor=>0.0171, Iris-virginica=>0.00619) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.000143, Iris-versicolor=>0.248, Iris-virginica=>0.752) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.97, Iris-versicolor=>0.0217, Iris-virginica=>0.00804) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00164, Iris-versicolor=>0.357, Iris-virginica=>0.641) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.124, Iris-versicolor=>0.525, Iris-virginica=>0.351) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(Iris-setosa=>0.00131, Iris-versicolor=>0.346, Iris-virginica=>0.652) ```` ### On learning curves @@ -1155,7 +1158,7 @@ curve = learning_curve( ```` ```` -(parameter_name = "epochs", parameter_scale = :log10, parameter_values = [1, 2, 3, 4, 5, 7, 9, 11, 14, 17, 22, 28, 36, 45, 57, 73, 92, 117, 149, 189, 240, 304, 386, 489, 621, 788, 1000], measurements = [0.996922160303613, 0.8577055250248313, 0.7981519638729986, 0.7620389845823741, 0.7323274239258524, 0.6767194293543304, 0.6456183342973714, 0.6019803869670477, 0.6148576306918716, 0.5858385363538815, 0.5420341283852436, 0.5502000369070692, 0.5408171567564948, 0.485989693319718, 0.4936070614522306, 0.4330706602760043, 0.4150256223224028, 0.3341514384124167, 0.2983904685917199, 0.2815979750419263, 0.27383412873432805, 0.2781026630466714, 0.2515121244169944, 0.25689560519828347, 0.2283805057524257, 0.2597360768185133, 0.24229163132135162]) +(parameter_name = "epochs", parameter_scale = :log10, parameter_values = [1, 2, 3, 4, 5, 7, 9, 11, 14, 17, 22, 28, 36, 45, 57, 73, 92, 117, 149, 189, 240, 304, 386, 489, 621, 788, 1000], measurements = [0.9792367389813159, 0.861137424251525, 0.795194663958999, 0.7178450923341597, 0.6672852524240639, 0.6298719732484978, 0.5722341032114211, 0.5468304476118365, 0.5176573197034343, 0.49085053804968687, 0.4656446207911134, 0.4495242488591051, 0.4099614796325163, 0.42257271926898354, 0.31717065032020536, 0.3321625563432286, 0.2845664781923225, 0.22536592494730678, 0.1853560701976897, 0.24943364437705945, 0.24673236950764432, 0.20234975793565377, 0.20691858169733499, 0.20911653361243762, 0.22259957311318937, 0.21965174517509872, 0.21279597941067163]) ```` ````@julia @@ -1218,16 +1221,16 @@ y4 = [n_devices(row.salary) for row in eachrow(X4)] ```` 10-element Vector{Int64}: + 3 1 - 1 - 2 - 1 - 2 + 3 0 + 2 + 6 0 0 + 2 1 - 4 ```` (b) What models can be applied if you coerce the salary to a @@ -1253,10 +1256,10 @@ pretty(data) │ Int64 │ Float64 │ Float64 │ CategoricalValue{String, UInt32} │ │ Count │ Continuous │ Continuous │ OrderedFactor{2} │ ├───────┼────────────┼────────────┼──────────────────────────────────┤ -│ 1 │ 0.303334 │ 0.803837 │ male │ -│ 2 │ 0.765226 │ 0.061673 │ female │ -│ 3 │ 0.88446 │ 0.716328 │ female │ -│ 4 │ 0.255475 │ 0.239904 │ male │ +│ 1 │ 0.754755 │ 0.208445 │ male │ +│ 2 │ 0.874161 │ 0.399919 │ female │ +│ 3 │ 0.906202 │ 0.19441 │ female │ +│ 4 │ 0.718781 │ 0.818259 │ male │ └───────┴────────────┴────────────┴──────────────────────────────────┘ ```` diff --git a/docs/src/notebooks/MLJTutorial/03_pipelines/notebook.md b/docs/src/notebooks/MLJTutorial/03_pipelines/notebook.md index 4f4cbf6..f8049ef 100644 --- a/docs/src/notebooks/MLJTutorial/03_pipelines/notebook.md +++ b/docs/src/notebooks/MLJTutorial/03_pipelines/notebook.md @@ -31,8 +31,8 @@ x = rand(100); ```` ```` -mean(x) = 0.52573272422357 -std(x) = 0.3020501201265377 +mean(x) = 0.47696749924138343 +std(x) = 0.29728842148044776 ```` @@ -46,7 +46,7 @@ xhat = transform(mach, x); ```` [ Info: Training machine(Standardizer(features = Symbol[], …), …). -mean(xhat) = -1.8596235662471373e-16 +mean(xhat) = 2.042810365310288e-16 std(xhat) = 1.0 ```` @@ -497,10 +497,11 @@ evaluate!(mach, measure=mae, resampling=Holdout()) # `CV(nfolds=6)` is `resampli ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-661 +Tag: DeterministicPipeline-190 Extract: ┌──────────┬───────────┬─────────────┐ │ measure │ operation │ measurement │ @@ -508,7 +509,7 @@ Extract: │ LPLoss( │ predict │ 176000.0 │ │ p = 1) │ │ │ └──────────┴───────────┴─────────────┘ - +Apply `describe` to this result for a named tuple summary. ```` ### Training of composite models is "smart" @@ -626,23 +627,24 @@ evaluate!(mach, measure=mae) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-106 +Tag: DeterministicPipeline-398 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 162000.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌──────────────────────────────────────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├──────────────────────────────────────────────────────────────┼─────────┤ -│ [160000.0, 170000.0, 163000.0, 156000.0, 163000.0, 162000.0] │ 4140.0 │ -└──────────────────────────────────────────────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 162300.0 │ 4100.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌──────────────────────────────────────────────────────────────┐ +│ per_fold │ +├──────────────────────────────────────────────────────────────┤ +│ [160000.0, 170000.0, 163000.0, 156000.0, 163000.0, 162000.0] │ +└──────────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` MLJ will also allow you to insert *learned* target transformations. For example, we @@ -666,23 +668,24 @@ evaluate!(mach, measure=mae) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-264 +Tag: DeterministicPipeline-611 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 509000.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌───────────────────────────────────────────────────────────┬──────────┐ -│ per_fold │ 1.96*SE │ -├───────────────────────────────────────────────────────────┼──────────┤ -│ [162000.0, 2.2e6, 181000.0, 161000.0, 176000.0, 172000.0] │ 728000.0 │ -└───────────────────────────────────────────────────────────┴──────────┘ - +┌──────────┬───────────┬─────────────┬──────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼──────────┤ +│ LPLoss( │ predict │ 510000.0 │ 730000.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴──────────┘ +┌───────────────────────────────────────────────────────────┐ +│ per_fold │ +├───────────────────────────────────────────────────────────┤ +│ [162000.0, 2.2e6, 181000.0, 161000.0, 176000.0, 172000.0] │ +└───────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ````@julia @@ -693,23 +696,24 @@ evaluate!(mach, measure=mae) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-300 +Tag: DeterministicPipeline-592 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 172000.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌──────────────────────────────────────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├──────────────────────────────────────────────────────────────┼─────────┤ -│ [171000.0, 172000.0, 173000.0, 170000.0, 173000.0, 171000.0] │ 1240.0 │ -└──────────────────────────────────────────────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 171500.0 │ 1200.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌──────────────────────────────────────────────────────────────┐ +│ per_fold │ +├──────────────────────────────────────────────────────────────┤ +│ [171000.0, 172000.0, 173000.0, 170000.0, 173000.0, 171000.0] │ +└──────────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ### Tutorial 3 Resources diff --git a/docs/src/notebooks/MLJTutorial/04_tuning/gamma_sampler.png b/docs/src/notebooks/MLJTutorial/04_tuning/gamma_sampler.png index 253f458e693d99542247096265e529fbaabdbcdf..8ddb87e85997807859d01a10d6bb417551fd3050 100644 GIT binary patch literal 9276 zcmcI~2T&AUyJjnjAP5L586^k`ND>JupddLZL2}MH#~Ff>RX{)nN0J{wB%4`gqQ5pX~h}l#-Op(i@&B{;`w{&ar!TcI(!6aX^5@T=uM2ACjdfo0Y&xqJi@7jZ2_;$xAk#@)*y1|a zV&pQ^5~j7>Iu9koU{7sqY^1`-%EeLpzV+5{BNY6|z4+&Gu5gG!j+2DX-g3-fvv8Y> zZ~ZhHjqVQ-iR~~q)fs7-Pc)%HDlN`!JMhtDsAb1F1d~x?J?{|zRwhL8*lQ)>yG~mm zs4P<5@74vVitN9=yJwZ6Zi{mM5eW$iO6>TSvGI6)9#vJ<%2@(gFcnidxw+8{qKzKj z4M*KhwzjtRxE}`FWyu`;rLAjg_Wcj-PEJl(Sy|VOD4`#5cTfRo`P@21=JAQ0wbHdn zi^;ucF2$>iYB!<4d(DTcBH2n$+!BNYDlBCEkiFwSqrMYCGZD^f^iZHexDy0%&dv}+ zBJsz^$2B#N6M2QWx%1K^*&}g;P!=VAj<2=Jh=rPVqoG^dpy`LWSZ~U4Pbj*KX^tLB zy4qCqMf!{_`K^6*UoCdk$>Fr6S{q?dVXq@2BcrU`JzWn=PfN4c34)^h2p|k$fJc^2 zspV9)n{3u#`Qgg>(&rzX;y1C?Zea}$0!YiABprSIwT%sw<5bOk^>3J^9D6Xrz}Kro z*@GkbdPfs&(YG6X_iYn+4A%4sz!1I;FZLwYI$=?vp@fp{2kR3{y{S%Tbbnv@*7fWB z?#q2aSIBQsQ>Ti$S&o<4928z?czOY{e($rpn6F)&R#oLakRk8y?|;Ax!N71F_$|9W z=j7z{N}R!al0*@$U0q!tK9H7;nzI}3oR9A&1|1(TKV|7Oh=~cRB zNC7?VFIS`n1>%6h6d2SBr9O3A>>4k(Lw9z9>9g7O5WCxaI2a=*ABY+Dtb0tQW^-r@ zmb=DteVoO1LMoa`EH*CA%+&Ogf(5K_RsLTy#dLmbVCc5cRG=Q1Q5T@^!C?KrQl78d zewnTVOy8uff|M}GfjS(VuL%ha<wG&3Nlph5cAbf}wBr5t)aH$!y#00Ql+gQCeD> z%Y3VjimpMmYo=~K4;L4@qhqWdb~sjQMa=Bs;ZbVUtD>sP%gvnxh6CHIw_0tx2+c@> z;c=)WiFmGQ-@cu?7>PevhaD{>qz0V&NVYeIkfQiI7jcHr6L9-CSMmkF6m{KUA{~wY$3uHW6-unVFe0ex*NM zc7K1ryu2K7@csMuw{PFxdYG$`Q&hxpD2F=8zkOaX8P^(s?M%>Vc^1(;^;NCYwj&CShyJQGAdIW=|E(ca!}))rO% zmr(v|t{A!Cb(!?yYl?VUQna9u(6}d3=%-7PN|u}ZO|sV0_H%eNEKKMyzZ&wOuug?B zS;a~Kd5^9{AzPz!q2Xii>lQC#W8+<-g4|q^t5>fQ5#0$M8XbkfV7rlT-n=>PI^3AD z<1ckaPStv0i%g{S^^+cjT$dDy+TGcqtf;N7zCN&X$T#w`{bm}Mt4P#58>M_(UbiD| z&2y&3{K>^?{U{tlj*Z4;>uj*n37e}`xKp_~SP!NSTKuVB%eBKBZ?w7_6>qLNo) zx7=!55q8Hw~+?)xi@{I;c))$Pab_k@KLCxUy6o8-5 zGV*&QB2sn2L<0r7!NzeYKm5U8QehV_Uc7O&1-#h6HmDM99~_AL?3nxd{`u7rm+E&g zw!b>M`;ZsZ$j+}0fG(IipYsw1#Vbwp>fQ}dVO8*>oecRnI2>N#gq>C)Cx(99e7at| zX*2jO!f?@Q=Eoa=kWT&g;uQcy*0}#3xU~+7_jEDQY5e6~(41y8uIyZZ^sfNu{15Q| z-!}U{IcYw;Ko|HK^JOImz@kZXVxk_X-6VT&Z)jPB0y}cQYNz)})~TDY5hk6>XKPB> z2<$~n!#&9Pa=6o~GrN43okzowL?aV6f8OAj@;%g7uvIAb?`gn4A`GtM7tG$je?K)f z6|xR?NO1ewOUTL;0OQWCbm+yyy**MgvI?nd&`fjE!}D#0KUgg!C8GBHZHMk+ovxI+ zHXe(2^RLvNogFR-ny+f5bC9ZMw1ps!#iVTqK{_$18ewkqjo{bNCi&qcT!>jo0RMULH3&Ik{vz z*iryED5_bNT2Z=UXH3TTSGZ4igpT9kGCcFo?r3HFKaK5;V6NY0r-i>?v+O z`n-$&e)sMjov5q%E^=^iQ0(c`TMu+!5RHL7>h0~VqoX4y7yjrn1XXcQFyK+3B|AE9 zfFgeIzzDwGQe@I(*_)CaAKy|rSr799Re8Mr_3PJZcE_(etv#O z9nH}+orPqy^-7M7H#Rec8pWz;BO%Jr`#zz!In zc;=PwTbJub&5!TB)XjIuwXwD?a$&s4(DvK?ajipGRD^{N!h$x`#hp1SV{{czLCL?I z&tDJz)<(oDWFHlWs%0OyWOWg4PYn%~1)TY#595MA%Ez&93x|?1`mT=TpWqMAde4rd z3kx^XWuv)wtp({MBqU}BKrj9B>64)G56FL@t*vchqPF~QwgzWJv$&SU9S1#SdL|7! z7Y2@ah5EWn33!*ga5z`If-Lo%+|`ha+YjOuz+~Qgx0z)UgcEoD72ePg0EgFEJ3V{$ z%*>3HIRkdMfeknl!(uQPZC%}QFn_bh0!VH_L5(s%pt~@E!_wU|qOjehI zN?D8~xy9Yy+w0}!b-1%L4wg-kxBdmu_VzYtdBSJ=mP-@wB`El~FZUP9bSR`9nSZWHQ1Ysrc>Y2drTn z&(fg4$Kg)34@+OL44+j?W~_F;8LtT#ufGFG%BkuV@b{mf*S|>3|KE3ZFd9ID>M3q3 zlN~b!y+p8#V20_#g^$cqYeqL9cct*ns26(ZP+2Xd36{0($HxzA_ts3NVuCLWrl$i`pk2wqmb97K{LTV; zG+)mPf8c-$u^8jhD)>Od|2CC@-?TY|gqnz%nnyo8I$G=Ajo^<}1r-&1^@gfMqlmB}-vgbX|xd*IwST%PiiP+2Uhn5kRPyat~!sCqvl24O_Mn~I=UY5{wf7fdh_Of zbii>?gP8NokGqNpr9>g+Yz<0Dn$X#aNSXAouq({XQ&R@%{n{ILd^9*wy=9*eFl9(0c?N#4z*SdN#_4E#_T3ol4bx zU8z5;=P-sLYL*v%t4aP1g_?H7|9C?HxJ`VAIb{SJKmT@aA&MoKhBTB_Hj0g%ol=rr z>9h0pTuWLSi)4Ex)_($&2fC0HU_^P|IR12RP!-V9X~3!h{Tdk>Vi0u=b}|A2643V7 zuV24<^~&0M82}nFGw2BCy(^5AmXeyIIxU#g6))pb|ZO6P~gv>|FlZ=WB2`a34&5$HMbumpn)I) zN3cQiV-GseS&Hga0j>G7S8SOHtP8$HVhGCoH%7&O+13BuNqf&sALkpTT_=j@>W}A`r5)tB zgstfP%-?h>eIYRa4l0=%EeAtc<8Ysf+T5`4dPAqPchHZ}t<8u3kKCB%YFEoXbS9S#eMOIf=2Rh;T_V3jE7uA6+bjF{v3;?NR zy6bs5r2~V5VPuR7MDohY^`7gx-rge2&49B|VgYbH0qS^bwuze8u+IO)_k;J+J$svj z*x{uS$7!CmogHU-YuxIS;?dyO7 zE%e{#vb!2su}jWjkRH;lMl1gE!m@GVVUA4tSe46si3J)bA0p}Udmt0g1%o$8kb*kxq5nGFd9g zo@*Ero3gdFOP4R-Wn-JHaxq&O$_8o}tZ7-7`m0yh>F7B1jbFZe>FVlwt{*k4Yip@k ziYJQ{(?BGsEcM8zv)0(X!Psq~EkreYA*VXnp* zuW~?ZsR>V1`UYrKr+Ep`PVfAQZ3K{xSQH~Yy&@5=Fz)Cm^V_#C-HSj3 zM$tX$1Wj$isbP3^|Djb6TX1u0Yio0}(r3O|peV;G940mc&S1xNe+8{P`s973!rpzC zjhyUB8XZJor3sIwdz7P?z=M#FSCA%(QOJ~+mj|YVy|wjcG1$SU`|8|hHB3)D?SaGw zu8)GUva*s=ndLA9@$}^2qN2|a>Ix2a_IKY()v}YGcojUdI|oE{#`Y_aGvnXR;6K8X ze|d-loLlbNP4g@7)+eNSqVLCb$V4+u!T0E)B7$b&r^)ZGSZF?aPiA=YfM&NKFE8Jq z%sS@e{Y7Vsa3^ntq#+6YhaR~_QZWe0erY8riSXQa`d==wHu`wEb>L=d4*WPI-~zJ& zCnqO^!JR<2#$u7n)$Vxns9|?+?}q2KwX@?d@#r-p%D};AaH%Jzq{`G!I4Dv- zA{afSMe%1*IKm>rIr?!OEI(V3EV^|7_Nx`k`085V?lX9hnX{#vfkASS0cZRl91hdK z1g+db{k*=S@Q%b)vR0gR%_Q>A0$N-rHr5_6+U2)6LRsTIwFYOu6~CB*%@H7HYN;R{Zk`M6h3pGVhsqt~e`) zJ1u*s@IeTT&;+g@s?J-N{-?U1ZFu!H^tpPet4EIRln!QJ$-Gh(F;f{jqf3E(c1Ja0#d#hQvimqAsrNxijjENtJ$Y{lmfC>>%Y_edc|L!ht{FV2kb35;v z0uy6LRZA3%$@5JR$1@4DxaM@KL!S`2QDlBdOXUmIgBUqB6biL&NdK(%IXp-m-B5D2 z@>f||dLz@f>}~8)4;igMGiWZDOLO!|batP}lCx7$nwa^MEF7>-6Fz+^v$mpHT=%l1 z-&amU!@Q>P6P1n?e#uaN7>s z{P5l562NSM`NuCVKHVRJi8Yvq0n6#O-`*>LNr2?Wu*#a6nN8S8U%PhAvJ{tiOaEHi zn4`|Pu8^tmeqv&Atj7-wrW2u zyw^I>!B_|-?icziWV}5b-!J6iEs$_u{+*E?}a?5x>*i-vor4LpD>= z`;$35Af-SBM@K|32X}pWWY?c24WvIn#R|O((7JqQ-99zDWtv#1KZKf^?CQ` zxk>cmMuT#(Qw%CS#30r|y)nh-H47$TscSL(_hUVwksB_A$=(ix4}A1FQG!vdcMs*_zFELD#oOp~ z@t0hZke?PPB4+eUOSpUNDUH5t@cT5r6N!t^5U23ThK{ivugv#6V2V~?tN}O%|joQm=!h=j9w}umJB^Z*}-1?Hy6##7XtRX zT}-spgBC+B5Zri&;AKLXD4-QsSuQo9DF!lzWiI8ivakdjJWeh!YVdE&bYbv1`Qrhs z+*t!UIyzE9sihv4rjLFB2NId_!;R#ek|Jj6>OjeD@g$8g+@xi~bNwv38vN9EZ+UI3 zG|eaiAf-z3W8#T!%hK1m`v|oW^pZ!r|Eb?DPaEjBiTF0n>_^8RJM9))+aQys&U(yIjnfWv|BkQp@Bg zwl491_$9A;NR@J@r};})l7600T>rfot_-!9%<@@cQ&QgueP(Jj9l`{d+n53W4Puty zaCYk4QBHHUS*m6ih9>!DY&3V1JlovnorfIpTrW}_8;fPq_4iKN1)BT5Q^4MW#6*dm z>ZB#O-`btZFT=7FsHA}H&!Q||)PCKH5!G6Uj;0osJwV_9yY|oSF2-r|2_N59Px2E) z2MFW(H+?$M==Il>5LCY=yeJRmyimWo5zBM`KA>FQ8`Vo|V5>tMW(`3c+-Pz9T4@}I zs{eeXFoM@;I$J%9x8Zm|JeLr17C)H4BhU-LSMmc{2OzVc4opOe$bbo8+yLS%sEC<^ zVYBw@*BQWOX#qFQO7*wpqhT#Ki{y^*jlVY)3g7jLI1sVZ-1HZj~SpYIBa&q6lQ_#~+>Hu?tltF|K z+&D-GxnAdCs$5vHDC_<-^(>Xlj0|@$P-f;gCNym$Hdg@cI-b9MR^^BHjQbt-GXTM3 z@FgWBB`IkVw1(@fGA|ooZK4SvbC#Bt-mV1h+|t5AX1_IqUaS}hnp!3%CIqraSBGu) zf$f3ocpgN!jwx<>o;fxFbgB>>=X~l37~H_X0B)k;biZ&kPiMZWX3cguN0W+OX+}gP zz5lyL4v@P$>p(EeMlrZp+pI^Kd~OV@E`Pxuh8-<5s!X~KoIR%y$(Xpfc}y(Bcc50b=i7n^ug!4)=^YybEUDPF z#nTh~G4c)26~G-Vs^io*VJ+g;)6z-+Ny^^xt5eb+Af*nGG&#yH;Ia<}6A7k(l!oue zKx{&Sl_1@N2P#F^lN{mzzy(B5{Ti~}kE9m{bK?g{sUxVT zNR#Tn4(;FeRA_X5dL5> z4N>swva+Fp_OPG@xLmoe!ONF3UoSc|jUAxs>+1G=>n+r>1|!IHW>|pA1XE-&AYB2H z>sFx8$QeZX(q+jd!xIvynVSVLEpal%&g|ft{@?}y#Y;`a0_91>47`)>Fnr%{_q24l z$d%Hu?k-!Ms`X?n`E5#etOZva9e*V zWOjCUFNipTEC8&ktGl%4@LaB9)Vxe2T{-;7HEJF#3f#}{fsvtTW?o(%=*hs{DYrhi zLynJjMWb@Fvw?vGWc3GvcclqH-<6Ja0Vw1<<|R5h07@Ms=SlNblau=K3Yyy5%k7-& zA!i$*Waa#NKG|zcNfI^gJ3m0JRCUM--lOJ#gCecr7J3FA6m9AxHjBwX>Q(H zzkK;JFe6t$DH|KlfcOmO?)DmN*lVPuAEKgw?Y%qmGYDjhxC0=ZLV|-q^nk*MCAO^!!vEzHnKI9hkp{z`dR54*a!guXCV2wqG=@ofO27B0T>4lT04bl zNJT|OK}KhD`skb6JeImUpvPE{4n)6_Qc}aV2vycE$w)B`^T!Q()XytK-T0!iby{{sW-85;lq literal 9095 zcma)i2UJtr*7ZTJQba_sON-JjAYDMZfP(bii}X(DMF^k>qBIo&ktQI$_Z|??&|9RJ z2%!W*?}YYk?*EPPz5l-P-nfUs2qY(a?{oHAbIm!|d8?`{dztJG83aL>pUKInLlDst z_~jru2i^%nTDO9ii)IS4GSKPS&)0^mXb56}p2Eqb2@nIF|=;vVX=2t+=Job4| z^MGpa@Lb)pqt%o|W!W1qFEvxRXdR~4gs+#cw+lWp*U2ry*NGL11PgK8kK_a|wdXf3 zLy}Cbf)Er#iGd(ZDZlfOD(QcIGd2AZcctTa<&&@*7cN}TtEMZ^W~X|qIPMity|=yr z@1l5n*I(tu81tp7acTF;Lr$qIG-<-)Q!y03kmzTeN zW9-BlWT?k?7p1cx#RjLM$ZwfX^lu+__wI@)`FT|y76xL4V&vx*7Z-6-G(lLQh^N@n zjc)F6?K}@JFE23C?~g?rf!@4!kuK(Qmt8#?@8=RhIEuu>i_ll9nY&P+d?*}(*kDK^ zDEsyPzrOj^z|M|sO+x102bp`*J91K~>tS5T;+e?0Sr3d=jy@ib&+PGUEtln>a#Zs5 z`Q7w#mx~Nq%|;?m)~8#_tb0A(VP5Ma=1~v%`E^}_L)T7iaZcn~K_450k65Umxla8Zo(> zo>MM=CMoX?C~(cq&hqo~(};RI_kDdll>hw?Fb$+j+`=uK}&(-A1!0C+^S1 znMg@VEgPN(m-)3b*>AHwKRbT4?GiO25C+a{W*_F%@m-jFMjUM=++}vCyg;-sLffwlp!aBF3d5%5BcxM@-RQC}Takz{{ z>~fJp!qjwhPS+|bDR1Ad@joV1R|^RW3L3!k_(Io2%=uR~-*UCR5w)36+rejOaEYYd zgPe=|&JI=7P~bjNWI%F&rMX&da48hP2g&!~APD;O|5B1g1qysY7TkOTQ*3TzS20%0 zE(11glCXoXvg-cw(0)j$_Pt2iYPuS@!pn`GtQ^ zfq%~nue+F=T*ediK3mPfmrf6Q13VoXJy*tBu1F-le_w3caxq3e13oL{go>nn_UxHP zrs5;F`Eb+ew6wGfmu`gL=CfKEt02H_E<%B;HRJm|Jw2RUTwEL+_5R1TscNaJiituz z;^HR_dsB7J)6F55?{2Tuf#E?m?XHa9EAE40Gb=;)A6leN{=)gV%Q*QOfeGd@z5x-ImSmzVRH z|K!jww;3z91$zwpD-<|7ktYQF{ioq=ZEZsCI8{f-g26pvsOgEW`>F`oD{*mS30i9E z^nM!)-gx~2q|a97be#wmi*<8zBZc|+_@v854GCT(CMLdkajM#-L z;O_0)wx4%4Hga-uKFDX#pmtU#4KO%7-atbm-Q+B^ZA{S~v9Yn7MzxyvKH#jSrKG}p zDlr+Al>%ie!byd9?eXNDrD9u*{?>JCWI~Y|Y%w&bN41ow=X%-&+I-|#0D#h`UDWuK zFCwoK5)$Ox-@pwjS&pA=|6cw6c|?Ayx^UjoNOh*Oh!t|`kxo`zn$6ssL?U2veMICy;Qz{ zXXRtnpY0}Aqm7IZgm>-PcXDuWP_-FL7B(GY^0qEL#uGus!VBSWDH0Widx(2o1(IhfDvf>zt~NHTFhDA9&G`O!cRfobMOj(dxXNkr4B)to9n@AJ zh_fqg-?-eSKkZ4_Habe2kgdsf{jCgPR4Y%rz__6n+Z-GmOdoLM0uWYGvMKSA8(qNB z#=+LYrQqrnYET>V-Bt}a1T+2UE5Q6F2JV=>-tbDQ4#8J7oZY0KyA8O$ePO+-*LvJk-J#I~&`pprD;g>jy_) zzkYr9?j6XCVH+*inxnQ4_iJiu2zn9{jh{Y!DjZ5jFAA^C*47H(XM}o|qe%)B@&?CD z*~OwySe8rAcpm#2QaKB4SE0NPZ;)j8y|w1-?U|16g zvw*;EU#j%he7E_nO!J3QseKM@xth5Nf^dO{4;QVarKPRAxP8{qN!;Yvkvip`aHjq zrn|d)=z>j!zKBDH^#H1>a}`YYnlf$@k)+H*Nn&WRSKz#QR(b$`J2^SYG$L;?tn*X~ zAeDD@-2}k{HWxO#G?=Gf;Mp2_H5N__H9gdOv0cz_LqR9%z5e^L^rg;Hu+`DbN}KEJ z`!iu&nrwp~nG{H18yg!gE-rjlU7v+LmOvpKE;M5QAjdHw3a%;)&$=|RKL!mTL4g{@a&mhqyO>dT&GI1 zsIBb*1Qm7+)Nuw&O$QR455L35$G5jS$sQwbZe|AfeH|Qw92pr2I6cIC|Nb3>)cx1> zAPy0SyA2+AZEx@D;0vJCGAr~Vh6e|q$jYLA{&bqCE@@~;q8tE(Urmk2Z{PX&9AMsc zK3mMNSwO}3_zC;_+xX$4uV0&*&qxwSSR^v&)vHLFN4yqo@4?j!=IMaZ5bNKGBO)SR zQ-W|k<^@;2v$OO1J-`GN6_plUKrTW;E}}hvgkWZ7zGL-|Y}k)TL~ zlmIXYahs!k^5hAvh$jaO_WffsBNZtuDL76{s=#0z=S~^byS5*<=HyH>JP1<;ij^*E z;V(c`W&I?_7|Genv%RqbC^=4Q`Mcdsigz;iVl5vz>)Uxr(Z)Qv$vSKU9#wV}3EEm*q8wL8U`y z>;u;tQ9;4IvR-lD&ewehyQsQh- zVINm85KD?5*p~qeCSd7TPEo_|7bF5~>GkW^&tD{8-`pIl^{{tv;5e&oG>=?obFz~4 zOU!=Y`qM!)6{sTFRoM?72aV_y#_w)ViIrt!{5ZKzB&qUT&5uj3)B;G#%oQgmCtnbE zbzVhkht)3FLL)wEX$b1msOrFHxOu1A#Goe_E(lU_ew^}?Uxa6 zWlO~?1D-+81w-NQc+Z9igAO12Y4nJ^-+s&O#)NLn9-x0KHt>sQ10&rMBjBa-qjz=ODb4pGbP);ieRivS)_VL#kzw(%=R z!yHup6+lTi+1LhD!hXzTD8wygX|W9g2H)Ay@idA~%({n0LEtj#{} ziT{D+TbZ)8{j2R?x}&u zGBZnKx*(~usDCC4e{GoD3;J+hR{0Bm`w0_~H2@8nci zSSarNyJ@~VAz*)+7}P+Z@&f~*miK@Ubacq0wFZCv0@d_Ku>}T+09;z5=sJbBGWT9eIv-l*uiq%j%S+(3?7*DtR&8KWcmsg%`6+!i{5(87 zhkI)(N=nzsg2`KMJ+-aS2YRelz{_h7hr@-2k{stZHa3p_a8d#HWndI<9W`9Bfb{Yf zokVoHOEcf8x|zg+am4+MswJKgQf8!jghINX?<^&6sa{^i2aS_@o;saC26r%*FJGo+ zQC9q`YBbmu>2lDCy>Nf_j1>0Gq|qPvLJ)ryqWA;Erl;V}+C%D22{ui9)%}O)dyUTI$^#MTDKSeC(WyW>N9~{gQZKU_iJs zZ_w^o#@-%(MdHZ0S8R_`3W&3_;20azmuhQ|W@)Iee;Urc<-rX>&Vn2bn~Ypsz2oD) zGp%o5Qw9W_=I3+7T7u<7M@Nf^iXsnJ>%eYv8dkrjq`H262XMIsouAUIq6W zEdy?AY#FQMnUDjKch~BN84v7r?}ACS^EBXzK&tkNZiP=9(@Duaefl#>;)FH2JsF() zJ&4$#)V8>^l3-wINo_4|Hg=zC-LqlmDX5Adg#*WnS&=QxXNTf=I&eYK(z+;~$)6pi z7U|Eay9ecI*EIVa&~#BnCV*Un^xWnbLJ$S~e)b|-%_`Y~fp8#wc=J5uHJf(x%{i!# z;-8SIW=9cc;xF@Kf}ZtTE<-SMq06(kow1xH0KgUmhtT5R;q_ zj9>ttOz=l%>|DVt+rBv!gn__V@fr z5LkJhq+gIDd5Y9Urn`bCo*_qx3t;7mlxJ4v|B}>1Q6drbib2g!j+3>67qcVbBzy}VXit~#?ZBHv z*4B=dgMG3COudrpav*Og9C!7T(!&IaV`Us-GCS$Ra5xU{KfNtTkwf;CMcPmL%6APF z%(qB+UMHl>yPD0|`%6T)Mdvxvzgy~J20Ps*J;^iiEjLLvx3*}F=mT4Rrz_w_f5|_# z5ZyYuk|{PN^3gzFzl#E7%9#RA>RrJ0FgV`{g@27HE%X~p^f4=G%Qi|^e<=7Xlt~>8 z)3>x=9b^Y3x=i6ER2jT~QJ*KgaB;dyZbS|n$yfJBMq~Cc0xtY*FX0*RdFg%-jnK{9 zwHXo#Ju70Lwmm@++YO20JTu1A;m$BSAH-O!;L-7M#MqYMOlP7j#LjioNld43$yMMXtwYN}M~WkDkyo#Ck|r@EazKN^&9?H{scqdi^$v*c7f zW7Q=!>R;{Dj#j=c&KxaBJ<1!hvbXL{K3>Wz0*!}xV1Z%xH?V*w1#JAdNoy1;HnKCZ zheR9#(|CtK z%ohOZPfv*}BL-8ylbQtDFV%V6v)Rk{3!ETc*^`dMNP? z`3e1nzIn0yTj9tAt+v%$qI!bX6|ExyvcRT()>uZP*qxlgOua)r}-usj!jrWXgj>?b>JmzHU$q z(fptsLK(}XB}j^yb>u3t(ke)T&oNJLfKN&~Zfr@Ds9Zh$oVF&3ja-{rC_5tjiC22F zF0PmHrTF81$Zapf{nuPYs^Z)A@i1cbwVDMww5OPfi3wkUlkuzfFwtmwUIP|<#R{#7 zM6}lhSa(Fkz*NJ5AGR!CIAY9z?cpJPNPViO;mj@cm8xrG!BVLtBqZFlX_{<+6ZtSp zSkxFAtEkV3ryUgCqAI8)=l;ilAJu)T>GSF7>uL z3JPJGcJsTFKY#uNyxF)xamx2~bxn;@JnxV4hP;d9w|)cP_|Vcjv7zUq`0+Lx7yv-Q z6GlOZT)r#Nl_WB3bGF5m<>j}Gub1dL>^^YyWsAxEny=0(u$9Xv{6K|aHlcorrju7V zNtHu@PbP1Uv`*+7x_~=8l1D@)YjCq0p3y0-`!ox^*xJywe6mv6%q{hyrT|xd)U%sW z2GcK)WaU2|oVgWZS|==nDo#|j@UuR04B-}j&{AyX)~K&NAzu|@b32kG*ywf|ns-(} zS|&3<`&L27TxfQ*VE8C3U%B&Ya?`}%AE1xGKouWey+Y|xr^@>KDj!@S_P$k-USDO^ z+u(D2Tud#+uZ*p#(gy8h<}q@2l+%|6I~C);kX&e(=w1m6$ppW|HSjp=KFfq{{|Z-9+TMo!Le)U{kidkzY`dT8WB zMNMs9J@yS$*v!m%fl~qz)U-0XHv3jN>n?Cn?njAk#x)*UqBE7$bGETjE~q<@q~Pcp z&Vcor|KUm%;C{eddlw$gpZEtc6zEip=?6xo?gwgd7ggry2PVj)@5Js@o2> z=ILwZ6giFSKY#i1B`(g|2B@nxV95dJ26-~pRiLH(%8m0*1oORnW|OrZ);)>00Tn&- zA93#-I0Xg0j`lYIVG)AKr1#$%!@R&30BQ(kTNx>7L7}n=3wvHZ8S9ayN0i$Q=7OFp zIKT-&e&fd*2}|sNOo65baH3Q}+n~nvM~TN+8@O(-<&lkUL5v@OSyEC`;G+#!(F-{< z@bhnjj+*c8GH7uyF)>9Jpff+o&jAt+z$TJTEEoRBXY*T8(HdZ5ASdb@8n}#VXGh2! zTSsROw|XV$`EB0&6MjbA1@`xY2M;12w2K3iQTV4|XDR3tNcbOB`+EX0K3?hQX6WJR z3JUf6_ms!mL08195r>|~+kmRYaha@E*DlY_wnsmH{FtzKW?UR$=?ROVDJ^z-v=y1x zIiKviS_9S&T9pHo7!VTU^*&|(gf(Dr0Lz|MC+YH|@%CtTokB)t=B2Fk^zcc8iS9t*REd=4nB6lngJ^pHR&}&ybQ$suV0MO?Z7q7 zYQ#jDey*<Rc_q@ zMz6>!65V)t-|r+>E065bB~YLlsotifq_8iafQ$mon4Ks+E;hC=AmuJ`)k~6)Ub^HK zJn2;##i^j60E7S;8EBmOE+l!)V6n7DAChh%+}m9yKwk!IMvDtj;YO;eeZZ5Y0WqMZ zqa!OT3kUKp`xx}HlSO@aTe#U`*f=>?LA%pk&ZkmaT|G@ZZwC!pluPKWD=pkKr~<

    LLZrts6H45CnJdSj331qXkG7U{1BXxdM);&`==+p&Z$e zn`<{xY$_c>PD)0mU+-O_oaN?_$moe6&^zIy@X) zTxS_NO1vq2I}-aP;q>qtHZnz*mwam+-Uwu%2R44OTs24YDaL+mqYVG*)wvSg3DE5G zjkA<>t*Ot??;0N$5fZA#vUqcJPSyLMJ3HAK8Cze@jYb%9#>p|JQADjj;NSoNIO#pZ x{~(bMw4@W0%Mk9QpxyS5o0.996, 2=>0.00197, 3=>0.00173) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000517, 2=>0.000196, 3=>0.999) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00154, 2=>0.994, 3=>0.00397) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00128, 2=>0.00405, 3=>0.995) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000393, 2=>0.000447, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.993, 2=>0.00372, 3=>0.00319) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000405, 2=>0.000195, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000309, 2=>0.000249, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00752, 2=>0.992, 3=>0.000947) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00012, 2=>1.96e-5, 3=>1.0) ```` **Step 1** - Edit your code as follows: @@ -119,15 +119,15 @@ yhat = predict(mach2, Xstand) ```` ```` -Node @316 → LogisticClassifier(…) +Node @440 → LogisticClassifier(…) args: - 1: Node @206 → Standardizer(…) + 1: Node @909 → Standardizer(…) formula: predict( machine(LogisticClassifier(lambda = 0.001, …), …), transform( machine(Standardizer(features = Symbol[], …), …), - Source @679, + Source @128, ), ) ```` @@ -150,11 +150,11 @@ Xstand() |> pretty │ Float64 │ Float64 │ Float64 │ │ Continuous │ Continuous │ Continuous │ ├────────────┼────────────┼────────────┤ -│ 1.06157 │ -1.56085 │ -1.46133 │ -│ -0.84203 │ -0.0942899 │ 0.781855 │ -│ 1.06519 │ 1.20404 │ -0.62893 │ -│ -0.322846 │ 0.266821 │ 0.683523 │ -│ -0.961878 │ 0.18428 │ 0.624879 │ +│ 1.54143 │ -0.508564 │ 0.563239 │ +│ -0.597675 │ 0.271238 │ -0.895631 │ +│ -0.704197 │ 0.909215 │ -0.136268 │ +│ 0.489404 │ -1.47949 │ 1.40741 │ +│ -0.728959 │ 0.807603 │ -0.938747 │ └────────────┴────────────┴────────────┘ ```` @@ -178,11 +178,11 @@ yhat() ```` 5-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, Int64, UInt32, Float64}: - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.996, 2=>0.00197, 3=>0.00173) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000517, 2=>0.000196, 3=>0.999) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00154, 2=>0.994, 3=>0.00397) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00128, 2=>0.00405, 3=>0.995) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000393, 2=>0.000447, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.993, 2=>0.00372, 3=>0.00319) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000405, 2=>0.000195, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000309, 2=>0.000249, 3=>0.999) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00752, 2=>0.992, 3=>0.000947) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00012, 2=>1.96e-5, 3=>1.0) ```` The node `yhat` is the "descendant" (in an associated DAG we have @@ -194,7 +194,7 @@ origins(yhat) ```` 1-element Vector{MLJBase.Source}: - Source @679 ⏎ `ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}` + Source @128 ⏎ `ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}` ```` The data at the source node is replaced by `Xnew` to obtain a @@ -207,8 +207,8 @@ yhat(Xnew) ```` 2-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, Int64, UInt32, Float64}: - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>2.72e-7, 2=>1.01e-7, 3=>1.0) - UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>2.8e-6, 2=>8.96e-5, 3=>1.0) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>1.13e-8, 2=>1.0, 3=>6.09e-8) + UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>2.89e-16, 2=>1.0, 3=>4.36e-13) ```` **Step 2** - Export the learning network as a new stand-alone model type @@ -238,15 +238,15 @@ yhat = predict(mach2, Xstand) ```` ```` -Node @855 → :classifier +Node @530 → :classifier args: - 1: Node @680 → :standardizer + 1: Node @804 → :standardizer formula: predict( machine(:classifier, …), transform( machine(:standardizer, …), - Source @679, + Source @128, ), ) ```` @@ -323,10 +323,10 @@ fitted_params(mach).classifier.coefs ```` 4-element Vector{Pair{Symbol, SubArray{Float64, 1, Matrix{Float64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}}: - :sepal_length => [-2.2884323728871254, 1.4607388496829115, 0.8276935232042247] - :sepal_width => [2.5849165395349716, -0.6646830918858928, -1.9202334476490799] - :petal_length => [-3.5313085896004583, -1.406180852406463, 4.937489442006912] - :petal_width => [-3.4480076498857017, -1.6977190775004856, 5.145726727386192] + :sepal_length => [-2.2884323728871574, 1.4607388496829223, 0.8276935232042454] + :sepal_width => [2.5849165395350733, -0.6646830918859379, -1.9202334476491332] + :petal_length => [-3.5313085896005036, -1.4061808524064117, 4.937489442006905] + :petal_width => [-3.448007649885748, -1.6977190775004702, 5.145726727386221] ```` ````@julia @@ -380,7 +380,7 @@ y = source(y) ```` ```` -Source @924 ⏎ `AbstractVector{ScientificTypesBase.Continuous}` +Source @555 ⏎ `AbstractVector{ScientificTypesBase.Continuous}` ```` **First layer and target transformation:** @@ -396,13 +396,13 @@ z = MLJ.transform(mach2, y) ```` ```` -Node @546 → UnivariateBoxCoxTransformer(…) +Node @022 → UnivariateBoxCoxTransformer(…) args: - 1: Source @924 + 1: Source @555 formula: transform( machine(UnivariateBoxCoxTransformer(n = 171, …), …), - Source @924, + Source @555, ) ```` @@ -419,10 +419,10 @@ zhat = 0.5*predict(mach3, W) + 0.5*predict(mach4, W) ```` ```` -Node @505 +Node @135 args: - 1: Node @084 - 2: Node @653 + 1: Node @083 + 2: Node @797 formula: +( var"#*##0#*##1"( @@ -430,7 +430,7 @@ Node @505 machine(RidgeRegressor(lambda = 0.1, …), …), transform( machine(Standardizer(features = Symbol[], …), …), - Source @511, + Source @100, ), ), ), @@ -439,7 +439,7 @@ Node @505 machine(RandomForestRegressor(max_depth = -1, …), …), transform( machine(Standardizer(features = Symbol[], …), …), - Source @511, + Source @100, ), ), ), @@ -453,9 +453,9 @@ yhat = inverse_transform(mach2, zhat) ```` ```` -Node @247 → UnivariateBoxCoxTransformer(…) +Node @539 → UnivariateBoxCoxTransformer(…) args: - 1: Node @505 + 1: Node @135 formula: inverse_transform( machine(UnivariateBoxCoxTransformer(n = 171, …), …), @@ -465,7 +465,7 @@ Node @247 → UnivariateBoxCoxTransformer(…) machine(RidgeRegressor(lambda = 0.1, …), …), transform( machine(Standardizer(features = Symbol[], …), …), - Source @511, + Source @100, ), ), ), @@ -474,7 +474,7 @@ Node @247 → UnivariateBoxCoxTransformer(…) machine(RandomForestRegressor(max_depth = -1, …), …), transform( machine(Standardizer(features = Symbol[], …), …), - Source @511, + Source @100, ), ), ), @@ -491,9 +491,9 @@ yhat(rows=1:3) ```` 3-element Vector{Float64}: - 0.38643611843841796 - 0.5723985261534488 - 0.44151392790057636 + 4.1049434111127105 + 4.674958202525163 + 4.112097582032496 ```` Now for the new model type: @@ -543,25 +543,26 @@ evaluate(composite, X, y; resampling=CV(nfolds=6, shuffle=true), measures=[rms, ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: CompositeModel-412 +Tag: CompositeModel-983 Extract: -┌───┬────────────────────────┬───────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼────────────────────────┼───────────┼─────────────┤ -│ A │ RootMeanSquaredError() │ predict │ 4.0 │ -│ B │ LPLoss( │ predict │ 2.51 │ -│ │ p = 1) │ │ │ -└───┴────────────────────────┴───────────┴─────────────┘ -┌───┬─────────────────────────────────────┬─────────┐ -│ │ per_fold │ 1.96*SE │ -├───┼─────────────────────────────────────┼─────────┤ -│ A │ [2.67, 2.9, 4.56, 3.85, 5.84, 3.28] │ 1.04 │ -│ B │ [1.81, 2.21, 2.81, 2.27, 3.6, 2.37] │ 0.547 │ -└───┴─────────────────────────────────────┴─────────┘ - +┌───┬────────────────────────┬───────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼────────────────────────┼───────────┼─────────────┼─────────┤ +│ A │ RootMeanSquaredError() │ predict │ 3.87 │ 0.46 │ +│ B │ LPLoss( │ predict │ 2.454 │ 0.04 │ +│ │ p = 1) │ │ │ │ +└───┴────────────────────────┴───────────┴─────────────┴─────────┘ +┌───┬──────────────────────────────────────┐ +│ │ per_fold │ +├───┼──────────────────────────────────────┤ +│ A │ [4.45, 4.28, 3.41, 3.3, 4.21, 3.37] │ +│ B │ [2.45, 2.46, 2.46, 2.38, 2.53, 2.45] │ +└───┴──────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ### Tutorial 5 Resources diff --git a/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/exercise_7c.png b/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/exercise_7c.png index 553fd96c4f5794c72d30d51efb7b82376237bee0..08ac21bab1bdaaf5d6a21a6aa79a63e3adce6876 100644 GIT binary patch literal 22708 zcmb6BbyStx_Xi4ZQ0dsxU4qgrom-F;q@=q=L1}4W3nG%z-Kk)JgoFap4F)aUAtlY7 zobT@)Pir)BJz@nwo=^W5Szt+bMBC z&w;xADJsc=bzAize*f;}ckBh;0#%A@*flv=md?(bE=^}Q-N}t^CUw8y{iggUVI=Y6 zN5S;Qs^GbrogLSlGJX;x*{_7RaWD{wN=$hi1mY&)|J$!<VEh3?dn8tPY<`4m;v@aB537zde6m^1apSI^AzB4#lJPp{5l)uaL_&SxhU80XSy*GwB#_0 zUlqs`-zx&_#-}R;_9Qqe3wEOB@s2Z*?tkO)N&; z{XLt)&+qh+HAZnO%yVY&_nN#3y(>*-ub8eP_{DYlpN`tP`+HqK$=$s%reA9|7AQ-3 zoU~f)8|u?Q1`}*yL}9W(ES)><`;sTK;&$=#2(&t}B_jc4zjOb2z(v+tfhNI*4@dBQ z&CZ3L++3ENG6ceU*Pi5J_g;XeOKix&BI^Cc7qjDkWw`|6xD(m}>qsw=af|XK7M%UQ z^RpA<&yO6PoW@IZZsj;(Ao5&ULo{T%Hv;tLZ712)+A7_w9w@$IAP|jsVttqHe96c0 z#gsXpB65{pR<`N)@87cLhj*4yh_A&a-FbOLWuF^ah&&%nyck~Z>6~X0IhtN8AQ$3& z60N#l{rn{pSx<(f*XikLylTPm_&ek3gU!h_Bnc8ZP@qPsrluxeb_J0aXQq0xxs>=K zqONYu;b@?1tw70)luI{%L*&Q%I}G8;^-<$8(5etbccc=xet`#s=SrW?@s<}M*#P`&t(n9u8ugK%{sjelfRry0( z$%Ags;0Ezf0%}2z)e+-rXBfWo-tyqjpFdH0n25XKc{V*-edD1izqI(q%(?IQcneBP zefPp&=@&iv^SG3OVI6K~^Sur+;e95u#M;`U!@qO-mX z2HDVi%a-GRTNlHcvX(!Y5Dwkn>5kh3^h}7KW@j|+6a0*LN1ZYf*C1k0pPje-=f-Mv ziB;P*-m3iT6_GiL?D@TEqWGqm2&#m6D+vqPYEEr3-es|#(39zX?ZGn^&tCe*Gf%x_ zTA|v=VuA9cTt(QSteNq#Qztc}8>7F{sQYWBvvGx*^4OoD4f!+=LWWIFHcBU>D^J%4 zSYX3r6>tuTn%!=WP8k{U)_$)|Dzu|Td(a~C$WP*4->_>Q@^Z^>i>cmaoPjBvb|3gv zLNfLhIrN1?=TW=kKg-DNkSmBFl^k@vTU^gc6oXc$iL;QBsbNjp3PILr}eL8eYCIosHTj|6((~0lo9KY}5pb&^FyZNsb z+TGkxqcUd7@;q{vFOHkV8}iLsja*zuQ;0ox!ZuSvAVRdhy~zpF&FSmu@f7oQA~aKl z_mjg-agP(S?(CmwtM_}oX*t4<2(mn!#q<)&VrXl9_V~`_y9FgwAJ$;+>B@e%tua0p zs!xWHo8C~LSz_D1b^OXi;A!La%No5mR{Ka|g|)i6x+aC^rkXIGi6a(bz+4o6Ip|91 zw9RZ%dQ5QSHJFO*B8Ki2r@s@y8o6ef%HQsyp=b{%DU#1ud2gOQSf}lMn(J)h?GNv$ zA+8xpr~9H-FzebJ_vG?6$rZ#hrD{j(TzWyWVQBcSM2Tahh1=F`ujAv%=zn1&ELfs!2cr9*d@(0K`u!)aSmTF?L0ey+^H9EueubIz z(^<6oP2=|#&zqVqf-tT`F-Vt|?Z8Jh@B2xtzh5Oz5o^89QS9UqP2+#KJ=5ra0?+8; zSFd)S+?=Q-B`3e_clac8$>TmFB52U}zxI4SY&dd-bm_%?YhBe~Vzm?(k>FJ9k z^8ji?%dwwcbTgH{CNE5PK{JQpJ|=@zs`RWl+{@?^64VeKIy^jF zL`006W9ov|vX2A};p|C7&YX~sY36Zy;=AB}$0D|?WTl+<)~ zFb=P(Xgua*w%J|m&AS_^T`gm3Vv@Nvia=0Bl`Pey3)WRsCoA@6XA5KJNL*u-_K8nU zK3pEWAu3v(oy}x!K7A~^sVRF_32zFw(PQl=kBm$Jd?RGnQ(V^qKd|dfB_}6W#7U9x zSdC*-8z+Sl~1^bus!VXzr03XWwQQ?v*dT&ps7+aW)-r@@{ot z;Q1Boa?>wAzkSQ;!|v_x4?NtqG%!d$9)JJ-J(qsD?s!p6&F{WGjkBtThBFv6XP|K4 zxu2xump_005X&5X9UD8Y%_K$yosTECR@8K6t@3>orK}?>=D?Zu|2GL!uaPG5cpzJT zb91w{w$}Gx{b(kPF_f6GG4SGCf>D@@E5ee>*VlKy8m1{FCFRG#%2?If@&1~*@5b$@ zx$~3hi^avo2fDh+DJhXOx4ZiLqqXaN_E%wbWy<&)asRxZkIL!j>Uucc=)aKobgu1Q zy4Z)teq+9b#6&>?-dG0=GnL1&-3y!vuWSXP@^W-9?%;@L_G-x7m|`n?l3kgvu-rv3 z;70$4BwWGzrEbAYI~yAtt;i#TN{7Mu&Nn}dQ*S%{%()Zx;GGd6y~N(P5Nr>#8&w}Z z&K)d{VIPGQ`5P*xdhdMl$6m5Smx_hI$T^_`ue{nCMHtPU!|lNh7r*}thkDh3zT9n z^q;L=n8{W4GWjrYgGOe#wx-70*H<=89k9uV4R=a5G4_FPG|-;V^4vN{zxk z>WIC8%lz!wvj_?vZO#OAOmj0TYL1(mduVu=lG|YW8#W^ng@=d7xumiF`@;=3wuRAW zln6v9dqw)nr|1)@Wcu|8iqDxjsjcnpqL0UfMMOT=)Ff+jCUwuv&H}s)xH$8QJds_S zX$gC0RP}JUFft;d`&O1;?v(fL;ysjdK!8kvFfVW4=x8l$jL)AxUoL8&FsEPTqp&bf zPEL-FU>uKrzt1Ub*SvRce`lxEULZ8LxLA3aip8XP=43t}3}oUi3$b91}RyZHEu z(Y&ox4rd5^_JaOGYd}`2s;}x+?5wSIw6#$<0zyJ%K8BtvLj~6DQSaESo@F+9ZDqcB z^XAhhw}rd=FRAu_!QC)4G`yE4QdL#;uI>)JT$y#};D93hA>D)Y#6*hg*JD#tjr!|F zalU{5u4j(tZRyQcrO2C<7k!^gM$$M|8zs8W-_AA|I(&>#{*1f#_XlgWRhpg=Km=o6 z7KR6-YsCp1$~aHmFCTuMVL?GbV~o^@uR-%1onyZ9>cY`vBbuop?_HN~A`rp@(Z6ct zWm;+9PxLYqQOMS-FwB^@yhtLlBp7FUq4dbdwC$IxR@Oq8ym3R&3mcnurQbL z&-aoNFc4JFZAA|q_(Zb92R!6@h0STs)+(B?h#A&bS0$`kalE|#rKhJiU7Y)F|M|xC z;H~$+NiPo*7DUig`)@H@6u|g|Ee0#n)g;z7-E*9rxcpZzern>dx&-utk@5Tu#WZ?el=0c@=H#q4m*E zW#|2^R)5pQ-C%J+RV?@}ON_918&7BGnsFgT#du@SH1Z< z?#0kEzdpYrEhPo$`rrDf8*~-h?nFZagXJ0pKsdYO(lP^4=-OOy2ZS*840IKEf~0+4 zdIFbz>DJwYAvM0oW0j78f+fm{5num2i9e0$s91UB+fR!{C~sh9)(DF!ie55P>X~+l zZfRJgp^?#QqZk5lXV0F5uZqR=X$9MroS9<}4UIlP#3R*rKE8kdv@IXZef3mDSfv zO;SR07+5r4ux~(r%MDf7wMYpL4z8>ehL-d4C61CccXnv*~v2qtsdh4odi zMDMr_e04Yvqn8b+ht4wA*4*42f=w(EpaQy(;}rR3w`ZDUzHED9R^M4O?48Wo=H2U}GD6TF!BIsMP?~5!F){i4n z)Pw{zr^3~|LxHxxhDnt-Q za?;np-$phObiy{NA=|y8(-W1BIsnn?ytd3dayJSHX@~7z-9*+9K_)Gqgpx zkLCeK1)KOVDze>*ovlh6dTw<26`H$uoGP-jv+L{YHM3;uDhgwDx92@E$m{SB{G8DX z?rwq?DoE$APTRs)b0Q)lU=IMF@-V-5qt;vjo9@r*6fq)uih^wi2y(`UiNu*GuTzI1V=Xoi9jDjIE`jg)5BT0jHNnlKgDs!Ub0+(TD*iHgcNYeB$)HulGlu^#M9zhSZQJKT7ogkpKPr z_u1OJYEDrsEUfdxx#%AcOoW&OIovKKAj11?P!|uWnH`8|$O$hrGM8Y9O9O(1g+fd# z^48VYz`y|3j*m|rQTK2m`&DLya8H-9UotJ}e9&UH)*uoM%>^ELO3~$?Cx84%K*nWc zyo;!#kyrD*+Zh}7y#s~tzP=!3=t|!!ey^>U|3j(d`+hxYih)Jd7V3A zm?g?JzW{HRV(whNc{OP1o0i}Orz{k*8+vmj@>*J2ASU#KLfO~%`O~M}?d<_u!$>9R z@MdB)6hf}yvC1Mn&r9`JyP;1@t=~z((BUx|pGyn#uqTi)fLfM-l64;i*qCn_y_h{H zSo`!V{BBu*Hxg6Me?xgjk>FVsM{3C3?)hN;Y}^}moo6gI%!rY0FWl3MD*NjpcYEf8 zV~|6W8REfkD@R%y-HnXokA2f-ONQK&X9M*YpTm&v^G&loF>h$im3Ej=7r@EO?};Vo zfze3N^=4DM3Xb%X^}o5PxGU@3oyY_7jdoy0TAHN8Kz8B1G$J}Nel9Mxu&Ub>=EiO5 zwwn~WS~!TRl%%h{kJ?|sg7btDFc&S`*V(BRc)5c7PY(9iM}?S~XCg(X1}q;(D#IrE zv&xPbSvL`BFjl^MEnD2-RQNO+A0J;xNJ!FsSq_JTk8c`q#cOJzpMa>Lr@?IKW97@Y zG~i-nd%BU<@SzOc=Cg4Jwlx#G7tk>P4v;9=-#WUw!WLf+BxmLo{LN_b(Xudgp8^aTUbL-HM zHoO+gg|^M8XM$Ib3^NGJtztV1JA&0fy-nBw2NkOLuf~G)zRg z{+(5XQ7|$xW>Ai@KT*4h7+8LQA)VQ&Nuc01pzCn1d9l;YH`f-~-PPrDarO_GnDKra z4Da8@_^+<6(oEuvgai)&K9!Y~sh;fY>|#%6TcN;}Jq`pY19+46ix9~tTnxFqEs{eH zc@_{-)({yvk=F2*8X6jxVvDEe`ASh%=A+Wby+GwGVBgjQ!(fdZ4PDIy6w#yIrM07M)qy^|m9xt08p%BF^@8X?(I|6^xgxIJ@gq~baOiD7Zc`|dd z)ubPI0v%7`?%ngh?Tmmmlf~T@X>=IbleBfBPWczCG0#^DZjMz{i~ zH20&G?rCU<2np$2SS?u4eK1&GSxEsHPD69No|PTXNrva)SNb9}RK)4$+kpOK#>vUf z1G4k8#QuG?$d^mcGTX;ojDqyFjny|2Ok%z^(RYZYI$fyjUgRyb-&X10S1rZ%{i$&W zxwx=UqRZ*x;(}84@j0$?`t{jmdaBZKxJ0KY06;MZ2L~*!iZgjwj-{amFJ8d37T1Lo z6fD5VWPSFQJKx;8#mYL-=>J7Af@0NN_m?AQuyeLm)vqjHep+giH#;6+84@Gd=Tlqn*)5o6kR8f)X3qxr(5{0v`_6DXiI zGfvfPEzy&31r~~5Pw{;SWdZHnM3a$|Uq`|6?ELr$Fe@(e>fhYejVd=UFMUl3-R{oL z*iM@t2ZnX-D^MRmPXVxW3phf1G-Fwy00&3ODjzJu)XcOuZ-#*ku5uSrV2HGiYA0x0 zPX5&5i;K~IXl;GNoxBZEk^lKUH&?8(Ix9|7?YD-};HxzwPrt8yFAllS5-Z#ml12jy zjb>^ZE(-3Gd8LDvmDMZ|&E}RC=t?-ut1n%>yef9(p=v$3I6v#@QAa5o8jiw7XlZUP z-Du(O0a*q*f~@O&2Xr5Ovu7`mL(s5?i!{r0Irp==dwU6SaUHka-V##KUxUI`9@jcG zC;QlmB$oJx!+8Y|c4#}itgQ3kqd?Qsa+&!2IIZaNVV^;~y0Rkfcep*6r{n_LZKPNW zlqIdDQP2iBNuPog18x~mWKem4q25!9q*26y`;}Hvu~qYQ4%UH;^F$3U=}j>SiHA$! zKM@g8m@Zy1F_@ z3ZgDkr$8j2lb48wqS3GEB_Ets!AE?Lpd_Xf+Xl^lYw}C=PVg4I5hf-kY&I$2y=-cU z_1mOnw^HJz6Teyi&_d`u(3O8MIV($i*KI&SAfCnsC%>RTQ9;4M_S*IXV?EkDg^hId z1*?IH$s;ExuK%05js;hpwc>vC7{NX zRD;?<00QG|Yq*MvN`lZrWQclKqi<<&9!sp2G!LTKm|7{IzHjRp`6rEgsf&^$S<=1- zMOgu*>hh?U(b1af>ODX}fClEZ?6`mX5vRhpqVgu2?gX=8%)$}$kklHma*c9L*t@glI$U=$!P`FMLvflmTkF!20n z3975Ty?xOMC#`Wj_Q?TmRNf^-y&0TmGjsd=SNSqMng*-e@9%d?x^^HV4!_DxS9ySp zfZ0-B&-Nnr=|?6rk)8w-n$_2=SNU;ic0TIaX_L!TbJYPio;#&a5}!Ya33;4*cEh^NX+ewp1y@Y?~u`84)HtS={SktrP} z9PSCQYFE%}9?A;NGR3Q;fT^&;42Y872%BNhkL_6lHIdt4(jrs6+fr1?{&f;mo!Y2q z)KhXPfffNwi~L2lCqm;N9EeeRsUZXG_PxJ4r_v8c1u5MesLG=-sq%zIg&X1t{E}F`gW-1Y!R=AMU!d)PzksqGYgOC{|+iK+S=Re zkAnap<2b}xpgnr1hl>T(fvT)qe9umXdYn!G=Z9k>;c!bB@eLN1(#OR3_*|TvKHGnu zYv7UI74B3xP+x|9uFU>pPQL?dk>G>@-QCRen9!R!pwz zm{c!n^$(-RUq0#qn!1;;$P};4D(ST;<+=VFfEl2^jP!IHK|>%;)NO~ievhWcCLZ+E z-m0W^*2S_o)K*)bc06t1Z)F-O{;akLf&f&{FMXVADv{ zPMD)@ej&R&ZgmP)BfL%8{Hv36tWMkT^6E2u*OKcDzU2Cht zbJZdISm&Mv6HX<#sg}npSfam^_CLtGAy*Gvkae}S(gCM6AZLDkxEGd?j208L|1`*c zm0t>Mt);Y6dj}%a*{NhSzo4KtODq5*VW*$me0(m=Xk2h5`p=u8d3YFP#*9?k_>5k; zQh`Cx117c1z0pNacxA&WEgJeUbX9XOIG|X9ny|65vNAu-jKGM5=UvbZr%R1@G&Vw{^W< zZuTK`19a%)A-||5*nA+DMyU;oKCs$}BXyIAIM{2we$9d^p6;msc7T%Rp2Jr+c(SLb zr)q&Km`dPj-P6k4DhuWl2F^UE2GiTST0QmiLfOnF?R|^P0F~n@$+p^6uTX zvWiBZjj`K+{?^yiK^!+TJKz6h-rn9`?J$UHXTxLf3d!)e4plZZC-@dV>Cr{kb|-me z^y18SaF)yf(SaldaQg1uP*lz#G$QdRS|PjFQEQ|RlDXV?qAB!oOtFCk25#+Q+*7Wh z1#0F8vWW@r1w1#2$5;fk(hmmeIF^PY5)wYZ6rlAftEdbQd{g28qoK9r{fWY!#qVDFW7a z0|L%LRg?mC^W(?!^OG&W)gVXwg7D8jJ-p%wE;pep3cZ+{cBmvw#O>ebF{0Kg6@?%J zMhr+jbO)bKTpV7}lOklVvCvjrTns||N-4j;B{K`XX|hgAWi(}!0781-AM+xR?}M1a z%y(E?TwG;<(m?#Z)B}r)JI2N&xVhhg3M#>?q1lnY9!t;|O<7)tSk4$BspCpW+|6$y zPW&B8sXM-E`51JD@z3vILmO*p+pUVfPQpN%WyV2jMRdo}ilHG>vtmidNq}@-E-@?u z?j&WXt?=5<&V*VWIsC40*!2YvCBe&_;K>J%IMUCDBW$YYR;?<{0NU|!f8|pPvt2uh z+V{?hD1F>eVtE5mw?ubWTMfIR#;L;r+<})2-50o09y@H5I zId}~AkB+h7-QL%1rY0$*DN>;hR~&S!e7Ec{8cgLTxl?`;!8PV>PC423TjVk>7ty%1 zkkQj;#Kc%Ys%>^Qm#FzinSK`3Vju!_@9vv-Nd3;Eu55}P@IWj-e>mxjTU!|}4i~QL zwN>@hf^~W7$`$a+!IasUot*`-hm-78moXn9U=ZlYK!RX9OT4{ruAq|gPa`J{sjj7U ztyW+8$GryVRI{X|es#Wlt>GUO|1nT;gHUc{R0sI`QlbUf`O?pw0BzGPb7^>Z_?{5X zos1Oh@4}iYbc6SBZq9|#=Kf}t(eJGKY#&UYB0{9DsrmWcmAO}=z7`6?S1m0qncJV9 zpPz$=rpt+Xsa-OnXnL&F6Zb@P(E#hhN64KtSfwWZj{c-Ak&+M(ee3e%l^caeBI*$G zfqGc+`3&d=OYHJ!kTAUB#QLH#rl#}rS_nzNMY z6YtKr?<@Wp4E5)WOqEqt(m-aw)0mx`b8~fl^!Tx@;ApW{UQh4Zm-Ri6oB{ynZvXcH zl-AnT)|V`hw@L)g1htg+6J1?0=6PTd$&y~q3(>g{g=1kc+oQhCa{W%TY06y6DH9A^ zE6pPoR9$yh%XwGW3=V#`HHs<26+*BGX@ngH@TDBg%*<3(Rg-v4N2bBB14*6h$3b{h z6sQvi4Zi=@H#S6_Cw{}=yBNW#_JU3a9EQ<as8i7WtFlXBi*!Pb0aPV3v*3Rt5L}Ebgbc`CxJ1 zaA$7&^B>pGWmwmaSTb;mpR18z)Oe@z=4QM$C=bs4V2{g4w%P$Ef$9y8CC~>j2^PTR zie{A6z$vK@9;+~y<>RX$y`@oqy3^f%jth`YNvR#+UKbCrjsf{yfr(Tbx-xe1QQY zxI$cfeEzuz2S;an(eaQCHZcR8xNCk@)$Q(&E)DxXLHsN2yMApbtt`6^v6!gIc27!VU zwFp}quxn|rSh&IvNS~4V9Jfgup1a{wsh<>mQUmpym~I;EE@P&@NRKXK^D_4O0O z!#b0=gFsC|w1f{6A{YjP>5Og*Ion~wJk#h?oB?L5uX-qBF2FZf3G zLkzDHY)nvdK=1$_s#jfBk&#-%&fz8;LiCRmF!$q_vW`7`zel36RN^={RRFluJYk0H z#I0NHAONwFBqStUNA>jdJbLs9iAEz+us_XG)Zuz+SNJtCv*d-cMye-; zQdlY~Dxy%$P&K5arE!_*+QFP90=6?Wv;#W(%c!XGi&uGVubA`GBz?I3?EgN&3uSCWD|*t)>0z#DocGa14TD`6_RMgE99#Q`8gF(tM#KXHXPZ*k~`}*pUW5 zKjC^2XrM0h2pDg&Fd=pjxNZ=M@$tA1fLh4DuehjT`ioaI(#p~@XO3iFCmy@-M~J0P zamUD~(~nPLsa=ukX`wgdSV^F<(o1>AAO&H*(?ne$|G9OEf6w-Rf$XY&M6|C5&G}WA zqYuYNB*BlGdd4Wcd9lAmaOj&Gd6mH6By6VJ4{~L!4!cA{um-w%CrNF{AT+7>p}Pyc&Z3JuHfWN;Gi%Q_b9 zfh>1~LA8Um&DTr1y5M@G#ORMEd;3Pci_Yh@Gm4)|r*DU``uforc-;RzrJ$`IQmrOlH%H33gDf*_yQ zJ<+BLB(GyJyi*s;gTY*WJXJ3)EIcweI5;)c_tvau9&_YFxAQYyf0^tzA zpb2Q{mm=6nFo6cXWaur{+T>$P1m<#CiG8h?@duvuZHj9Ha(5VWv`g~i8=>&mRkX>0 zlY0F%{WX6N!8e&*xx;q`L~*iHAZy_1Nkm8}?Y8)OvrOQ=AmDKSg4xcnqFOdn11A%T zYsJqOL#&T>7GO@_6&EjqI(Zq8V-yz?Th8ad1E|-|93{%UoVOtQj8aXon7U9MF`^6s z04jbaE-t68ct9ZPB?oyA)uJ((Tbur8!O;v{_2g7tHemUJ zQPN&ORk^v@6Wfl18sHo5cbhYQ&$IFO^W{^R=pddIwZ`wR#nAIjX>TJD=6hSV=+p8{ z``#i_7#m|XDD%ojj7rK4YJlzj-*k354V^9Rf3~SFYa{linzAUs|ETiZ7z1vlq@;BD zKkwh)pW{0P-L(q;3nso?|6H%N`A3tiOiw%Ny zj_MfwWDj@6s$c&`*ZD`F_A1(gBkVk* z?P6NZKnW|m5g2fVSpx|vY1w0tk_iY2AD_k%BqZIo=CO08WWtl@mttc~U!-s+8BM`A z#S5D8j%su-&<>Z>8sWu@hzidBIklijKusDXAc*LnDG|R5?lY(W=os*>fC=bEHb5Me zoa+H-3+nCr)D%#Lxn49*8!bzGWnJLv)TgX%Xn?Nt=SYei4avG>36Ij_HITz~$MslZ zFB9rY9@K^G96}r1O5r&6ECUUONj1!kk0fu^pCB^(b>zjoxBmd+BSD1{(jEZ#t#&?h z87>4UgMonop}{)dfo$O_yY-%ww4=(%D;jsRb$;zQOk6o{L-u50dpm1h%Ykfgb+st7 z4e;)nnHg~JF!gM|LeW*m3P!rWBH5X7bT%vQz|A;B^lQ2m`9MOnw9vxJ$_n}l3Kd*l zDrlj_Py;J<%;leV?W@-=&*4t+<+iPmezNP;4s3w+4VVpZQ{$a5B9JDAEgxfX0PXhg zv6f(7R@b_g@+|g39Mw82c=anj82Av@REnZo8O#ei9*5=N1vLRM7{mYFx%5(%E2tSabkUdUNw0sDi*5E=3n0e{i*m(oQUJ8UQ)FxLCMG zL)hERu>P08ofMS^5m3tbH`T^guLA|p=@vx@$H0!|w~%vn-GL4bu_WldKWIT( zwzjfzH%P;Xh!a#2M4|^?E}G21`3lp?zYJF#*n1)Q`u9IhQCwF5`O--9=vdg&vT@7t zOi(;;cvQrcf9W=-M|;>HDzn?BUabA9Xgb;a5*r%}>tdA$9Kg|!wh`rcLXUvdNrf!I zri+}L6PkFNq@c|O@4$}-!VWM$(4Z%p0-IjEP<;5X)F%(lg~|BW5jhE2lqa?hr`D!M zuwLDW@cmkp5cki>@-;z_r-MwQ%;|QEjQ=q>d?(P1ODBGVzz0Cr#>Qs3RHUWQ%w2bx z^XqSR-TC3&)1Cjixty9XojSmCpeX0F5qvQ5)Qi*Q-uvEKS1qNXCJz#rUB0A8?GMWY zRSKCk4C?M3W4)x|PGPdY;Y6xO1-}3ls z3qOF7(?21^PyisKsmu8TP%;FkDU=e=Cy5ST;lW&enTld5UkLcabx$P zj)^H^pmD6+ZES3;5qPcy*Y`s)qZm7po&~zGQSv1|@24ef?xTd8&4@i!gfO149*urH zwjC+z&OMnHOHjhYFU#rpaVYypZs!r zTqe6AEAHs%xVix<8u*6e(X!{iA=V9xOj!Gw#IR{sW?$~YOO!YBf)of0)xzqm-*}cW zXow$Q1@raC4-Y`l@k7An4gZ$_7zZDD9*i2Tz{#e-3%FN0ssMa>czCv^n>0BfKnu$p z4lvSOD8S+cDb$**!egUaKfP~u?UOy^YOE=(Sza$Dobo=WeQ(ZYfdP5r+w>vZCcq@f z@Lmk)QQ?ABt67BucCbPTg+!%BrW6NBI4~-C99&3a5)e#$ve&HghY;}Fx95gZ(_LMv zEV0N-lX>)mwNhYUAW9jocz1UfY{Q9dA2gUrN(^~v$V_7&4pYBK|}t3%O^-^XpD}&_!NARH1rHfyKeF?Q>wr0S9J)M zV3fY=s^NLagSXMQ7oL5SQKoL^Dj`0+fOC=7+a?vP-9OK?c1oB|SIa6M09dv77W@LA z@?d$;`t@H#nKHokh#D%l$DkB^*w=v&COf4E_d^&xh^{y z%>`P!1Tu<{k1w6bz0VG6c~-S9{jyDedM^ceVxr;5kKR08&fcCUWqLgP!TLM8B5u6K zr)upx_C)j<+KjdP;t6sax|ilYNgdvezb4oy~F}!>C?$0Jw$nHW8yvrp^>KO?+xt<5M2kAD*dXTbVLK_$w!Yybr-Ib_-VsW_l#%fozoff|*&cf5oKbCrAoc6wfao;Bv?Hjj-NoxKC{r(ME|M!$HG0S5uj zYFEIx2Tuxwa;HB+BiMlRw0dr$eT;oVl8oXoWL`2NOpE_2>qL8wf@TPUKAe<*iVB2# z22wLw*!$btRN}7lFR$PD{Ag4V5=Zd34AXbOkwHn1RRx?6A3xHFJ^6#iB?U*n5lS6U z`)HcmppH>fuWW5yW;l3^KSPqr4jgGHq((l4s5lkF%WLS;j=82^o7F5}i7BVFdfowt zG}oZv%*=e_SFcLhsxkn31XaXI4=|lHRA8=_x_95ds_Nli>IVUZMde`RKakz&ZXMs%EtRBf}kk=s-7gsA+&8l zOdG=6r6;SQ5T$~&(`91gV`J+7{QvyILOfGte~+mgqu+)j?|wQr15SXDZ$^SmE5W}6 zsRTtOhDB#Wa^to3=3LiWI)Aj$a~TxR5MkM@WMq0ZULD~Wilk1xL(XXx zPJEKB@;TVg_k|=~o|LGW)ju!zljhRmdWUQ6vjb{!1OLv ztnvOk3xT;wkP=7FFt1z*BW4VMAlRj#P%2_gy6~;9q1k`ZY+2&9skmn27hVz3rTRxy zd`@lMp@+ZukEKr0Me~zyCYt-+UY?>vBk4$Xos9o=WcY9>-wWv-pY0?EiBc9|T2aDOy*VTk`)_4ByxLa;QrWJR76B{Tn-2C``$|WuI|dp;^S~Kp5{L7s;a^Q&mXd;zpRd6Y z5BU6gN84kg#I`VL(lm@~vbq}qi z*jGCiQFhxE-q~ErC#rh+7Xdb`a%eagrFHpt0TM=Dg>T%{j(>1b5_0Y7dy6qKB&zh5 zMq?Z+aI{n)e-?k`+(neMUi!lYQH_m4*J-7Fd0SsPLuFN|0Z)%V(Ckn=t+JtxTmW! zdbcR;3=iGD-=}9~@#u?9rHM`WWyH4-)9xyBM}wgo#{c)d)#;vSL^(Eh`%c`Yz;%jH;b8`nZHS6Rn1( z!o?F5eV@&0Cwtm(D)&p5V#xGN+@Caka$a=(NL`LoJnn(&u&Rm??mZ4>QM~0w_f#X4 zwrVl8TK&i!kGItO4jEzJ(a4><_@G%Jf9uGzzv+Y%jK`g~vM#MYZ7>#ReVE%e-qL@&X7S8>}>c_dt6Kx1>Bx29-@vZA%pXF!& zR8atv%uS-KZCnnO;Y>NO`K306kNFyg9R3rA>p|*WR)d3-i7{$<=MQ=bZEsm5<02fc zXs}`aCoVW#Z+xjHE-YbIC1oO9E4{H)#6;az8 z=ovLVtu#W>9fu)jjY8tx#QTKnbkKe8{7c#C-^N@KB4v4mg9#4m8~8JEuN)e+mnHW4 ziNjBGola*QVgpI@nGnJfS92^{xmRYp(mPsz3>IOdmUPW=Tu%i(VH7Q&cSKq zSAzPOajx0TPNK=BsGlZxYY+H0eE$p|hs1&m+gj6fh923Rk?`1uTHpavs6j(2R+YG-Y6=0o65Vx~(dY4eDd~wEUkGmVDZWH>C z34w!mi+QF~O1&)Y{rmL2v9cYf=2KqM@V{7B5jitbtWs}`+=b0md^Yr6hzXXlD01KI zy%NMAhs3MMAbrDJWQdou=HNGq$8utPnZ;6gi3~<$-$J6B;RA;<9yzj?|0m@Zj_s3$)W zjnr}#kUo3~f$5G$VLy7s&`>x6ZfW7uOJ2p^ zz78W78cCvFRl{MXc6AkukWEpU7Vn(q%Fi%?OT9T1xv|=3u~;!&fW$%x z_74@w)f|jBMzJ}KZzae+kjp1pq!lpXKW5@ozZ5x(}>mLPfNK2(KWU`=Mnkd55>KN`7(ViCHNUy$4y3z+PH#@M% z;I7=RrntGo<@`4+S25snC-E!epl;Z+-GocamlTpl>;0X7$RM4fQq4N-rSU*4`C?c* zUWM8vjWYbhHT@$GuOm`_#s79Zzb+1nl*LX3+rBVFNJ#wU6W#pG(i+JG4axR~ue#qy zlWN|f>W`5uAXL)ZBXC@>kHwMpS+Nr?}YGPsn&~U(gw^McN z(2)H`aCkU(!NvS2umk$zBs+zHKYBBlYR($rG{$6pl=-`zN=Z=f ze9Ndx%I3uKpsO4y`QV>usp<#Z3M?jEI09Tv{p>(VKlc9;<`h;>a9o&BOI?z%RwTp2 ze(hTEhSmR%{63E_ZTg3lltZK<=ZKSFpi;V(ni9fK?yr8yd6d_b#%H3HWv(oi*`{eC z0|_|@65m2WnR`HC?dCLX@rcuV=aYt&~!0 zvB|)u!p$M@v?K0r?nxWVGb7N#Wq#!1;=~>QN0$$}=sIIofk`_1ds3G)bU{r`M!hR| zRVcB!IB8GMJ_{ZP6$xzEyC1JoCo^?C+btjg&9yo*1uW@5HVVmD3NVal$=%f)GhDdp#5Jjk z#~*UCvR;CDb_SzW2(~v4Qb9h#Nq?I?bPw7zBcUg z{-(19IYs z#fhSPd@R4TGduIM$%%=o@2#zC&W)!jZk`J0H`V`rU$-PN+a*AQq~%gKpeEQ)s!Wdd zv~cVwxGbSkd*em_!u6HO@QnsV>SDn3%9oj29~-9T;1z9ov*I>%B?_b|aBaKeRg%x1 zKZg%t(l6+!7#lGF5GHIu3arE$@CiP8v<-+RiGrmRA_r}0VUq_q7x=UgOR?O(w8RYj zY910&XI7x(u3H|$%gc+E3NLvKvhi&qB2@V{XT?-gE%!!?UN;U=?$Q)#NGUEaf9|yhm_+Kut=qOW zfwclJKm=6_Vq^=?6F~&KI3{LDJD!gj(2d$=KFuYtj>_lJkBKwpRLoYd6J>kG^mXCkS0%2u8@2ZJw2U1(`adD({d z?Y{u`)QbGnnd{#e0<}==3}>I{J3TilwK{V|F{kI>-@Zi#zB)ilYHU0N;465WleU~e z_S?6oMdDh#_LPXQJ`@%et%VwD$a)(Be^DFErz;tlsz|JV{;cU5S00b3GlHX}wv{A- zaQS463EZa5g~kZ4A<+62Me)umz01N}4&>@!a)150^yLg!j*LZ= zbe#9r^sc+sD(ma(;Smw=`Tg){o3LYIU!P4Z<+QF0uDYbruqerbYzst*!?A{VC+3M_ zH$OkWtgI}eElYgH^+__h zIXOtK(2M+fDJUq&&20qxbaC>v^7A7N+eswrQ>O~7Re+5|f-@=I`Ooz7axIZK@$K8U zi;L&KoatJ|0!@=f>{{=w12wqRb#(`P4{a&5viA4#Vs&|m)&Dv$G(;woK>;~`d&`?= z>hA82r;gnW#JJ_1LnnjvE(EVm+hmxnt@IkT7#&rA3^yad9Gop88?C_=_yAxSJN~1x z`nS(}c<^*EqYwW!=o5c`gG52ia0U@erx*4yxaxf9IbgU>Gujf$8W>-?G%@0=prAlb z?+QF>zi9G|9t5{C%4X+xZJIZ?U%YD-fCPMa_}u3#y-QRW%Jgq1>*XQ3;URP9gC&ko zLfi7{YC{Ewq5IUUn#J18;_U4;SZSB_w{viCSy5d~4hZh*?lw0w3!RO8I%ad1S^{tv zgrAes_Q5L7pCAj-2rO^ww|VV3OQ%Cv3TsZouyYRh_KArJpx8Bz_s}2`q;^x~Mdxsl zaxKV#r+lCR0!`x5ueeHdNQFEJiWCU>sPe?0w~B~xRa{UT*zr@s@NS804u|vpy-%I1 zf}~{TojX>+))(-+kP+snA(_x-Yio;w%zfgmzrUG~`Bj+DGcs}{hmU>yq4UJll!Lwf z)5$WeH;nfwgWFEB`X_-$9VAIG)qIiO7QFSw91lttn>ZI#i{;XJn;di^e)c3!af8+ z-072UzSr8`0qT4Au)aRGdlKFnYgw_A5ClQ(`wapJIDd_aYpbix|7MuC)3iHvlK5hx zpU|2E7wunzdS@_-EyxE0#^5O;r_YTxjpp#}5T!2sYc%p;$npei0653-Y49P{KYB!! zkC9#JtF8=J={tB6-LYTv7wC1NK#Qt5yFZ=s#xPCWwBQ8}c?Lu7a*+VO(|5={Y9k{jC-md|&ojC2XJ>J2j^(Fn z^!6Pl@2CckeVX_=?~L z=pRQPGV|-}>tXytJAzysj6eJa;F(O6Z2bLqI|0mhSCOc8)If_umI^?g^e~AVSy@4K z!#_lQO}%$3J$-zBJ`#m4)^mDxR#-rQi6ijKFUd(s@(K!#6%;13>agP2(j`|dsTOX= zPD&OOj`&=KgoOSY9{v=5=z2$ez*`>X-(0H%{Nth&p6JWnlYw=^XnF5m;HZOBJs>BN z)xEa11`gLeJe*+NDNJw_k-l12Umq72*X>mecJb=fs}OP~w7t7=;|97Bco+d=27zs+D3X!T~$@;6Pkpiq?TOs#Kc|j=hiCs zN=j_hG61zG+bf;-^ZQXP9DE!Uyf}@1etzItV0_0t|3s{Lr6CrN6Rsod3rN`Ce^=&D zTI%Y1b9=aAnefF;l;YCTQc%{7jUxj%rVg9Ra9ry!~hk&S~9H-Zd-D(`;s@`cdQP*_{>{ql%XqwYXCGKe{bR0l+cF9 zte5Qjz*$LDiKQ)^keGO{tgQRlGuLOo_+ms0KQvn1+TWjUU|%*!WJ9lZu>Sl0J69`--8VAs!WUmvDbgEnV`cx+!P7uClvfkpfQf;>G<=%V0OsI=~ZQ zr9c9RP%Wpg&-j8|*BfN$<@OvSf=b`PS`FP$YLp}B;zJ?`ZszCT$wGqC1Lrw(F3>C* zMaMAf{@UgdeHe30E4?P_(_8)mYIa7(gMeJ{U~(e6;Cij8u`L^fq5-XLxfFE*2M;P< zq&B#_-*Kyem&FqG`oswVrw-(=f$PM%0>=RnXJDPd&Ydw2YlCO(SsW?%aPC$yG(_K% zow?*qIIL58e1=eYj3&*sj4t1}>Afq}v@;Ebmt`Fmc95Bqp6==Yy=j4rJuhCT03uF$ z&1Z-apsB%Tp}1fe_3bNYAq1XCB^$G?Qx>*hsA>GR?(3x!;Q@~axIL565Q|=v)*INi*@8G(_tr*ua&JDMzd#x*zWLA*Y<%QIXk^Nd%LF>Q#>n}fTX-GF7 zH#BrEr=*bB*6_NO+m|7j30u+FoMNSD3L^Z7PRbON@-goin|uQ2L`1{}WCbykm^7e& z|Jr|@|MEUUC^9-u1O!xC7KK$Bpw_@_nv)4L99Uls2TZlc4*%~|OIycq``=AyV_QdO Wf2D}VCUbm^aN?+?NjZgf<$nN#i#kyN literal 22694 zcmZ_0bySsmv<13BP*S=}8UX?65~RCRBt&TiB&18(ASK;h(jhHUigc$aDTq?ijl^4= zd+&IEyzxGV=Qv=0^^3LUoNKNfs;RDki$#frKp=3H6lJv$2vkD^0wo#)6@GGI5IPM1 zyZKZ_K^Ad!{U^JnFb;vZgHV!{()CFHJLmZXXXWbFj2){>&sAFjvW-}-j3SjW?+%6K5jMh zx3^+#%2IXvQ!~sryyD3wg^z(i1mxrj(7;0ku+S5rA_Amj7|;-iz+hSU)lJ+`d;|iA zl8_#OAm?Y2Mj&`>6p;wTlaXi)6vXH3|2biNQkE^Le`O#gZ;dTJ;+L@6+icdCObJ0N zp4b0ElluHcm-Kzs&rhis*x1-Y4zoq@F;%Xl=b~s2UNaVRCL|<$^x;77cAA`=OpTP* zV7ow@n!LVyHu3-WUjM%j7g}%w-oZoA^iI`}_S91GPNjxmj{Srfc+Q8VzxARIr%qR& zH{VxdsV(UKSl#FAd;K;e@~HGxkKp6yUsU}9I9k|4)>fnmr2G2Nezy`_zqCz~m)d8& zmVCi3b|10-&cMPIK+*eeSol>3)joS&55z-0{z=|9eurJZoye;hpQzCtTGe`apYqCmT_d7pUI?uLnXc zr|Ew$`cq_8>i>RE++zE`zt_m~|G%52w+fI`tiKUCV zZ*8|-{qgks`@M=r%uV-&ag&?9xp~e~Ra4Wy34NmA2CY4oBF%=Im$cTlV@m zOifMK*47?)ANic@n77b8efpGvf#L7pzee>hC@3kt7TeJ~kAC+jnSF9CF8c2%D^rzQ zgvAlJ$KDf5+G%=Nk1gW)%_ukN+Wg_GVh`Q+M;#6(7$ws>8fu4HXD z@2k(zkaC-fyKl|nNt^`WYd?B)I_JJ%KV7S|JF(mw7x?ClE-^U*5pu?rU>4sl=QWll zow9Z0m3h9dtrY&Dtn6=l5cb8tV}tYWm1b>3tm+dptgNhsdZqed-S8IOku;^gcJSW? z-{ZL)nYX$GR4`p%u`TRSkGs__pXOWGJAUnZBP(~WJw2T|EEnF9SW;9}w5z*&YNpU- zWk?~CnnK)zhkkiy@$2d7DQCYj6QQJ}WU;Q-)#ZiIf=`Y6_Pqb)Y2oT)T|GS#5)!pk z{$jWO&#!Nod95+)or~D%XlZ@ni73b$xg{Li$B@+TnOSJ7Hz#wSnzO%}vwy$2;0A$K zXN>!0!g61bjPEI_Zk=((T0rLSM)jABjEten6e;S=p(8y#iuh=ezw)*3(&rZxbbbGL zsh9`jiA%}P#?1Ww)2Hd#+1c;k4R_z#&wkXZ`dVu{Zq(qAk(~U!#rv3bL7F{*l7iy& z-@hpb+o_7Vxb57>C8edWE;L36;i1}{(yOZO6PoYe$N9<=LF&gDUaGY0H2*DmS~8k< zG{!M%_|+SsKjWpbva8F>vg5A$`iTtPF2rY){(vG3juHoC0r z?d`FxS9N`?si`R}T*(k~mnEWC8~^j?PaLa;zTA0guCN|IO_0Rr7+uoN7+HoF5>ohR zQX}IPX z%RK~%me1Yy_xD|#osC+3PD?9WOTMT467Hi%o09Ajt-Sq};i!<1j~=^A@2G@e_=w%Nx8|Cr+WfC_b8{2#J-$4gGHbg$D^+Fr zcf8==^lIa1RDq5_n#;8LWUrqarriV|`ThHM4atkz`c?=6@yzsQu z(4;g+RaRD3vV@)5r>8}Qg$+j|%FD`{UDvdyDoiMXi%UvGUH+&#I8-z=G-PFEEg7Im zDXOS6`CW*rjkmV8W@cvMws^?P%WGlGIXU}z z?J04frK+kbKMe*w^6#hqHka6cjIz9+^;jRjQI`ComyDr@6^fsUs-XQS8Y4l@n{Y8I z>6E$HM^$)yQlGWvj@B2<$VPb-1kRg>J?Hzg-)7p2=_B3-Rai;sJ%q6v@t+h=As?P& zCt13s@|S;sYm)0aka=yH<5H0HQ}XE=BRhxDdv8`FXv|9%|koEQ;lS?WnhMPC(fCrX*|CPx;55Uu6S4Af+e?MU>_~c*X*Z9O=)i)HmkF0qF3v4@x%;#P$ zLjn+^=_O>i{Xr6qCvGN3Z`^gM+hO*j)7I7&pY>3)_i=^SFBC+8{11;nI`@dVNi*JR zc+{E@x>kt82b(kQzxxs-BqRcF;YEgqE()s;#=s?I^i7vmvR0z-6469*WojGYUYeRC zbpGctX=(rZx!Wu(EO2NVO-tXI~y;+x# zFgfGcmJu84ywnjI9UaZAlIY>>jYlO2Y2ev979IjYTfEehEWG6v`qDY1qm%lPrby;o zjSpP%#;x1bVr~(+MH0SekSy-;^Sd6bj*1VL(o4Xt*h&~Kj%vDaen@JLp?1FM8-K!J z@yHDpbl^?gy`!Vc-$`cjQMA7BZXZ8>hg~$&;20Giu4A2zKm`AFNWmxn+G?X> zk0D=`9UDqFgpN)6v8hQxUY?YcR9->B+S0PzuqszAO%RD6mTT(yo5a!4ah4kmp&6wj zC!O_RHec|d4-qW?wx~LESm5m6b{ifg|0<-6!~K1txy6mjwylZcIQAO=45Tqa`$@Z~ zE&VZ`$b0877ZEOn@(2h_wzn_cZ4^GFSU2W9DLn!WWEf{M zz?0O})P#iJtE(_z-TLK5#X^Drkm~mc+YT7~pG0>|-F&1O1k)oY*BO6TN6=x`(A{Pg zl7#=iO`+${pTFw7iFlHYCM}_|d}N7H_?Fj!El%_PF6~Nh9P1TiKA2J`Y3X)|jzbw@ z$CCzTG$PIxFwS<7c!#4g~(>;oU}MG|8^a@HJ`TIh*EyNGseKl&29b2?dRv$hxK^m zJdj~^b@L-X0>IO#H^!LjKbzK^IpghlW5*bavAMZ9H#f&^{87{KUl2AKCnqQIB=VEo zo%|x@aVdZalfN10f6_UmflDPQ?sF22udSt31vyh_-s9Jl znZJOLkb!sfV1hmqa;tMM43XWkR;rz?EqEC{f`4eF9&%DU-{LxLG>-SdnWDI*1n2MO-GR-L}C%jhd!L-a!1k!bA>84%@ffOQ&r2Fua3pH&F9farg!0I+BOt6~9705jo-^V~Hp4 zCAs?@oe(-sDbk6un?&X>Uc9ieDXOchd;i`N&=Bg48-RO?hkD{m<8bw~@8wI%a-ksx zy$xl8X`a%3i_Rk;6x;n5oGq~wx&tWgHXdG^*Wo%xWizkieDjp4_dOvY_l^*PQvGsm z7oR?90>ZzlbTWDP2>yEwZ_u_aEv+VwS|f^2Ht%Y3>}usfu$iBqZ*t$Rp!wBN{kJ2O z=-Jb!kgQ>RC#dp6RwxBZKCEM1ZTl`BOH4n`537podjrEtE>OZ*mnZL-fE9S#`B_V` zC%4^ponrsp3pBD*p9M&l-^w4G`JGzM#k3NJT6}qf-r3n{P+_dKG*haR91%7s9S#q~ zvfDbs5u7m^Do;AWj_Ws?S9B)im^GXG*z`j#nzTbA0~u<-&A+{O0`aOh&J^eE!b$7h z2(q3I_V+(uY!3pCP)FsNd9_|zSyoo&TI92%lryAjEC@lnXy%((``gfg*4MEZLURsU ztIWGpbqjv~HjA|Knp;}l;?Z%%_3rKOA49;sI6H)GHwrjuke*#DIYKo0Zgct9Vrqoz zaObUdg{xR!UlB>M;(})=1WTbWVPO9qZ<~F1@})$t)G+bA?WgBc6_4v*gocG(k2VQQ zo`hIGCi05`FNCIQN+2tHFBtgq+u^TAG=4)oq>224A!+ib z;m|UIM_IQi^xU6`us4E&kCfAJ=jWFjhqv_~`-mzS2b-AzL$OASgn1QTBJF!S=UH)^h0z13hv6v^Uhn1=o; zc=hzG>8EJk4Tyz4KFy8S$7vl5l}S^?*juuI>bok_~%R6_t&JwzhQ{?Z!m;jwbX3 z54syxh=(b}b^8mitVMIR9~Fu#96zzNw46I+b2TDop+QHyFDcK+8lh}3aUFPej4?hw z4s1dlqqw3%!hI{g5fk@8AFa^lD>*I4SJVDc3ZVh~4nb?;MouQ+SlC+cKNY>Rz09 z5j0};8d(y^=$)M%i0^J&vyDf0b#%rd>8uSJKPS~5>kU@!A#1l-?Yk9V)ysB0VN^0q zMHy3CQOwc`{~gjjFy21Cz5p=!i4#PTy+wHcruokI&vs`NEVdvm5C^x#EKA=O9zt$yQ0q~Y@b@lS{ zGBQey`tMl14GT^~O|7n~YHhmi`P}9;`u9F|SQ*N&P94Qyh+m9HSgV<*-D3 z8292+Cta`nX7dn{2C@<$HGpdvA|h&PWB1(Au9=Tczq;xCXJNmL7i##QtrgZidua!m z?fKqvukYDGqw{a&e@B}D=gU0uJoo>|DJTH=z5s4mYET&&5%HqENhlao=KOC_T%$CS zO5!VGXBLLHihyupXhVDvQ%_G1K(v|v#bKO=fVQRO`@le%7ca{0U?g#y2?_~ub8?n7 z-RQ5budly<|9)7*^O^e1R^NYUQawFAfEHSO|JC2YXmZ=!+u3>YNdMs8*!cL|>})D^ zPP@TMqsCB3TdRD0Vxryu${N3a|EBJ?6~(gGHM&Mj$c&tA_8bRtxf@PPOB?1(9_-}g z)Z5dubA0T-cMnYpxc&D}ubyv`nwc#?n3pt2zpP)@FOKG3*{prmPMx{pNvo61juM$) z{Zlc_D4pG+KS)w$wTq7W@~W7h=fr zpdVyT5xXM#nl7|1$4{H$-k(`_yD=RB?b@f+Sb^}FP*48WS?d{Wa+Y4=# zt1RWue^gt3`SN9DI13LOyQZ#gu(y{UDJ&u~_t7bDi(6DoEW9-+G&Hold^3_pER$kj z{LtmMh9thJZ63)alDZoPA6COPJdtH#VNpcZ@%9CU7XElgPe|9a@~_~%BGQY`q{iwx znQ9zC1rg zMh6500Jfi)m{5E_MkVAxOhO{S8qtLjr&UR#V2zFtr?Oi)Bw@y2Un5Buc3K)t6Dm|q z*;~o-ccv~R=d1xVvAVhnkc~h(H!rWWq$J|qJJ9NE92{C57bH3xhO)Eoq^Jr$?Fb>^ z&__c<3$fCaR{TFo>c#hK8U5*1TWUY!dvzloPvb|AQr^FR|NlxIqYPyNzloEWS^bZ# zQ=@N=GUzgZA~<~FvoC(NjMeh43T{upC{Hc!@xe`&jlN@H!5_AKSU2o_V8Lg_I)$d* zn{~;_$U@1tdT%Ev>ql8>pyQ*w`fHGQMM9QrRmBwi4;dP#^k+ zO`pDm_#_bHZx1q+sp&ju7eBv3+znf|S=uJ)R7TxeCpsH&>ck}i4^XJdkltWjS)2P-4lr-213lI~Ctc=I~B<``@RjWwC;5J-aG`ocu#;)7H zDcAhv8WqaCy}c{m=C(O~3y;?4@^l5VEo^Oa5$F8WUpiC~H0S|FhVJv({K6!*Ha0e( zFq)g2`}ws3@lqxWpSqbdG%=53?kuL(jeA%+fe8U;7( zR&X&17a4Pg08#B=^vAWnWtQ?jsX4}7YSx?>!gGD!<@KxG#&GA#j|m4-#PQ`zl0{cTQdS`+^cKnyBgR;2xAAcpk%M})U>uO(tS~WBc{Z%U>1s{i|KTc_P zwmLJSl!6|atkTm=mcl2)tb{IInDKWW$5%{2Q_S%v@R54^85l0&DA3hGAv7^D0g#)S zlvLRDogdMXhd@T7lL@C#2+XfkZOA<5U}rJjQ@OLh)@FQl@i*b=0}&lfF+AWC5NkQv z*fhm2|JDrOrWPK7DO+A%22!E+uFCdd5#KBR?$2GlDB@evIUoM8w-q#;lvn$4={o0I ze4v^W3(iRWobatkMD%xB>gOLhZu1{P&$m^Sm7QH&Kq-^I22S(voDin`A##DLOe5-A zBq+819^d-RH8Z=?T?&y|9L88lvyymhk*-QLtDqcLOtTiwn4dsx+m>PClCbis-l?c4 zVLPBK~mAAN5@))~^ha_lHcf;GC-{W8$t7lHIArI_C z$>hHhU$InrXDfRVP($%(F8s7fvgT5BVV~|uwRI~)0JdJn1^L7fm2gZHY`8#O_Li56 zsPqLfr@TymOz~M*V#VDgEE=knJv6U>+NCm^Y? zs3u1zB?(#f#aFhTDOc@6769Ey5ER&F9+V^9p~~#|eUD(PWShDSIc2;k(kf3)Cg;qz zm=7Na{Std(+&X^SEz2#*5O#J^I3Zg3WwRg?tQ|HsML8i(?``w8)soj$6R<@S0m}I4 zv>+Kzn+f-VdcM9;&YxJ^{gZ9XC3OG3)&Zt~sj8^tCeA_y^CODS9*^uyx}O=>^IVcA z39FXt1ztp#T}@1`#Vg#vgG9C!oFv&oxeaca|P4~I?v{4 zo`S1uE#%K(;S-^dMfjdwP7g+**~p>pdON?Ihg4H-yb!lrKOWpIXBTnwGG119uyJZ z^eE8rY(ElfKSp71e9Xd^Cm77rbaubZbkTdu>T1$8wDT=ydIsj96Dv;i>+TSrJG9~+ zpR3E7n#8Wf@%^RF@bLmQmshVcKuLoV#r9lN;nAJR!4Q@?nj#N=EUAUNqH_K?`SuZd zCtR0D6UFqg$J66q@oI=7olYs)tG>I2N6Bo8hd{gs@f;aFHl{7z1Bp#$-1X&49(vjP zrn*%o+d-mvA*crng`vidtacyY_B~eQmxFRwWkhV>fD)@(ykWK*ZKm`x+K1ZO7jvHu z0bl6o=m27;At$%7w^zexe6>MJ-_hHvGDay@4ju?*##;36X<>)KNj5= zf45tcw-t|xp3reSFGGqhp=P<9Xg1X}qK;imPOh#h_Xsz-BY%H8V`=(vX>*749Sd!7 zvc=$OPn+ly)@Q&}0pve$`}=)!ra?wpnwZM_c&o*BoaHd6K#MT-z&SrM$DR0K;(*WC zj6$GC?b*G2`mKWmt#!7lhcMA`9+2OceqJA%?kXU;tp6Hi$l*P+iR7tH0Q#s||NX_H zW@0qqkDqImwDSBZ(=oKa%Oxernm1Jx8uq$w( zwea41P4!nD_vK{8?8scEGC6z7L@@0~GZALeqKvm)K&W&IRH5M41i2a%I{=ODM}L0+ z5Om#|J%vCtG1{n<7Eqx0fzKCb5}B=V+Zt2B1u1tq?eVp&zN39Q*>zT{W{+@K@E7d^ znAY76gCa?nKLdD@=OaKUr{4kO866!3k*d*cGnMk|_y+#}X0m z-CV=M>}+NAZ}PAPE8e!)%W$i{m8Sdj^|Yr4*AmAB_M07xmh74uKKlITW@8#B+qa;Q zF=2<|A9UO*Y)pDk{KT53=eB)W(Zvns1-HV52c5Vg9rJg+BggR}1T@0~HB?po_+T#6 zvPmWQB3k)9$XIrmsvNb?m8wuZE8_4Uigo7i7w#d0=4z{SYUyQW*ZLWmTBz4DuRib% zJL6GlT|bHqYRv99&R3uF-R>Ie;Z|U^pbuH_{(xKR9~O+g_fOW2p>t( zJypBFI4oB~bCMLF*ahk?LmMXuEjpN|14*vOjbat7L~K~{r!JlfSHi{jPTP^{bC zhZU~QUc|JWx%BPGO7)cl9-;Q*r}wq(>{UfyR6g&h5S94xnxa+iYng$2ypSB_uMmehCsbJBg*yVk$gM6a&QjtR?Yu8X|qEPpw zgTFsqV4YrGUIIJTipTrr^>4Fbk}#PRGeJv-NZ9mAyf__C06Vw>q6j}GqYys37zn%y z-ODvQ%C`;s`4`Con+bHx-`A0Pp)D&uUU&QM4OJvBU*CMZvhN+&s?NrnqR;y7Zk5;9 zW@TLZ+q}2IlHLztZ3kw%NNa8qmcSN`A!Tsx6xa4BOd2&nomDa36EZ$NURFUtBTdkLxhDoxPCGnp4nt)n@8|2P5?%^YRCC^J10rwLwE_z8xw8yE zzFa)qn6e%K%R(0pWH32v+bjuRkjUQ8@6@U!16)nRbNaRsD*EsiNUAQeX`>XSqp18r z9t`~V??b)CKax(8mW(VZEe(n@RD^_MZT|jYysP7!y7Jr~KMzj1k|Pe!W4n@ci6$CR z`3J{YGBE%Qg1q{eZJN6$MwyGa@6pr9Q_+XdXYUgI$ykZROMEh@8c^`Xpl9U?3qAL= z7ORU>8h$N0ceKz_;}4VMqkAzoQBpBNcTSO_<7~Nz&D~#?t--8xuAf73MqGvcGx7S? zrMzdkh+35vza5*8+{GH9E>T8;HuN<-mEuZ@q7oYV@8II({0lKY zY;$pOasJ8?b_zRIWv`d8=2Tb^gG{>B?OJT^=&2|G;k@PL(QyCYS`6YKD>QrV6Gy?Q zrHKR%keT^NX-!<09cjdlU}UaM9P_tf?E!HAHrM37zqbd%d7*kn$@k2Zlt!qJe+7ae z;o||?#9*itNrcVV`0KD?pqe({_f(pZ4BK@Ig|{ENJP-}9Fl#eyT9Wnf@UXM9i!+N~ zIfc^tP`U^xOhv#4;W`V~f!vpukI(z)udTWH7arAxR$o3ozUNu7y%yIz*wmCarL6aU zk$O>tL~7u?)=2(y=Jr@(u1`W|vMBLCwP*tJ@}2>w2WAfil+@E)Fl9jg1bU%){vEUt zkXt!+f)|6wb|MR9gg}#BB+`g8@Gb>n4_~ z15F{tTu3`P!fPMR2$_Aq`Qg0ymwN}#njhc6LuLDtC;PqBfSZVj2oDcWLt_|Nt!ldP zDAW>r&4eL8@|o$zWDN`sl1SFuj7s0vdu+nR#l_vc)1E5u9C$Ol=uEvm$qW=me&op9 zlET-oFs7iT%}h*W5iNcUkD#s|nUrL(nYacTXqw<=N|k^a6;bT&$f}!EV_<0POb{rG zHzh1zSFaw&7`w?a5ENBaB-kY*%(#+jI90}11MInuu53l|eRqsmKY}17yB9IGt)TJP z$Ov?&ibpkn0C527d*$R*Raa+k6c0oAAEK1wC@(7N`SK-aUV_f=BnO^BRTYQc0)T5w zOiUPh=GpOf<#+SmSZ1hvh`e4ISN>NSB%$%>qNGp!wbFa`Pv@BdM4ZMSq zYbNp2b8R=KS)23RIpcwt>Qh_uvfze1+v$!Dw9pTy#E(WgI%LORuigMDDC)P8yZbTV z7DDOt^mHgt;FFVouO+Jk=SV|?hPwJ~OiX%Yet!Oh;EA0{o8JQl2I<}J!^7MU9<+e` zF<&T)kG6SkN(aKnAVF?(W(#WoVU@V&rf@?~mQ%w>*FNe)g;4)d5O0kjW2T+w63c-- z#};5e9H5%O8ua}2P6&e@fF0}yE^cmCA*1}jw^DgSNHKLDc-o?(+5=6;5-+xCC;|2y)!H)!c zEs|QK`lx<^&v&v>CA z+kl;ffbf8}uZ=TQ2AXk`J9oF!uda7GOWqr`niN5H(wb?YN?7yd{^7)Uaz}QE1LRy7 zN+`$eCK#A%MTow)6=h&}3Zc(>IJ3olJ99d4Lf)L0T@Zlc1bw6C$stx7I|hgJjUuEX z11A@k)uV4yQw{S+Tqd9JrJ=kxNs#pI`7=38K znmydn&#GT$sHLe11;jVtahPzx!Nt9}I9`|}Nxr?xk!O|p8|_VKloWG`Fz=%9vCN%c zHYIDCh=WwfSbTglU^?gwC;Rp57Z`p92M0k#JioY@q)tzCX~^b$uWJ-D@`bV(Ki<)S zaB4J_auO>e7g2IpTdXMdYW*O-lP=)=ikMFkneA%57bNMhb9a>(UGss@j~Aq!`FW|4 zs0V7y3M`#2lIXm*|N9M7yVV*;Uexa$YHfP9$+DW+aJLDJ>!Cm&sGvxw6|u^5&S}x}iNj0z-=7Me8+qcsu1voXrXb z2rm)suMb{6MJ>_@%#(kV_`rF&2kLaNz9m4VmF|J+;^4plPmSsW`k@1=?ncT?MW@X- ze?(Fl&wVu%-}y`%Iv_MfcXhbH*atR55s@EInwg(Bjlq~Id$bQiY^LZdE}IkfI#>lh2Ya-L zLokEjPs9vz#F#E?vE5?gj*w3Vs-{qktf8r?A6O@@)n0`tTH?diYZ5Imybp`X4o5=> z*=IA$g8UOQND3y81-EWREd(yU5)u*;b+@Ot_teZ#H79+3Nr^Iz(^|ZQ`oxca(&(jY z!ViTKzC|nT8oN3cypJi6LE%vZpq7%B215V08Coi;ex89iOhV55{5*k#=%{%un=o6< zS(U8`gaOQnP4Ee?Dwr-(L|h6$v3MHfC4~rnU=#0iXyjJi5IH|JaaK?FfZM+d=(o}T_$cccJsyBdy@ zROAHc=eD@EuQ@3Br{7}E_Gjvb@QKQy@E3o%O%ZI)0ICxs<0QqqEnf4GESgum1|gq! z+-=g6T`w)Tu0N*H968e4yLPZPekjPo#57S}5a88raWMLVSnFY)uJiK6`1#qBE>sf? zMCuXs_!}oEV~E|zm9{iY!>8O>;h!IlT;D`GCL&U&zIW*B>AeDvptZGif_&b%*LNKu zm7sR^r{Q<2e-_hzi?$?dWF$B-!ptV2eNIqeSoJ2CEi4!O5O3bRDG~)k-JjvCYsY~d zyXGcqj#4%wNz`4dpO0Eqe&75!w)r?$1Ora|r#=l;NZ6@F(xhC*!hYu-cz99Mg>W`d zKTr`1wh*tBi}-x`6+C|Ej5`I8auhL17Vd_A9i>JaA&ts-2l)jKWHh*qyh8CQ8CB(5 z+^FwKd!z>S&VKw>o5Y+Nh?tf>f4=rVE)>7jrz*dDf@ZOnEFfAuDtw*d@6!4y0MpJqsW3-5&GDAKW>*$7H@QI#IN3GrMx1(_-GBoi%L4b)^H zAdCzQP-O&f=~o_*-;wymx_0K~uVqhM!x+NuC#O(nzATshp7q5nO4e z7?>gk1_lhwd=Gy`NhbFH26z+KYteRblm-fhh&eCk81|A?#&2s^D;`X`fO-O30w`NC zv9fN@Gz@ok2H}J2w77V!Cx#KKqh?U%2?@a%5I!TODJ$NiT%4Er@)b3Y{%1_haZty< z_AGkCEVPFkV1ok9`%$7E4u$<_t{gy_r{0Q+ist5v5JDy?vnz5iFk?m)M%j?vK+p%h zV+7?!(E^S@KRY@)(k|u~5=z2f+S|qtEG{hQw(Hm|1`PpeP*^dL1reX|HO+PAEF_ zbCse3M6e-2-E2mQ8wewCeK}JL!5Cl?v7Ddp4}&nd9;GCS6wz_|p@h0ghWcJo!qXNV_89 zZ*^^=vDkYxl1{nJI-c+&Xzn=YY7-xhYe)`eJ`lTmSLp&}4fmGLl9f~j_d^zb*##z- zXTBfKxB?vT+g=mU(t@m`58iAlsx)vIeDht0iZv)jbZq*$I}P2FBOfdhu>EJ3xg$UG z5+SZlDC$rW1)%%-_3QG-4PDE*YEVME@Cv#bAiH=L{5zb<&sl65vveDE6$281zW&ja znLoJ5-Lri5LeFlDu|l>wd1g$a5TT`L#REmK`)lWQS~j;)S8-yxl!2Mw!GMz1dX)UI zAw`U!zV4FmEDQggtgK(KCCWzMp=OK866R!NWMpNTNlOC|8a2wr|Je@c{VtY{Sc7e_OWjPemsfJ~O~ z^-@v_UF9x&+@Nk!wHAuFweB3FX(VHmU8|~b;S|^)$PzdUY}a)xPc}Iz$sH`Ou&#)t zfqCj^YGyYU9faMN0g}8c(*DHVnC(zgK!eoyr_j?w6GX-#8v)>7IM``HPcV|%T`knK zKC`(k`JHA;`N6?+%7rhfSncag2vL=DT5~(Qbs(`D_TZ^>Joh4o@vE^K#KObN5`9%- z^Glm6i5s|HPc%JLet|C(&t+z0Jbx4Gr!3W$w_|gEpPnX^&etw(YrBFT zf_3J0FyVllR|7aDC@2V?;p$JCBL+Pvlps%r*>Zvpgqns1h$sxh%xs%BHsF>#k@g;a z@Mp$bYw-$AhaWk-{&Oe5!JCu7VY=4#KUWqsh?F&yhKBY6EvMYi1^xY1Arts90b-Xo z48sPW1K-zm^q81vh7|~+h5a%&yRNS8^2&-k_!8Xu^OfSRT?t*tLV^cnMXMb0mX>>k zfA^PtAK6d}Q1_-|?fnz@HGZR{uTS?vKUEi2f{sq$-EXNH_dAW$)ZP``7`+Ak6yV%f zd_P=d0?!snem@u1J2D!~HlRb-K~4Xt7%H77Cnpe*`Tx8FwTzvEZVo@L|F^~vwh(wg zz$n(y(GeRP3z_KjnlbWOM+ODS`T1Rr`)h%bW@38U3m&%a@$B4OooDXQLu_qwGaco} zACo1^QE~?T-#BO;#-wKsw2VjAI!)|fk$vS!ey)3$XR64#_+^Eb`+tXYPcSYOI8;e| z)6{@LYk_xu@5Gch&>TU^@Vv27fbE^Kaxu^oI#XS$42kHFPmKp(+- zlYDvc4}{FeyC(qht;mD%rN<9IT4nik6$h1+XV%tJHP#xCQo!TPCU^;*ACR8IxUH@r+ZU6YBBp2$!HMRq(|8a|fi-zdhE%uGEtho3-AD@`d z$xB5=4ElVKPr)XAjVXX`e*%sQ_Fe0F4_!8H?lz}i@?Z}Gi!AYD;^Q#55Q!vU;JHt) z`T?rV&9!W)g#apqN?ILQ$=!xo*ZRDWhaX=&)wpeL^eF5&^63L zoraD{PZ*l-;J_y+=&;b*Qdnr+GCnXp{Rv(`qxa1xjt-~{tkx4XwjUDmR^bxf2)M)- z$pRvXK@VXS39*PgwSYuT#>K@&T2ZD{`{6TW-^(=3hC%Uuz*+C2Z=oE7^kB~A%3(V? zZi71s{Nk2_sRCjDe2~%5a6lvt(P5G%j}|jd%jt(WZ%*X2ZxnYaeWeFIF6t`D?vEVU z<$}NfK~;v$2s*q%l@rV*Z_2>rq~5#5H7ahEzT}(=5^Vd&xgpZsJE*6;p=QYIr5XvD zP4LBaVUpXl8SI`4WMpKWU7)Xl6SMTb;05s?)lN%Bg7wU+x#fHL%gkl1*xBxs>D-1a z0FbLcJtnTnmf{QQE1Ek--rdYSQT-s=g8__Nhu}7LU5&1)ayxEm0V|&)r=|4+$rPNQ zuR04F+CMyLQ9r#T1bNN96B3}8G7|wQF|l!r*Q;-3k04_L8(ubuAd|bM``25ma*Zup zE3Bjk>@TNtn!G644YKNvnJcgilrEh3!_Z?X8xP!!dD(Wc`iYUE_S#&A{t1 zO;TjIPAacn=|zMW&oCH6^BUNepf)f;mXTR4z>-O-tBvBNP-haEe(v@zYYu%-3gOG5 zEdfe4Ox*&cJ*bXzb8{zJK6~~IY7^C9JaTM3{nMMs!2fzl&|KqIEnn?4v^S;M+fEf$ zyvIR&QWq~(%M?Fd{Q4F;!Te8l=;`wz*T8#a`5yg%G+Q#ibD&=oakj{NC6Y>M^jALg z!!NGunBk8rxGizI9OpjoxdJ>DU^Y}QASnUKSdUO`3lbi$r2Wk<^7PI9)9Ll_{c~4X z08bE^--bc67J2d+Q5W=7lk2{D}Dr}eR=%ZL*c~1%C#WKR+lnU>0QX25b1DO z=!0v0h75S9#|%h-_i$qm*^aXVSVa|8;oP< z6Ht@}vJE{#Ams@|TOe5audT-mtrxM(D#X~D0fbOJ%M)3pnQX>N%qp9Bfu8WZaDPIczAhJK|x7NOA`&`Lo>`bQ1S6k!E?p3kR3;zG(+J1w!VamhWoscYm1vPaukM zadDC-KLaNwbnE1-?vRHaxaCIWg-GWm=D+0j9)oZqg7G9omSt|JxAzB_!$ARtGoWDF z{`D)XYBCQ5`eI@t?h@H+50*m6-k2+OFuQ_!Zsi^sL)8dNz-zC+%L^^u?yjx}UjlDE zO5B`n+z%t+05%IkM`7_#99dYjhge2P%9@|3LRGk%!LY$_CZC1AT%Rb0@)AHG0OmfR zU-Og1<*dDsNzlbk`TZuLfbsrA_devN`w54kFWKjS2Z|y8XYuuDXs4H~&_B;RT8T?OqX4QU1IakJ`qXIiS=ZJwh4{Xto0mP)HX+QzQ+B7d}IYvY38#V^a+)Z^M8b2%{kBy ztlYm`FUzee*%Lr#Er7|usI9SO{o(~=Ksv`C)#&e$B_my7gcyzc_`d`uR&+C87Kb#y zyME{SywU(;F35^qI$b>=?5k8o8KLqCgSBv1+&F&HdProe)S>o%f$j|+kEn`fSiNuV zB&VbZ)Rr#YZ8WZk{D`Ab6cIM8z@QuFOCjxm4x1k#z0T4OnS+vw>RLq0F{w0ZlE-jz zbJJHa4lZoNCP`AE>yD-(}o_BCj5%eT7m;KO$FLV990EajG2T}z%1Oxdmwr2RQYhqZ9) z=hJeZ=asv(q)e*8GT4Z#V#(1R=-*h#Q;33o1?W-96mbd0e`;$>K}sqJjt4+g&l#ix zf7vQ!(6UoFhxlFyH`EI;%H7J`0|jrx!q3|bhXg%9LDp*a`1p9C+7g>ymMdvr+!6ZH zkBV?I-rX1BSL78<^j*8tl$V7@vd~w(M@aYc;o-}NF8ByqPa~JF0|PjZG<61;E!;nL zjK{7wM75X|uh|?(D+xUE3r$XXY#^{z%ln6i_kYg!;6B>15t!`Pnu@O&-<(! zA?hRPO-|$@G-*Kg5wpDNj0+L+napBtr<&E?UiDq%v;W!*7dQI%!7KKnwX^VDM>Zn@ z|FufV5Qb9$@$aL4Er+Z0KPfPKCxfO785j7`H(_ul8kEqd;`Kw{4{bdqx|;JpAlxDZ_dyR3HW&`X!_m}V5T4D+qQAbD@eD(*yl=q*Mp?KXq2)GJ5%v9DY%SvR)$;gDj=N?ML zN(%sG^gE?LGAO^4GdRcNHXEXeZuGL?V%JPtzthHIga{3>mft$ynx-tHB|O?%^%vELwIc zQnY{qEv0ZCF|o(^Xi$Xdi~=(P#0=ccW{0^?P^tPhFYD^&2AxvIPoFM5c5FEWV0oQ8 za@#tqUjXWvvxcv6Kt)B3KUT1L2+CMY4Da@f!S#nm%0&|1NRyoA%6EijhF{fJbj(=d z-YDBea?`Czv9;^zJ*W>Hi^~53&7q*p7vQcr)YyHvHkr!V{@I}HcB1q&dk+mF6(uEJ z6-37D2@?A^%3@_tmL;2V{QW^W0c9YHPSUiASx``a_b0$y`1%P7J}XrqY;yhv_())| z({!G+mNyvY?HPMigL0}SuFj(8BSjbZRXMX;n;$wz=1Xj{HE(^TS{*S&XtF0TNkfvA zRs?M=EsctvNl2)n{*~b{@XVH$mW%WAOrH|)I)g(EdW%Takc6QfeSPaq+pRiwKPmOW zsqLs3qb*yb!qNK=Cy=9oGh#+r!a8JcTAJfE*TT9we&FY&VCDN^0%u$j z5r9>+UV0dSa%1V&ug$-IL3eOFJFu0bFDxzHxOS0)t>Pa!71ed=V1B;!r~zs#*H$|N z0|UUR_dsd;{y|D^k%Faj{fN~$NT}BD)xQX5b34?PW1;8vsufYOet%RLu~AnWpw z!^g%(037mzj^*v8DzFwd=H{W=ugi~10=oi^^{poWFUC~3TMoulO2Yd-PbSWQ*}DO$ zRLj#eTglYV<@~&I&$0f!qpVW7Qz71G9&!o_@NnRQO9nIpJHO|%FYXJ6IeT04vAb5f zl{;Kzhe4dIo6*W-?M64l1r zd`=Lsez$KEz762b@rQr6cxg!~Tc4eL2tH5m;%@3``KweYt86^*YUwshz-^9ldh`Ul zlES?Y+3S(Y(Y+S@YTMtdenC`WMnXRdH(Xawg9%rPfnPIha&uGlB%9npLwLMXpRZA2 zD0a232mKfKR99TOPHlwlu@iZWsCCuHR( z^sdF*!eUv0&KE=wQYtDcO3HPp!&hly&?_SUCy&Z?R!QFE(XzYSU{IFCzb(E!_on0j zDdWn+q2Am6SkgF_#*!FIrjT8AkZdu893d&Y(s7tamPjgRLdRZ5DkMi^y_B3HStdEw zk}PGf>?yL#B;pXyXRh-+@AY2q>mPr(t{>)if4}$VzCTOUj~acHO`PIlAmwpC7rHQ^ zttrgDsM*6CPg+-DhEK{m*dDrD>foXNpwhkb?~`{EMetu-4jpHl?_&l8czf@|*wt_g zTSwe=0H@eXK?LCF=m^c=QG+_fL{JJen2n)_5Ty_sYo4m^hNY^b-glS4Ko%B-L<+E} zONm!PZ0u%XN`89IK-f#lQo={qQacCsW!b9V0*|G&L#Vxq{S1Ln0bv73RwnFxmB8D- zVB{e4@;Jy}Fg@ab!dA&t#$)V(u=~7cA?a}{Y!sJ1{j|&65(N=KVv|dteXKGK-HNG^ zFGlLQ6M+Eq(zKZa>sHn60o(P~b@shUx0a?z^jtCXRwO9?^=;@^B_$;`aK6j6g506* zMqAIr0bbW7xBeAd^=-aFB|+Vg1X)K%plkBBOFy&Ec&UHuyZ~Y917ye(6NqJ!sWYIG zI!c39RT}D353R9W*(dX7;?Fs|RI)fCGE8_o>wgUjlSW@JBMP9Dm|Z+`*ka%(907W8KtW30t-D5qh;cA_ZFR0x!>PCz z9E6FjuHA(HiM=w}(Jdy`i0RX@>&Qer$bf&gc6IFBn}r1}_=X#v8;MD(6?(0TqIh_x zZE|zB9`ECcJ(B*vqq>&DX9q7?9d+2gd4A`?(!BN#OC^o6U>(*k4j-=6J3gz8IH%#& zYeCZ%G5h4zlbz+&%X3@Sbo@-01=0W2V{Ymvu{(TGsSopldz7tA+WmjmedE;0D|$yQ zxjeaZY2~9~h0S&bzuH33?B|8IRpS{MVg0nQfuXZ)c~i}Mc_KNUJsQvW_1cUq-;1#z zx2n)%s$xmfqk&mHKKmd;@RfKFPR@s4b-%9eAB`VxNJo9a96jiNqQpy}*I2R5tK|FN z(?!4q3o~qj9bC=%IYWE3DEA^JVsF~UlcYWCmg&_nWr<60AT312YhqD>z@EYKUTSX5rl-tA$JFOQGLHz)|& z$OXhlIX^6a{P+vp8lJmy3Lr61*tLsUT1wOW1B%KFz{43+`@Oq&*PbfI5*(&0hV`zF zu(9T?{Li|577=fO6&M~Q^0KmaR%8?xZEn{jB;CYp?5Nh(*5c!NHxAHix%Q9~uqr)) zjb2dD7vQ*;*G!IqPWds-3JD5oYe)IpF`-L~i5+)vh$ZQ=z1U~Rk$+?nqS$;!N9y>W zJ=2R{R_N)kUU=?hEkRd$L{%*H)~%<5bKV&j??UYlME13^R)Vtf#8usWlMoe}JDJ4B z>BXCp@lWZ7nN|vV7r)4{w)5!U!9ob;8QXhZ|0;9dsCix&`QeO?U@X^gdH*bSHM0A* zlaWjw|X0r za(sQ|^!TEk9=?I~d>Y6kSQpPow^B&4+_a4;V~sk5f&hR^i~Z?=q$ zn&lK;xpD=S0EH7D1t3T6h6S+iHet?Plr39MYzm{(r8ZLPeYJ;I7bdwbQ=cf_ffc*B z7#o!ED6NZJloZROc6RfARqi;OpwyZYOpid-gO3xg(N&bN5EVYT=WVv{jXP+Xr>t{Z z#&SAnkEG}k5H-L_T4<|?U8r=_b!s+yA1=#9rN|AlDM}1Epr|TxaySF_!~7ZhjG`qf z?;#%JAznS$*GJyIJ(IyW>@PQ^tfA4?*2e7h6^u1ePsM8MIwP(C;OILB2^Uo{U55M7 zqdSwDvPiEp8$b3|o-Z2oHz&?QrQhWFVc+c$2X{5G7Wh;##&D`eomu5x;n}>o$EPg3 z1>~4@>&{JE%pV@dkpx$jHX|zD}W zK;FIR+f|Kf*$@*G6V~=LA*z9a0c@wLVt+n*w7a(AiS0jk%9E3P@M`26Q@9BPQg;@K zlRzN){O4!?@4iWVI5Of<;cBR@y%fMd7gRACw1PjnW{(abQ5d(9j0;;mW2I0D$>)0$ zZ1Y>jf|&h>OLe3K)2!}x&^WhC5rWviX|1HNP(97+Q&7cALC)IxNGh?GGo8VZmXurw zfBLv(?6LF1@c!P(V61PwJ-7y{FR`ziYl09Hid`KGGe`CJ`Ys(4Gkbh58JSnsw&XkK?Er$EWyu{6OIYaR?%~}xRhY*vk};# zA}*L2A3Z9P{tD--DE>9n;z+crfP;HL;_uZ})KhM5E~a>S;U<{P-K0_4%XGukmS0MG z#bex_oVl!%%wV_yjk&oYPw3T0))Ut`$e>v!Y<1}7?eqxJhlLei?ete(Hda>eV`0_@ z$bNRJ_?>SEu4HQVg_lKJ2*N7_0w3BEXrp4s2eyj1rfz3s$Vf;irC0)0-S6?irWf1* z2KTsi=<@^w1TeDcGUh;bL#6@kGbOMzD=VVOb80a(v+bVUeVV58xpRvH9FYG4i9}I7 zTwKzv@970kkgA0PZEUs-jbM@X123->)llp2!j4=!3+v(o%#R<=IV!RI)^zWD>uD*@kAv_`bdWyGNH%;Im87 z-%oKYGx4%ddYlx0;Z>q0##jC8WT$+Tt(TWVNA}aFWQykg{VPlj4J9R~iXi!7JuR&= ztW3aCBGDnK@nh9|J)e8g`8fjKAu$_xf~1CVM)(L3Pl~KlR99EHhHJ?%^Mt*9)aNUv zJID!MyT#)Kcifd1uoba5~LDr8XrCs zJ3Bkn6q!t3XHi1X6X}NGSgV-+1OtGJi)8h~9v^8zbIfAi-n%-CWl8KI=J_0U+z{U! z+DOHh9>ZFf5mvKru-s3wxC#>QHLlkTk%$H!G_0yJgCkj!;W570^3l7FW@_Ib*G+62ntPAUEh{1SX-s@{dlr{*} zlpEdf;#odGn=edleDNZ*Y6$KNsM1&Hp8ES+RW~JbMUx1^!ROAQ$nxycl9D2sQqXP{ z-mBe*?6lzElj0#rvRYfCXWQz0yN!?$BrMDauxe5=8ctE7AgDyA{=BaqeB$Rz7p<&X z|1uONByTD)k4AfFASLk8DCrM3fz-Pg_Zca|U13!cOd*t{4>d5u=u=H2QU0eXm zEu|8!URA*~1|@2K{%&%D^}PkWB`m2I7Z*)4kvs?Et~fu>pFPeHrXK$j7V&u#(EH}Z zOF+H|8^a-#N%ZEZ5g=HYE%{0lWJJVcs#q^1pR0%6bMy04>*yRBx<+E3qVq7JNRj^@ zJ}rXKj^#U@ELLD<8xx!}ThLk=l5k-b@cNdjQ|G{d{&}7RGZWWsR@T;Rd;9g_SrJ$Y zkRnvqHhTt(D*(skBR4F*_MI-8?Je0{6K9dzAt<9i!MYGvr=qNk<4)5gQ(vAceqUnN zc4xmyaIFiZ$_ufpoE)+(!kKN~h%M@b_u-JIETDs;AE&4|sCVD?XS&?$>X`frEzW?J z1`j0hxo@Z4)FJJ6rPK90pz%=_{UQC2KU%#O+S{)i$zsDRYhi>7U3S!g*o3P*mj`b> zOE}MNzn k|MZ&yL!jON+mbdo_IBr14@u`}0{jvV9kezrq@Mik-(z$PtN;K2 diff --git a/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/notebook.md b/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/notebook.md index 7624a2d..5cca734 100644 --- a/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/notebook.md +++ b/docs/src/notebooks/MLJTutorial/99_solution_to_exercises/notebook.md @@ -50,8 +50,8 @@ A = rand(2, 3) ```` 2×3 Matrix{Float64}: - 0.441502 0.871795 0.685263 - 0.535322 0.850152 0.76627 + 0.301055 0.691879 0.504642 + 0.572222 0.788816 0.738401 ```` ````@julia @@ -77,8 +77,8 @@ Asparse = sparse(A) ```` 2×3 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: - 0.441502 0.871795 0.685263 - 0.535322 0.850152 0.76627 + 0.301055 0.691879 0.504642 + 0.572222 0.788816 0.738401 ```` ````@julia @@ -95,8 +95,8 @@ C = coerce(A, Multiclass) ```` 2×3 CategoricalArrays.CategoricalArray{Float64,2,UInt32}: - 0.441502 0.871795 0.685263 - 0.535322 0.850152 0.76627 + 0.301055 0.691879 0.504642 + 0.572222 0.788816 0.738401 ```` ````@julia @@ -326,15 +326,15 @@ y4 = [n_devices(row.salary) for row in eachrow(X4)] ```` 10-element Vector{Int64}: - 4 - 3 2 - 4 - 6 - 1 3 2 + 3 + 3 + 0 2 + 1 + 1 2 ```` @@ -411,10 +411,10 @@ pretty(X) │ Float64 │ Float64 │ │ Continuous │ Continuous │ ├────────────┼────────────┤ -│ 0.256611 │ 0.672213 │ -│ 0.385614 │ 0.149492 │ -│ 0.873735 │ 0.269848 │ -│ 0.753635 │ 0.206412 │ +│ 0.11433 │ 0.538742 │ +│ 0.0704153 │ 0.19053 │ +│ 0.130344 │ 0.0186654 │ +│ 0.760926 │ 0.454443 │ └────────────┴────────────┘ ```` @@ -518,7 +518,7 @@ fitted_params(mach) ```` ```` -(classes = CategoricalArrays.CategoricalValue{Int64, UInt32}[CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 1), CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 2), CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 3)], coefs = Pair{Symbol, SubArray{Float64, 1, Matrix{Float64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}[:rectal_temperature => [0.01679940181352405, -0.006529534963799633, -0.010269866849724429], :pulse => [-0.002078922183617693, 0.00248508277505068, -0.00040616059143293954], :respiratory_rate => [-0.002078922183617693, 0.00248508277505068, -0.00040616059143293954], :packed_cell_volume => [0.0026345470622363147, 0.002707772252298626, -0.005342319314534946], :total_protein => [0.009110258500266177, -0.01679787903626504, 0.0076876205359988755]], intercept = [0.00043563961897106053, -0.00016706727792654562, -0.0005913867104289383]) +(classes = CategoricalArrays.CategoricalValue{Int64, UInt32}[CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 1), CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 2), CategoricalValue(CategoricalArrays.CategoricalPool{Int64, UInt32}([1, 2, 3]), 3)], coefs = Pair{Symbol, SubArray{Float64, 1, Matrix{Float64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}[:rectal_temperature => [0.016799401813523955, -0.0065295349637995905, -0.010269866849724361], :pulse => [-0.0020789221836177217, 0.0024850827750506855, -0.0004061605914329617], :respiratory_rate => [-0.0020789221836177217, 0.0024850827750506855, -0.0004061605914329617], :packed_cell_volume => [0.0026345470622364743, 0.002707772252298548, -0.005342319314535014], :total_protein => [0.009110258500266097, -0.016797879036265145, 0.0076876205359990195]], intercept = [0.00043563961897105766, -0.00016706727792654296, -0.0005913867104289052]) ```` ````@julia @@ -532,7 +532,7 @@ err = log_loss(yhat, y[test]) ```` ```` -0.8334775485441969 +0.8334775485441962 ```` 6(b)(iii) @@ -586,23 +586,24 @@ evaluate!(mach, resampling=CV(nfolds=6), measure=log_loss) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: RandomForestClassifier-372 +Tag: RandomForestClassifier-355 Extract: -┌──────────────────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────────────────┼───────────┼─────────────┤ -│ LogLoss( │ predict │ 1.1 │ -│ tol = 2.22045e-16) │ │ │ -└──────────────────────┴───────────┴─────────────┘ -┌─────────────────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├─────────────────────────────────────────┼─────────┤ -│ [0.762, 1.39, 1.79, 1.31, 0.697, 0.629] │ 0.411 │ -└─────────────────────────────────────────┴─────────┘ - +┌──────────────────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────────────────┼───────────┼─────────────┼─────────┤ +│ LogLoss( │ predict │ 1.27 │ 0.33 │ +│ tol = 2.22045e-16) │ │ │ │ +└──────────────────────┴───────────┴─────────────┴─────────┘ +┌──────────────────────────────────────┐ +│ per_fold │ +├──────────────────────────────────────┤ +│ [1.27, 1.4, 1.79, 1.27, 1.26, 0.616] │ +└──────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ````@julia @@ -660,7 +661,7 @@ err_forest = ```` ```` -0.987927309743752 +1.290842672700936 ```` #### Exercise 7 @@ -722,23 +723,24 @@ evaluate!(mach, resampling=CV(nfolds=6), measure=log_loss) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: ProbabilisticPipeline-616 +Tag: ProbabilisticPipeline-524 Extract: -┌──────────────────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────────────────┼───────────┼─────────────┤ -│ LogLoss( │ predict │ 0.847 │ -│ tol = 2.22045e-16) │ │ │ -└──────────────────────┴───────────┴─────────────┘ -┌───────────────────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├───────────────────────────────────────────┼─────────┤ -│ [0.888, 1.15, 0.847, 0.812, 0.782, 0.608] │ 0.154 │ -└───────────────────────────────────────────┴─────────┘ - +┌──────────────────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────────────────┼───────────┼─────────────┼─────────┤ +│ LogLoss( │ predict │ 0.85 │ 0.11 │ +│ tol = 2.22045e-16) │ │ │ │ +└──────────────────────┴───────────┴─────────────┴─────────┘ +┌──────────────────────────────────────────┐ +│ per_fold │ +├──────────────────────────────────────────┤ +│ [0.927, 1.02, 0.773, 0.85, 0.875, 0.665] │ +└──────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` 7(c) @@ -825,23 +827,24 @@ best_err = evaluate!(best_mach, resampling=CV(nfolds=3), measure=mae) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-942 +Tag: DeterministicPipeline-491 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 66800.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌─────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├─────────────────────────────┼─────────┤ -│ [66200.0, 66700.0, 67500.0] │ 941.0 │ -└─────────────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 66780.0 │ 940.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌─────────────────────────────┐ +│ per_fold │ +├─────────────────────────────┤ +│ [66200.0, 66700.0, 67500.0] │ +└─────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ````@julia @@ -851,23 +854,24 @@ tuned_err = evaluate!(tuned_mach, resampling=CV(nfolds=3), measure=mae) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicTunedModel-175 +Tag: DeterministicTunedModel-859 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 67600.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌─────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├─────────────────────────────┼─────────┤ -│ [67600.0, 67800.0, 67500.0] │ 214.0 │ -└─────────────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 67620.0 │ 210.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌─────────────────────────────┐ +│ per_fold │ +├─────────────────────────────┤ +│ [67600.0, 67800.0, 67500.0] │ +└─────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` --- diff --git a/docs/src/notebooks/UsingMLJ/01_basics/notebook.md b/docs/src/notebooks/UsingMLJ/01_basics/notebook.md index 74a57b8..afb15fd 100644 --- a/docs/src/notebooks/UsingMLJ/01_basics/notebook.md +++ b/docs/src/notebooks/UsingMLJ/01_basics/notebook.md @@ -21,11 +21,12 @@ Pkg.status() ```` Status `~/work/MLJTutorial.jl/MLJTutorial.jl/docs/src/notebooks/UsingMLJ/01_basics/Project.toml` - [336ed68f] CSV v0.10.17 +⌃ [336ed68f] CSV v0.10.17 [add582a8] MLJ v0.23.3 [c6f25543] MLJDecisionTreeInterface v0.5.0 - [b8865327] UnicodePlots v3.8.4 + [b8865327] UnicodePlots v3.9.0 [44cfe95a] Pkg v1.12.1 +Info Packages marked with ⌃ have new versions available and may be upgradable. ```` @@ -186,8 +187,8 @@ fitted_params(mach) ```` (forest = Ensemble of Decision Trees Trees: 100 -Avg Leaves: 147.07 -Avg Depth: 14.67,) +Avg Leaves: 147.25 +Avg Depth: 14.49,) ```` ````@julia @@ -206,9 +207,9 @@ predict(mach, X)[1:3] ```` 3-element Vector{Float64}: - 26.394000000000002 - 22.337 - 34.722 + 26.984 + 22.682999999999993 + 35.31 ```` Predict in the `test` rows: @@ -224,7 +225,7 @@ mae(ypred, y[test]) ```` ```` -4.623326732673267 +4.410905940594058 ```` `mae` is actually just an alias: @@ -267,19 +268,20 @@ evaluate!(mach; resampling=[(train, test),], measures=[mae, RSquared()]) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: RandomForestRegressor-413 +Tag: RandomForestRegressor-162 Extract: ┌────────────┬───────────┬─────────────┐ │ measure │ operation │ measurement │ ├────────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 4.62 │ +│ LPLoss( │ predict │ 4.41 │ │ p = 1) │ │ │ -│ RSquared() │ predict │ 0.323 │ +│ RSquared() │ predict │ 0.369 │ └────────────┴───────────┴─────────────┘ - +Apply `describe` to this result for a named tuple summary. ```` Something fancier: @@ -295,25 +297,26 @@ evaluate( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: RandomForestRegressor-666 +Tag: RandomForestRegressor-267 Extract: -┌───┬────────────┬───────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼────────────┼───────────┼─────────────┤ -│ A │ LPLoss( │ predict │ 3.0 │ -│ │ p = 1) │ │ │ -│ B │ RSquared() │ predict │ 0.656 │ -└───┴────────────┴───────────┴─────────────┘ -┌───┬──────────────────────────────────────────┬─────────┐ -│ │ per_fold │ 1.96*SE │ -├───┼──────────────────────────────────────────┼─────────┤ -│ A │ [2.24, 2.29, 3.32, 2.41, 4.73, 3.03] │ 0.834 │ -│ B │ [0.726, 0.838, 0.7, 0.819, 0.472, 0.376] │ 0.166 │ -└───┴──────────────────────────────────────────┴─────────┘ - +┌───┬────────────┬───────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼────────────┼───────────┼─────────────┼─────────┤ +│ A │ LPLoss( │ predict │ 2.99 │ 0.92 │ +│ │ p = 1) │ │ │ │ +│ B │ RSquared() │ predict │ 0.67 │ 0.17 │ +└───┴────────────┴───────────┴─────────────┴─────────┘ +┌───┬──────────────────────────────────────────┐ +│ │ per_fold │ +├───┼──────────────────────────────────────────┤ +│ A │ [2.14, 2.11, 3.32, 2.5, 4.9, 2.96] │ +│ B │ [0.74, 0.871, 0.73, 0.813, 0.454, 0.392] │ +└───┴──────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Dietterich's 5 x 2 test: @@ -331,26 +334,26 @@ e = evaluate( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: RandomForestRegressor-199 +Tag: RandomForestRegressor-760 Extract: -┌───┬────────────┬───────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼────────────┼───────────┼─────────────┤ -│ A │ LPLoss( │ predict │ 2.45 │ -│ │ p = 1) │ │ │ -│ B │ RSquared() │ predict │ 0.842 │ -└───┴────────────┴───────────┴─────────────┘ -┌───┬───────────────────────────────────────────────────────────────────────┬─── -│ │ per_fold │ ⋯ -├───┼───────────────────────────────────────────────────────────────────────┼─── -│ A │ [2.49, 2.49, 2.6, 2.47, 2.35, 2.52, 2.43, 2.35, 2.52, 2.24] │ ⋯ -│ B │ [0.817, 0.855, 0.835, 0.83, 0.853, 0.808, 0.863, 0.856, 0.835, 0.867] │ ⋯ -└───┴───────────────────────────────────────────────────────────────────────┴─── - 1 column omitted - +┌───┬────────────┬───────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼────────────┼───────────┼─────────────┼─────────┤ +│ A │ LPLoss( │ predict │ 2.49 │ 0.14 │ +│ │ p = 1) │ │ │ │ +│ B │ RSquared() │ predict │ 0.823 │ 0.033 │ +└───┴────────────┴───────────┴─────────────┴─────────┘ +┌───┬────────────────────────────────────────────────────────────────────────┐ +│ │ per_fold │ +├───┼────────────────────────────────────────────────────────────────────────┤ +│ A │ [2.64, 2.35, 2.17, 2.66, 2.86, 2.27, 2.67, 2.47, 2.54, 2.28] │ +│ B │ [0.752, 0.874, 0.897, 0.758, 0.777, 0.853, 0.835, 0.792, 0.824, 0.864] │ +└───┴────────────────────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` ````@julia @@ -359,8 +362,8 @@ e.uncertainty_radius_95 ```` 2-element Vector{Float64}: - 0.06904375445230268 - 0.013098268581533108 + 0.14383105341435692 + 0.032989951374955834 ```` # Interlude on scientific types @@ -605,8 +608,8 @@ first(yprob, 5) 5-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{2}, InlineStrings.String7, UInt32, Float64}: UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>1.0, >50K=>0.0) UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>1.0, >50K=>0.0) - UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>0.98, >50K=>0.02) - UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>0.24, >50K=>0.76) + UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>0.91, >50K=>0.09) + UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>0.33, >50K=>0.67) UnivariateFinite{ScientificTypesBase.Multiclass{2}}(<=50K=>1.0, >50K=>0.0) ```` @@ -617,8 +620,8 @@ yprob[3] ```` UnivariateFinite{ScientificTypesBase.Multiclass{2}} ┌ ┐ - <=50K ┤■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.98 - >50K ┤■ 0.02 + <=50K ┤■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.91 + >50K ┤■■■ 0.09 └ ┘ ```` @@ -629,7 +632,7 @@ accuracy(ypoint, y[test]) ```` ```` -0.7822081179300814 +0.7823616727235502 ```` ````@julia @@ -637,7 +640,7 @@ log_loss(yprob, y[test]) ```` ```` -1.5659651402227337 +1.5655836383744342 ```` Evaluate with one command: @@ -653,19 +656,20 @@ evaluate( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: RandomForestClassifier-400 +Tag: RandomForestClassifier-308 Extract: ┌──────────────────────┬──────────────┬─────────────┐ │ measure │ operation │ measurement │ ├──────────────────────┼──────────────┼─────────────┤ │ Accuracy() │ predict_mode │ 0.781 │ -│ LogLoss( │ predict │ 1.53 │ +│ LogLoss( │ predict │ 1.55 │ │ tol = 2.22045e-16) │ │ │ └──────────────────────┴──────────────┴─────────────┘ - +Apply `describe` to this result for a named tuple summary. ```` --- diff --git a/docs/src/notebooks/UsingMLJ/02_model_composition/notebook.md b/docs/src/notebooks/UsingMLJ/02_model_composition/notebook.md index 47968ab..54350fb 100644 --- a/docs/src/notebooks/UsingMLJ/02_model_composition/notebook.md +++ b/docs/src/notebooks/UsingMLJ/02_model_composition/notebook.md @@ -142,24 +142,25 @@ e1 = evaluate(pipe, X, y; resampling=CV(nfolds=4, rng=123), repeats=2, measure=m ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicPipeline-381 +Tag: DeterministicPipeline-639 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 180000.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 179980.0 │ 630.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ ┌─────────────────────────────────────────────────────────────────────────────── │ per_fold ⋯ ├─────────────────────────────────────────────────────────────────────────────── │ [180000.0, 180000.0, 181000.0, 179000.0, 179000.0, 180000.0, 180000.0, 18100 ⋯ └─────────────────────────────────────────────────────────────────────────────── - 2 columns omitted - + 1 column omitted +Apply `describe` to this result for a named tuple summary. ```` Notice the target very large on the current scale: @@ -206,24 +207,24 @@ e2 = evaluate(norm_pipe, X, y; resampling=CV(nfolds=4, rng=123), repeats=2, meas ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: TransformedTargetModelDeterministic-404 +Tag: TransformedTargetModelDeterministic-399 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 19800.0 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌──────────────────────────────────────────────────────────────────────────┬──── -│ per_fold │ 1 ⋯ -├──────────────────────────────────────────────────────────────────────────┼──── -│ [18100.0, 19800.0, 21400.0, 19600.0, 20300.0, 19500.0, 18700.0, 20900.0] │ 8 ⋯ -└──────────────────────────────────────────────────────────────────────────┴──── - 1 column omitted - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 19780.0 │ 800.0 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌──────────────────────────────────────────────────────────────────────────┐ +│ per_fold │ +├──────────────────────────────────────────────────────────────────────────┤ +│ [18100.0, 19800.0, 21400.0, 19600.0, 20300.0, 19500.0, 18700.0, 20900.0] │ +└──────────────────────────────────────────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Changing the regularization parameter `lambda` of ridge regressor, we can arrange that @@ -248,9 +249,9 @@ evaluations = evaluate( ```` 3-element Vector{PerformanceEvaluation{M, Vector{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasuresBase.FussyMeasure{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasuresBase.Multimeasure{StatisticalMeasuresBase.SupportsMissingsMeasure{StatisticalMeasures.LPLossOnScalars{Int64}}, Nothing, StatisticalMeasuresBase.Mean, typeof(identity)}}, Nothing}}}, Vector{Float64}, Vector{Float64}, Vector{typeof(predict)}, Vector{Vector{Float64}}, Vector{Vector{Vector{Float64}}}, FittedParamsPerFold, ReportPerFold, CV} where {M, FittedParamsPerFold, ReportPerFold}}: - PerformanceEvaluation("default lambda", 180000.0 ± 635.0) - PerformanceEvaluation("new lambda", 180000.0 ± 1470.0) - PerformanceEvaluation("new lambda & normalized target", 18500.0 ± 443.0) + PerformanceEvaluation("default lambda", 179980.0 ± 630.0) + PerformanceEvaluation("new lambda", 180100.0 ± 1500.0) + PerformanceEvaluation("new lambda & normalized target", 18460.0 ± 440.0) ```` Here's a pretty view of these results: diff --git a/docs/src/notebooks/UsingMLJ/03_model_tuning/learning_curve.svg b/docs/src/notebooks/UsingMLJ/03_model_tuning/learning_curve.svg index 55dc002..1687109 100644 --- a/docs/src/notebooks/UsingMLJ/03_model_tuning/learning_curve.svg +++ b/docs/src/notebooks/UsingMLJ/03_model_tuning/learning_curve.svg @@ -1,46 +1,46 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/src/notebooks/UsingMLJ/03_model_tuning/notebook.md b/docs/src/notebooks/UsingMLJ/03_model_tuning/notebook.md index 13d89b2..286e160 100644 --- a/docs/src/notebooks/UsingMLJ/03_model_tuning/notebook.md +++ b/docs/src/notebooks/UsingMLJ/03_model_tuning/notebook.md @@ -77,23 +77,24 @@ ebase = evaluate(baseline, X, y; options...) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: ProbabilisticPipeline-112 +Tag: ProbabilisticPipeline-908 Extract: -┌──────────┬──────────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼──────────────┼─────────────┤ -│ LPLoss( │ predict_mean │ 0.568 │ -│ p = 1) │ │ │ -└──────────┴──────────────┴─────────────┘ -┌──────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├──────────────────────┼─────────┤ -│ [0.544, 0.57, 0.592] │ 0.0332 │ -└──────────────────────┴─────────┘ - +┌──────────┬──────────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼──────────────┼─────────────┼─────────┤ +│ LPLoss( │ predict_mean │ 0.568 │ 0.033 │ +│ p = 1) │ │ │ │ +└──────────┴──────────────┴─────────────┴─────────┘ +┌──────────────────────┐ +│ per_fold │ +├──────────────────────┤ +│ [0.544, 0.57, 0.592] │ +└──────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Inspect hyper-parameters: @@ -147,7 +148,7 @@ savefig("learning_curve.svg"); ```` [ Info: Training machine(DeterministicTunedModel(model = DeterministicPipeline(continuous_encoder = ContinuousEncoder(drop_last = false, …), …), …), …). [ Info: Attempting to evaluate 30 models. - Evaluating over 30 metamodels: 7%[=> ] ETA: 0:00:01 Evaluating over 30 metamodels: 10%[==> ] ETA: 0:00:01 Evaluating over 30 metamodels: 13%[===> ] ETA: 0:00:01 Evaluating over 30 metamodels: 17%[====> ] ETA: 0:00:01 Evaluating over 30 metamodels: 20%[=====> ] ETA: 0:00:01 Evaluating over 30 metamodels: 23%[=====> ] ETA: 0:00:02 Evaluating over 30 metamodels: 27%[======> ] ETA: 0:00:02 Evaluating over 30 metamodels: 30%[=======> ] ETA: 0:00:02 Evaluating over 30 metamodels: 33%[========> ] ETA: 0:00:03 Evaluating over 30 metamodels: 37%[=========> ] ETA: 0:00:03 Evaluating over 30 metamodels: 40%[==========> ] ETA: 0:00:03 Evaluating over 30 metamodels: 43%[==========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 47%[===========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 50%[============> ] ETA: 0:00:04 Evaluating over 30 metamodels: 53%[=============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 57%[==============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 60%[===============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 63%[===============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 67%[================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 70%[=================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 73%[==================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 77%[===================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 80%[====================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 83%[====================> ] ETA: 0:00:04 Evaluating over 30 metamodels: 87%[=====================> ] ETA: 0:00:04 Evaluating over 30 metamodels: 90%[======================> ] ETA: 0:00:03 Evaluating over 30 metamodels: 93%[=======================> ] ETA: 0:00:02 Evaluating over 30 metamodels: 97%[========================>] ETA: 0:00:01 Evaluating over 30 metamodels: 100%[=========================] Time: 0:00:36 + Evaluating over 30 metamodels: 7%[=> ] ETA: 0:00:00 Evaluating over 30 metamodels: 10%[==> ] ETA: 0:00:01 Evaluating over 30 metamodels: 13%[===> ] ETA: 0:00:01 Evaluating over 30 metamodels: 17%[====> ] ETA: 0:00:02 Evaluating over 30 metamodels: 20%[=====> ] ETA: 0:00:02 Evaluating over 30 metamodels: 23%[=====> ] ETA: 0:00:04 Evaluating over 30 metamodels: 27%[======> ] ETA: 0:00:04 Evaluating over 30 metamodels: 30%[=======> ] ETA: 0:00:04 Evaluating over 30 metamodels: 33%[========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 37%[=========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 40%[==========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 43%[==========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 47%[===========> ] ETA: 0:00:04 Evaluating over 30 metamodels: 50%[============> ] ETA: 0:00:04 Evaluating over 30 metamodels: 53%[=============> ] ETA: 0:00:04 Evaluating over 30 metamodels: 57%[==============> ] ETA: 0:00:04 Evaluating over 30 metamodels: 60%[===============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 63%[===============> ] ETA: 0:00:05 Evaluating over 30 metamodels: 67%[================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 70%[=================> ] ETA: 0:00:05 Evaluating over 30 metamodels: 73%[==================> ] ETA: 0:00:04 Evaluating over 30 metamodels: 77%[===================> ] ETA: 0:00:04 Evaluating over 30 metamodels: 80%[====================> ] ETA: 0:00:04 Evaluating over 30 metamodels: 83%[====================> ] ETA: 0:00:03 Evaluating over 30 metamodels: 87%[=====================> ] ETA: 0:00:03 Evaluating over 30 metamodels: 90%[======================> ] ETA: 0:00:02 Evaluating over 30 metamodels: 93%[=======================> ] ETA: 0:00:02 Evaluating over 30 metamodels: 97%[========================>] ETA: 0:00:01 Evaluating over 30 metamodels: 100%[=========================] Time: 0:00:28 ```` @@ -224,7 +225,7 @@ savefig("tuned_model_grid.svg"); ```` [ Info: Training machine(DeterministicTunedModel(model = DeterministicPipeline(continuous_encoder = ContinuousEncoder(drop_last = false, …), …), …), …). [ Info: Attempting to evaluate 36 models. - Evaluating over 36 metamodels: 6%[=> ] ETA: 0:00:33 Evaluating over 36 metamodels: 8%[==> ] ETA: 0:00:45 Evaluating over 36 metamodels: 11%[==> ] ETA: 0:00:42 Evaluating over 36 metamodels: 14%[===> ] ETA: 0:00:37 Evaluating over 36 metamodels: 17%[====> ] ETA: 0:00:42 Evaluating over 36 metamodels: 19%[====> ] ETA: 0:00:39 Evaluating over 36 metamodels: 22%[=====> ] ETA: 0:00:36 Evaluating over 36 metamodels: 25%[======> ] ETA: 0:00:36 Evaluating over 36 metamodels: 28%[======> ] ETA: 0:00:34 Evaluating over 36 metamodels: 31%[=======> ] ETA: 0:00:33 Evaluating over 36 metamodels: 33%[========> ] ETA: 0:00:30 Evaluating over 36 metamodels: 36%[=========> ] ETA: 0:00:29 Evaluating over 36 metamodels: 39%[=========> ] ETA: 0:00:28 Evaluating over 36 metamodels: 42%[==========> ] ETA: 0:00:28 Evaluating over 36 metamodels: 44%[===========> ] ETA: 0:00:28 Evaluating over 36 metamodels: 47%[===========> ] ETA: 0:00:27 Evaluating over 36 metamodels: 50%[============> ] ETA: 0:00:26 Evaluating over 36 metamodels: 53%[=============> ] ETA: 0:00:25 Evaluating over 36 metamodels: 56%[=============> ] ETA: 0:00:24 Evaluating over 36 metamodels: 58%[==============> ] ETA: 0:00:22 Evaluating over 36 metamodels: 61%[===============> ] ETA: 0:00:21 Evaluating over 36 metamodels: 64%[===============> ] ETA: 0:00:19 Evaluating over 36 metamodels: 67%[================> ] ETA: 0:00:18 Evaluating over 36 metamodels: 69%[=================> ] ETA: 0:00:16 Evaluating over 36 metamodels: 72%[==================> ] ETA: 0:00:15 Evaluating over 36 metamodels: 75%[==================> ] ETA: 0:00:13 Evaluating over 36 metamodels: 78%[===================> ] ETA: 0:00:12 Evaluating over 36 metamodels: 81%[====================> ] ETA: 0:00:10 Evaluating over 36 metamodels: 83%[====================> ] ETA: 0:00:09 Evaluating over 36 metamodels: 86%[=====================> ] ETA: 0:00:08 Evaluating over 36 metamodels: 89%[======================> ] ETA: 0:00:06 Evaluating over 36 metamodels: 92%[======================> ] ETA: 0:00:04 Evaluating over 36 metamodels: 94%[=======================> ] ETA: 0:00:03 Evaluating over 36 metamodels: 97%[========================>] ETA: 0:00:01 Evaluating over 36 metamodels: 100%[=========================] Time: 0:00:53 + Evaluating over 36 metamodels: 6%[=> ] ETA: 0:00:43 Evaluating over 36 metamodels: 8%[==> ] ETA: 0:00:50 Evaluating over 36 metamodels: 11%[==> ] ETA: 0:00:43 Evaluating over 36 metamodels: 14%[===> ] ETA: 0:00:44 Evaluating over 36 metamodels: 17%[====> ] ETA: 0:00:42 Evaluating over 36 metamodels: 19%[====> ] ETA: 0:00:40 Evaluating over 36 metamodels: 22%[=====> ] ETA: 0:00:39 Evaluating over 36 metamodels: 25%[======> ] ETA: 0:00:35 Evaluating over 36 metamodels: 28%[======> ] ETA: 0:00:34 Evaluating over 36 metamodels: 31%[=======> ] ETA: 0:00:34 Evaluating over 36 metamodels: 33%[========> ] ETA: 0:00:33 Evaluating over 36 metamodels: 36%[=========> ] ETA: 0:00:30 Evaluating over 36 metamodels: 39%[=========> ] ETA: 0:00:30 Evaluating over 36 metamodels: 42%[==========> ] ETA: 0:00:29 Evaluating over 36 metamodels: 44%[===========> ] ETA: 0:00:28 Evaluating over 36 metamodels: 47%[===========> ] ETA: 0:00:26 Evaluating over 36 metamodels: 50%[============> ] ETA: 0:00:25 Evaluating over 36 metamodels: 53%[=============> ] ETA: 0:00:23 Evaluating over 36 metamodels: 56%[=============> ] ETA: 0:00:22 Evaluating over 36 metamodels: 58%[==============> ] ETA: 0:00:20 Evaluating over 36 metamodels: 61%[===============> ] ETA: 0:00:19 Evaluating over 36 metamodels: 64%[===============> ] ETA: 0:00:17 Evaluating over 36 metamodels: 67%[================> ] ETA: 0:00:16 Evaluating over 36 metamodels: 69%[=================> ] ETA: 0:00:15 Evaluating over 36 metamodels: 72%[==================> ] ETA: 0:00:13 Evaluating over 36 metamodels: 75%[==================> ] ETA: 0:00:12 Evaluating over 36 metamodels: 78%[===================> ] ETA: 0:00:11 Evaluating over 36 metamodels: 81%[====================> ] ETA: 0:00:09 Evaluating over 36 metamodels: 83%[====================> ] ETA: 0:00:08 Evaluating over 36 metamodels: 86%[=====================> ] ETA: 0:00:07 Evaluating over 36 metamodels: 89%[======================> ] ETA: 0:00:05 Evaluating over 36 metamodels: 92%[======================> ] ETA: 0:00:04 Evaluating over 36 metamodels: 94%[=======================> ] ETA: 0:00:03 Evaluating over 36 metamodels: 97%[========================>] ETA: 0:00:01 Evaluating over 36 metamodels: 100%[=========================] Time: 0:00:48 ```` @@ -271,18 +272,18 @@ CompactPerformanceEvaluation object with these fields: resampling, repeats Tag: Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 0.171 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌─────────────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├─────────────────────────────┼─────────┤ -│ [0.174, 0.16, 0.156, 0.192] │ 0.0181 │ -└─────────────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 0.171 │ 0.018 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌─────────────────────────────┐ +│ per_fold │ +├─────────────────────────────┤ +│ [0.174, 0.16, 0.156, 0.192] │ +└─────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Instead we can use a random search strategy: @@ -306,7 +307,7 @@ savefig("tuned_model_random.svg"); ```` [ Info: Training machine(DeterministicTunedModel(model = DeterministicPipeline(continuous_encoder = ContinuousEncoder(drop_last = false, …), …), …), …). [ Info: Attempting to evaluate 40 models. - Evaluating over 40 metamodels: 0%[> ] ETA: N/A Evaluating over 40 metamodels: 2%[> ] ETA: 0:00:32 Evaluating over 40 metamodels: 5%[=> ] ETA: 0:00:47 Evaluating over 40 metamodels: 8%[=> ] ETA: 0:00:44 Evaluating over 40 metamodels: 10%[==> ] ETA: 0:00:46 Evaluating over 40 metamodels: 12%[===> ] ETA: 0:00:53 Evaluating over 40 metamodels: 15%[===> ] ETA: 0:00:49 Evaluating over 40 metamodels: 18%[====> ] ETA: 0:00:47 Evaluating over 40 metamodels: 20%[=====> ] ETA: 0:00:46 Evaluating over 40 metamodels: 22%[=====> ] ETA: 0:00:47 Evaluating over 40 metamodels: 25%[======> ] ETA: 0:00:46 Evaluating over 40 metamodels: 28%[======> ] ETA: 0:00:43 Evaluating over 40 metamodels: 30%[=======> ] ETA: 0:00:42 Evaluating over 40 metamodels: 32%[========> ] ETA: 0:00:40 Evaluating over 40 metamodels: 35%[========> ] ETA: 0:00:40 Evaluating over 40 metamodels: 38%[=========> ] ETA: 0:00:39 Evaluating over 40 metamodels: 40%[==========> ] ETA: 0:00:37 Evaluating over 40 metamodels: 42%[==========> ] ETA: 0:00:36 Evaluating over 40 metamodels: 45%[===========> ] ETA: 0:00:34 Evaluating over 40 metamodels: 48%[===========> ] ETA: 0:00:33 Evaluating over 40 metamodels: 50%[============> ] ETA: 0:00:32 Evaluating over 40 metamodels: 52%[=============> ] ETA: 0:00:30 Evaluating over 40 metamodels: 55%[=============> ] ETA: 0:00:28 Evaluating over 40 metamodels: 58%[==============> ] ETA: 0:00:27 Evaluating over 40 metamodels: 60%[===============> ] ETA: 0:00:25 Evaluating over 40 metamodels: 62%[===============> ] ETA: 0:00:24 Evaluating over 40 metamodels: 65%[================> ] ETA: 0:00:22 Evaluating over 40 metamodels: 68%[================> ] ETA: 0:00:21 Evaluating over 40 metamodels: 70%[=================> ] ETA: 0:00:19 Evaluating over 40 metamodels: 72%[==================> ] ETA: 0:00:17 Evaluating over 40 metamodels: 75%[==================> ] ETA: 0:00:15 Evaluating over 40 metamodels: 78%[===================> ] ETA: 0:00:14 Evaluating over 40 metamodels: 80%[====================> ] ETA: 0:00:12 Evaluating over 40 metamodels: 82%[====================> ] ETA: 0:00:11 Evaluating over 40 metamodels: 85%[=====================> ] ETA: 0:00:09 Evaluating over 40 metamodels: 88%[=====================> ] ETA: 0:00:08 Evaluating over 40 metamodels: 90%[======================> ] ETA: 0:00:06 Evaluating over 40 metamodels: 92%[=======================> ] ETA: 0:00:05 Evaluating over 40 metamodels: 95%[=======================> ] ETA: 0:00:03 Evaluating over 40 metamodels: 98%[========================>] ETA: 0:00:02 Evaluating over 40 metamodels: 100%[=========================] Time: 0:01:01 + Evaluating over 40 metamodels: 5%[=> ] ETA: 0:00:56 Evaluating over 40 metamodels: 8%[=> ] ETA: 0:00:48 Evaluating over 40 metamodels: 10%[==> ] ETA: 0:00:48 Evaluating over 40 metamodels: 12%[===> ] ETA: 0:00:48 Evaluating over 40 metamodels: 15%[===> ] ETA: 0:00:44 Evaluating over 40 metamodels: 18%[====> ] ETA: 0:00:46 Evaluating over 40 metamodels: 20%[=====> ] ETA: 0:00:44 Evaluating over 40 metamodels: 22%[=====> ] ETA: 0:00:42 Evaluating over 40 metamodels: 25%[======> ] ETA: 0:00:41 Evaluating over 40 metamodels: 28%[======> ] ETA: 0:00:38 Evaluating over 40 metamodels: 30%[=======> ] ETA: 0:00:39 Evaluating over 40 metamodels: 32%[========> ] ETA: 0:00:37 Evaluating over 40 metamodels: 35%[========> ] ETA: 0:00:36 Evaluating over 40 metamodels: 38%[=========> ] ETA: 0:00:35 Evaluating over 40 metamodels: 40%[==========> ] ETA: 0:00:33 Evaluating over 40 metamodels: 42%[==========> ] ETA: 0:00:33 Evaluating over 40 metamodels: 45%[===========> ] ETA: 0:00:31 Evaluating over 40 metamodels: 48%[===========> ] ETA: 0:00:29 Evaluating over 40 metamodels: 50%[============> ] ETA: 0:00:28 Evaluating over 40 metamodels: 52%[=============> ] ETA: 0:00:27 Evaluating over 40 metamodels: 55%[=============> ] ETA: 0:00:26 Evaluating over 40 metamodels: 58%[==============> ] ETA: 0:00:24 Evaluating over 40 metamodels: 60%[===============> ] ETA: 0:00:23 Evaluating over 40 metamodels: 62%[===============> ] ETA: 0:00:22 Evaluating over 40 metamodels: 65%[================> ] ETA: 0:00:20 Evaluating over 40 metamodels: 68%[================> ] ETA: 0:00:19 Evaluating over 40 metamodels: 70%[=================> ] ETA: 0:00:17 Evaluating over 40 metamodels: 72%[==================> ] ETA: 0:00:15 Evaluating over 40 metamodels: 75%[==================> ] ETA: 0:00:14 Evaluating over 40 metamodels: 78%[===================> ] ETA: 0:00:12 Evaluating over 40 metamodels: 80%[====================> ] ETA: 0:00:11 Evaluating over 40 metamodels: 82%[====================> ] ETA: 0:00:10 Evaluating over 40 metamodels: 85%[=====================> ] ETA: 0:00:08 Evaluating over 40 metamodels: 88%[=====================> ] ETA: 0:00:07 Evaluating over 40 metamodels: 90%[======================> ] ETA: 0:00:06 Evaluating over 40 metamodels: 92%[=======================> ] ETA: 0:00:04 Evaluating over 40 metamodels: 95%[=======================> ] ETA: 0:00:03 Evaluating over 40 metamodels: 98%[========================>] ETA: 0:00:01 Evaluating over 40 metamodels: 100%[=========================] Time: 0:00:55 ```` @@ -336,23 +337,24 @@ e1 = evaluate(tuned_pipe, X, y; options...) ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicTunedModel-250 +Tag: DeterministicTunedModel-569 Extract: -┌──────────┬───────────┬─────────────┐ -│ measure │ operation │ measurement │ -├──────────┼───────────┼─────────────┤ -│ LPLoss( │ predict │ 0.175 │ -│ p = 1) │ │ │ -└──────────┴───────────┴─────────────┘ -┌───────────────────────┬─────────┐ -│ per_fold │ 1.96*SE │ -├───────────────────────┼─────────┤ -│ [0.166, 0.166, 0.192] │ 0.021 │ -└───────────────────────┴─────────┘ - +┌──────────┬───────────┬─────────────┬─────────┐ +│ measure │ operation │ measurement │ 1.96*SE │ +├──────────┼───────────┼─────────────┼─────────┤ +│ LPLoss( │ predict │ 0.175 │ 0.021 │ +│ p = 1) │ │ │ │ +└──────────┴───────────┴─────────────┴─────────┘ +┌───────────────────────┐ +│ per_fold │ +├───────────────────────┤ +│ [0.166, 0.166, 0.192] │ +└───────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Comparing with the baseline computed earlier: @@ -362,9 +364,9 @@ Comparing with the baseline computed earlier: ```` ```` -ebase = PerformanceEvaluation("ProbabilisticPipeline-112", 0.568 ± 0.0332) -e0 = PerformanceEvaluation("DeterministicPipeline-658", 0.184 ± 0.0281) -e1 = PerformanceEvaluation("DeterministicTunedModel-250", 0.175 ± 0.021) +ebase = PerformanceEvaluation("ProbabilisticPipeline-908", 0.568 ± 0.033) +e0 = PerformanceEvaluation("DeterministicPipeline-823", 0.184 ± 0.028) +e1 = PerformanceEvaluation("DeterministicTunedModel-569", 0.175 ± 0.021) ```` @@ -380,8 +382,8 @@ describe.([e0, e1]) |> pretty │ String │ Measurement{Float64} │ │ Textual │ Continuous │ ├─────────────────────────────┼──────────────────────┤ -│ DeterministicPipeline-658 │ 0.184±0.028 │ -│ DeterministicTunedModel-250 │ 0.175±0.021 │ +│ DeterministicPipeline-823 │ 0.184±0.028 │ +│ DeterministicTunedModel-569 │ 0.175±0.021 │ └─────────────────────────────┴──────────────────────┘ ```` diff --git a/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_grid.svg b/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_grid.svg index c80b3f8..94ad1ac 100644 --- a/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_grid.svg +++ b/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_grid.svg @@ -1,294 +1,294 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_random.svg b/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_random.svg index 9911fd3..33a2310 100644 --- a/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_random.svg +++ b/docs/src/notebooks/UsingMLJ/03_model_tuning/tuned_model_random.svg @@ -1,269 +1,269 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/src/notebooks/lightning_tour/notebook.md b/docs/src/notebooks/lightning_tour/notebook.md index ba6449b..c912761 100644 --- a/docs/src/notebooks/lightning_tour/notebook.md +++ b/docs/src/notebooks/lightning_tour/notebook.md @@ -246,8 +246,8 @@ mach = machine(self_tuning_pipe, X, y) untrained Machine; does not cache data model: DeterministicTunedModel(model = DeterministicPipeline(continuous_encoder = ContinuousEncoder(drop_last = false, …), …), …) args: - 1: Source @477 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}} - 2: Source @252 ⏎ AbstractVector{ScientificTypesBase.Continuous} + 1: Source @634 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}} + 2: Source @471 ⏎ AbstractVector{ScientificTypesBase.Continuous} ```` @@ -261,9 +261,9 @@ first(yhat, 3) ```` 3-element Vector{Float32}: - -0.42176783 - 0.82785225 - 0.09056717 + -2.8776388 + -0.45805833 + -0.8760577 ```` Evaluating the "self-tuning" pipeline model's performance using all data and 5-fold @@ -281,25 +281,26 @@ evaluate!( ```` PerformanceEvaluation object with these fields: model, tag, measure, operation, - measurement, uncertainty_radius_95, per_fold, per_observation, + measurement (per-fold aggregate), uncertainty_radius_95 (1.96*SE), + per_fold, per_observation, fitted_params_per_fold, report_per_fold, train_test_rows, resampling, repeats -Tag: DeterministicTunedModel-354 +Tag: DeterministicTunedModel-157 Extract: -┌───┬────────────┬───────────┬─────────────┐ -│ │ measure │ operation │ measurement │ -├───┼────────────┼───────────┼─────────────┤ -│ A │ LPLoss( │ predict │ 0.309 │ -│ │ p = 1) │ │ │ -│ B │ RSquared() │ predict │ 0.946 │ -└───┴────────────┴───────────┴─────────────┘ -┌───┬─────────────────────────────────────┬─────────┐ -│ │ per_fold │ 1.96*SE │ -├───┼─────────────────────────────────────┼─────────┤ -│ A │ [0.211, 0.279, 0.414, 0.334, 0.305] │ 0.0729 │ -│ B │ [0.967, 0.957, 0.902, 0.944, 0.962] │ 0.0258 │ -└───┴─────────────────────────────────────┴─────────┘ - +┌───┬────────────┬───────────┬─────────────┬─────────┐ +│ │ measure │ operation │ measurement │ 1.96*SE │ +├───┼────────────┼───────────┼─────────────┼─────────┤ +│ A │ LPLoss( │ predict │ 0.17 │ 0.023 │ +│ │ p = 1) │ │ │ │ +│ B │ RSquared() │ predict │ 0.9699 │ 0.009 │ +└───┴────────────┴───────────┴─────────────┴─────────┘ +┌───┬────────────────────────────────────┐ +│ │ per_fold │ +├───┼────────────────────────────────────┤ +│ A │ [0.155, 0.159, 0.2, 0.19, 0.146] │ +│ B │ [0.96, 0.981, 0.962, 0.969, 0.977] │ +└───┴────────────────────────────────────┘ +Apply `describe` to this result for a named tuple summary. ```` Compare to a dummy model: @@ -319,7 +320,7 @@ describe.(evaluations) |> pretty ```` [ Info: Performing evaluations using 1 thread. - Evaluating over 5 folds: 40%[==========> ] ETA: 0:00:27 Evaluating over 5 folds: 60%[===============> ] ETA: 0:00:15 Evaluating over 5 folds: 80%[====================> ] ETA: 0:00:07 Evaluating over 5 folds: 100%[=========================] Time: 0:00:33 + Evaluating over 5 folds: 40%[==========> ] ETA: 0:00:11 Evaluating over 5 folds: 60%[===============> ] ETA: 0:00:07 Evaluating over 5 folds: 80%[====================> ] ETA: 0:00:04 Evaluating over 5 folds: 100%[=========================] Time: 0:00:18 [ Info: Performing evaluations using 1 thread. Evaluating over 5 folds: 40%[==========> ] ETA: 0:00:01 Evaluating over 5 folds: 100%[=========================] Time: 0:00:00 ┌─────────┬──────────────────────┬──────────────────────┐ @@ -327,8 +328,8 @@ describe.(evaluations) |> pretty │ String │ Measurement{Float64} │ Measurement{Float64} │ │ Textual │ Continuous │ Continuous │ ├─────────┼──────────────────────┼──────────────────────┤ -│ booster │ 0.309±0.073 │ 0.946±0.026 │ -│ dummy │ 1.48±0.15 │ -0.014±0.021 │ +│ booster │ 0.17±0.023 │ 0.9699±0.009 │ +│ dummy │ 1.13±0.15 │ -0.0044±0.0051 │ └─────────┴──────────────────────┴──────────────────────┘ ```` diff --git a/learning_curve.svg b/learning_curve.svg index dd9bd14..42b059d 100644 --- a/learning_curve.svg +++ b/learning_curve.svg @@ -1,46 +1,46 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tuned_model_grid.svg b/tuned_model_grid.svg index 358a08e..e3ed916 100644 --- a/tuned_model_grid.svg +++ b/tuned_model_grid.svg @@ -1,294 +1,294 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tuned_model_random.svg b/tuned_model_random.svg index f06d6fe..7f4925e 100644 --- a/tuned_model_random.svg +++ b/tuned_model_random.svg @@ -1,269 +1,269 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +