From 2a145e0c4326b3fae2796af82f0658ddb6985008 Mon Sep 17 00:00:00 2001 From: Alex Murphy Date: Sat, 17 May 2025 10:46:48 +0100 Subject: [PATCH 1/3] Add Tutorial 4 changes and denote as bonus --- .../W1D3_Tutorial4.ipynb | 81 ++++++++++++++++++- 1 file changed, 77 insertions(+), 4 deletions(-) diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb index f2a33e3b5..4952feb92 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb @@ -17,7 +17,7 @@ "execution": {} }, "source": [ - "# Tutorial 4: Representational geometry & noise\n", + "# (Bonus) Tutorial 4: Representational geometry & noise\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", @@ -25,9 +25,9 @@ "\n", "__Content creators:__ Wenxuan Guo, Heiko Schütt\n", "\n", - "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk\n", + "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n", + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n", "\n", "Acknowledgments: the tutorial outline was written by Heiko Schütt. The content was greatly improved by discussions with Heiko, Hlib, and Alish, and the insightful illustrations presented in the paper by Walther et al. (2016)\n" ] @@ -61,7 +61,7 @@ "\n", "5. Using random projections to estimate distances. This section introduces the Johnson–Lindenstrauss Lemma, which states that random projections maintain the integrity of distance estimates in a lower-dimensional space. This concept is crucial for reducing dimensionality while preserving the relational structure of the data.\n", "\n", - "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures. " + "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures." ] }, { @@ -644,6 +644,67 @@ "display(tabs)" ] }, + { + "cell_type": "markdown", + "id": "b64eaea5", + "metadata": {}, + "source": [ + "The video below is additional information in more detail which was previously part of the introductory video for this course day. It provides some useful further information on the technical details mentioned during these tutorials. Please feel free to check it out and use it as a resource if you want to learn more or if you want to get a deeper understanding on some of the important details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "200235dc", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Video 2 (BONUS): Extended Intro Video\n", + "\n", + "from ipywidgets import widgets\n", + "from IPython.display import YouTubeVideo\n", + "from IPython.display import IFrame\n", + "from IPython.display import display\n", + "\n", + "class PlayVideo(IFrame):\n", + " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", + " self.id = id\n", + " if source == 'Bilibili':\n", + " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", + " elif source == 'Osf':\n", + " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", + " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "\n", + "def display_videos(video_ids, W=400, H=300, fs=1):\n", + " tab_contents = []\n", + " for i, video_id in enumerate(video_ids):\n", + " out = widgets.Output()\n", + " with out:\n", + " if video_ids[i][0] == 'Youtube':\n", + " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", + " height=H, fs=fs, rel=0)\n", + " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", + " else:\n", + " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", + " height=H, fs=fs, autoplay=False)\n", + " if video_ids[i][0] == 'Bilibili':\n", + " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", + " elif video_ids[i][0] == 'Osf':\n", + " print(f'Video available at https://osf.io/{video.id}')\n", + " display(video)\n", + " tab_contents.append(out)\n", + " return tab_contents\n", + "\n", + "#assert 1 == 0, \"Upload this video\"\n", + "video_ids = [('Youtube', 'm9srqTx5ci0'), ('Bilibili', 'BV1meVjz3Eeh')]\n", + "tab_contents = display_videos(video_ids, W=854, H=480)\n", + "tabs = widgets.Tab()\n", + "tabs.children = tab_contents\n", + "for i in range(len(tab_contents)):\n", + " tabs.set_title(i, video_ids[i][0])\n", + "display(tabs)" + ] + }, { "cell_type": "code", "execution_count": null, @@ -1503,6 +1564,18 @@ "3. Cross-validated distance estimators (cross-validated Euclidean or Mahalanobis distance) can remove the positive bias introduced by noise.\n", "4. The Johnson–Lindenstrauss Lemma shows that random projections preserve the Euclidean distance with some distortions. Crucially, the distortion does not depend on the dimensionality of the original space." ] + }, + { + "cell_type": "markdown", + "id": "40936ec4", + "metadata": {}, + "source": [ + "# The Big Picture\n", + "\n", + "The goal of this tutorial is to provide you with some mathematical tools for your NeuroAI researcher toolkit. What happens when you pull out the Euclidean metric from your toolkit and, while this has worked well in the past, suddenly in different scenarios it doesn't seem to perform so well. Aha, you spot the potential for correlated noise and you reach deeper into your toolkit and pull out the Mahalanobis metric, which implicitly undoes the correlated noise in the model. Perhaps you can't even tell if there is any correlated noise in your data and you try with both metrics, and Mahalanobis works well but Euclidean does not, that can be a hunch that leads you to confirm the presence of correlated noise. \n", + "\n", + "Sometimes you might be faced with dimensionalities that are just too high to practically deal with in your use case. Then, why not recall what you learned about how random projections can reduce the dimensionality of a feature space and be largely resistant to corrupting the applicability of distance metrics. These metrics also might work better in this lower dimensional space. If you apply this idea and need to justify it, just reach into your NeuroAI toolkit and pull out the Johnson-Lindenstrauss Lemma as your justification." + ] } ], "metadata": { From 5faf4bf9dc3f9976b92d567a1d6e075e165b56e8 Mon Sep 17 00:00:00 2001 From: Alex Murphy Date: Sat, 17 May 2025 11:06:55 +0100 Subject: [PATCH 2/3] Add (new) Tutorial 5 --- .../W1D3_Tutorial5.ipynb | 2483 ++++++++++++++++- 1 file changed, 2476 insertions(+), 7 deletions(-) diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb index 77480d80e..97fd20d3f 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb @@ -17,17 +17,17 @@ "execution": {} }, "source": [ - "# Bonus Material: Dynamical similarity analysis (DSA)\n", + "# Tutorial 5: Dynamical Similarity Analysis (DSA)\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", "**By Neuromatch Academy**\n", "\n", - "__Content creators:__ Mitchell Ostrow\n", + "__Content creators:__ Mitchell Ostrow, Alex Murphy\n", "\n", - "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk\n", + "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n" + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n" ] }, { @@ -52,7 +52,7 @@ "source": [ "# @title Install and import feedback gadget\n", "\n", - "!pip install vibecheck --quiet\n", + "!pip install vibecheck rsatoolbox --quiet\n", "\n", "from vibecheck import DatatopsContentReviewContainer\n", "def content_review(notebook_section: str):\n", @@ -67,7 +67,2069 @@ " ).render()\n", "\n", "\n", - "feedback_prefix = \"W1D3_Bonus\"" + "feedback_prefix = \"W1D5_DSA\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ef9abaa3", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Helper functions\n", + "\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "\n", + "def generate_2d_random_process(A, B, T=1000):\n", + " \"\"\"\n", + " Generates a 2D random process with the equation x(t+1) = A.x(t) + B.noise.\n", + "\n", + " Args:\n", + " A: 2x2 transition matrix.\n", + " B: 2x2 noise scaling matrix.\n", + " T: Number of time steps.\n", + "\n", + " Returns:\n", + " A NumPy array of shape (T+1, 2) representing the trajectory.\n", + " \"\"\"\n", + " # Assuming equilibrium distribution is zero mean and identity covariance for simplicity.\n", + " # You may adjust this according to your actual equilibrium distribution\n", + " x = np.zeros(2)\n", + "\n", + " trajectory = [x.copy()] # Initialize with x(0)\n", + " for t in range(T):\n", + " noise = np.random.normal(size=2) # Standard normal noise\n", + " x = np.dot(A, x) + np.dot(B, noise)\n", + " trajectory.append(x.copy())\n", + " return np.array(trajectory)\n", + "\n", + "\"\"\"This module computes the Havok DMD model for a given dataset.\"\"\"\n", + "import torch\n", + "\n", + "def embed_signal_torch(data, n_delays, delay_interval=1):\n", + " \"\"\"\n", + " Create a delay embedding from the provided tensor data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : torch.tensor\n", + " The data from which to create the delay embedding. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + " \"\"\"\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " device = data.device\n", + "\n", + " if data.shape[int(data.ndim==3)] - (n_delays - 1)*delay_interval < 1:\n", + " raise ValueError(\"The number of delays is too large for the number of time points in the data!\")\n", + "\n", + " # initialize the embedding\n", + " if data.ndim == 3:\n", + " embedding = torch.zeros((data.shape[0], data.shape[1] - (n_delays - 1)*delay_interval, data.shape[2]*n_delays)).to(device)\n", + " else:\n", + " embedding = torch.zeros((data.shape[0] - (n_delays - 1)*delay_interval, data.shape[1]*n_delays)).to(device)\n", + "\n", + " for d in range(n_delays):\n", + " index = (n_delays - 1 - d)*delay_interval\n", + " ddelay = d*delay_interval\n", + "\n", + " if data.ndim == 3:\n", + " ddata = d*data.shape[2]\n", + " embedding[:,:, ddata: ddata + data.shape[2]] = data[:,index:data.shape[1] - ddelay]\n", + " else:\n", + " ddata = d*data.shape[1]\n", + " embedding[:, ddata:ddata + data.shape[1]] = data[index:data.shape[0] - ddelay]\n", + "\n", + " return embedding\n", + "\n", + "class DMD:\n", + " \"\"\"DMD class for computing and predicting with DMD models.\n", + " \"\"\"\n", + " def __init__(\n", + " self,\n", + " data,\n", + " n_delays,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=False,\n", + " lamb=0,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=False,\n", + " steps_ahead=1\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " reduced_rank_reg : bool\n", + " Determines whether to use reduced rank regression (True) or principal component regression (False)\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0.\n", + "\n", + " device: string, int, or torch.device\n", + " A string, int or torch.device object to indicate the device to torch.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + " \"\"\"\n", + "\n", + " self.device = device\n", + " self._init_data(data)\n", + "\n", + " self.n_delays = n_delays\n", + " self.delay_interval = delay_interval\n", + " self.rank = rank\n", + " self.rank_thresh = rank_thresh\n", + " self.rank_explained_variance = rank_explained_variance\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.lamb = lamb\n", + " self.verbose = verbose\n", + " self.send_to_cpu = send_to_cpu\n", + " self.steps_ahead = steps_ahead\n", + "\n", + " # Hankel matrix\n", + " self.H = None\n", + "\n", + " # SVD attributes\n", + " self.U = None\n", + " self.S = None\n", + " self.V = None\n", + " self.S_mat = None\n", + " self.S_mat_inv = None\n", + "\n", + " # DMD attributes\n", + " self.A_v = None\n", + " self.A_havok_dmd = None\n", + "\n", + " def _init_data(self, data):\n", + " # check if the data is an np.ndarry - if so, convert it to Torch\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " self.data = data\n", + " # create attributes for the data dimensions\n", + " if self.data.ndim == 3:\n", + " self.ntrials = self.data.shape[0]\n", + " self.window = self.data.shape[1]\n", + " self.n = self.data.shape[2]\n", + " else:\n", + " self.window = self.data.shape[0]\n", + " self.n = self.data.shape[1]\n", + " self.ntrials = 1\n", + "\n", + " def compute_hankel(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " ):\n", + " \"\"\"\n", + " Computes the Hankel matrix from the provided data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing Hankel matrix ...\")\n", + "\n", + " # if parameters are provided, overwrite them from the init\n", + " self.data = self.data if data is None else self._init_data(data)\n", + " self.n_delays = self.n_delays if n_delays is None else n_delays\n", + " self.delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " self.data = self.data.to(self.device)\n", + "\n", + " self.H = embed_signal_torch(self.data, self.n_delays, self.delay_interval)\n", + "\n", + " if self.verbose:\n", + " print(\"Hankel matrix computed!\")\n", + "\n", + " def compute_svd(self):\n", + " \"\"\"\n", + " Computes the SVD of the Hankel matrix.\n", + " \"\"\"\n", + "\n", + " if self.verbose:\n", + " print(\"Computing SVD on Hankel matrix ...\")\n", + " if self.H.ndim == 3: #flatten across trials for 3d\n", + " H = self.H.reshape(self.H.shape[0] * self.H.shape[1], self.H.shape[2])\n", + " else:\n", + " H = self.H\n", + " # compute the SVD\n", + " U, S, Vh = torch.linalg.svd(H.T, full_matrices=False)\n", + "\n", + " # update attributes\n", + " V = Vh.T\n", + " self.U = U\n", + " self.S = S\n", + " self.V = V\n", + "\n", + " # construct the singuar value matrix and its inverse\n", + " # dim = self.n_delays * self.n\n", + " # s = len(S)\n", + " # self.S_mat = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " # self.S_mat_inv = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " self.S_mat = torch.diag(S).to(self.device)\n", + " self.S_mat_inv= torch.diag(1 / S).to(self.device)\n", + "\n", + " # compute explained variance\n", + " exp_variance_inds = self.S**2 / ((self.S**2).sum())\n", + " cumulative_explained = torch.cumsum(exp_variance_inds, 0)\n", + " self.cumulative_explained_variance = cumulative_explained\n", + "\n", + " #make the X and Y components of the regression by staggering the hankel eigen-time delay coordinates by time\n", + " if self.reduced_rank_reg:\n", + " V = self.V\n", + " else:\n", + " V = self.V\n", + "\n", + " if self.ntrials > 1:\n", + " if V.numel() < self.H.numel():\n", + " raise ValueError(\"The dimension of the SVD of the Hankel matrix is smaller than the dimension of the Hankel matrix itself. \\n \\\n", + " This is likely due to the number of time points being smaller than the number of dimensions. \\n \\\n", + " Please reduce the number of delays.\")\n", + "\n", + " V = V.reshape(self.H.shape)\n", + "\n", + " #first reshape back into Hankel shape, separated by trials\n", + " newshape = (self.H.shape[0]*(self.H.shape[1]-self.steps_ahead),self.H.shape[2])\n", + " self.Vt_minus = V[:,:-self.steps_ahead].reshape(newshape)\n", + " self.Vt_plus = V[:,self.steps_ahead:].reshape(newshape)\n", + " else:\n", + " self.Vt_minus = V[:-self.steps_ahead]\n", + " self.Vt_plus = V[self.steps_ahead:]\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"SVD complete!\")\n", + "\n", + " def recalc_rank(self,rank,rank_thresh,rank_explained_variance):\n", + " '''\n", + " Parameters\n", + " ----------\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used. Provide only if you want to override the value from the init.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want\n", + " to override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + " '''\n", + " # if an argument was provided, overwrite the stored rank information\n", + " none_vars = (rank is None) + (rank_thresh is None) + (rank_explained_variance is None)\n", + " if none_vars != 3:\n", + " self.rank = None\n", + " self.rank_thresh = None\n", + " self.rank_explained_variance = None\n", + "\n", + " self.rank = self.rank if rank is None else rank\n", + " self.rank_thresh = self.rank_thresh if rank_thresh is None else rank_thresh\n", + " self.rank_explained_variance = self.rank_explained_variance if rank_explained_variance is None else rank_explained_variance\n", + "\n", + " none_vars = (self.rank is None) + (self.rank_thresh is None) + (self.rank_explained_variance is None)\n", + " if none_vars < 2:\n", + " raise ValueError(\"More than one value was provided between rank, rank_thresh, and rank_explained_variance. Please provide only one of these, and ensure the others are None!\")\n", + " elif none_vars == 3:\n", + " self.rank = len(self.S)\n", + "\n", + " if self.reduced_rank_reg:\n", + " S = self.proj_mat_S\n", + " else:\n", + " S = self.S\n", + "\n", + " if rank_thresh is not None:\n", + " if S[-1] > rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < rank_thresh))\n", + "\n", + " if rank_explained_variance is not None:\n", + " self.rank = int(torch.argmax((self.cumulative_explained_variance > rank_explained_variance).type(torch.int)).cpu().numpy())\n", + "\n", + " if self.rank > self.H.shape[-1]:\n", + " self.rank = self.H.shape[-1]\n", + "\n", + " if self.rank is None:\n", + " if S[-1] > self.rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < self.rank_thresh))\n", + "\n", + " def compute_havok_dmd(self,lamb=None):\n", + " \"\"\"\n", + " Computes the Havok DMD matrix (Principal Component Regression)\n", + "\n", + " Parameters\n", + " ----------\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0 - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing least squares fits to HAVOK DMD ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " A_v = (torch.linalg.inv(self.Vt_minus[:, :self.rank].T @ self.Vt_minus[:, :self.rank] + self.lamb*torch.eye(self.rank).to(self.device)) \\\n", + " @ self.Vt_minus[:, :self.rank].T @ self.Vt_plus[:, :self.rank]).T\n", + " self.A_v = A_v\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1], :self.rank] @ self.A_v @ self.S_mat_inv[:self.rank, :self.U.shape[1]] @ self.U.T\n", + "\n", + " if self.verbose:\n", + " print(\"Least squares complete! \\n\")\n", + "\n", + " def compute_proj_mat(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Projector Matrix for Reduced Rank Regression\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " self.proj_mat = self.Vt_plus.T @ self.Vt_minus @ torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus +\n", + " self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ \\\n", + " self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.proj_mat_S, self.proj_mat_V = torch.linalg.eigh(self.proj_mat)\n", + " #todo: more efficient to flip ranks (negative index) in compute_reduced_rank_regression but also less interpretable\n", + " self.proj_mat_S = torch.flip(self.proj_mat_S, dims=(0,))\n", + " self.proj_mat_V = torch.flip(self.proj_mat_V, dims=(1,))\n", + "\n", + " if self.verbose:\n", + " print(\"Projector Matrix computed! \\n\")\n", + "\n", + " def compute_reduced_rank_regression(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Reduced Rank Regression ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + " proj_mat = self.proj_mat_V[:,:self.rank] @ self.proj_mat_V[:,:self.rank].T\n", + " B_ols = torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus + self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.A_v = B_ols @ proj_mat\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1],:self.A_v.shape[1]] @ self.A_v.T @ self.S_mat_inv[:self.A_v.shape[0], :self.U.shape[1]] @ self.U.T\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"Reduced Rank Regression complete! \\n\")\n", + "\n", + " def fit(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " lamb=None,\n", + " device=None,\n", + " verbose=None,\n", + " steps_ahead=None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults to None -\n", + " provide only if you want to override the value from the init.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_thresh : int\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " device: string or int\n", + " A string or int to indicate the device to torch. For example, can be 'cpu' or 'cuda',\n", + " or alternatively 0 if the intenion is to use GPU device 0. Defaults to None - provide only\n", + " if you want to override the value from the init.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + " Defaults to None - provide only if you want to override the value from the init.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + "\n", + " \"\"\"\n", + " # if parameters are provided, overwrite them from the init\n", + " self.steps_ahead = self.steps_ahead if steps_ahead is None else steps_ahead\n", + " self.device = self.device if device is None else device\n", + " self.verbose = self.verbose if verbose is None else verbose\n", + "\n", + " self.compute_hankel(data, n_delays, delay_interval)\n", + " self.compute_svd()\n", + "\n", + " if self.reduced_rank_reg:\n", + " self.compute_proj_mat(lamb)\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_reduced_rank_regression(lamb)\n", + " else:\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_havok_dmd(lamb)\n", + "\n", + " if self.send_to_cpu:\n", + " self.all_to_device('cpu') #send back to the cpu to save memory\n", + "\n", + " def predict(\n", + " self,\n", + " test_data=None,\n", + " reseed=None,\n", + " full_return=False\n", + " ):\n", + " \"\"\"\n", + " Returns\n", + " -------\n", + " pred_data : torch.tensor\n", + " The predictions generated by the HAVOK model. Of the same shape as test_data. Note that the first\n", + " (self.n_delays - 1)*self.delay_interval + 1 time steps of the generated predictions are by construction\n", + " identical to the test_data.\n", + "\n", + " H_test_havok_dmd : torch.tensor (Optional)\n", + " Returned if full_return=True. The predicted Hankel matrix generated by the HAVOK model.\n", + " H_test : torch.tensor (Optional)\n", + " Returned if full_return=True. The true Hankel matrix\n", + " \"\"\"\n", + " # initialize test_data\n", + " if test_data is None:\n", + " test_data = self.data\n", + " if isinstance(test_data, np.ndarray):\n", + " test_data = torch.from_numpy(test_data).to(self.device)\n", + " ndim = test_data.ndim\n", + " if ndim == 2:\n", + " test_data = test_data.unsqueeze(0)\n", + " H_test = embed_signal_torch(test_data, self.n_delays, self.delay_interval)\n", + " steps_ahead = self.steps_ahead if self.steps_ahead is not None else 1\n", + "\n", + " if reseed is None:\n", + " reseed = 1\n", + "\n", + " H_test_havok_dmd = torch.zeros(H_test.shape).to(self.device)\n", + " H_test_havok_dmd[:, :steps_ahead] = H_test[:, :steps_ahead]\n", + "\n", + " A = self.A_havok_dmd.unsqueeze(0)\n", + " for t in range(steps_ahead, H_test.shape[1]):\n", + " if t % reseed == 0:\n", + " H_test_havok_dmd[:, t] = (A @ H_test[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " else:\n", + " H_test_havok_dmd[:, t] = (A @ H_test_havok_dmd[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " pred_data = torch.hstack([test_data[:, :(self.n_delays - 1)*self.delay_interval + steps_ahead], H_test_havok_dmd[:, steps_ahead:, :self.n]])\n", + "\n", + " if ndim == 2:\n", + " pred_data = pred_data[0]\n", + "\n", + " if full_return:\n", + " return pred_data, H_test_havok_dmd, H_test\n", + " else:\n", + " return pred_data\n", + "\n", + " def all_to_device(self,device='cpu'):\n", + " for k,v in self.__dict__.items():\n", + " if isinstance(v, torch.Tensor):\n", + " self.__dict__[k] = v.to(device)\n", + "\n", + "from typing import Literal\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "from typing import Literal\n", + "import torch.nn.utils.parametrize as parametrize\n", + "from scipy.stats import wasserstein_distance\n", + "\n", + "def pad_zeros(A,B,device):\n", + "\n", + " with torch.no_grad():\n", + " dim = max(A.shape[0],B.shape[0])\n", + " A1 = torch.zeros((dim,dim)).float()\n", + " A1[:A.shape[0],:A.shape[1]] += A\n", + " A = A1.float().to(device)\n", + "\n", + " B1 = torch.zeros((dim,dim)).float()\n", + " B1[:B.shape[0],:B.shape[1]] += B\n", + " B = B1.float().to(device)\n", + "\n", + " return A,B\n", + "\n", + "class LearnableSimilarityTransform(nn.Module):\n", + " \"\"\"\n", + " Computes the similarity transform for a learnable orthonormal matrix C\n", + " \"\"\"\n", + " def __init__(self, n,orthog=True):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + " n : int\n", + " dimension of the C matrix\n", + " \"\"\"\n", + " super(LearnableSimilarityTransform, self).__init__()\n", + " #initialize orthogonal matrix as identity\n", + " self.C = nn.Parameter(torch.eye(n).float())\n", + " self.orthog = orthog\n", + "\n", + " def forward(self, B):\n", + " if self.orthog:\n", + " return self.C @ B @ self.C.transpose(-1, -2)\n", + " else:\n", + " return self.C @ B @ torch.linalg.inv(self.C)\n", + "\n", + "class Skew(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a skew-symmetric matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X - X.transpose(-1, -2)\n", + "\n", + "class Matrix(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X\n", + "\n", + "class CayleyMap(nn.Module):\n", + " \"\"\"\n", + " Maps a skew-symmetric matrix to an orthogonal matrix in O(n)\n", + " \"\"\"\n", + " def __init__(self, n, device):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " n : int\n", + " dimension of the matrix we want to map\n", + "\n", + " device : {'cpu','cuda'} or int\n", + " hardware device on which to send the matrix\n", + " \"\"\"\n", + " super().__init__()\n", + " self.register_buffer(\"Id\", torch.eye(n,device = device))\n", + "\n", + " def forward(self, X):\n", + " # (I + X)(I - X)^{-1}\n", + " return torch.linalg.solve(self.Id + X, self.Id - X)\n", + "\n", + "class SimilarityTransformDist:\n", + " \"\"\"\n", + " Computes the Procrustes Analysis over Vector Fields\n", + " \"\"\"\n", + " def __init__(self,\n", + " iters = 200,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 0.01,\n", + " device: Literal[\"cpu\",\"cuda\"] = 'cpu',\n", + " verbose = False,\n", + " group: Literal[\"O(n)\",\"SO(n)\",\"GL(n)\"] = \"O(n)\",\n", + " wasserstein_compare = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " _________\n", + " iters : int\n", + " number of iterations to perform gradient descent\n", + "\n", + " score_method : {\"angular\",\"euclidean\",\"wasserstein\"}\n", + " specifies the type of metric to use\n", + " \"wasserstein\" will compare the singular values or eigenvalues\n", + " of the two matrices as in Redman et al., (2023)\n", + "\n", + " lr : float\n", + " learning rate\n", + "\n", + " device : {'cpu','cuda'} or int\n", + "\n", + " verbose : bool\n", + " prints when finished optimizing\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + "\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.verbose = verbose\n", + " self.device = device\n", + " self.C_star = None\n", + " self.A = None\n", + " self.B = None\n", + " self.group = group\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " def fit(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " group = None,\n", + " ):\n", + " \"\"\"\n", + " Computes the optimal matrix C over specified group\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " Returns\n", + " _______\n", + " None\n", + " \"\"\"\n", + " assert A.shape[0] == A.shape[1]\n", + " assert B.shape[0] == B.shape[1]\n", + "\n", + " A = A.to(self.device)\n", + " B = B.to(self.device)\n", + " self.A,self.B = A,B\n", + " lr = self.lr if lr is None else lr\n", + " iters = self.iters if iters is None else iters\n", + " group = self.group if group is None else group\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if group == \"O(n)\":\n", + " #permute the first row and column of B then rerun the optimization\n", + " P = torch.eye(B.shape[0],device=self.device)\n", + " if P.shape[0] > 1:\n", + " P[[0, 1], :] = P[[1, 0], :]\n", + " losses, C_star, sim_net = self.optimize_C(A,\n", + " P @ B @ P.T,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if losses[-1] < self.losses[-1]:\n", + " self.losses = losses\n", + " self.C_star = C_star @ P\n", + " self.sim_net = sim_net\n", + " if group == \"GL(n)\":\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=False,\n", + " verbose=self.verbose)\n", + "\n", + " def optimize_C(self,A,B,lr,iters,orthog,verbose):\n", + " #parameterize mapping to be orthogonal\n", + " n = A.shape[0]\n", + " sim_net = LearnableSimilarityTransform(n,orthog=orthog).to(self.device)\n", + " if orthog:\n", + " parametrize.register_parametrization(sim_net, \"C\", Skew(n,self.device))\n", + " parametrize.register_parametrization(sim_net, \"C\", CayleyMap(n,self.device))\n", + " else:\n", + " parametrize.register_parametrization(sim_net, \"C\", Matrix(n,self.device))\n", + "\n", + " simdist_loss = nn.MSELoss(reduction = 'sum')\n", + "\n", + " optimizer = optim.Adam(sim_net.parameters(), lr=lr)\n", + " # scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.999)\n", + "\n", + " losses = []\n", + " A /= torch.linalg.norm(A)\n", + " B /= torch.linalg.norm(B)\n", + " for _ in range(iters):\n", + " # Zero the gradients of the optimizer.\n", + " optimizer.zero_grad()\n", + " # Compute the Frobenius norm between A and the product.\n", + " loss = simdist_loss(A, sim_net(B))\n", + "\n", + " loss.backward()\n", + "\n", + " optimizer.step()\n", + " # if _ % 99:\n", + " # scheduler.step()\n", + " losses.append(loss.item())\n", + "\n", + " if verbose:\n", + " print(\"Finished optimizing C\")\n", + "\n", + " C_star = sim_net.C.detach()\n", + " return losses, C_star,sim_net\n", + "\n", + " def score(self,A=None,B=None,score_method=None,group=None):\n", + " \"\"\"\n", + " Given an optimal C already computed, calculate the metric\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor or None\n", + " first data matrix, if None defaults to the saved matrix in fit\n", + " B : np.array or torch.tensor or None\n", + " second data matrix if None, defaults to the savec matrix in fit\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + " \"\"\"\n", + " assert self.C_star is not None\n", + " A = self.A if A is None else A\n", + " B = self.B if B is None else B\n", + " assert A is not None\n", + " assert B is not None\n", + " assert A.shape == self.C_star.shape\n", + " assert B.shape == self.C_star.shape\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + " with torch.no_grad():\n", + " if not isinstance(A,torch.Tensor):\n", + " A = torch.from_numpy(A).float().to(self.device)\n", + " if not isinstance(B,torch.Tensor):\n", + " B = torch.from_numpy(B).float().to(self.device)\n", + " C = self.C_star.to(self.device)\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " Cinv = C.T\n", + " elif group in {\"GL(n)\"}:\n", + " Cinv = torch.linalg.inv(C)\n", + " else:\n", + " raise AssertionError(\"Need proper group name\")\n", + " if score_method == 'angular':\n", + " num = torch.trace(A.T @ C @ B @ Cinv)\n", + " den = torch.norm(A,p = 'fro')*torch.norm(B,p = 'fro')\n", + " score = torch.arccos(num/den).cpu().numpy()\n", + " if np.isnan(score): #around -1 and 1, we sometimes get NaNs due to arccos\n", + " if num/den < 0:\n", + " score = np.pi\n", + " else:\n", + " score = 0\n", + " else:\n", + " score = torch.norm(A - C @ B @ Cinv,p='fro').cpu().numpy().item() #/ A.numpy().size\n", + "\n", + " return score\n", + "\n", + " def fit_score(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " score_method = None,\n", + " zero_pad = True,\n", + " group = None):\n", + " \"\"\"\n", + " for efficiency, computes the optimal matrix and returns the score\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : {'angular','euclidean'} or None\n", + " overwrites parameter in the class\n", + " zero_pad : bool\n", + " if True, then the smaller matrix will be zero padded so its the same size\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + "\n", + " \"\"\"\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + "\n", + " if isinstance(A,np.ndarray):\n", + " A = torch.from_numpy(A).float()\n", + " if isinstance(B,np.ndarray):\n", + " B = torch.from_numpy(B).float()\n", + "\n", + " assert A.shape[0] == B.shape[1] or self.wasserstein_compare is not None\n", + " if A.shape[0] != B.shape[0]:\n", + " if self.wasserstein_compare is None:\n", + " raise AssertionError(\"Matrices must be the same size unless using wasserstein distance\")\n", + " else: #otherwise resort to L2 Wasserstein over singular or eigenvalues\n", + " print(f\"resorting to wasserstein distance over {self.wasserstein_compare}\")\n", + "\n", + " if self.score_method == \"wasserstein\":\n", + " assert self.wasserstein_compare in {\"sv\",\"eig\"}\n", + " if self.wasserstein_compare == \"sv\":\n", + " a = torch.svd(A).S.view(-1,1)\n", + " b = torch.svd(B).S.view(-1,1)\n", + " elif self.wasserstein_compare == \"eig\":\n", + " a = torch.linalg.eig(A).eigenvalues\n", + " a = torch.vstack([a.real,a.imag]).T\n", + "\n", + " b = torch.linalg.eig(B).eigenvalues\n", + " b = torch.vstack([b.real,b.imag]).T\n", + " else:\n", + " raise AssertionError(\"wasserstein_compare must be 'sv' or 'eig'\")\n", + " device = a.device\n", + " a = a#.cpu()\n", + " b = b#.cpu()\n", + " M = ot.dist(a,b)#.numpy()\n", + " a,b = torch.ones(a.shape[0])/a.shape[0],torch.ones(b.shape[0])/b.shape[0]\n", + " a,b = a.to(device),b.to(device)\n", + "\n", + " score_star = ot.emd2(a,b,M)\n", + " #wasserstein_distance(A.cpu().numpy(),B.cpu().numpy())\n", + "\n", + " else:\n", + "\n", + " self.fit(A, B,iters,lr,group)\n", + " score_star = self.score(self.A,self.B,score_method=score_method,group=group)\n", + "\n", + " return score_star\n", + "\n", + "class DSA:\n", + " \"\"\"\n", + " Computes the Dynamical Similarity Analysis (DSA) for two data matrices\n", + " \"\"\"\n", + " def __init__(self,\n", + " X,\n", + " Y=None,\n", + " n_delays=1,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance = None,\n", + " lamb = 0.0,\n", + " send_to_cpu = True,\n", + " iters = 1500,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 5e-3,\n", + " group: Literal[\"GL(n)\", \"O(n)\", \"SO(n)\"] = \"O(n)\",\n", + " zero_pad = False,\n", + " device = 'cpu',\n", + " verbose = False,\n", + " reduced_rank_reg = False,\n", + " kernel=None,\n", + " num_centers=0.1,\n", + " svd_solver='arnoldi',\n", + " wasserstein_compare: Literal['sv','eig',None] = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " X : np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " first data matrix/matrices\n", + "\n", + " Y : None or np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " second data matrix/matrices.\n", + " * If Y is None, X is compared to itself pairwise\n", + " (must be a list)\n", + " * If Y is a single matrix, all matrices in X are compared to Y\n", + " * If Y is a list, all matrices in X are compared to all matrices in Y\n", + "\n", + " DMD parameters:\n", + "\n", + " n_delays : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " number of delays to use in constructing the Hankel matrix\n", + "\n", + " delay_interval : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " interval between samples taken in constructing Hankel matrix\n", + "\n", + " rank : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " rank of DMD matrix fit in reduced-rank regression\n", + "\n", + " rank_thresh : float or list or tuple/list: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float or list or tuple: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " lamb : float\n", + " L-1 regularization parameter in DMD fit\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " NOTE: for all of these above, they can be single values or lists or tuples,\n", + " depending on the corresponding dimensions of the data\n", + " If at least one of X and Y are lists, then if they are a single value\n", + " it will default to the rank of all DMD matrices.\n", + " If they are (int,int), then they will correspond to an individual dmd matrix\n", + " OR to X and Y respectively across all matrices\n", + " If it is (list,list), then each element will correspond to an individual\n", + " dmd matrix indexed at the same position\n", + "\n", + " SimDist parameters:\n", + "\n", + " iters : int\n", + " number of optimization iterations in Procrustes over vector fields\n", + "\n", + " score_method : {'angular','euclidean'}\n", + " type of metric to compute, angular vs euclidean distance\n", + "\n", + " lr : float\n", + " learning rate of the Procrustes over vector fields optimization\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " zero_pad : bool\n", + " whether or not to zero-pad if the dimensions are different\n", + "\n", + " device : 'cpu' or 'cuda' or int\n", + " hardware to use in both DMD and PoVF\n", + "\n", + " verbose : bool\n", + " whether or not print when sections of the analysis is completed\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + " self.X = X\n", + " self.Y = Y\n", + " if self.X is None and isinstance(self.Y,list):\n", + " self.X, self.Y = self.Y, self.X #swap so code is easy\n", + "\n", + " self.check_method()\n", + " if self.method == 'self-pairwise':\n", + " self.data = [self.X]\n", + " else:\n", + " self.data = [self.X, self.Y]\n", + "\n", + " self.n_delays = self.broadcast_params(n_delays,cast=int)\n", + " self.delay_interval = self.broadcast_params(delay_interval,cast=int)\n", + " self.rank = self.broadcast_params(rank,cast=int)\n", + " self.rank_thresh = self.broadcast_params(rank_thresh)\n", + " self.rank_explained_variance = self.broadcast_params(rank_explained_variance)\n", + " self.lamb = self.broadcast_params(lamb)\n", + " self.send_to_cpu = send_to_cpu\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.device = device\n", + " self.verbose = verbose\n", + " self.zero_pad = zero_pad\n", + " self.group = group\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.kernel = kernel\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " if kernel is None:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[DMD(Xi,\n", + " self.n_delays[i][j],\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " rank_thresh=self.rank_thresh[i][j],\n", + " rank_explained_variance=self.rank_explained_variance[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " device=self.device,\n", + " verbose=self.verbose,\n", + " send_to_cpu=self.send_to_cpu) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + " else:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[KernelDMD(Xi,\n", + " self.n_delays[i][j],\n", + " kernel=self.kernel,\n", + " num_centers=num_centers,\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " verbose=self.verbose,\n", + " svd_solver=svd_solver,\n", + " ) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + "\n", + " self.simdist = SimilarityTransformDist(iters,score_method,lr,device,verbose,group,wasserstein_compare)\n", + "\n", + " def check_method(self):\n", + " '''\n", + " helper function to identify what type of dsa we're running\n", + " '''\n", + " tensor_or_np = lambda x: isinstance(x,(np.ndarray,torch.Tensor))\n", + "\n", + " if isinstance(self.X,list):\n", + " if self.Y is None:\n", + " self.method = 'self-pairwise'\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'bipartite-pairwise'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'list-to-one'\n", + " self.Y = [self.Y] #wrap in a list for iteration\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " elif tensor_or_np(self.X):\n", + " self.X = [self.X]\n", + " if self.Y is None:\n", + " raise ValueError('only one element provided')\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'one-to-list'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'default'\n", + " self.Y = [self.Y]\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " else:\n", + " raise ValueError('unknown type of X')\n", + "\n", + " def broadcast_params(self,param,cast=None):\n", + " '''\n", + " aligns the dimensionality of the parameters with the data so it's one-to-one\n", + " '''\n", + " out = []\n", + " if isinstance(param,(int,float,np.integer)) or param is None: #self.X has already been mapped to [self.X]\n", + " out.append([param] * len(self.X))\n", + " if self.Y is not None:\n", + " out.append([param] * len(self.Y))\n", + " elif isinstance(param,(tuple,list,np.ndarray)):\n", + " if self.method == 'self-pairwise' and len(param) >= len(self.X):\n", + " out = [param]\n", + " else:\n", + " assert len(param) <= 2 #only 2 elements max\n", + "\n", + " #if the inner terms are singly valued, we broadcast, otherwise needs to be the same dimensions\n", + " for i,data in enumerate([self.X,self.Y]):\n", + " if data is None:\n", + " continue\n", + " if isinstance(param[i],(int,float)):\n", + " out.append([param[i]] * len(data))\n", + " elif isinstance(param[i],(list,np.ndarray,tuple)):\n", + " assert len(param[i]) >= len(data)\n", + " out.append(param[i][:len(data)])\n", + " else:\n", + " raise ValueError(\"unknown type entered for parameter\")\n", + "\n", + " if cast is not None and param is not None:\n", + " out = [[cast(x) for x in dat] for dat in out]\n", + "\n", + " return out\n", + "\n", + " def fit_dmds(self,\n", + " X=None,\n", + " Y=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh = None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=None,\n", + " lamb = None,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=True\n", + " ):\n", + " \"\"\"\n", + " Recomputes only the DMDs with a single set of hyperparameters. This will not compare, that will need to be done with the full procedure\n", + " \"\"\"\n", + " X = self.X if X is None else X\n", + " Y = self.Y if Y is None else Y\n", + " n_delays = self.n_delays if n_delays is None else n_delays\n", + " delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " rank = self.rank if rank is None else rank\n", + " lamb = self.lamb if lamb is None else lamb\n", + " data = []\n", + " if isinstance(X,list):\n", + " data.append(X)\n", + " else:\n", + " data.append([X])\n", + " if Y is not None:\n", + " if isinstance(Y,list):\n", + " data.append(Y)\n", + " else:\n", + " data.append([Y])\n", + "\n", + " dmds = [[DMD(Xi,n_delays,delay_interval,\n", + " rank,rank_thresh,rank_explained_variance,reduced_rank_reg,\n", + " lamb,device,verbose,send_to_cpu) for Xi in dat] for dat in data]\n", + "\n", + " for dmd_sets in dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return dmds\n", + "\n", + " def fit_score(self):\n", + " \"\"\"\n", + " Standard fitting function for both DMDs and PoVF\n", + "\n", + " Parameters\n", + " __________\n", + "\n", + " Returns\n", + " _______\n", + "\n", + " sims : np.array\n", + " data matrix of the similarity scores between the specific sets of data\n", + " \"\"\"\n", + " for dmd_sets in self.dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return self.score()\n", + "\n", + " def score(self,iters=None,lr=None,score_method=None):\n", + " \"\"\"\n", + " Rescore DSA with precomputed dmds if you want to try again\n", + "\n", + " Parameters\n", + " __________\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + "\n", + " Returns\n", + " ________\n", + " score : float\n", + " similarity score of the two precomputed DMDs\n", + " \"\"\"\n", + "\n", + " iters = self.iters if iters is None else iters\n", + " lr = self.lr if lr is None else lr\n", + " score_method = self.score_method if score_method is None else score_method\n", + "\n", + " ind2 = 1 - int(self.method == 'self-pairwise')\n", + " # 0 if self.pairwise (want to compare the set to itself)\n", + "\n", + " self.sims = np.zeros((len(self.dmds[0]),len(self.dmds[ind2])))\n", + " for i,dmd1 in enumerate(self.dmds[0]):\n", + " for j,dmd2 in enumerate(self.dmds[ind2]):\n", + " if self.method == 'self-pairwise':\n", + " if j >= i:\n", + " continue\n", + " if self.verbose:\n", + " print(f'computing similarity between DMDs {i} and {j}')\n", + "\n", + " self.sims[i,j] = self.simdist.fit_score(dmd1.A_v,dmd2.A_v,iters,lr,score_method,zero_pad=self.zero_pad)\n", + "\n", + " if self.method == 'self-pairwise':\n", + " self.sims[j,i] = self.sims[i,j]\n", + "\n", + "\n", + " if self.method == 'default':\n", + " return self.sims[0,0]\n", + "\n", + " return self.sims" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eced3162", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Helper functions (Bonus Section)\n", + "\n", + "import contextlib\n", + "import io\n", + "import argparse\n", + "# Standard library imports\n", + "from collections import OrderedDict\n", + "import logging\n", + "\n", + "# External libraries: General utilities\n", + "import argparse\n", + "import numpy as np\n", + "\n", + "# PyTorch related imports\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import torch.optim as optim\n", + "from torch.optim.lr_scheduler import StepLR\n", + "from torchvision import datasets, transforms\n", + "from torchvision.models.feature_extraction import create_feature_extractor, get_graph_node_names\n", + "from torchvision.utils import make_grid\n", + "\n", + "# Matplotlib for plotting\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "# SciPy for statistical functions\n", + "from scipy import stats\n", + "\n", + "# Scikit-Learn for machine learning utilities\n", + "from sklearn.decomposition import PCA\n", + "from sklearn import manifold\n", + "\n", + "# RSA toolbox specific imports\n", + "import rsatoolbox\n", + "from rsatoolbox.data import Dataset\n", + "from rsatoolbox.rdm.calc import calc_rdm\n", + "\n", + "class Net(nn.Module):\n", + " \"\"\"\n", + " A neural network model for image classification, consisting of two convolutional layers,\n", + " followed by two fully connected layers with dropout regularization.\n", + "\n", + " Methods:\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self):\n", + " \"\"\"\n", + " Initializes the network layers.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 32 output channels, and a 3x3 kernel.\n", + " - conv2: Second convolutional layer with 32 input channels, 64 output channels, and a 3x3 kernel.\n", + " - dropout1: Dropout layer with a dropout probability of 0.25.\n", + " - dropout2: Dropout layer with a dropout probability of 0.5.\n", + " - fc1: First fully connected layer with 9216 input features and 128 output features.\n", + " - fc2: Second fully connected layer with 128 input features and 10 output features.\n", + " \"\"\"\n", + " super(Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 32, 3, 1)\n", + " self.conv2 = nn.Conv2d(32, 64, 3, 1)\n", + " self.dropout1 = nn.Dropout(0.25)\n", + " self.dropout2 = nn.Dropout(0.5)\n", + " self.fc1 = nn.Linear(9216, 128)\n", + " self.fc2 = nn.Linear(128, 10)\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " x = self.conv1(input)\n", + " x = F.relu(x)\n", + " x = self.conv2(x)\n", + " x = F.relu(x)\n", + " x = F.max_pool2d(x, 2)\n", + " x = self.dropout1(x)\n", + " x = torch.flatten(x, 1)\n", + " x = self.fc1(x)\n", + " x = F.relu(x)\n", + " x = self.dropout2(x)\n", + " x = self.fc2(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "class recurrent_Net(nn.Module):\n", + " \"\"\"\n", + " A recurrent neural network model for image classification, consisting of two convolutional layers\n", + " with recurrent connections and a readout layer.\n", + "\n", + " Methods:\n", + " - __init__(time_steps=5): Initializes the network layers and sets the number of time steps for recurrence.\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self, time_steps=5):\n", + " \"\"\"\n", + " Initializes the network layers and sets the number of time steps for recurrence.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 16 output channels, and a 3x3 kernel with a stride of 3.\n", + " - conv2: Second convolutional layer with 16 input channels, 16 output channels, and a 3x3 kernel with padding of 1.\n", + " - readout: A sequential layer containing:\n", + " - dropout: Dropout layer with a dropout probability of 0.25.\n", + " - avgpool: Adaptive average pooling layer to reduce spatial dimensions to 1x1.\n", + " - flatten: Flatten layer to convert the 2D pooled output to 1D.\n", + " - linear: Fully connected layer with 16 input features and 10 output features.\n", + " - time_steps (int): Number of time steps for the recurrent connection.\n", + " \"\"\"\n", + " super(recurrent_Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 16, 3, 3)\n", + " self.conv2 = nn.Conv2d(16, 16, 3, 1, padding=1)\n", + " self.readout = nn.Sequential(OrderedDict([\n", + " ('dropout', nn.Dropout(0.25)),\n", + " ('avgpool', nn.AdaptiveAvgPool2d(1)),\n", + " ('flatten', nn.Flatten()),\n", + " ('linear', nn.Linear(16, 10))\n", + " ]))\n", + " self.time_steps = time_steps\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " input = self.conv1(input)\n", + " x = input\n", + " for t in range(0, self.time_steps):\n", + " x = input + self.conv2(x)\n", + " x = F.relu(x)\n", + "\n", + " x = self.readout(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "\n", + "def train_one_epoch(args, model, device, train_loader, optimizer, epoch):\n", + " \"\"\"\n", + " Trains the model for one epoch.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Arguments for training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - device (torch.device): The device to use for training (CPU/GPU).\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + " - epoch (int): The current epoch number.\n", + " \"\"\"\n", + " model.train()\n", + " for batch_idx, (data, target) in enumerate(train_loader):\n", + " data, target = data.to(device), target.to(device)\n", + " optimizer.zero_grad()\n", + " output = model(data)\n", + " output = torch.log(output) # to make it a log_softmax\n", + " loss = F.nll_loss(output, target)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if batch_idx % args.log_interval == 0:\n", + " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", + " epoch, batch_idx * len(data), len(train_loader.dataset),\n", + " 100. * batch_idx / len(train_loader), loss.item()))\n", + " if args.dry_run:\n", + " break\n", + "\n", + "def test(model, device, test_loader, return_features=False):\n", + " \"\"\"\n", + " Evaluates the model on the test dataset.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be evaluated.\n", + " - device (torch.device): The device to use for evaluation (CPU/GPU).\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " - return_features (bool): If True, returns the features from the model. Default is False.\n", + " \"\"\"\n", + " model.eval()\n", + " test_loss = 0\n", + " correct = 0\n", + " with torch.no_grad():\n", + " for data, target in test_loader:\n", + " data, target = data.to(device), target.to(device)\n", + " output = model(data)\n", + " output = torch.log(output)\n", + " test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(target.view_as(pred)).sum().item()\n", + "\n", + " test_loss /= len(test_loader.dataset)\n", + "\n", + " print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n", + " test_loss, correct, len(test_loader.dataset),\n", + " 100. * correct / len(test_loader.dataset)))\n", + "\n", + "def build_args():\n", + " \"\"\"\n", + " Builds and parses command-line arguments for training.\n", + " \"\"\"\n", + " parser = argparse.ArgumentParser(description='PyTorch MNIST Example')\n", + " parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n", + " help='input batch size for training (default: 64)')\n", + " parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n", + " help='input batch size for testing (default: 1000)')\n", + " parser.add_argument('--epochs', type=int, default=2, metavar='N',\n", + " help='number of epochs to train (default: 14)')\n", + " parser.add_argument('--lr', type=float, default=1.0, metavar='LR',\n", + " help='learning rate (default: 1.0)')\n", + " parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n", + " help='Learning rate step gamma (default: 0.7)')\n", + " parser.add_argument('--no-cuda', action='store_true', default=False,\n", + " help='disables CUDA training')\n", + " parser.add_argument('--no-mps', action='store_true', default=False,\n", + " help='disables macOS GPU training')\n", + " parser.add_argument('--dry-run', action='store_true', default=False,\n", + " help='quickly check a single pass')\n", + " parser.add_argument('--seed', type=int, default=1, metavar='S',\n", + " help='random seed (default: 1)')\n", + " parser.add_argument('--log-interval', type=int, default=50, metavar='N',\n", + " help='how many batches to wait before logging training status')\n", + " parser.add_argument('--save-model', action='store_true', default=False,\n", + " help='For Saving the current Model')\n", + " args = parser.parse_args('')\n", + "\n", + " use_cuda = torch.cuda.is_available() #not args.no_cuda and\n", + "\n", + " if use_cuda:\n", + " device = torch.device(\"cuda\")\n", + " else:\n", + " device = torch.device(\"cpu\")\n", + "\n", + " args.use_cuda = use_cuda\n", + " args.device = device\n", + " return args\n", + "\n", + "def fetch_dataloaders(args):\n", + " \"\"\"\n", + " Fetches the data loaders for training and testing datasets.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + "\n", + " Outputs:\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " \"\"\"\n", + " train_kwargs = {'batch_size': args.batch_size}\n", + " test_kwargs = {'batch_size': args.test_batch_size}\n", + " if args.use_cuda:\n", + " cuda_kwargs = {'num_workers': 1,\n", + " 'pin_memory': True,\n", + " 'shuffle': True}\n", + " train_kwargs.update(cuda_kwargs)\n", + " test_kwargs.update(cuda_kwargs)\n", + "\n", + " transform=transforms.Compose([\n", + " transforms.ToTensor(),\n", + " transforms.Normalize((0.1307,), (0.3081,))\n", + " ])\n", + " with contextlib.redirect_stdout(io.StringIO()): #to suppress output\n", + " dataset1 = datasets.MNIST('../data', train=True, download=True,\n", + " transform=transform)\n", + " dataset2 = datasets.MNIST('../data', train=False,\n", + " transform=transform)\n", + " train_loader = torch.utils.data.DataLoader(dataset1, **train_kwargs)\n", + " test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)\n", + " return train_loader, test_loader\n", + "\n", + "def train_model(args, model, optimizer):\n", + " \"\"\"\n", + " Trains the model using the specified arguments and optimizer.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + "\n", + " Outputs:\n", + " - None: The function trains the model and optionally saves it.\n", + " \"\"\"\n", + " scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n", + " for epoch in range(1, args.epochs + 1):\n", + " train_one_epoch(args, model, args.device, train_loader, optimizer, epoch)\n", + " test(model, args.device, test_loader)\n", + " scheduler.step()\n", + "\n", + " if args.save_model:\n", + " torch.save(model.state_dict(), \"mnist_cnn.pt\")\n", + "\n", + "\n", + "def calc_rdms(model_features, method='correlation'):\n", + " \"\"\"\n", + " Calculates representational dissimilarity matrices (RDMs) for model features.\n", + "\n", + " Inputs:\n", + " - model_features (dict): A dictionary where keys are layer names and values are features of the layers.\n", + " - method (str): The method to calculate RDMs, e.g., 'correlation'. Default is 'correlation'.\n", + "\n", + " Outputs:\n", + " - rdms (pyrsa.rdm.RDMs): RDMs object containing dissimilarity matrices.\n", + " - rdms_dict (dict): A dictionary with layer names as keys and their corresponding RDMs as values.\n", + " \"\"\"\n", + " ds_list = []\n", + " for l in range(len(model_features)):\n", + " layer = list(model_features.keys())[l]\n", + " feats = model_features[layer]\n", + "\n", + " if type(feats) is list:\n", + " feats = feats[-1]\n", + "\n", + " if args.use_cuda:\n", + " feats = feats.cpu()\n", + "\n", + " if len(feats.shape) > 2:\n", + " feats = feats.flatten(1)\n", + "\n", + " feats = feats.detach().numpy()\n", + " ds = Dataset(feats, descriptors=dict(layer=layer))\n", + " ds_list.append(ds)\n", + "\n", + " rdms = calc_rdm(ds_list, method=method)\n", + " rdms_dict = {list(model_features.keys())[i]: rdms.get_matrices()[i] for i in range(len(model_features))}\n", + "\n", + " return rdms, rdms_dict\n", + "\n", + "def fgsm_attack(image, epsilon, data_grad):\n", + " \"\"\"\n", + " Performs FGSM attack on an image.\n", + "\n", + " Inputs:\n", + " - image (torch.Tensor): Original image.\n", + " - epsilon (float): Perturbation magnitude.\n", + " - data_grad (torch.Tensor): Gradient of the data.\n", + "\n", + " Outputs:\n", + " - perturbed_image (torch.Tensor): Perturbed image after FGSM attack.\n", + " \"\"\"\n", + " sign_data_grad = data_grad.sign()\n", + " perturbed_image = image + epsilon * sign_data_grad\n", + " perturbed_image = torch.clamp(perturbed_image, 0, 1)\n", + " return perturbed_image\n", + "\n", + "def denorm(batch, mean=[0.1307], std=[0.3081]):\n", + " \"\"\"\n", + " Converts a batch of normalized tensors to their original scale.\n", + "\n", + " Inputs:\n", + " - batch (torch.Tensor): Batch of normalized tensors.\n", + " - mean (torch.Tensor or list): Mean used for normalization.\n", + " - std (torch.Tensor or list): Standard deviation used for normalization.\n", + "\n", + " Outputs:\n", + " - torch.Tensor: Batch of tensors without normalization applied to them.\n", + " \"\"\"\n", + " if isinstance(mean, list):\n", + " mean = torch.tensor(mean).to(batch.device)\n", + " if isinstance(std, list):\n", + " std = torch.tensor(std).to(batch.device)\n", + "\n", + " return batch * std.view(1, -1, 1, 1) + mean.view(1, -1, 1, 1)\n", + "\n", + "def generate_adversarial(model, imgs, targets, epsilon):\n", + " \"\"\"\n", + " Generates adversarial examples using FGSM attack.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to attack.\n", + " - imgs (torch.Tensor): Batch of images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " - epsilon (float): Perturbation magnitude.\n", + "\n", + " Outputs:\n", + " - adv_imgs (torch.Tensor): Batch of adversarial images.\n", + " \"\"\"\n", + " adv_imgs = []\n", + "\n", + " for img, target in zip(imgs, targets):\n", + " img = img.unsqueeze(0)\n", + " target = target.unsqueeze(0)\n", + " img.requires_grad = True\n", + "\n", + " output = model(img)\n", + " output = torch.log(output)\n", + " loss = F.nll_loss(output, target)\n", + "\n", + " model.zero_grad()\n", + " loss.backward()\n", + "\n", + " data_grad = img.grad.data\n", + " data_denorm = denorm(img)\n", + " perturbed_data = fgsm_attack(data_denorm, epsilon, data_grad)\n", + " perturbed_data_normalized = transforms.Normalize((0.1307,), (0.3081,))(perturbed_data)\n", + "\n", + " adv_imgs.append(perturbed_data_normalized.detach())\n", + "\n", + " return torch.cat(adv_imgs)\n", + "\n", + "def test_adversarial(model, imgs, targets):\n", + " \"\"\"\n", + " Tests the model on adversarial examples and prints the accuracy.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be tested.\n", + " - imgs (torch.Tensor): Batch of adversarial images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " \"\"\"\n", + " correct = 0\n", + " output = model(imgs)\n", + " output = torch.log(output)\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(targets.view_as(pred)).sum().item()\n", + "\n", + " final_acc = correct / float(len(imgs))\n", + " print(f\"adversarial test accuracy = {correct} / {len(imgs)} = {final_acc}\")\n", + "\n", + "def extract_features(model, imgs, return_layers, plot='none'):\n", + " \"\"\"\n", + " Extracts features from specified layers of the model.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model from which to extract features.\n", + " - imgs (torch.Tensor): Batch of input images.\n", + " - return_layers (list): List of layer names from which to extract features.\n", + " - plot (str): Option to plot the features. Default is 'none'.\n", + "\n", + " Outputs:\n", + " - model_features (dict): A dictionary with layer names as keys and extracted features as values.\n", + " \"\"\"\n", + " if return_layers == 'all':\n", + " return_layers, _ = get_graph_node_names(model)\n", + " elif return_layers == 'layers':\n", + " layers, _ = get_graph_node_names(model)\n", + " return_layers = [l for l in layers if 'input' in l or 'conv' in l or 'fc' in l]\n", + "\n", + " feature_extractor = create_feature_extractor(model, return_nodes=return_layers)\n", + " model_features = feature_extractor(imgs)\n", + "\n", + " return model_features" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be4a4946", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Plotting functions (Bonus)\n", + "\n", + "def sample_images(data_loader, n=5, plot=False):\n", + " \"\"\"\n", + " Samples a specified number of images from a data loader.\n", + "\n", + " Inputs:\n", + " - data_loader (torch.utils.data.DataLoader): Data loader containing images and labels.\n", + " - n (int): Number of images to sample per class.\n", + " - plot (bool): Whether to plot the sampled images using matplotlib.\n", + "\n", + " Outputs:\n", + " - imgs (torch.Tensor): Sampled images.\n", + " - labels (torch.Tensor): Corresponding labels for the sampled images.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " imgs, targets = next(iter(data_loader))\n", + "\n", + " imgs_o = []\n", + " labels = []\n", + " for value in range(10):\n", + " cat_imgs = imgs[np.where(targets == value)][0:n]\n", + " imgs_o.append(cat_imgs)\n", + " labels.append([value]*len(cat_imgs))\n", + "\n", + " imgs = torch.cat(imgs_o, dim=0)\n", + " labels = torch.tensor(labels).flatten()\n", + "\n", + " if plot:\n", + " plt.imshow(torch.moveaxis(make_grid(imgs, nrow=5, padding=0, normalize=False, pad_value=0), 0,-1))\n", + " plt.axis('off')\n", + "\n", + " return imgs, labels\n", + "\n", + "\n", + "def plot_rdms(model_rdms):\n", + " \"\"\"\n", + " Plots the Representational Dissimilarity Matrices (RDMs) for each layer of a model.\n", + "\n", + " Inputs:\n", + " - model_rdms (dict): A dictionary where keys are layer names and values are the corresponding RDMs.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, len(model_rdms))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " for l in range(len(model_rdms)):\n", + "\n", + " layer = list(model_rdms.keys())[l]\n", + " rdm = np.squeeze(model_rdms[layer])\n", + "\n", + " if len(rdm.shape) < 2:\n", + " rdm = rdm.reshape( (int(np.sqrt(rdm.shape[0])), int(np.sqrt(rdm.shape[0]))) )\n", + "\n", + " rdm = rdm / np.max(rdm)\n", + "\n", + " ax = plt.subplot(gs[0,l])\n", + " ax_ = ax.imshow(rdm, cmap='magma_r')\n", + " ax.set_title(f'{layer}')\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " cbar_ax.text(-2.3, 0.05, 'Normalized euclidean distance', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax)\n", + "\n", + " plt.show()\n", + "\n", + "def rep_path(model_features, model_colors, labels=None, rdm_calc_method='euclidean', rdm_comp_method='cosine'):\n", + " \"\"\"\n", + " Represents paths of model features in a reduced-dimensional space.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each model.\n", + " - model_colors (dict): Dictionary mapping model names to colors for visualization.\n", + " - labels (array-like, optional): Array of labels corresponding to the model features.\n", + " - rdm_calc_method (str, optional): Method for calculating RDMS ('euclidean' or 'correlation').\n", + " - rdm_comp_method (str, optional): Method for comparing RDMS ('cosine' or 'corr').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + " path_len = []\n", + " path_colors = []\n", + " rdms_list = []\n", + " ax_ticks = []\n", + " tick_colors = []\n", + " model_names = list(model_features.keys())\n", + " for m in range(len(model_names)):\n", + " model_name = model_names[m]\n", + " features = model_features[model_name]\n", + " path_colors.append(model_colors[model_name])\n", + " path_len.append(len(features))\n", + " ax_ticks.append(list(features.keys()))\n", + " tick_colors.append([model_colors[model_name]]*len(features))\n", + " rdms, _ = calc_rdms(features, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + "\n", + " path_len = np.insert(np.cumsum(path_len),0,0)\n", + "\n", + " if labels is not None:\n", + " rdms, _ = calc_rdms({'labels' : F.one_hot(labels).float().to(device)}, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + " ax_ticks.append(['labels'])\n", + " tick_colors.append(['m'])\n", + " idx_labels = -1\n", + "\n", + " rdms = rsatoolbox.rdm.concat(rdms_list)\n", + "\n", + " #Flatten the list\n", + " ax_ticks = [l for model_layers in ax_ticks for l in model_layers]\n", + " tick_colors = [l for model_layers in tick_colors for l in model_layers]\n", + " tick_colors = ['k' if tick == 'input' else color for tick, color in zip(ax_ticks, tick_colors)]\n", + "\n", + " rdms_comp = rsatoolbox.rdm.compare(rdms, rdms, method=rdm_comp_method)\n", + " if rdm_comp_method == 'cosine':\n", + " rdms_comp = np.arccos(rdms_comp)\n", + " rdms_comp = np.nan_to_num(rdms_comp, nan=0.0)\n", + "\n", + " # Symmetrize\n", + " rdms_comp = (rdms_comp + rdms_comp.T) / 2.0\n", + "\n", + " # reduce dim to 2\n", + " transformer = manifold.MDS(n_components = 2, max_iter=1000, n_init=10, normalized_stress='auto', dissimilarity=\"precomputed\")\n", + " dims= transformer.fit_transform(rdms_comp)\n", + "\n", + " # remove duplicates of the input layer from multiple models\n", + " remove_duplicates = np.where(np.array(ax_ticks) == 'input')[0][1:]\n", + " for index in remove_duplicates:\n", + " del ax_ticks[index]\n", + " del tick_colors[index]\n", + " rdms_comp = np.delete(np.delete(rdms_comp, index, axis=0), index, axis=1)\n", + "\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, 2)\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " ax = plt.subplot(gs[0,0])\n", + " ax_ = ax.imshow(rdms_comp, cmap='viridis_r')\n", + " fig.subplots_adjust(left=0.2)\n", + " cbar_ax = fig.add_axes([-0.01, 0.2, 0.01, 0.5])\n", + " #cbar_ax.text(-7, 0.05, 'dissimilarity between rdms', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax,location='left')\n", + " ax.set_title('Dissimilarity between layer rdms', fontdict = {'fontsize': 14})\n", + " ax.set_xticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7, rotation=83)\n", + " ax.set_yticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7)\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.xaxis.get_ticklabels())]\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.yaxis.get_ticklabels())]\n", + "\n", + " ax = plt.subplot(gs[0,1])\n", + " amin, amax = dims.min(), dims.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + "\n", + " for i in range(len(rdms_list)-1):\n", + "\n", + " path_indices = np.arange(path_len[i], path_len[i+1])\n", + " ax.plot(dims[path_indices, 0], dims[path_indices, 1], color=path_colors[i], marker='.')\n", + " ax.set_title('Representational geometry path', fontdict = {'fontsize': 14})\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.set_xlabel(f\"dim 1\")\n", + " ax.set_ylabel(f\"dim 2\")\n", + "\n", + " # if idx_input is not None:\n", + " idx_input = 0\n", + " ax.plot(dims[idx_input, 0], dims[idx_input, 1], color='k', marker='s')\n", + "\n", + " if labels is not None:\n", + " ax.plot(dims[idx_labels, 0], dims[idx_labels, 1], color='m', marker='*')\n", + "\n", + " ax.legend(model_names, fontsize=8)\n", + " fig.tight_layout()\n", + "\n", + "def plot_dim_reduction(model_features, labels, transformer_funcs):\n", + " \"\"\"\n", + " Plots the dimensionality reduction results for model features using various transformers.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each layer.\n", + " - labels (array-like): Array of labels corresponding to the model features.\n", + " - transformer_funcs (list): List of dimensionality reduction techniques to apply ('PCA', 'MDS', 't-SNE').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + "\n", + " transformers = []\n", + " for t in transformer_funcs:\n", + " if t == 'PCA': transformers.append(PCA(n_components=2))\n", + " if t == 'MDS': transformers.append(manifold.MDS(n_components = 2, normalized_stress='auto'))\n", + " if t == 't-SNE': transformers.append(manifold.TSNE(n_components = 2, perplexity=40, verbose=0))\n", + "\n", + " fig = plt.figure(figsize=(8, 2.5*len(transformers)))\n", + " # and we add one plot per reference point\n", + " gs = fig.add_gridspec(len(transformers), len(model_features))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " return_layers = list(model_features.keys())\n", + "\n", + " for f in range(len(transformer_funcs)):\n", + "\n", + " for l in range(len(return_layers)):\n", + " layer = return_layers[l]\n", + " feats = model_features[layer].detach().cpu().flatten(1)\n", + " feats_transformed= transformers[f].fit_transform(feats)\n", + "\n", + " amin, amax = feats_transformed.min(), feats_transformed.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + " ax = plt.subplot(gs[f,l])\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.axis(\"off\")\n", + " #ax.set_title(f'{layer}')\n", + " if f == 0: ax.text(0.5, 1.12, f'{layer}', size=16, ha=\"center\", transform=ax.transAxes)\n", + " if l == 0: ax.text(-0.3, 0.5, transformer_funcs[f], size=16, ha=\"center\", transform=ax.transAxes)\n", + " # Create a discrete color map based on unique labels\n", + " num_colors = len(np.unique(labels))\n", + " cmap = plt.get_cmap('viridis_r', num_colors) # 10 discrete colors\n", + " norm = mpl.colors.BoundaryNorm(np.arange(-0.5,num_colors), cmap.N)\n", + " ax_ = ax.scatter(feats_transformed[:, 0], feats_transformed[:, 1], c=labels, cmap=cmap, norm=norm)\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " fig.colorbar(ax_, cax=cbar_ax, ticks=np.linspace(0,9,10))\n", + " plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21f68945", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Data retrieval\n", + "\n", + "import os\n", + "import requests\n", + "import hashlib\n", + "\n", + "# Variables for file and download URL\n", + "fnames = [\"standard_model.pth\", \"adversarial_model.pth\", \"recurrent_model.pth\"] # The names of the files to be downloaded\n", + "urls = [\"https://osf.io/s5rt6/download\", \"https://osf.io/qv5eb/download\", \"https://osf.io/6hnwk/download\"] # URLs from where the files will be downloaded\n", + "expected_md5s = [\"2e63c2cd77bc9f1fa67673d956ec910d\", \"25fb34497377921b54368317f68a7aa7\", \"ee5cea3baa264cb78300102fa6ed66e8\"] # MD5 hashes for verifying files integrity\n", + "\n", + "for fname, url, expected_md5 in zip(fnames, urls, expected_md5s):\n", + " if not os.path.isfile(fname):\n", + " try:\n", + " # Attempt to download the file\n", + " r = requests.get(url) # Make a GET request to the specified URL\n", + " except requests.ConnectionError:\n", + " # Handle connection errors during the download\n", + " print(\"!!! Failed to download data !!!\")\n", + " else:\n", + " # No connection errors, proceed to check the response\n", + " if r.status_code != requests.codes.ok:\n", + " # Check if the HTTP response status code indicates a successful download\n", + " print(\"!!! Failed to download data !!!\")\n", + " elif hashlib.md5(r.content).hexdigest() != expected_md5:\n", + " # Verify the integrity of the downloaded file using MD5 checksum\n", + " print(\"!!! Data download appears corrupted !!!\")\n", + " else:\n", + " # If download is successful and data is not corrupted, save the file\n", + " with open(fname, \"wb\") as fid:\n", + " fid.write(r.content) # Write the downloaded content to a file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "93aeca0a", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Figure settings\n", + "\n", + "logging.getLogger('matplotlib.font_manager').disabled = True\n", + "\n", + "%matplotlib inline\n", + "%config InlineBackend.figure_format = 'retina' # perfrom high definition rendering for images and plots\n", + "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/main/nma.mplstyle\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dd8052d5", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Set device (GPU or CPU)\n", + "\n", + "# inform the user if the notebook uses GPU or CPU.\n", + "\n", + "def set_device():\n", + " \"\"\"\n", + " Determines and sets the computational device for PyTorch operations based on the availability of a CUDA-capable GPU.\n", + "\n", + " Outputs:\n", + " - device (str): The device that PyTorch will use for computations ('cuda' or 'cpu'). This string can be directly used\n", + " in PyTorch operations to specify the device.\n", + " \"\"\"\n", + "\n", + " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + " if device != \"cuda\":\n", + " print(\"GPU is not enabled in this notebook. \\n\"\n", + " \"If you want to enable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `GPU` from the dropdown menu\")\n", + " else:\n", + " print(\"GPU is enabled in this notebook. \\n\"\n", + " \"If you want to disable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `None` from the dropdown menu\")\n", + "\n", + " return device\n", + "\n", + "device = set_device()" ] }, { @@ -80,7 +2142,7 @@ }, "outputs": [], "source": [ - "# @title Bonus material slides\n", + "# @title Load Slides\n", "\n", "from IPython.display import IFrame\n", "from ipywidgets import widgets\n", @@ -94,6 +2156,22 @@ "display(out)" ] }, + { + "cell_type": "markdown", + "id": "407ace26", + "metadata": {}, + "source": [ + "---\n", + "\n", + "# Intro\n", + "\n", + "Welcome to Tutorial 5 of Day 3 (W1D3) of the NeuroAI course. In this tutorial we are going to look at an exciting method that measures similarity from a slightly different perspective, a temporal one. The prior methods we have looked at were centeed around geometry and spatial representations, where we looked at metrics such as the Euclidean and Mahalanobis distance metrics. However, one thing we often want to study in neuroscience and in AI separately - is the temporal domain. Even more so in our own field of NeuroAI, we often deal with time series of neuronal / biological recordings. One thing you should already have a broad level of awareness of is that end structures can end up looking the same even though the paths taken to arrive at those end structures were very different.\n", + "\n", + "In NeuroAI, we're often confronted with systems that seem to have some sort of overlap and we want to study whether this implies there is a shared computation pairs up with the shared task (we looked at this in detail yesterday in our *Comparing Tasks* day). Today, we will begin by watching a short intro video by Mitchell Ostrow, who will describe his method to compare representations over temporal sequences (the method is called Dynamic Similarity Analysis). Then we are going to introduce three simple dynamical systems and we will explore them from the perspective of Dynamic Similarity Analysis and also describe the conceptual relationship to Representational Similarity Analysis. You will have a short coding exercise on the topic of temporal similarity analysis on three different types of trajectories. \n", + "\n", + "At the end of the tutorial, we will finally look at a further aspect of temporal sequences using RNNs. This is an adaptation of the ideas introduced in Tutorial 2 but now based around recurrent representations from RNNs. We hope you enjoy this tutorial today and that it gets you thinking not just what similarity values mean, but which ones are appropriate (here, from a spatial or temporal perspective). We aim to continually expand the tools necessary in the NeuroAI researcher's toolkit. Complementary tools, when applicable, can often tell a far richer story than just using a single method." + ] + }, { "cell_type": "code", "execution_count": null, @@ -162,6 +2240,397 @@ "# @title Submit your feedback\n", "content_review(f\"{feedback_prefix}_DSA_video\")" ] + }, + { + "cell_type": "markdown", + "id": "937041e9", + "metadata": {}, + "source": [ + "## Section 1: Visualization of Three Temporal Sequences\n", + "\n", + "We are going to be working with the analysis of three temporal sequences today:\n", + "\n", + "* The circular time series (`Circle`)\n", + "* The oval time series (`Oval`)\n", + "* The random walk (`R-Walk`)\n", + "\n", + "The random walk is going to be broadly *oval shaped*. Now, what do you think, from a geometric perspective, might result from a spatial analysis of these three different *representations*? You will probably assume because the random walk has an oval shape and there is also an oval time series (that's not a random walk) that these would result in a higher spatial similarity. You'd be right to assume this. However, what we're going to do with the `Circle` and `Oval` time series is to include an oscillator at a specific frequency, shared amongst these two time series. In effect, this means that although when plotted in totality the shapes are different, during the dynamic (temporal) evolution of these time series, a very similar shared pattern is emerging. We want methods that are sensitive to these changes to give higher scores for time series sharing similar temporal patterns (e.g. both containing oscillating patterns at similar frequences) rather than just a random walk that resembles (geometrically) one of the other shapes (`R-Walk`). Before we continue, we'll just define this random walk in a little more detail. A random walk at a specific location / timepoint takes a random step of fixed length in a specific direction, but this can be broadly controlled to resemble geometric shapes. We've taken a random walk and then reframed it to be similar in shape to `Oval`. \n", + "\n", + "Let's now visualize these three temporal sequences, to make the previous paragraph a little clearer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b57dfe1a", + "metadata": {}, + "outputs": [], + "source": [ + "# Circle\n", + "r = .1; # rotation\n", + "A = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_circle = trajectory\n", + "\n", + "# Oval\n", + "r = .1; # rotation\n", + "s = 4; # scaling\n", + "S = np.array([[1, 0], [0, s]])\n", + "Si = np.array([[1, 0], [0, 1/s]])\n", + "V = np.array([[1, 1], [-1, 1]])/np.sqrt(2)\n", + "Vi = np.array([[1, -1], [1, 1]])/np.sqrt(2)\n", + "R = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "A = np.linalg.multi_dot([V,Si,R,S,Vi])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_oval = trajectory\n", + "\n", + "# R-Walk (random walk)\n", + "r = .1; # rotation\n", + "A = np.array([[.9, 0], [0, .9]])\n", + "c = -.95; # correlation coefficient\n", + "B = np.array([[1, c], [0, np.sqrt(1-c*c)]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_walk = trajectory" + ] + }, + { + "cell_type": "markdown", + "id": "113a0dee", + "metadata": {}, + "source": [ + "Can you see how the spatial / geometric similarity of `R-Walk` and `Oval` are more similar, but the oscillations during the temporal sequence are shared between `Circle` and `Oval`? Let's run Dynamic Similarity Analysis on these temporal sequences and see what scores are returned.\n", + "\n", + "We calcularted `trajectory_oval` and `trajectory_circle` above, so let's plug these into the `DSA` function imported earlier (in the helper function cell) and see what the similarity score is." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3e36d59", + "metadata": {}, + "outputs": [], + "source": [ + "# Define the DSA computation class\n", + "dsa = DSA(X=trajectory_oval, Y=trajectory_circle, n_delays=1)\n", + "\n", + "# Call the fit method and save the result\n", + "similarities_oval_circle = dsa.fit_score()\n", + "\n", + "print(f\"DSA similarity between Oval and Circle: {similarities_oval_circle:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9f1fb622", + "metadata": {}, + "source": [ + "## Multi-way Comparison\n", + "\n", + "We're now going to run DSA on our three trajectories and fit the model, returning the scores which we can investigate by plotting a confusion matrix with a heatmap to show the DSA scores." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6ee9e8e8", + "metadata": {}, + "outputs": [], + "source": [ + "n_delays = 1\n", + "delay_interval = 1\n", + "\n", + "models = [trajectory_circle, trajectory_oval, trajectory_walk]\n", + "dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval)\n", + "similarities = dsa.fit_score()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18318ddb", + "metadata": {}, + "outputs": [], + "source": [ + "labels = ['Circle', 'Oval', 'Walk']\n", + "data = np.random.rand(len(labels), len(labels))\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "ffd49b4b", + "metadata": {}, + "source": [ + "This heatmap across the three model comparisons shows that the DSA scores between (`Walk` and `Circle`) and (`Walk` and `Oval`) to be (relatively) high, while the comparison between (`Circle` and `Oval`) is very low. Please note that this confusion matrix is symmetrical, meaning that the analysis between `trajectory_A` and `trajectory_B` returns the same dynamic similarity score as `trajectory_B` and `trajectory_A`. This is a common feature we have also seen in comparison metrics in standard RSA. One thing to note in the calculation of DSA is that comparisons among identical trajectories is `0`. This is unlike in RSA where we expect the correlation among the same stimuli to be `1.0`. This is why we see black squares along the diagonal.\n", + "\n", + "Let's put our thinking caps on for a moment: This isn't really the result we would have expected, right? What do you think might be going on here? Have a look back at the *hyperparameters* and try to make an educated guess!" + ] + }, + { + "cell_type": "markdown", + "id": "d0ff5faa", + "metadata": {}, + "source": [ + "## DSA Hyperparameters (`n_delays` and `delay_interval`)\n", + "\n", + "We'll now give you a hint as to why the setting of these hyperparameters is important when considering dynamic similarity analysis. The oscillators we have placed in the trajectories of `Circle` and `Oval` are not immediately apparent if you study only the previous time step for each element. It's only when considering the recurring pattern across a few different temporal delays and at what delay interval you want those to be, that we would expect to be able to detect recurring oscillations that provide us with the information we need to conclude that `Oval` and `Circle` are actually *dynamically* similar.\n", + "\n", + "You should change the values below to be more sensible hyperparameter settings and re-run the model and plot the new confusion matrix. Try using `n_delays` equal to `20` and `delay_interval` equal to `10`. Don't forget to define `models` (see above example if you get stuck)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9d8d4c03", + "metadata": {}, + "outputs": [], + "source": [ + "#################################################\n", + "## TODO for students: fill in the missing parts ##\n", + "raise NotImplementedError(\"Student exercise\")\n", + "#################################################\n", + "\n", + "n_delays = ...\n", + "delay_interval = ...\n", + "\n", + "models = ...\n", + "dsa = DSA(...)\n", + "similarities = ...\n", + "\n", + "labels = ['Circle', 'Oval', 'Walk']\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a6377c65", + "metadata": {}, + "outputs": [], + "source": [ + "# to_remove solution\n", + "\n", + "n_delays = 20\n", + "delay_interval = 10\n", + "\n", + "models = [trajectory_circle, trajectory_oval, trajectory_walk]\n", + "dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval)\n", + "similarities = dsa.fit_score()\n", + "\n", + "labels = ['Circle', 'Oval', 'Walk']\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "04b0e32f", + "metadata": {}, + "source": [ + "What do you see now? We now see a much more sensible result. The DSA scores have now correctly identified that `Oval` and `Circle` are very dynamically similar! They have the highest color score according to the colorbar on the side. As is always good practice in science, let's have a look inside the `similarities` variable to look at the exact values and confirm what we see in the figure above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "55fa4065", + "metadata": {}, + "outputs": [], + "source": [ + "similarities" + ] + }, + { + "cell_type": "markdown", + "id": "59cb799f", + "metadata": {}, + "source": [ + "## Comparison with RSA\n", + "\n", + "At the start of this exercise, we saw three different trajectories and pointed out that the random walk and oval shapes were most similar from a geometric perspective, both ellipse-like but not similar in their dynamic similarity. To better show the difference between DSA and RSA, we encourage you to run another comparison where we consider each time step to be a pair in the X,Y space and we will look at the the similarity between of `Oval` with both `Circle` and `Walk`. If our understanding is correct, then RSA should indicate a higher geometric similarity between (`Oval` and `Walk`) than with (`Oval` and `Circle`)." + ] + }, + { + "cell_type": "markdown", + "id": "87cf4e6e", + "metadata": {}, + "source": [ + "---\n", + "# (Bonus) Representational Geometry of Recurrent Models\n", + "\n", + "Transformations of representations can occur across space and time, e.g., layers of a neural network and steps of recurrent computation. We've looked at the temporal dimension today and earlier today in the other tutorials we looked mainly at spatial representations.\n", + "\n", + "Just as the layers in a feedforward DNN can change the representational geometry to perform a task, steps in a recurrent network can reuse the same layer to reach the same computational depth.\n", + "\n", + "In this section, we look at a very simple recurrent network with only 2650 trainable parameters." + ] + }, + { + "cell_type": "markdown", + "id": "3d613edd", + "metadata": {}, + "source": [ + "Here is a diagram of this network:\n", + "\n", + "![Recurrent convolutional neural network](https://github.com/neuromatch/NeuroAI_Course/blob/main/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/rcnn_tutorial.png?raw=true)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2f0443d3", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Grab a recurrent model\n", + "\n", + "args = build_args()\n", + "train_loader, test_loader = fetch_dataloaders(args)\n", + "path = \"recurrent_model.pth\"\n", + "model_recurrent = torch.load(path, map_location=args.device, weights_only=False)" + ] + }, + { + "cell_type": "markdown", + "id": "d463c3a9", + "metadata": {}, + "source": [ + "
We can first look at the computational steps in this network. As we see below, the `conv2` operation is repeated for 5 times." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6bfabacd", + "metadata": {}, + "outputs": [], + "source": [ + "train_nodes, _ = get_graph_node_names(model_recurrent)\n", + "print('The computational steps in the network are: \\n', train_nodes)" + ] + }, + { + "cell_type": "markdown", + "id": "1d410c3a", + "metadata": {}, + "source": [ + "Plotting the RDMs after each application of the `conv2` operation shows the same progressive emergence of the blockwise structure around the diagonal, mediating the correct classification in this task." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "30249608", + "metadata": {}, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=20)\n", + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", + "\n", + "rdms, rdms_dict = calc_rdms(model_features)\n", + "plot_rdms(rdms_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "248329c3", + "metadata": {}, + "source": [ + "We can also look at how the different dimensionality reduction techniques capture the dynamics of changing geometry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b0e2cdf", + "metadata": {}, + "outputs": [], + "source": [ + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", + "\n", + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", + "\n", + "plot_dim_reduction(model_features, labels, transformer_funcs =['PCA', 'MDS', 't-SNE'])" + ] + }, + { + "cell_type": "markdown", + "id": "1aaf5f4a", + "metadata": {}, + "source": [ + "## Representational geometry paths for recurrent models\n", + "\n", + "We can look at the model's recurrent computational steps as a path in the representational geometry space." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7f88274a", + "metadata": {}, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features_recurrent = extract_features(model_recurrent, imgs.to(device), return_layers='all')\n", + "\n", + "#rdms, rdms_dict = calc_rdms(model_features)\n", + "features = {'recurrent model': model_features_recurrent}\n", + "model_colors = {'recurrent model': 'y'}\n", + "\n", + "rep_path(features, model_colors, labels)" + ] + }, + { + "cell_type": "markdown", + "id": "5c3fbd44", + "metadata": {}, + "source": [ + "We can also look at the paths taken by the feedforward and the recurrent models and compare them." + ] + }, + { + "cell_type": "markdown", + "id": "b25a8cc6", + "metadata": {}, + "source": [ + "If you recall back to Tutorial 2, we compared a standard feedward model's representations. We can extend our analysis of the analysis of the recurrent model's representations by making a side-by-side comparison. We can also look at the paths taken by the feedforward and the recurrent models and compare them. What we see above in the case of the recurrent model is the fast-shifting path through the geometric space from inputs to labels. This illustration serves to show that models take many different paths and can have very diverse underlying mechanisms but still arrive at a superficially similar output at the end of training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c904e840", + "metadata": {}, + "outputs": [], + "source": [ + "# @title Submit your feedback\n", + "content_review(f\"{feedback_prefix}_recurrent_models\")" + ] + }, + { + "cell_type": "markdown", + "id": "3ed56061", + "metadata": {}, + "source": [ + "# The Big Picture\n", + "\n", + "Today, you've looked at what it means to measure representations from different systems. These systems can be of the same type (multiple brain systems, multiple artificial models) as well as with representations between these systems. In NeuroAI, we're especially interested in such comparisons, comparing representational systems in deep learning networks, for instance, to brain recordings recorded while those biological systems experienced / perceived the same set of stimuli. Comparisons can be geometric / spatial or they can be temporal. Today, we looked at Dynamic Similarity Analysis, a method used to be able to capture the dependencies among trajectories, not just capturing the similarity of the full temporal sequence upon completion of the temporal sequence. It's often important to take into account multiple dimensions of representational similarity. A combination of tools is definitely required in the NeuroAI researcher's toolkit. We hope you have many chances to use these tools in your future work as NeuroAI researchers." + ] } ], "metadata": { From b3b9d1407242b3bd1e187abdabc625c00e2fc5b6 Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Mon, 19 May 2025 14:31:12 +0000 Subject: [PATCH 3/3] Process tutorial notebooks --- .../W1D3_Tutorial4.ipynb | 15 +- .../W1D3_Tutorial5.ipynb | 154 +- .../instructor/W1D3_Tutorial4.ipynb | 90 +- .../instructor/W1D3_Tutorial5.ipynb | 2669 ++++++++++++++++- .../W1D3_Tutorial5_Solution_0467919d.py | 13 + .../W1D3_Tutorial4_Solution_1ac2083f_0.png | Bin 147893 -> 131259 bytes .../W1D3_Tutorial5_Solution_0467919d_0.png | Bin 0 -> 72195 bytes .../student/W1D3_Tutorial4.ipynb | 90 +- .../student/W1D3_Tutorial5.ipynb | 2658 +++++++++++++++- 9 files changed, 5520 insertions(+), 169 deletions(-) create mode 100644 tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/solutions/W1D3_Tutorial5_Solution_0467919d.py create mode 100644 tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/W1D3_Tutorial5_Solution_0467919d_0.png diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb index 4952feb92..064782b50 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial4.ipynb @@ -647,7 +647,9 @@ { "cell_type": "markdown", "id": "b64eaea5", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "The video below is additional information in more detail which was previously part of the introductory video for this course day. It provides some useful further information on the technical details mentioned during these tutorials. Please feel free to check it out and use it as a resource if you want to learn more or if you want to get a deeper understanding on some of the important details." ] @@ -656,7 +658,10 @@ "cell_type": "code", "execution_count": null, "id": "200235dc", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Video 2 (BONUS): Extended Intro Video\n", @@ -1568,7 +1573,9 @@ { "cell_type": "markdown", "id": "40936ec4", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "# The Big Picture\n", "\n", @@ -1606,7 +1613,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4, diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb index 97fd20d3f..cf185d05d 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/W1D3_Tutorial5.ipynb @@ -74,7 +74,10 @@ "cell_type": "code", "execution_count": null, "id": "ef9abaa3", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Helper functions\n", @@ -1356,7 +1359,10 @@ "cell_type": "code", "execution_count": null, "id": "eced3162", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Helper functions (Bonus Section)\n", @@ -1813,7 +1819,10 @@ "cell_type": "code", "execution_count": null, "id": "be4a4946", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Plotting functions (Bonus)\n", @@ -2045,7 +2054,10 @@ "cell_type": "code", "execution_count": null, "id": "21f68945", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Data retrieval\n", @@ -2085,7 +2097,10 @@ "cell_type": "code", "execution_count": null, "id": "93aeca0a", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Figure settings\n", @@ -2101,7 +2116,10 @@ "cell_type": "code", "execution_count": null, "id": "dd8052d5", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Set device (GPU or CPU)\n", @@ -2159,7 +2177,9 @@ { "cell_type": "markdown", "id": "407ace26", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "---\n", "\n", @@ -2244,7 +2264,9 @@ { "cell_type": "markdown", "id": "937041e9", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "## Section 1: Visualization of Three Temporal Sequences\n", "\n", @@ -2263,7 +2285,9 @@ "cell_type": "code", "execution_count": null, "id": "b57dfe1a", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "# Circle\n", @@ -2301,7 +2325,9 @@ { "cell_type": "markdown", "id": "113a0dee", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "Can you see how the spatial / geometric similarity of `R-Walk` and `Oval` are more similar, but the oscillations during the temporal sequence are shared between `Circle` and `Oval`? Let's run Dynamic Similarity Analysis on these temporal sequences and see what scores are returned.\n", "\n", @@ -2312,7 +2338,9 @@ "cell_type": "code", "execution_count": null, "id": "c3e36d59", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "# Define the DSA computation class\n", @@ -2327,7 +2355,9 @@ { "cell_type": "markdown", "id": "9f1fb622", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "## Multi-way Comparison\n", "\n", @@ -2338,7 +2368,9 @@ "cell_type": "code", "execution_count": null, "id": "6ee9e8e8", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "n_delays = 1\n", @@ -2353,7 +2385,9 @@ "cell_type": "code", "execution_count": null, "id": "18318ddb", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "labels = ['Circle', 'Oval', 'Walk']\n", @@ -2367,7 +2401,9 @@ { "cell_type": "markdown", "id": "ffd49b4b", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "This heatmap across the three model comparisons shows that the DSA scores between (`Walk` and `Circle`) and (`Walk` and `Oval`) to be (relatively) high, while the comparison between (`Circle` and `Oval`) is very low. Please note that this confusion matrix is symmetrical, meaning that the analysis between `trajectory_A` and `trajectory_B` returns the same dynamic similarity score as `trajectory_B` and `trajectory_A`. This is a common feature we have also seen in comparison metrics in standard RSA. One thing to note in the calculation of DSA is that comparisons among identical trajectories is `0`. This is unlike in RSA where we expect the correlation among the same stimuli to be `1.0`. This is why we see black squares along the diagonal.\n", "\n", @@ -2377,7 +2413,9 @@ { "cell_type": "markdown", "id": "d0ff5faa", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "## DSA Hyperparameters (`n_delays` and `delay_interval`)\n", "\n", @@ -2390,7 +2428,9 @@ "cell_type": "code", "execution_count": null, "id": "9d8d4c03", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "#################################################\n", @@ -2416,7 +2456,9 @@ "cell_type": "code", "execution_count": null, "id": "a6377c65", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "# to_remove solution\n", @@ -2438,7 +2480,9 @@ { "cell_type": "markdown", "id": "04b0e32f", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "What do you see now? We now see a much more sensible result. The DSA scores have now correctly identified that `Oval` and `Circle` are very dynamically similar! They have the highest color score according to the colorbar on the side. As is always good practice in science, let's have a look inside the `similarities` variable to look at the exact values and confirm what we see in the figure above." ] @@ -2447,7 +2491,9 @@ "cell_type": "code", "execution_count": null, "id": "55fa4065", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "similarities" @@ -2456,7 +2502,9 @@ { "cell_type": "markdown", "id": "59cb799f", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "## Comparison with RSA\n", "\n", @@ -2466,7 +2514,9 @@ { "cell_type": "markdown", "id": "87cf4e6e", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "---\n", "# (Bonus) Representational Geometry of Recurrent Models\n", @@ -2481,7 +2531,9 @@ { "cell_type": "markdown", "id": "3d613edd", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "Here is a diagram of this network:\n", "\n", @@ -2492,7 +2544,10 @@ "cell_type": "code", "execution_count": null, "id": "2f0443d3", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Grab a recurrent model\n", @@ -2506,7 +2561,9 @@ { "cell_type": "markdown", "id": "d463c3a9", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "
We can first look at the computational steps in this network. As we see below, the `conv2` operation is repeated for 5 times." ] @@ -2515,7 +2572,9 @@ "cell_type": "code", "execution_count": null, "id": "6bfabacd", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "train_nodes, _ = get_graph_node_names(model_recurrent)\n", @@ -2525,7 +2584,9 @@ { "cell_type": "markdown", "id": "1d410c3a", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "Plotting the RDMs after each application of the `conv2` operation shows the same progressive emergence of the blockwise structure around the diagonal, mediating the correct classification in this task." ] @@ -2534,7 +2595,9 @@ "cell_type": "code", "execution_count": null, "id": "30249608", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "imgs, labels = sample_images(test_loader, n=20)\n", @@ -2548,7 +2611,9 @@ { "cell_type": "markdown", "id": "248329c3", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "We can also look at how the different dimensionality reduction techniques capture the dynamics of changing geometry." ] @@ -2557,7 +2622,9 @@ "cell_type": "code", "execution_count": null, "id": "1b0e2cdf", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", @@ -2571,7 +2638,9 @@ { "cell_type": "markdown", "id": "1aaf5f4a", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "## Representational geometry paths for recurrent models\n", "\n", @@ -2582,7 +2651,9 @@ "cell_type": "code", "execution_count": null, "id": "7f88274a", - "metadata": {}, + "metadata": { + "execution": {} + }, "outputs": [], "source": [ "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", @@ -2598,7 +2669,9 @@ { "cell_type": "markdown", "id": "5c3fbd44", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "We can also look at the paths taken by the feedforward and the recurrent models and compare them." ] @@ -2606,7 +2679,9 @@ { "cell_type": "markdown", "id": "b25a8cc6", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "If you recall back to Tutorial 2, we compared a standard feedward model's representations. We can extend our analysis of the analysis of the recurrent model's representations by making a side-by-side comparison. We can also look at the paths taken by the feedforward and the recurrent models and compare them. What we see above in the case of the recurrent model is the fast-shifting path through the geometric space from inputs to labels. This illustration serves to show that models take many different paths and can have very diverse underlying mechanisms but still arrive at a superficially similar output at the end of training." ] @@ -2615,7 +2690,10 @@ "cell_type": "code", "execution_count": null, "id": "c904e840", - "metadata": {}, + "metadata": { + "cellView": "form", + "execution": {} + }, "outputs": [], "source": [ "# @title Submit your feedback\n", @@ -2625,7 +2703,9 @@ { "cell_type": "markdown", "id": "3ed56061", - "metadata": {}, + "metadata": { + "execution": {} + }, "source": [ "# The Big Picture\n", "\n", @@ -2660,7 +2740,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4, diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial4.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial4.ipynb index 139d831e9..3566eb171 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial4.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial4.ipynb @@ -17,7 +17,7 @@ "execution": {} }, "source": [ - "# Tutorial 4: Representational geometry & noise\n", + "# (Bonus) Tutorial 4: Representational geometry & noise\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", @@ -25,9 +25,9 @@ "\n", "__Content creators:__ Wenxuan Guo, Heiko Schütt\n", "\n", - "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk\n", + "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n", + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n", "\n", "Acknowledgments: the tutorial outline was written by Heiko Schütt. The content was greatly improved by discussions with Heiko, Hlib, and Alish, and the insightful illustrations presented in the paper by Walther et al. (2016)\n" ] @@ -61,7 +61,7 @@ "\n", "5. Using random projections to estimate distances. This section introduces the Johnson–Lindenstrauss Lemma, which states that random projections maintain the integrity of distance estimates in a lower-dimensional space. This concept is crucial for reducing dimensionality while preserving the relational structure of the data.\n", "\n", - "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures. " + "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures." ] }, { @@ -644,6 +644,72 @@ "display(tabs)" ] }, + { + "cell_type": "markdown", + "id": "b64eaea5", + "metadata": { + "execution": {} + }, + "source": [ + "The video below is additional information in more detail which was previously part of the introductory video for this course day. It provides some useful further information on the technical details mentioned during these tutorials. Please feel free to check it out and use it as a resource if you want to learn more or if you want to get a deeper understanding on some of the important details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "200235dc", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Video 2 (BONUS): Extended Intro Video\n", + "\n", + "from ipywidgets import widgets\n", + "from IPython.display import YouTubeVideo\n", + "from IPython.display import IFrame\n", + "from IPython.display import display\n", + "\n", + "class PlayVideo(IFrame):\n", + " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", + " self.id = id\n", + " if source == 'Bilibili':\n", + " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", + " elif source == 'Osf':\n", + " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", + " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "\n", + "def display_videos(video_ids, W=400, H=300, fs=1):\n", + " tab_contents = []\n", + " for i, video_id in enumerate(video_ids):\n", + " out = widgets.Output()\n", + " with out:\n", + " if video_ids[i][0] == 'Youtube':\n", + " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", + " height=H, fs=fs, rel=0)\n", + " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", + " else:\n", + " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", + " height=H, fs=fs, autoplay=False)\n", + " if video_ids[i][0] == 'Bilibili':\n", + " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", + " elif video_ids[i][0] == 'Osf':\n", + " print(f'Video available at https://osf.io/{video.id}')\n", + " display(video)\n", + " tab_contents.append(out)\n", + " return tab_contents\n", + "\n", + "#assert 1 == 0, \"Upload this video\"\n", + "video_ids = [('Youtube', 'm9srqTx5ci0'), ('Bilibili', 'BV1meVjz3Eeh')]\n", + "tab_contents = display_videos(video_ids, W=854, H=480)\n", + "tabs = widgets.Tab()\n", + "tabs.children = tab_contents\n", + "for i in range(len(tab_contents)):\n", + " tabs.set_title(i, video_ids[i][0])\n", + "display(tabs)" + ] + }, { "cell_type": "code", "execution_count": null, @@ -1509,6 +1575,20 @@ "3. Cross-validated distance estimators (cross-validated Euclidean or Mahalanobis distance) can remove the positive bias introduced by noise.\n", "4. The Johnson–Lindenstrauss Lemma shows that random projections preserve the Euclidean distance with some distortions. Crucially, the distortion does not depend on the dimensionality of the original space." ] + }, + { + "cell_type": "markdown", + "id": "40936ec4", + "metadata": { + "execution": {} + }, + "source": [ + "# The Big Picture\n", + "\n", + "The goal of this tutorial is to provide you with some mathematical tools for your NeuroAI researcher toolkit. What happens when you pull out the Euclidean metric from your toolkit and, while this has worked well in the past, suddenly in different scenarios it doesn't seem to perform so well. Aha, you spot the potential for correlated noise and you reach deeper into your toolkit and pull out the Mahalanobis metric, which implicitly undoes the correlated noise in the model. Perhaps you can't even tell if there is any correlated noise in your data and you try with both metrics, and Mahalanobis works well but Euclidean does not, that can be a hunch that leads you to confirm the presence of correlated noise. \n", + "\n", + "Sometimes you might be faced with dimensionalities that are just too high to practically deal with in your use case. Then, why not recall what you learned about how random projections can reduce the dimensionality of a feature space and be largely resistant to corrupting the applicability of distance metrics. These metrics also might work better in this lower dimensional space. If you apply this idea and need to justify it, just reach into your NeuroAI toolkit and pull out the Johnson-Lindenstrauss Lemma as your justification." + ] } ], "metadata": { @@ -1539,7 +1619,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4, diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial5.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial5.ipynb index 77480d80e..8c9385c4b 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial5.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/instructor/W1D3_Tutorial5.ipynb @@ -17,17 +17,17 @@ "execution": {} }, "source": [ - "# Bonus Material: Dynamical similarity analysis (DSA)\n", + "# Tutorial 5: Dynamical Similarity Analysis (DSA)\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", "**By Neuromatch Academy**\n", "\n", - "__Content creators:__ Mitchell Ostrow\n", + "__Content creators:__ Mitchell Ostrow, Alex Murphy\n", "\n", - "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk\n", + "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n" + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n" ] }, { @@ -52,7 +52,7 @@ "source": [ "# @title Install and import feedback gadget\n", "\n", - "!pip install vibecheck --quiet\n", + "!pip install vibecheck rsatoolbox --quiet\n", "\n", "from vibecheck import DatatopsContentReviewContainer\n", "def content_review(notebook_section: str):\n", @@ -67,7 +67,2087 @@ " ).render()\n", "\n", "\n", - "feedback_prefix = \"W1D3_Bonus\"" + "feedback_prefix = \"W1D5_DSA\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ef9abaa3", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Helper functions\n", + "\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "\n", + "def generate_2d_random_process(A, B, T=1000):\n", + " \"\"\"\n", + " Generates a 2D random process with the equation x(t+1) = A.x(t) + B.noise.\n", + "\n", + " Args:\n", + " A: 2x2 transition matrix.\n", + " B: 2x2 noise scaling matrix.\n", + " T: Number of time steps.\n", + "\n", + " Returns:\n", + " A NumPy array of shape (T+1, 2) representing the trajectory.\n", + " \"\"\"\n", + " # Assuming equilibrium distribution is zero mean and identity covariance for simplicity.\n", + " # You may adjust this according to your actual equilibrium distribution\n", + " x = np.zeros(2)\n", + "\n", + " trajectory = [x.copy()] # Initialize with x(0)\n", + " for t in range(T):\n", + " noise = np.random.normal(size=2) # Standard normal noise\n", + " x = np.dot(A, x) + np.dot(B, noise)\n", + " trajectory.append(x.copy())\n", + " return np.array(trajectory)\n", + "\n", + "\"\"\"This module computes the Havok DMD model for a given dataset.\"\"\"\n", + "import torch\n", + "\n", + "def embed_signal_torch(data, n_delays, delay_interval=1):\n", + " \"\"\"\n", + " Create a delay embedding from the provided tensor data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : torch.tensor\n", + " The data from which to create the delay embedding. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + " \"\"\"\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " device = data.device\n", + "\n", + " if data.shape[int(data.ndim==3)] - (n_delays - 1)*delay_interval < 1:\n", + " raise ValueError(\"The number of delays is too large for the number of time points in the data!\")\n", + "\n", + " # initialize the embedding\n", + " if data.ndim == 3:\n", + " embedding = torch.zeros((data.shape[0], data.shape[1] - (n_delays - 1)*delay_interval, data.shape[2]*n_delays)).to(device)\n", + " else:\n", + " embedding = torch.zeros((data.shape[0] - (n_delays - 1)*delay_interval, data.shape[1]*n_delays)).to(device)\n", + "\n", + " for d in range(n_delays):\n", + " index = (n_delays - 1 - d)*delay_interval\n", + " ddelay = d*delay_interval\n", + "\n", + " if data.ndim == 3:\n", + " ddata = d*data.shape[2]\n", + " embedding[:,:, ddata: ddata + data.shape[2]] = data[:,index:data.shape[1] - ddelay]\n", + " else:\n", + " ddata = d*data.shape[1]\n", + " embedding[:, ddata:ddata + data.shape[1]] = data[index:data.shape[0] - ddelay]\n", + "\n", + " return embedding\n", + "\n", + "class DMD:\n", + " \"\"\"DMD class for computing and predicting with DMD models.\n", + " \"\"\"\n", + " def __init__(\n", + " self,\n", + " data,\n", + " n_delays,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=False,\n", + " lamb=0,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=False,\n", + " steps_ahead=1\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " reduced_rank_reg : bool\n", + " Determines whether to use reduced rank regression (True) or principal component regression (False)\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0.\n", + "\n", + " device: string, int, or torch.device\n", + " A string, int or torch.device object to indicate the device to torch.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + " \"\"\"\n", + "\n", + " self.device = device\n", + " self._init_data(data)\n", + "\n", + " self.n_delays = n_delays\n", + " self.delay_interval = delay_interval\n", + " self.rank = rank\n", + " self.rank_thresh = rank_thresh\n", + " self.rank_explained_variance = rank_explained_variance\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.lamb = lamb\n", + " self.verbose = verbose\n", + " self.send_to_cpu = send_to_cpu\n", + " self.steps_ahead = steps_ahead\n", + "\n", + " # Hankel matrix\n", + " self.H = None\n", + "\n", + " # SVD attributes\n", + " self.U = None\n", + " self.S = None\n", + " self.V = None\n", + " self.S_mat = None\n", + " self.S_mat_inv = None\n", + "\n", + " # DMD attributes\n", + " self.A_v = None\n", + " self.A_havok_dmd = None\n", + "\n", + " def _init_data(self, data):\n", + " # check if the data is an np.ndarry - if so, convert it to Torch\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " self.data = data\n", + " # create attributes for the data dimensions\n", + " if self.data.ndim == 3:\n", + " self.ntrials = self.data.shape[0]\n", + " self.window = self.data.shape[1]\n", + " self.n = self.data.shape[2]\n", + " else:\n", + " self.window = self.data.shape[0]\n", + " self.n = self.data.shape[1]\n", + " self.ntrials = 1\n", + "\n", + " def compute_hankel(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " ):\n", + " \"\"\"\n", + " Computes the Hankel matrix from the provided data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing Hankel matrix ...\")\n", + "\n", + " # if parameters are provided, overwrite them from the init\n", + " self.data = self.data if data is None else self._init_data(data)\n", + " self.n_delays = self.n_delays if n_delays is None else n_delays\n", + " self.delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " self.data = self.data.to(self.device)\n", + "\n", + " self.H = embed_signal_torch(self.data, self.n_delays, self.delay_interval)\n", + "\n", + " if self.verbose:\n", + " print(\"Hankel matrix computed!\")\n", + "\n", + " def compute_svd(self):\n", + " \"\"\"\n", + " Computes the SVD of the Hankel matrix.\n", + " \"\"\"\n", + "\n", + " if self.verbose:\n", + " print(\"Computing SVD on Hankel matrix ...\")\n", + " if self.H.ndim == 3: #flatten across trials for 3d\n", + " H = self.H.reshape(self.H.shape[0] * self.H.shape[1], self.H.shape[2])\n", + " else:\n", + " H = self.H\n", + " # compute the SVD\n", + " U, S, Vh = torch.linalg.svd(H.T, full_matrices=False)\n", + "\n", + " # update attributes\n", + " V = Vh.T\n", + " self.U = U\n", + " self.S = S\n", + " self.V = V\n", + "\n", + " # construct the singuar value matrix and its inverse\n", + " # dim = self.n_delays * self.n\n", + " # s = len(S)\n", + " # self.S_mat = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " # self.S_mat_inv = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " self.S_mat = torch.diag(S).to(self.device)\n", + " self.S_mat_inv= torch.diag(1 / S).to(self.device)\n", + "\n", + " # compute explained variance\n", + " exp_variance_inds = self.S**2 / ((self.S**2).sum())\n", + " cumulative_explained = torch.cumsum(exp_variance_inds, 0)\n", + " self.cumulative_explained_variance = cumulative_explained\n", + "\n", + " #make the X and Y components of the regression by staggering the hankel eigen-time delay coordinates by time\n", + " if self.reduced_rank_reg:\n", + " V = self.V\n", + " else:\n", + " V = self.V\n", + "\n", + " if self.ntrials > 1:\n", + " if V.numel() < self.H.numel():\n", + " raise ValueError(\"The dimension of the SVD of the Hankel matrix is smaller than the dimension of the Hankel matrix itself. \\n \\\n", + " This is likely due to the number of time points being smaller than the number of dimensions. \\n \\\n", + " Please reduce the number of delays.\")\n", + "\n", + " V = V.reshape(self.H.shape)\n", + "\n", + " #first reshape back into Hankel shape, separated by trials\n", + " newshape = (self.H.shape[0]*(self.H.shape[1]-self.steps_ahead),self.H.shape[2])\n", + " self.Vt_minus = V[:,:-self.steps_ahead].reshape(newshape)\n", + " self.Vt_plus = V[:,self.steps_ahead:].reshape(newshape)\n", + " else:\n", + " self.Vt_minus = V[:-self.steps_ahead]\n", + " self.Vt_plus = V[self.steps_ahead:]\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"SVD complete!\")\n", + "\n", + " def recalc_rank(self,rank,rank_thresh,rank_explained_variance):\n", + " '''\n", + " Parameters\n", + " ----------\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used. Provide only if you want to override the value from the init.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want\n", + " to override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + " '''\n", + " # if an argument was provided, overwrite the stored rank information\n", + " none_vars = (rank is None) + (rank_thresh is None) + (rank_explained_variance is None)\n", + " if none_vars != 3:\n", + " self.rank = None\n", + " self.rank_thresh = None\n", + " self.rank_explained_variance = None\n", + "\n", + " self.rank = self.rank if rank is None else rank\n", + " self.rank_thresh = self.rank_thresh if rank_thresh is None else rank_thresh\n", + " self.rank_explained_variance = self.rank_explained_variance if rank_explained_variance is None else rank_explained_variance\n", + "\n", + " none_vars = (self.rank is None) + (self.rank_thresh is None) + (self.rank_explained_variance is None)\n", + " if none_vars < 2:\n", + " raise ValueError(\"More than one value was provided between rank, rank_thresh, and rank_explained_variance. Please provide only one of these, and ensure the others are None!\")\n", + " elif none_vars == 3:\n", + " self.rank = len(self.S)\n", + "\n", + " if self.reduced_rank_reg:\n", + " S = self.proj_mat_S\n", + " else:\n", + " S = self.S\n", + "\n", + " if rank_thresh is not None:\n", + " if S[-1] > rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < rank_thresh))\n", + "\n", + " if rank_explained_variance is not None:\n", + " self.rank = int(torch.argmax((self.cumulative_explained_variance > rank_explained_variance).type(torch.int)).cpu().numpy())\n", + "\n", + " if self.rank > self.H.shape[-1]:\n", + " self.rank = self.H.shape[-1]\n", + "\n", + " if self.rank is None:\n", + " if S[-1] > self.rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < self.rank_thresh))\n", + "\n", + " def compute_havok_dmd(self,lamb=None):\n", + " \"\"\"\n", + " Computes the Havok DMD matrix (Principal Component Regression)\n", + "\n", + " Parameters\n", + " ----------\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0 - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing least squares fits to HAVOK DMD ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " A_v = (torch.linalg.inv(self.Vt_minus[:, :self.rank].T @ self.Vt_minus[:, :self.rank] + self.lamb*torch.eye(self.rank).to(self.device)) \\\n", + " @ self.Vt_minus[:, :self.rank].T @ self.Vt_plus[:, :self.rank]).T\n", + " self.A_v = A_v\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1], :self.rank] @ self.A_v @ self.S_mat_inv[:self.rank, :self.U.shape[1]] @ self.U.T\n", + "\n", + " if self.verbose:\n", + " print(\"Least squares complete! \\n\")\n", + "\n", + " def compute_proj_mat(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Projector Matrix for Reduced Rank Regression\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " self.proj_mat = self.Vt_plus.T @ self.Vt_minus @ torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus +\n", + " self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ \\\n", + " self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.proj_mat_S, self.proj_mat_V = torch.linalg.eigh(self.proj_mat)\n", + " #todo: more efficient to flip ranks (negative index) in compute_reduced_rank_regression but also less interpretable\n", + " self.proj_mat_S = torch.flip(self.proj_mat_S, dims=(0,))\n", + " self.proj_mat_V = torch.flip(self.proj_mat_V, dims=(1,))\n", + "\n", + " if self.verbose:\n", + " print(\"Projector Matrix computed! \\n\")\n", + "\n", + " def compute_reduced_rank_regression(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Reduced Rank Regression ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + " proj_mat = self.proj_mat_V[:,:self.rank] @ self.proj_mat_V[:,:self.rank].T\n", + " B_ols = torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus + self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.A_v = B_ols @ proj_mat\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1],:self.A_v.shape[1]] @ self.A_v.T @ self.S_mat_inv[:self.A_v.shape[0], :self.U.shape[1]] @ self.U.T\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"Reduced Rank Regression complete! \\n\")\n", + "\n", + " def fit(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " lamb=None,\n", + " device=None,\n", + " verbose=None,\n", + " steps_ahead=None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults to None -\n", + " provide only if you want to override the value from the init.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_thresh : int\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " device: string or int\n", + " A string or int to indicate the device to torch. For example, can be 'cpu' or 'cuda',\n", + " or alternatively 0 if the intenion is to use GPU device 0. Defaults to None - provide only\n", + " if you want to override the value from the init.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + " Defaults to None - provide only if you want to override the value from the init.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + "\n", + " \"\"\"\n", + " # if parameters are provided, overwrite them from the init\n", + " self.steps_ahead = self.steps_ahead if steps_ahead is None else steps_ahead\n", + " self.device = self.device if device is None else device\n", + " self.verbose = self.verbose if verbose is None else verbose\n", + "\n", + " self.compute_hankel(data, n_delays, delay_interval)\n", + " self.compute_svd()\n", + "\n", + " if self.reduced_rank_reg:\n", + " self.compute_proj_mat(lamb)\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_reduced_rank_regression(lamb)\n", + " else:\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_havok_dmd(lamb)\n", + "\n", + " if self.send_to_cpu:\n", + " self.all_to_device('cpu') #send back to the cpu to save memory\n", + "\n", + " def predict(\n", + " self,\n", + " test_data=None,\n", + " reseed=None,\n", + " full_return=False\n", + " ):\n", + " \"\"\"\n", + " Returns\n", + " -------\n", + " pred_data : torch.tensor\n", + " The predictions generated by the HAVOK model. Of the same shape as test_data. Note that the first\n", + " (self.n_delays - 1)*self.delay_interval + 1 time steps of the generated predictions are by construction\n", + " identical to the test_data.\n", + "\n", + " H_test_havok_dmd : torch.tensor (Optional)\n", + " Returned if full_return=True. The predicted Hankel matrix generated by the HAVOK model.\n", + " H_test : torch.tensor (Optional)\n", + " Returned if full_return=True. The true Hankel matrix\n", + " \"\"\"\n", + " # initialize test_data\n", + " if test_data is None:\n", + " test_data = self.data\n", + " if isinstance(test_data, np.ndarray):\n", + " test_data = torch.from_numpy(test_data).to(self.device)\n", + " ndim = test_data.ndim\n", + " if ndim == 2:\n", + " test_data = test_data.unsqueeze(0)\n", + " H_test = embed_signal_torch(test_data, self.n_delays, self.delay_interval)\n", + " steps_ahead = self.steps_ahead if self.steps_ahead is not None else 1\n", + "\n", + " if reseed is None:\n", + " reseed = 1\n", + "\n", + " H_test_havok_dmd = torch.zeros(H_test.shape).to(self.device)\n", + " H_test_havok_dmd[:, :steps_ahead] = H_test[:, :steps_ahead]\n", + "\n", + " A = self.A_havok_dmd.unsqueeze(0)\n", + " for t in range(steps_ahead, H_test.shape[1]):\n", + " if t % reseed == 0:\n", + " H_test_havok_dmd[:, t] = (A @ H_test[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " else:\n", + " H_test_havok_dmd[:, t] = (A @ H_test_havok_dmd[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " pred_data = torch.hstack([test_data[:, :(self.n_delays - 1)*self.delay_interval + steps_ahead], H_test_havok_dmd[:, steps_ahead:, :self.n]])\n", + "\n", + " if ndim == 2:\n", + " pred_data = pred_data[0]\n", + "\n", + " if full_return:\n", + " return pred_data, H_test_havok_dmd, H_test\n", + " else:\n", + " return pred_data\n", + "\n", + " def all_to_device(self,device='cpu'):\n", + " for k,v in self.__dict__.items():\n", + " if isinstance(v, torch.Tensor):\n", + " self.__dict__[k] = v.to(device)\n", + "\n", + "from typing import Literal\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "from typing import Literal\n", + "import torch.nn.utils.parametrize as parametrize\n", + "from scipy.stats import wasserstein_distance\n", + "\n", + "def pad_zeros(A,B,device):\n", + "\n", + " with torch.no_grad():\n", + " dim = max(A.shape[0],B.shape[0])\n", + " A1 = torch.zeros((dim,dim)).float()\n", + " A1[:A.shape[0],:A.shape[1]] += A\n", + " A = A1.float().to(device)\n", + "\n", + " B1 = torch.zeros((dim,dim)).float()\n", + " B1[:B.shape[0],:B.shape[1]] += B\n", + " B = B1.float().to(device)\n", + "\n", + " return A,B\n", + "\n", + "class LearnableSimilarityTransform(nn.Module):\n", + " \"\"\"\n", + " Computes the similarity transform for a learnable orthonormal matrix C\n", + " \"\"\"\n", + " def __init__(self, n,orthog=True):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + " n : int\n", + " dimension of the C matrix\n", + " \"\"\"\n", + " super(LearnableSimilarityTransform, self).__init__()\n", + " #initialize orthogonal matrix as identity\n", + " self.C = nn.Parameter(torch.eye(n).float())\n", + " self.orthog = orthog\n", + "\n", + " def forward(self, B):\n", + " if self.orthog:\n", + " return self.C @ B @ self.C.transpose(-1, -2)\n", + " else:\n", + " return self.C @ B @ torch.linalg.inv(self.C)\n", + "\n", + "class Skew(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a skew-symmetric matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X - X.transpose(-1, -2)\n", + "\n", + "class Matrix(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X\n", + "\n", + "class CayleyMap(nn.Module):\n", + " \"\"\"\n", + " Maps a skew-symmetric matrix to an orthogonal matrix in O(n)\n", + " \"\"\"\n", + " def __init__(self, n, device):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " n : int\n", + " dimension of the matrix we want to map\n", + "\n", + " device : {'cpu','cuda'} or int\n", + " hardware device on which to send the matrix\n", + " \"\"\"\n", + " super().__init__()\n", + " self.register_buffer(\"Id\", torch.eye(n,device = device))\n", + "\n", + " def forward(self, X):\n", + " # (I + X)(I - X)^{-1}\n", + " return torch.linalg.solve(self.Id + X, self.Id - X)\n", + "\n", + "class SimilarityTransformDist:\n", + " \"\"\"\n", + " Computes the Procrustes Analysis over Vector Fields\n", + " \"\"\"\n", + " def __init__(self,\n", + " iters = 200,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 0.01,\n", + " device: Literal[\"cpu\",\"cuda\"] = 'cpu',\n", + " verbose = False,\n", + " group: Literal[\"O(n)\",\"SO(n)\",\"GL(n)\"] = \"O(n)\",\n", + " wasserstein_compare = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " _________\n", + " iters : int\n", + " number of iterations to perform gradient descent\n", + "\n", + " score_method : {\"angular\",\"euclidean\",\"wasserstein\"}\n", + " specifies the type of metric to use\n", + " \"wasserstein\" will compare the singular values or eigenvalues\n", + " of the two matrices as in Redman et al., (2023)\n", + "\n", + " lr : float\n", + " learning rate\n", + "\n", + " device : {'cpu','cuda'} or int\n", + "\n", + " verbose : bool\n", + " prints when finished optimizing\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + "\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.verbose = verbose\n", + " self.device = device\n", + " self.C_star = None\n", + " self.A = None\n", + " self.B = None\n", + " self.group = group\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " def fit(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " group = None,\n", + " ):\n", + " \"\"\"\n", + " Computes the optimal matrix C over specified group\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " Returns\n", + " _______\n", + " None\n", + " \"\"\"\n", + " assert A.shape[0] == A.shape[1]\n", + " assert B.shape[0] == B.shape[1]\n", + "\n", + " A = A.to(self.device)\n", + " B = B.to(self.device)\n", + " self.A,self.B = A,B\n", + " lr = self.lr if lr is None else lr\n", + " iters = self.iters if iters is None else iters\n", + " group = self.group if group is None else group\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if group == \"O(n)\":\n", + " #permute the first row and column of B then rerun the optimization\n", + " P = torch.eye(B.shape[0],device=self.device)\n", + " if P.shape[0] > 1:\n", + " P[[0, 1], :] = P[[1, 0], :]\n", + " losses, C_star, sim_net = self.optimize_C(A,\n", + " P @ B @ P.T,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if losses[-1] < self.losses[-1]:\n", + " self.losses = losses\n", + " self.C_star = C_star @ P\n", + " self.sim_net = sim_net\n", + " if group == \"GL(n)\":\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=False,\n", + " verbose=self.verbose)\n", + "\n", + " def optimize_C(self,A,B,lr,iters,orthog,verbose):\n", + " #parameterize mapping to be orthogonal\n", + " n = A.shape[0]\n", + " sim_net = LearnableSimilarityTransform(n,orthog=orthog).to(self.device)\n", + " if orthog:\n", + " parametrize.register_parametrization(sim_net, \"C\", Skew(n,self.device))\n", + " parametrize.register_parametrization(sim_net, \"C\", CayleyMap(n,self.device))\n", + " else:\n", + " parametrize.register_parametrization(sim_net, \"C\", Matrix(n,self.device))\n", + "\n", + " simdist_loss = nn.MSELoss(reduction = 'sum')\n", + "\n", + " optimizer = optim.Adam(sim_net.parameters(), lr=lr)\n", + " # scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.999)\n", + "\n", + " losses = []\n", + " A /= torch.linalg.norm(A)\n", + " B /= torch.linalg.norm(B)\n", + " for _ in range(iters):\n", + " # Zero the gradients of the optimizer.\n", + " optimizer.zero_grad()\n", + " # Compute the Frobenius norm between A and the product.\n", + " loss = simdist_loss(A, sim_net(B))\n", + "\n", + " loss.backward()\n", + "\n", + " optimizer.step()\n", + " # if _ % 99:\n", + " # scheduler.step()\n", + " losses.append(loss.item())\n", + "\n", + " if verbose:\n", + " print(\"Finished optimizing C\")\n", + "\n", + " C_star = sim_net.C.detach()\n", + " return losses, C_star,sim_net\n", + "\n", + " def score(self,A=None,B=None,score_method=None,group=None):\n", + " \"\"\"\n", + " Given an optimal C already computed, calculate the metric\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor or None\n", + " first data matrix, if None defaults to the saved matrix in fit\n", + " B : np.array or torch.tensor or None\n", + " second data matrix if None, defaults to the savec matrix in fit\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + " \"\"\"\n", + " assert self.C_star is not None\n", + " A = self.A if A is None else A\n", + " B = self.B if B is None else B\n", + " assert A is not None\n", + " assert B is not None\n", + " assert A.shape == self.C_star.shape\n", + " assert B.shape == self.C_star.shape\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + " with torch.no_grad():\n", + " if not isinstance(A,torch.Tensor):\n", + " A = torch.from_numpy(A).float().to(self.device)\n", + " if not isinstance(B,torch.Tensor):\n", + " B = torch.from_numpy(B).float().to(self.device)\n", + " C = self.C_star.to(self.device)\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " Cinv = C.T\n", + " elif group in {\"GL(n)\"}:\n", + " Cinv = torch.linalg.inv(C)\n", + " else:\n", + " raise AssertionError(\"Need proper group name\")\n", + " if score_method == 'angular':\n", + " num = torch.trace(A.T @ C @ B @ Cinv)\n", + " den = torch.norm(A,p = 'fro')*torch.norm(B,p = 'fro')\n", + " score = torch.arccos(num/den).cpu().numpy()\n", + " if np.isnan(score): #around -1 and 1, we sometimes get NaNs due to arccos\n", + " if num/den < 0:\n", + " score = np.pi\n", + " else:\n", + " score = 0\n", + " else:\n", + " score = torch.norm(A - C @ B @ Cinv,p='fro').cpu().numpy().item() #/ A.numpy().size\n", + "\n", + " return score\n", + "\n", + " def fit_score(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " score_method = None,\n", + " zero_pad = True,\n", + " group = None):\n", + " \"\"\"\n", + " for efficiency, computes the optimal matrix and returns the score\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : {'angular','euclidean'} or None\n", + " overwrites parameter in the class\n", + " zero_pad : bool\n", + " if True, then the smaller matrix will be zero padded so its the same size\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + "\n", + " \"\"\"\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + "\n", + " if isinstance(A,np.ndarray):\n", + " A = torch.from_numpy(A).float()\n", + " if isinstance(B,np.ndarray):\n", + " B = torch.from_numpy(B).float()\n", + "\n", + " assert A.shape[0] == B.shape[1] or self.wasserstein_compare is not None\n", + " if A.shape[0] != B.shape[0]:\n", + " if self.wasserstein_compare is None:\n", + " raise AssertionError(\"Matrices must be the same size unless using wasserstein distance\")\n", + " else: #otherwise resort to L2 Wasserstein over singular or eigenvalues\n", + " print(f\"resorting to wasserstein distance over {self.wasserstein_compare}\")\n", + "\n", + " if self.score_method == \"wasserstein\":\n", + " assert self.wasserstein_compare in {\"sv\",\"eig\"}\n", + " if self.wasserstein_compare == \"sv\":\n", + " a = torch.svd(A).S.view(-1,1)\n", + " b = torch.svd(B).S.view(-1,1)\n", + " elif self.wasserstein_compare == \"eig\":\n", + " a = torch.linalg.eig(A).eigenvalues\n", + " a = torch.vstack([a.real,a.imag]).T\n", + "\n", + " b = torch.linalg.eig(B).eigenvalues\n", + " b = torch.vstack([b.real,b.imag]).T\n", + " else:\n", + " raise AssertionError(\"wasserstein_compare must be 'sv' or 'eig'\")\n", + " device = a.device\n", + " a = a#.cpu()\n", + " b = b#.cpu()\n", + " M = ot.dist(a,b)#.numpy()\n", + " a,b = torch.ones(a.shape[0])/a.shape[0],torch.ones(b.shape[0])/b.shape[0]\n", + " a,b = a.to(device),b.to(device)\n", + "\n", + " score_star = ot.emd2(a,b,M)\n", + " #wasserstein_distance(A.cpu().numpy(),B.cpu().numpy())\n", + "\n", + " else:\n", + "\n", + " self.fit(A, B,iters,lr,group)\n", + " score_star = self.score(self.A,self.B,score_method=score_method,group=group)\n", + "\n", + " return score_star\n", + "\n", + "class DSA:\n", + " \"\"\"\n", + " Computes the Dynamical Similarity Analysis (DSA) for two data matrices\n", + " \"\"\"\n", + " def __init__(self,\n", + " X,\n", + " Y=None,\n", + " n_delays=1,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance = None,\n", + " lamb = 0.0,\n", + " send_to_cpu = True,\n", + " iters = 1500,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 5e-3,\n", + " group: Literal[\"GL(n)\", \"O(n)\", \"SO(n)\"] = \"O(n)\",\n", + " zero_pad = False,\n", + " device = 'cpu',\n", + " verbose = False,\n", + " reduced_rank_reg = False,\n", + " kernel=None,\n", + " num_centers=0.1,\n", + " svd_solver='arnoldi',\n", + " wasserstein_compare: Literal['sv','eig',None] = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " X : np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " first data matrix/matrices\n", + "\n", + " Y : None or np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " second data matrix/matrices.\n", + " * If Y is None, X is compared to itself pairwise\n", + " (must be a list)\n", + " * If Y is a single matrix, all matrices in X are compared to Y\n", + " * If Y is a list, all matrices in X are compared to all matrices in Y\n", + "\n", + " DMD parameters:\n", + "\n", + " n_delays : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " number of delays to use in constructing the Hankel matrix\n", + "\n", + " delay_interval : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " interval between samples taken in constructing Hankel matrix\n", + "\n", + " rank : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " rank of DMD matrix fit in reduced-rank regression\n", + "\n", + " rank_thresh : float or list or tuple/list: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float or list or tuple: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " lamb : float\n", + " L-1 regularization parameter in DMD fit\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " NOTE: for all of these above, they can be single values or lists or tuples,\n", + " depending on the corresponding dimensions of the data\n", + " If at least one of X and Y are lists, then if they are a single value\n", + " it will default to the rank of all DMD matrices.\n", + " If they are (int,int), then they will correspond to an individual dmd matrix\n", + " OR to X and Y respectively across all matrices\n", + " If it is (list,list), then each element will correspond to an individual\n", + " dmd matrix indexed at the same position\n", + "\n", + " SimDist parameters:\n", + "\n", + " iters : int\n", + " number of optimization iterations in Procrustes over vector fields\n", + "\n", + " score_method : {'angular','euclidean'}\n", + " type of metric to compute, angular vs euclidean distance\n", + "\n", + " lr : float\n", + " learning rate of the Procrustes over vector fields optimization\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " zero_pad : bool\n", + " whether or not to zero-pad if the dimensions are different\n", + "\n", + " device : 'cpu' or 'cuda' or int\n", + " hardware to use in both DMD and PoVF\n", + "\n", + " verbose : bool\n", + " whether or not print when sections of the analysis is completed\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + " self.X = X\n", + " self.Y = Y\n", + " if self.X is None and isinstance(self.Y,list):\n", + " self.X, self.Y = self.Y, self.X #swap so code is easy\n", + "\n", + " self.check_method()\n", + " if self.method == 'self-pairwise':\n", + " self.data = [self.X]\n", + " else:\n", + " self.data = [self.X, self.Y]\n", + "\n", + " self.n_delays = self.broadcast_params(n_delays,cast=int)\n", + " self.delay_interval = self.broadcast_params(delay_interval,cast=int)\n", + " self.rank = self.broadcast_params(rank,cast=int)\n", + " self.rank_thresh = self.broadcast_params(rank_thresh)\n", + " self.rank_explained_variance = self.broadcast_params(rank_explained_variance)\n", + " self.lamb = self.broadcast_params(lamb)\n", + " self.send_to_cpu = send_to_cpu\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.device = device\n", + " self.verbose = verbose\n", + " self.zero_pad = zero_pad\n", + " self.group = group\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.kernel = kernel\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " if kernel is None:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[DMD(Xi,\n", + " self.n_delays[i][j],\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " rank_thresh=self.rank_thresh[i][j],\n", + " rank_explained_variance=self.rank_explained_variance[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " device=self.device,\n", + " verbose=self.verbose,\n", + " send_to_cpu=self.send_to_cpu) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + " else:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[KernelDMD(Xi,\n", + " self.n_delays[i][j],\n", + " kernel=self.kernel,\n", + " num_centers=num_centers,\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " verbose=self.verbose,\n", + " svd_solver=svd_solver,\n", + " ) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + "\n", + " self.simdist = SimilarityTransformDist(iters,score_method,lr,device,verbose,group,wasserstein_compare)\n", + "\n", + " def check_method(self):\n", + " '''\n", + " helper function to identify what type of dsa we're running\n", + " '''\n", + " tensor_or_np = lambda x: isinstance(x,(np.ndarray,torch.Tensor))\n", + "\n", + " if isinstance(self.X,list):\n", + " if self.Y is None:\n", + " self.method = 'self-pairwise'\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'bipartite-pairwise'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'list-to-one'\n", + " self.Y = [self.Y] #wrap in a list for iteration\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " elif tensor_or_np(self.X):\n", + " self.X = [self.X]\n", + " if self.Y is None:\n", + " raise ValueError('only one element provided')\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'one-to-list'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'default'\n", + " self.Y = [self.Y]\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " else:\n", + " raise ValueError('unknown type of X')\n", + "\n", + " def broadcast_params(self,param,cast=None):\n", + " '''\n", + " aligns the dimensionality of the parameters with the data so it's one-to-one\n", + " '''\n", + " out = []\n", + " if isinstance(param,(int,float,np.integer)) or param is None: #self.X has already been mapped to [self.X]\n", + " out.append([param] * len(self.X))\n", + " if self.Y is not None:\n", + " out.append([param] * len(self.Y))\n", + " elif isinstance(param,(tuple,list,np.ndarray)):\n", + " if self.method == 'self-pairwise' and len(param) >= len(self.X):\n", + " out = [param]\n", + " else:\n", + " assert len(param) <= 2 #only 2 elements max\n", + "\n", + " #if the inner terms are singly valued, we broadcast, otherwise needs to be the same dimensions\n", + " for i,data in enumerate([self.X,self.Y]):\n", + " if data is None:\n", + " continue\n", + " if isinstance(param[i],(int,float)):\n", + " out.append([param[i]] * len(data))\n", + " elif isinstance(param[i],(list,np.ndarray,tuple)):\n", + " assert len(param[i]) >= len(data)\n", + " out.append(param[i][:len(data)])\n", + " else:\n", + " raise ValueError(\"unknown type entered for parameter\")\n", + "\n", + " if cast is not None and param is not None:\n", + " out = [[cast(x) for x in dat] for dat in out]\n", + "\n", + " return out\n", + "\n", + " def fit_dmds(self,\n", + " X=None,\n", + " Y=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh = None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=None,\n", + " lamb = None,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=True\n", + " ):\n", + " \"\"\"\n", + " Recomputes only the DMDs with a single set of hyperparameters. This will not compare, that will need to be done with the full procedure\n", + " \"\"\"\n", + " X = self.X if X is None else X\n", + " Y = self.Y if Y is None else Y\n", + " n_delays = self.n_delays if n_delays is None else n_delays\n", + " delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " rank = self.rank if rank is None else rank\n", + " lamb = self.lamb if lamb is None else lamb\n", + " data = []\n", + " if isinstance(X,list):\n", + " data.append(X)\n", + " else:\n", + " data.append([X])\n", + " if Y is not None:\n", + " if isinstance(Y,list):\n", + " data.append(Y)\n", + " else:\n", + " data.append([Y])\n", + "\n", + " dmds = [[DMD(Xi,n_delays,delay_interval,\n", + " rank,rank_thresh,rank_explained_variance,reduced_rank_reg,\n", + " lamb,device,verbose,send_to_cpu) for Xi in dat] for dat in data]\n", + "\n", + " for dmd_sets in dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return dmds\n", + "\n", + " def fit_score(self):\n", + " \"\"\"\n", + " Standard fitting function for both DMDs and PoVF\n", + "\n", + " Parameters\n", + " __________\n", + "\n", + " Returns\n", + " _______\n", + "\n", + " sims : np.array\n", + " data matrix of the similarity scores between the specific sets of data\n", + " \"\"\"\n", + " for dmd_sets in self.dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return self.score()\n", + "\n", + " def score(self,iters=None,lr=None,score_method=None):\n", + " \"\"\"\n", + " Rescore DSA with precomputed dmds if you want to try again\n", + "\n", + " Parameters\n", + " __________\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + "\n", + " Returns\n", + " ________\n", + " score : float\n", + " similarity score of the two precomputed DMDs\n", + " \"\"\"\n", + "\n", + " iters = self.iters if iters is None else iters\n", + " lr = self.lr if lr is None else lr\n", + " score_method = self.score_method if score_method is None else score_method\n", + "\n", + " ind2 = 1 - int(self.method == 'self-pairwise')\n", + " # 0 if self.pairwise (want to compare the set to itself)\n", + "\n", + " self.sims = np.zeros((len(self.dmds[0]),len(self.dmds[ind2])))\n", + " for i,dmd1 in enumerate(self.dmds[0]):\n", + " for j,dmd2 in enumerate(self.dmds[ind2]):\n", + " if self.method == 'self-pairwise':\n", + " if j >= i:\n", + " continue\n", + " if self.verbose:\n", + " print(f'computing similarity between DMDs {i} and {j}')\n", + "\n", + " self.sims[i,j] = self.simdist.fit_score(dmd1.A_v,dmd2.A_v,iters,lr,score_method,zero_pad=self.zero_pad)\n", + "\n", + " if self.method == 'self-pairwise':\n", + " self.sims[j,i] = self.sims[i,j]\n", + "\n", + "\n", + " if self.method == 'default':\n", + " return self.sims[0,0]\n", + "\n", + " return self.sims" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eced3162", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Helper functions (Bonus Section)\n", + "\n", + "import contextlib\n", + "import io\n", + "import argparse\n", + "# Standard library imports\n", + "from collections import OrderedDict\n", + "import logging\n", + "\n", + "# External libraries: General utilities\n", + "import argparse\n", + "import numpy as np\n", + "\n", + "# PyTorch related imports\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import torch.optim as optim\n", + "from torch.optim.lr_scheduler import StepLR\n", + "from torchvision import datasets, transforms\n", + "from torchvision.models.feature_extraction import create_feature_extractor, get_graph_node_names\n", + "from torchvision.utils import make_grid\n", + "\n", + "# Matplotlib for plotting\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "# SciPy for statistical functions\n", + "from scipy import stats\n", + "\n", + "# Scikit-Learn for machine learning utilities\n", + "from sklearn.decomposition import PCA\n", + "from sklearn import manifold\n", + "\n", + "# RSA toolbox specific imports\n", + "import rsatoolbox\n", + "from rsatoolbox.data import Dataset\n", + "from rsatoolbox.rdm.calc import calc_rdm\n", + "\n", + "class Net(nn.Module):\n", + " \"\"\"\n", + " A neural network model for image classification, consisting of two convolutional layers,\n", + " followed by two fully connected layers with dropout regularization.\n", + "\n", + " Methods:\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self):\n", + " \"\"\"\n", + " Initializes the network layers.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 32 output channels, and a 3x3 kernel.\n", + " - conv2: Second convolutional layer with 32 input channels, 64 output channels, and a 3x3 kernel.\n", + " - dropout1: Dropout layer with a dropout probability of 0.25.\n", + " - dropout2: Dropout layer with a dropout probability of 0.5.\n", + " - fc1: First fully connected layer with 9216 input features and 128 output features.\n", + " - fc2: Second fully connected layer with 128 input features and 10 output features.\n", + " \"\"\"\n", + " super(Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 32, 3, 1)\n", + " self.conv2 = nn.Conv2d(32, 64, 3, 1)\n", + " self.dropout1 = nn.Dropout(0.25)\n", + " self.dropout2 = nn.Dropout(0.5)\n", + " self.fc1 = nn.Linear(9216, 128)\n", + " self.fc2 = nn.Linear(128, 10)\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " x = self.conv1(input)\n", + " x = F.relu(x)\n", + " x = self.conv2(x)\n", + " x = F.relu(x)\n", + " x = F.max_pool2d(x, 2)\n", + " x = self.dropout1(x)\n", + " x = torch.flatten(x, 1)\n", + " x = self.fc1(x)\n", + " x = F.relu(x)\n", + " x = self.dropout2(x)\n", + " x = self.fc2(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "class recurrent_Net(nn.Module):\n", + " \"\"\"\n", + " A recurrent neural network model for image classification, consisting of two convolutional layers\n", + " with recurrent connections and a readout layer.\n", + "\n", + " Methods:\n", + " - __init__(time_steps=5): Initializes the network layers and sets the number of time steps for recurrence.\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self, time_steps=5):\n", + " \"\"\"\n", + " Initializes the network layers and sets the number of time steps for recurrence.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 16 output channels, and a 3x3 kernel with a stride of 3.\n", + " - conv2: Second convolutional layer with 16 input channels, 16 output channels, and a 3x3 kernel with padding of 1.\n", + " - readout: A sequential layer containing:\n", + " - dropout: Dropout layer with a dropout probability of 0.25.\n", + " - avgpool: Adaptive average pooling layer to reduce spatial dimensions to 1x1.\n", + " - flatten: Flatten layer to convert the 2D pooled output to 1D.\n", + " - linear: Fully connected layer with 16 input features and 10 output features.\n", + " - time_steps (int): Number of time steps for the recurrent connection.\n", + " \"\"\"\n", + " super(recurrent_Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 16, 3, 3)\n", + " self.conv2 = nn.Conv2d(16, 16, 3, 1, padding=1)\n", + " self.readout = nn.Sequential(OrderedDict([\n", + " ('dropout', nn.Dropout(0.25)),\n", + " ('avgpool', nn.AdaptiveAvgPool2d(1)),\n", + " ('flatten', nn.Flatten()),\n", + " ('linear', nn.Linear(16, 10))\n", + " ]))\n", + " self.time_steps = time_steps\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " input = self.conv1(input)\n", + " x = input\n", + " for t in range(0, self.time_steps):\n", + " x = input + self.conv2(x)\n", + " x = F.relu(x)\n", + "\n", + " x = self.readout(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "\n", + "def train_one_epoch(args, model, device, train_loader, optimizer, epoch):\n", + " \"\"\"\n", + " Trains the model for one epoch.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Arguments for training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - device (torch.device): The device to use for training (CPU/GPU).\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + " - epoch (int): The current epoch number.\n", + " \"\"\"\n", + " model.train()\n", + " for batch_idx, (data, target) in enumerate(train_loader):\n", + " data, target = data.to(device), target.to(device)\n", + " optimizer.zero_grad()\n", + " output = model(data)\n", + " output = torch.log(output) # to make it a log_softmax\n", + " loss = F.nll_loss(output, target)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if batch_idx % args.log_interval == 0:\n", + " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", + " epoch, batch_idx * len(data), len(train_loader.dataset),\n", + " 100. * batch_idx / len(train_loader), loss.item()))\n", + " if args.dry_run:\n", + " break\n", + "\n", + "def test(model, device, test_loader, return_features=False):\n", + " \"\"\"\n", + " Evaluates the model on the test dataset.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be evaluated.\n", + " - device (torch.device): The device to use for evaluation (CPU/GPU).\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " - return_features (bool): If True, returns the features from the model. Default is False.\n", + " \"\"\"\n", + " model.eval()\n", + " test_loss = 0\n", + " correct = 0\n", + " with torch.no_grad():\n", + " for data, target in test_loader:\n", + " data, target = data.to(device), target.to(device)\n", + " output = model(data)\n", + " output = torch.log(output)\n", + " test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(target.view_as(pred)).sum().item()\n", + "\n", + " test_loss /= len(test_loader.dataset)\n", + "\n", + " print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n", + " test_loss, correct, len(test_loader.dataset),\n", + " 100. * correct / len(test_loader.dataset)))\n", + "\n", + "def build_args():\n", + " \"\"\"\n", + " Builds and parses command-line arguments for training.\n", + " \"\"\"\n", + " parser = argparse.ArgumentParser(description='PyTorch MNIST Example')\n", + " parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n", + " help='input batch size for training (default: 64)')\n", + " parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n", + " help='input batch size for testing (default: 1000)')\n", + " parser.add_argument('--epochs', type=int, default=2, metavar='N',\n", + " help='number of epochs to train (default: 14)')\n", + " parser.add_argument('--lr', type=float, default=1.0, metavar='LR',\n", + " help='learning rate (default: 1.0)')\n", + " parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n", + " help='Learning rate step gamma (default: 0.7)')\n", + " parser.add_argument('--no-cuda', action='store_true', default=False,\n", + " help='disables CUDA training')\n", + " parser.add_argument('--no-mps', action='store_true', default=False,\n", + " help='disables macOS GPU training')\n", + " parser.add_argument('--dry-run', action='store_true', default=False,\n", + " help='quickly check a single pass')\n", + " parser.add_argument('--seed', type=int, default=1, metavar='S',\n", + " help='random seed (default: 1)')\n", + " parser.add_argument('--log-interval', type=int, default=50, metavar='N',\n", + " help='how many batches to wait before logging training status')\n", + " parser.add_argument('--save-model', action='store_true', default=False,\n", + " help='For Saving the current Model')\n", + " args = parser.parse_args('')\n", + "\n", + " use_cuda = torch.cuda.is_available() #not args.no_cuda and\n", + "\n", + " if use_cuda:\n", + " device = torch.device(\"cuda\")\n", + " else:\n", + " device = torch.device(\"cpu\")\n", + "\n", + " args.use_cuda = use_cuda\n", + " args.device = device\n", + " return args\n", + "\n", + "def fetch_dataloaders(args):\n", + " \"\"\"\n", + " Fetches the data loaders for training and testing datasets.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + "\n", + " Outputs:\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " \"\"\"\n", + " train_kwargs = {'batch_size': args.batch_size}\n", + " test_kwargs = {'batch_size': args.test_batch_size}\n", + " if args.use_cuda:\n", + " cuda_kwargs = {'num_workers': 1,\n", + " 'pin_memory': True,\n", + " 'shuffle': True}\n", + " train_kwargs.update(cuda_kwargs)\n", + " test_kwargs.update(cuda_kwargs)\n", + "\n", + " transform=transforms.Compose([\n", + " transforms.ToTensor(),\n", + " transforms.Normalize((0.1307,), (0.3081,))\n", + " ])\n", + " with contextlib.redirect_stdout(io.StringIO()): #to suppress output\n", + " dataset1 = datasets.MNIST('../data', train=True, download=True,\n", + " transform=transform)\n", + " dataset2 = datasets.MNIST('../data', train=False,\n", + " transform=transform)\n", + " train_loader = torch.utils.data.DataLoader(dataset1, **train_kwargs)\n", + " test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)\n", + " return train_loader, test_loader\n", + "\n", + "def train_model(args, model, optimizer):\n", + " \"\"\"\n", + " Trains the model using the specified arguments and optimizer.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + "\n", + " Outputs:\n", + " - None: The function trains the model and optionally saves it.\n", + " \"\"\"\n", + " scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n", + " for epoch in range(1, args.epochs + 1):\n", + " train_one_epoch(args, model, args.device, train_loader, optimizer, epoch)\n", + " test(model, args.device, test_loader)\n", + " scheduler.step()\n", + "\n", + " if args.save_model:\n", + " torch.save(model.state_dict(), \"mnist_cnn.pt\")\n", + "\n", + "\n", + "def calc_rdms(model_features, method='correlation'):\n", + " \"\"\"\n", + " Calculates representational dissimilarity matrices (RDMs) for model features.\n", + "\n", + " Inputs:\n", + " - model_features (dict): A dictionary where keys are layer names and values are features of the layers.\n", + " - method (str): The method to calculate RDMs, e.g., 'correlation'. Default is 'correlation'.\n", + "\n", + " Outputs:\n", + " - rdms (pyrsa.rdm.RDMs): RDMs object containing dissimilarity matrices.\n", + " - rdms_dict (dict): A dictionary with layer names as keys and their corresponding RDMs as values.\n", + " \"\"\"\n", + " ds_list = []\n", + " for l in range(len(model_features)):\n", + " layer = list(model_features.keys())[l]\n", + " feats = model_features[layer]\n", + "\n", + " if type(feats) is list:\n", + " feats = feats[-1]\n", + "\n", + " if args.use_cuda:\n", + " feats = feats.cpu()\n", + "\n", + " if len(feats.shape) > 2:\n", + " feats = feats.flatten(1)\n", + "\n", + " feats = feats.detach().numpy()\n", + " ds = Dataset(feats, descriptors=dict(layer=layer))\n", + " ds_list.append(ds)\n", + "\n", + " rdms = calc_rdm(ds_list, method=method)\n", + " rdms_dict = {list(model_features.keys())[i]: rdms.get_matrices()[i] for i in range(len(model_features))}\n", + "\n", + " return rdms, rdms_dict\n", + "\n", + "def fgsm_attack(image, epsilon, data_grad):\n", + " \"\"\"\n", + " Performs FGSM attack on an image.\n", + "\n", + " Inputs:\n", + " - image (torch.Tensor): Original image.\n", + " - epsilon (float): Perturbation magnitude.\n", + " - data_grad (torch.Tensor): Gradient of the data.\n", + "\n", + " Outputs:\n", + " - perturbed_image (torch.Tensor): Perturbed image after FGSM attack.\n", + " \"\"\"\n", + " sign_data_grad = data_grad.sign()\n", + " perturbed_image = image + epsilon * sign_data_grad\n", + " perturbed_image = torch.clamp(perturbed_image, 0, 1)\n", + " return perturbed_image\n", + "\n", + "def denorm(batch, mean=[0.1307], std=[0.3081]):\n", + " \"\"\"\n", + " Converts a batch of normalized tensors to their original scale.\n", + "\n", + " Inputs:\n", + " - batch (torch.Tensor): Batch of normalized tensors.\n", + " - mean (torch.Tensor or list): Mean used for normalization.\n", + " - std (torch.Tensor or list): Standard deviation used for normalization.\n", + "\n", + " Outputs:\n", + " - torch.Tensor: Batch of tensors without normalization applied to them.\n", + " \"\"\"\n", + " if isinstance(mean, list):\n", + " mean = torch.tensor(mean).to(batch.device)\n", + " if isinstance(std, list):\n", + " std = torch.tensor(std).to(batch.device)\n", + "\n", + " return batch * std.view(1, -1, 1, 1) + mean.view(1, -1, 1, 1)\n", + "\n", + "def generate_adversarial(model, imgs, targets, epsilon):\n", + " \"\"\"\n", + " Generates adversarial examples using FGSM attack.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to attack.\n", + " - imgs (torch.Tensor): Batch of images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " - epsilon (float): Perturbation magnitude.\n", + "\n", + " Outputs:\n", + " - adv_imgs (torch.Tensor): Batch of adversarial images.\n", + " \"\"\"\n", + " adv_imgs = []\n", + "\n", + " for img, target in zip(imgs, targets):\n", + " img = img.unsqueeze(0)\n", + " target = target.unsqueeze(0)\n", + " img.requires_grad = True\n", + "\n", + " output = model(img)\n", + " output = torch.log(output)\n", + " loss = F.nll_loss(output, target)\n", + "\n", + " model.zero_grad()\n", + " loss.backward()\n", + "\n", + " data_grad = img.grad.data\n", + " data_denorm = denorm(img)\n", + " perturbed_data = fgsm_attack(data_denorm, epsilon, data_grad)\n", + " perturbed_data_normalized = transforms.Normalize((0.1307,), (0.3081,))(perturbed_data)\n", + "\n", + " adv_imgs.append(perturbed_data_normalized.detach())\n", + "\n", + " return torch.cat(adv_imgs)\n", + "\n", + "def test_adversarial(model, imgs, targets):\n", + " \"\"\"\n", + " Tests the model on adversarial examples and prints the accuracy.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be tested.\n", + " - imgs (torch.Tensor): Batch of adversarial images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " \"\"\"\n", + " correct = 0\n", + " output = model(imgs)\n", + " output = torch.log(output)\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(targets.view_as(pred)).sum().item()\n", + "\n", + " final_acc = correct / float(len(imgs))\n", + " print(f\"adversarial test accuracy = {correct} / {len(imgs)} = {final_acc}\")\n", + "\n", + "def extract_features(model, imgs, return_layers, plot='none'):\n", + " \"\"\"\n", + " Extracts features from specified layers of the model.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model from which to extract features.\n", + " - imgs (torch.Tensor): Batch of input images.\n", + " - return_layers (list): List of layer names from which to extract features.\n", + " - plot (str): Option to plot the features. Default is 'none'.\n", + "\n", + " Outputs:\n", + " - model_features (dict): A dictionary with layer names as keys and extracted features as values.\n", + " \"\"\"\n", + " if return_layers == 'all':\n", + " return_layers, _ = get_graph_node_names(model)\n", + " elif return_layers == 'layers':\n", + " layers, _ = get_graph_node_names(model)\n", + " return_layers = [l for l in layers if 'input' in l or 'conv' in l or 'fc' in l]\n", + "\n", + " feature_extractor = create_feature_extractor(model, return_nodes=return_layers)\n", + " model_features = feature_extractor(imgs)\n", + "\n", + " return model_features" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be4a4946", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Plotting functions (Bonus)\n", + "\n", + "def sample_images(data_loader, n=5, plot=False):\n", + " \"\"\"\n", + " Samples a specified number of images from a data loader.\n", + "\n", + " Inputs:\n", + " - data_loader (torch.utils.data.DataLoader): Data loader containing images and labels.\n", + " - n (int): Number of images to sample per class.\n", + " - plot (bool): Whether to plot the sampled images using matplotlib.\n", + "\n", + " Outputs:\n", + " - imgs (torch.Tensor): Sampled images.\n", + " - labels (torch.Tensor): Corresponding labels for the sampled images.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " imgs, targets = next(iter(data_loader))\n", + "\n", + " imgs_o = []\n", + " labels = []\n", + " for value in range(10):\n", + " cat_imgs = imgs[np.where(targets == value)][0:n]\n", + " imgs_o.append(cat_imgs)\n", + " labels.append([value]*len(cat_imgs))\n", + "\n", + " imgs = torch.cat(imgs_o, dim=0)\n", + " labels = torch.tensor(labels).flatten()\n", + "\n", + " if plot:\n", + " plt.imshow(torch.moveaxis(make_grid(imgs, nrow=5, padding=0, normalize=False, pad_value=0), 0,-1))\n", + " plt.axis('off')\n", + "\n", + " return imgs, labels\n", + "\n", + "\n", + "def plot_rdms(model_rdms):\n", + " \"\"\"\n", + " Plots the Representational Dissimilarity Matrices (RDMs) for each layer of a model.\n", + "\n", + " Inputs:\n", + " - model_rdms (dict): A dictionary where keys are layer names and values are the corresponding RDMs.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, len(model_rdms))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " for l in range(len(model_rdms)):\n", + "\n", + " layer = list(model_rdms.keys())[l]\n", + " rdm = np.squeeze(model_rdms[layer])\n", + "\n", + " if len(rdm.shape) < 2:\n", + " rdm = rdm.reshape( (int(np.sqrt(rdm.shape[0])), int(np.sqrt(rdm.shape[0]))) )\n", + "\n", + " rdm = rdm / np.max(rdm)\n", + "\n", + " ax = plt.subplot(gs[0,l])\n", + " ax_ = ax.imshow(rdm, cmap='magma_r')\n", + " ax.set_title(f'{layer}')\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " cbar_ax.text(-2.3, 0.05, 'Normalized euclidean distance', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax)\n", + "\n", + " plt.show()\n", + "\n", + "def rep_path(model_features, model_colors, labels=None, rdm_calc_method='euclidean', rdm_comp_method='cosine'):\n", + " \"\"\"\n", + " Represents paths of model features in a reduced-dimensional space.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each model.\n", + " - model_colors (dict): Dictionary mapping model names to colors for visualization.\n", + " - labels (array-like, optional): Array of labels corresponding to the model features.\n", + " - rdm_calc_method (str, optional): Method for calculating RDMS ('euclidean' or 'correlation').\n", + " - rdm_comp_method (str, optional): Method for comparing RDMS ('cosine' or 'corr').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + " path_len = []\n", + " path_colors = []\n", + " rdms_list = []\n", + " ax_ticks = []\n", + " tick_colors = []\n", + " model_names = list(model_features.keys())\n", + " for m in range(len(model_names)):\n", + " model_name = model_names[m]\n", + " features = model_features[model_name]\n", + " path_colors.append(model_colors[model_name])\n", + " path_len.append(len(features))\n", + " ax_ticks.append(list(features.keys()))\n", + " tick_colors.append([model_colors[model_name]]*len(features))\n", + " rdms, _ = calc_rdms(features, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + "\n", + " path_len = np.insert(np.cumsum(path_len),0,0)\n", + "\n", + " if labels is not None:\n", + " rdms, _ = calc_rdms({'labels' : F.one_hot(labels).float().to(device)}, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + " ax_ticks.append(['labels'])\n", + " tick_colors.append(['m'])\n", + " idx_labels = -1\n", + "\n", + " rdms = rsatoolbox.rdm.concat(rdms_list)\n", + "\n", + " #Flatten the list\n", + " ax_ticks = [l for model_layers in ax_ticks for l in model_layers]\n", + " tick_colors = [l for model_layers in tick_colors for l in model_layers]\n", + " tick_colors = ['k' if tick == 'input' else color for tick, color in zip(ax_ticks, tick_colors)]\n", + "\n", + " rdms_comp = rsatoolbox.rdm.compare(rdms, rdms, method=rdm_comp_method)\n", + " if rdm_comp_method == 'cosine':\n", + " rdms_comp = np.arccos(rdms_comp)\n", + " rdms_comp = np.nan_to_num(rdms_comp, nan=0.0)\n", + "\n", + " # Symmetrize\n", + " rdms_comp = (rdms_comp + rdms_comp.T) / 2.0\n", + "\n", + " # reduce dim to 2\n", + " transformer = manifold.MDS(n_components = 2, max_iter=1000, n_init=10, normalized_stress='auto', dissimilarity=\"precomputed\")\n", + " dims= transformer.fit_transform(rdms_comp)\n", + "\n", + " # remove duplicates of the input layer from multiple models\n", + " remove_duplicates = np.where(np.array(ax_ticks) == 'input')[0][1:]\n", + " for index in remove_duplicates:\n", + " del ax_ticks[index]\n", + " del tick_colors[index]\n", + " rdms_comp = np.delete(np.delete(rdms_comp, index, axis=0), index, axis=1)\n", + "\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, 2)\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " ax = plt.subplot(gs[0,0])\n", + " ax_ = ax.imshow(rdms_comp, cmap='viridis_r')\n", + " fig.subplots_adjust(left=0.2)\n", + " cbar_ax = fig.add_axes([-0.01, 0.2, 0.01, 0.5])\n", + " #cbar_ax.text(-7, 0.05, 'dissimilarity between rdms', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax,location='left')\n", + " ax.set_title('Dissimilarity between layer rdms', fontdict = {'fontsize': 14})\n", + " ax.set_xticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7, rotation=83)\n", + " ax.set_yticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7)\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.xaxis.get_ticklabels())]\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.yaxis.get_ticklabels())]\n", + "\n", + " ax = plt.subplot(gs[0,1])\n", + " amin, amax = dims.min(), dims.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + "\n", + " for i in range(len(rdms_list)-1):\n", + "\n", + " path_indices = np.arange(path_len[i], path_len[i+1])\n", + " ax.plot(dims[path_indices, 0], dims[path_indices, 1], color=path_colors[i], marker='.')\n", + " ax.set_title('Representational geometry path', fontdict = {'fontsize': 14})\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.set_xlabel(f\"dim 1\")\n", + " ax.set_ylabel(f\"dim 2\")\n", + "\n", + " # if idx_input is not None:\n", + " idx_input = 0\n", + " ax.plot(dims[idx_input, 0], dims[idx_input, 1], color='k', marker='s')\n", + "\n", + " if labels is not None:\n", + " ax.plot(dims[idx_labels, 0], dims[idx_labels, 1], color='m', marker='*')\n", + "\n", + " ax.legend(model_names, fontsize=8)\n", + " fig.tight_layout()\n", + "\n", + "def plot_dim_reduction(model_features, labels, transformer_funcs):\n", + " \"\"\"\n", + " Plots the dimensionality reduction results for model features using various transformers.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each layer.\n", + " - labels (array-like): Array of labels corresponding to the model features.\n", + " - transformer_funcs (list): List of dimensionality reduction techniques to apply ('PCA', 'MDS', 't-SNE').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + "\n", + " transformers = []\n", + " for t in transformer_funcs:\n", + " if t == 'PCA': transformers.append(PCA(n_components=2))\n", + " if t == 'MDS': transformers.append(manifold.MDS(n_components = 2, normalized_stress='auto'))\n", + " if t == 't-SNE': transformers.append(manifold.TSNE(n_components = 2, perplexity=40, verbose=0))\n", + "\n", + " fig = plt.figure(figsize=(8, 2.5*len(transformers)))\n", + " # and we add one plot per reference point\n", + " gs = fig.add_gridspec(len(transformers), len(model_features))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " return_layers = list(model_features.keys())\n", + "\n", + " for f in range(len(transformer_funcs)):\n", + "\n", + " for l in range(len(return_layers)):\n", + " layer = return_layers[l]\n", + " feats = model_features[layer].detach().cpu().flatten(1)\n", + " feats_transformed= transformers[f].fit_transform(feats)\n", + "\n", + " amin, amax = feats_transformed.min(), feats_transformed.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + " ax = plt.subplot(gs[f,l])\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.axis(\"off\")\n", + " #ax.set_title(f'{layer}')\n", + " if f == 0: ax.text(0.5, 1.12, f'{layer}', size=16, ha=\"center\", transform=ax.transAxes)\n", + " if l == 0: ax.text(-0.3, 0.5, transformer_funcs[f], size=16, ha=\"center\", transform=ax.transAxes)\n", + " # Create a discrete color map based on unique labels\n", + " num_colors = len(np.unique(labels))\n", + " cmap = plt.get_cmap('viridis_r', num_colors) # 10 discrete colors\n", + " norm = mpl.colors.BoundaryNorm(np.arange(-0.5,num_colors), cmap.N)\n", + " ax_ = ax.scatter(feats_transformed[:, 0], feats_transformed[:, 1], c=labels, cmap=cmap, norm=norm)\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " fig.colorbar(ax_, cax=cbar_ax, ticks=np.linspace(0,9,10))\n", + " plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21f68945", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Data retrieval\n", + "\n", + "import os\n", + "import requests\n", + "import hashlib\n", + "\n", + "# Variables for file and download URL\n", + "fnames = [\"standard_model.pth\", \"adversarial_model.pth\", \"recurrent_model.pth\"] # The names of the files to be downloaded\n", + "urls = [\"https://osf.io/s5rt6/download\", \"https://osf.io/qv5eb/download\", \"https://osf.io/6hnwk/download\"] # URLs from where the files will be downloaded\n", + "expected_md5s = [\"2e63c2cd77bc9f1fa67673d956ec910d\", \"25fb34497377921b54368317f68a7aa7\", \"ee5cea3baa264cb78300102fa6ed66e8\"] # MD5 hashes for verifying files integrity\n", + "\n", + "for fname, url, expected_md5 in zip(fnames, urls, expected_md5s):\n", + " if not os.path.isfile(fname):\n", + " try:\n", + " # Attempt to download the file\n", + " r = requests.get(url) # Make a GET request to the specified URL\n", + " except requests.ConnectionError:\n", + " # Handle connection errors during the download\n", + " print(\"!!! Failed to download data !!!\")\n", + " else:\n", + " # No connection errors, proceed to check the response\n", + " if r.status_code != requests.codes.ok:\n", + " # Check if the HTTP response status code indicates a successful download\n", + " print(\"!!! Failed to download data !!!\")\n", + " elif hashlib.md5(r.content).hexdigest() != expected_md5:\n", + " # Verify the integrity of the downloaded file using MD5 checksum\n", + " print(\"!!! Data download appears corrupted !!!\")\n", + " else:\n", + " # If download is successful and data is not corrupted, save the file\n", + " with open(fname, \"wb\") as fid:\n", + " fid.write(r.content) # Write the downloaded content to a file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "93aeca0a", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Figure settings\n", + "\n", + "logging.getLogger('matplotlib.font_manager').disabled = True\n", + "\n", + "%matplotlib inline\n", + "%config InlineBackend.figure_format = 'retina' # perfrom high definition rendering for images and plots\n", + "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/main/nma.mplstyle\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dd8052d5", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Set device (GPU or CPU)\n", + "\n", + "# inform the user if the notebook uses GPU or CPU.\n", + "\n", + "def set_device():\n", + " \"\"\"\n", + " Determines and sets the computational device for PyTorch operations based on the availability of a CUDA-capable GPU.\n", + "\n", + " Outputs:\n", + " - device (str): The device that PyTorch will use for computations ('cuda' or 'cpu'). This string can be directly used\n", + " in PyTorch operations to specify the device.\n", + " \"\"\"\n", + "\n", + " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + " if device != \"cuda\":\n", + " print(\"GPU is not enabled in this notebook. \\n\"\n", + " \"If you want to enable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `GPU` from the dropdown menu\")\n", + " else:\n", + " print(\"GPU is enabled in this notebook. \\n\"\n", + " \"If you want to disable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `None` from the dropdown menu\")\n", + "\n", + " return device\n", + "\n", + "device = set_device()" ] }, { @@ -75,84 +2155,543 @@ "execution_count": null, "id": "c28a92e7-e76c-48de-b574-15a1272717cf", "metadata": { - "cellView": "form", + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Load Slides\n", + "\n", + "from IPython.display import IFrame\n", + "from ipywidgets import widgets\n", + "out = widgets.Output()\n", + "\n", + "link_id = \"8fx23\"\n", + "\n", + "with out:\n", + " print(f\"If you want to download the slides: https://osf.io/download/{link_id}/\")\n", + " display(IFrame(src=f\"https://mfr.ca-1.osf.io/render?url=https://osf.io/{link_id}/?direct%26mode=render%26action=download%26mode=render\", width=730, height=410))\n", + "display(out)" + ] + }, + { + "cell_type": "markdown", + "id": "407ace26", + "metadata": { + "execution": {} + }, + "source": [ + "---\n", + "\n", + "# Intro\n", + "\n", + "Welcome to Tutorial 5 of Day 3 (W1D3) of the NeuroAI course. In this tutorial we are going to look at an exciting method that measures similarity from a slightly different perspective, a temporal one. The prior methods we have looked at were centeed around geometry and spatial representations, where we looked at metrics such as the Euclidean and Mahalanobis distance metrics. However, one thing we often want to study in neuroscience and in AI separately - is the temporal domain. Even more so in our own field of NeuroAI, we often deal with time series of neuronal / biological recordings. One thing you should already have a broad level of awareness of is that end structures can end up looking the same even though the paths taken to arrive at those end structures were very different.\n", + "\n", + "In NeuroAI, we're often confronted with systems that seem to have some sort of overlap and we want to study whether this implies there is a shared computation pairs up with the shared task (we looked at this in detail yesterday in our *Comparing Tasks* day). Today, we will begin by watching a short intro video by Mitchell Ostrow, who will describe his method to compare representations over temporal sequences (the method is called Dynamic Similarity Analysis). Then we are going to introduce three simple dynamical systems and we will explore them from the perspective of Dynamic Similarity Analysis and also describe the conceptual relationship to Representational Similarity Analysis. You will have a short coding exercise on the topic of temporal similarity analysis on three different types of trajectories. \n", + "\n", + "At the end of the tutorial, we will finally look at a further aspect of temporal sequences using RNNs. This is an adaptation of the ideas introduced in Tutorial 2 but now based around recurrent representations from RNNs. We hope you enjoy this tutorial today and that it gets you thinking not just what similarity values mean, but which ones are appropriate (here, from a spatial or temporal perspective). We aim to continually expand the tools necessary in the NeuroAI researcher's toolkit. Complementary tools, when applicable, can often tell a far richer story than just using a single method." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b5d6178f-ddf5-41ae-b676-15e452dc8b78", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Video 1: Dynamical Similarity Analysis\n", + "\n", + "from ipywidgets import widgets\n", + "from IPython.display import YouTubeVideo\n", + "from IPython.display import IFrame\n", + "from IPython.display import display\n", + "\n", + "class PlayVideo(IFrame):\n", + " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", + " self.id = id\n", + " if source == 'Bilibili':\n", + " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", + " elif source == 'Osf':\n", + " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", + " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "\n", + "def display_videos(video_ids, W=400, H=300, fs=1):\n", + " tab_contents = []\n", + " for i, video_id in enumerate(video_ids):\n", + " out = widgets.Output()\n", + " with out:\n", + " if video_ids[i][0] == 'Youtube':\n", + " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", + " height=H, fs=fs, rel=0)\n", + " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", + " else:\n", + " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", + " height=H, fs=fs, autoplay=False)\n", + " if video_ids[i][0] == 'Bilibili':\n", + " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", + " elif video_ids[i][0] == 'Osf':\n", + " print(f'Video available at https://osf.io/{video.id}')\n", + " display(video)\n", + " tab_contents.append(out)\n", + " return tab_contents\n", + "\n", + "video_ids = [('Youtube', 'FHikIsQFQvM'), ('Bilibili', 'BV1qm421g7hV')]\n", + "tab_contents = display_videos(video_ids, W=854, H=480)\n", + "tabs = widgets.Tab()\n", + "tabs.children = tab_contents\n", + "for i in range(len(tab_contents)):\n", + " tabs.set_title(i, video_ids[i][0])\n", + "display(tabs)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d2ce83bc-7e86-44d3-a40a-4ad46fd5a6df", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Submit your feedback\n", + "content_review(f\"{feedback_prefix}_DSA_video\")" + ] + }, + { + "cell_type": "markdown", + "id": "937041e9", + "metadata": { + "execution": {} + }, + "source": [ + "## Section 1: Visualization of Three Temporal Sequences\n", + "\n", + "We are going to be working with the analysis of three temporal sequences today:\n", + "\n", + "* The circular time series (`Circle`)\n", + "* The oval time series (`Oval`)\n", + "* The random walk (`R-Walk`)\n", + "\n", + "The random walk is going to be broadly *oval shaped*. Now, what do you think, from a geometric perspective, might result from a spatial analysis of these three different *representations*? You will probably assume because the random walk has an oval shape and there is also an oval time series (that's not a random walk) that these would result in a higher spatial similarity. You'd be right to assume this. However, what we're going to do with the `Circle` and `Oval` time series is to include an oscillator at a specific frequency, shared amongst these two time series. In effect, this means that although when plotted in totality the shapes are different, during the dynamic (temporal) evolution of these time series, a very similar shared pattern is emerging. We want methods that are sensitive to these changes to give higher scores for time series sharing similar temporal patterns (e.g. both containing oscillating patterns at similar frequences) rather than just a random walk that resembles (geometrically) one of the other shapes (`R-Walk`). Before we continue, we'll just define this random walk in a little more detail. A random walk at a specific location / timepoint takes a random step of fixed length in a specific direction, but this can be broadly controlled to resemble geometric shapes. We've taken a random walk and then reframed it to be similar in shape to `Oval`. \n", + "\n", + "Let's now visualize these three temporal sequences, to make the previous paragraph a little clearer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b57dfe1a", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "# Circle\n", + "r = .1; # rotation\n", + "A = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_circle = trajectory\n", + "\n", + "# Oval\n", + "r = .1; # rotation\n", + "s = 4; # scaling\n", + "S = np.array([[1, 0], [0, s]])\n", + "Si = np.array([[1, 0], [0, 1/s]])\n", + "V = np.array([[1, 1], [-1, 1]])/np.sqrt(2)\n", + "Vi = np.array([[1, -1], [1, 1]])/np.sqrt(2)\n", + "R = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "A = np.linalg.multi_dot([V,Si,R,S,Vi])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_oval = trajectory\n", + "\n", + "# R-Walk (random walk)\n", + "r = .1; # rotation\n", + "A = np.array([[.9, 0], [0, .9]])\n", + "c = -.95; # correlation coefficient\n", + "B = np.array([[1, c], [0, np.sqrt(1-c*c)]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_walk = trajectory" + ] + }, + { + "cell_type": "markdown", + "id": "113a0dee", + "metadata": { + "execution": {} + }, + "source": [ + "Can you see how the spatial / geometric similarity of `R-Walk` and `Oval` are more similar, but the oscillations during the temporal sequence are shared between `Circle` and `Oval`? Let's run Dynamic Similarity Analysis on these temporal sequences and see what scores are returned.\n", + "\n", + "We calcularted `trajectory_oval` and `trajectory_circle` above, so let's plug these into the `DSA` function imported earlier (in the helper function cell) and see what the similarity score is." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3e36d59", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "# Define the DSA computation class\n", + "dsa = DSA(X=trajectory_oval, Y=trajectory_circle, n_delays=1)\n", + "\n", + "# Call the fit method and save the result\n", + "similarities_oval_circle = dsa.fit_score()\n", + "\n", + "print(f\"DSA similarity between Oval and Circle: {similarities_oval_circle:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9f1fb622", + "metadata": { + "execution": {} + }, + "source": [ + "## Multi-way Comparison\n", + "\n", + "We're now going to run DSA on our three trajectories and fit the model, returning the scores which we can investigate by plotting a confusion matrix with a heatmap to show the DSA scores." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6ee9e8e8", + "metadata": { "execution": {} }, "outputs": [], "source": [ - "# @title Bonus material slides\n", + "n_delays = 1\n", + "delay_interval = 1\n", "\n", - "from IPython.display import IFrame\n", - "from ipywidgets import widgets\n", - "out = widgets.Output()\n", + "models = [trajectory_circle, trajectory_oval, trajectory_walk]\n", + "dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval)\n", + "similarities = dsa.fit_score()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18318ddb", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "labels = ['Circle', 'Oval', 'Walk']\n", + "data = np.random.rand(len(labels), len(labels))\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "ffd49b4b", + "metadata": { + "execution": {} + }, + "source": [ + "This heatmap across the three model comparisons shows that the DSA scores between (`Walk` and `Circle`) and (`Walk` and `Oval`) to be (relatively) high, while the comparison between (`Circle` and `Oval`) is very low. Please note that this confusion matrix is symmetrical, meaning that the analysis between `trajectory_A` and `trajectory_B` returns the same dynamic similarity score as `trajectory_B` and `trajectory_A`. This is a common feature we have also seen in comparison metrics in standard RSA. One thing to note in the calculation of DSA is that comparisons among identical trajectories is `0`. This is unlike in RSA where we expect the correlation among the same stimuli to be `1.0`. This is why we see black squares along the diagonal.\n", "\n", - "link_id = \"8fx23\"\n", + "Let's put our thinking caps on for a moment: This isn't really the result we would have expected, right? What do you think might be going on here? Have a look back at the *hyperparameters* and try to make an educated guess!" + ] + }, + { + "cell_type": "markdown", + "id": "d0ff5faa", + "metadata": { + "execution": {} + }, + "source": [ + "## DSA Hyperparameters (`n_delays` and `delay_interval`)\n", "\n", - "with out:\n", - " print(f\"If you want to download the slides: https://osf.io/download/{link_id}/\")\n", - " display(IFrame(src=f\"https://mfr.ca-1.osf.io/render?url=https://osf.io/{link_id}/?direct%26mode=render%26action=download%26mode=render\", width=730, height=410))\n", - "display(out)" + "We'll now give you a hint as to why the setting of these hyperparameters is important when considering dynamic similarity analysis. The oscillators we have placed in the trajectories of `Circle` and `Oval` are not immediately apparent if you study only the previous time step for each element. It's only when considering the recurring pattern across a few different temporal delays and at what delay interval you want those to be, that we would expect to be able to detect recurring oscillations that provide us with the information we need to conclude that `Oval` and `Circle` are actually *dynamically* similar.\n", + "\n", + "You should change the values below to be more sensible hyperparameter settings and re-run the model and plot the new confusion matrix. Try using `n_delays` equal to `20` and `delay_interval` equal to `10`. Don't forget to define `models` (see above example if you get stuck)." + ] + }, + { + "cell_type": "markdown", + "id": "9d8d4c03", + "metadata": { + "colab_type": "text", + "execution": {} + }, + "source": [ + "```python\n", + "#################################################\n", + "## TODO for students: fill in the missing parts ##\n", + "raise NotImplementedError(\"Student exercise\")\n", + "#################################################\n", + "\n", + "n_delays = ...\n", + "delay_interval = ...\n", + "\n", + "models = ...\n", + "dsa = DSA(...)\n", + "similarities = ...\n", + "\n", + "labels = ['Circle', 'Oval', 'Walk']\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");\n", + "\n", + "```" ] }, { "cell_type": "code", "execution_count": null, - "id": "b5d6178f-ddf5-41ae-b676-15e452dc8b78", + "id": "a6377c65", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "# to_remove solution\n", + "\n", + "n_delays = 20\n", + "delay_interval = 10\n", + "\n", + "models = [trajectory_circle, trajectory_oval, trajectory_walk]\n", + "dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval)\n", + "similarities = dsa.fit_score()\n", + "\n", + "labels = ['Circle', 'Oval', 'Walk']\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "04b0e32f", + "metadata": { + "execution": {} + }, + "source": [ + "What do you see now? We now see a much more sensible result. The DSA scores have now correctly identified that `Oval` and `Circle` are very dynamically similar! They have the highest color score according to the colorbar on the side. As is always good practice in science, let's have a look inside the `similarities` variable to look at the exact values and confirm what we see in the figure above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "55fa4065", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "similarities" + ] + }, + { + "cell_type": "markdown", + "id": "59cb799f", + "metadata": { + "execution": {} + }, + "source": [ + "## Comparison with RSA\n", + "\n", + "At the start of this exercise, we saw three different trajectories and pointed out that the random walk and oval shapes were most similar from a geometric perspective, both ellipse-like but not similar in their dynamic similarity. To better show the difference between DSA and RSA, we encourage you to run another comparison where we consider each time step to be a pair in the X,Y space and we will look at the the similarity between of `Oval` with both `Circle` and `Walk`. If our understanding is correct, then RSA should indicate a higher geometric similarity between (`Oval` and `Walk`) than with (`Oval` and `Circle`)." + ] + }, + { + "cell_type": "markdown", + "id": "87cf4e6e", + "metadata": { + "execution": {} + }, + "source": [ + "---\n", + "# (Bonus) Representational Geometry of Recurrent Models\n", + "\n", + "Transformations of representations can occur across space and time, e.g., layers of a neural network and steps of recurrent computation. We've looked at the temporal dimension today and earlier today in the other tutorials we looked mainly at spatial representations.\n", + "\n", + "Just as the layers in a feedforward DNN can change the representational geometry to perform a task, steps in a recurrent network can reuse the same layer to reach the same computational depth.\n", + "\n", + "In this section, we look at a very simple recurrent network with only 2650 trainable parameters." + ] + }, + { + "cell_type": "markdown", + "id": "3d613edd", + "metadata": { + "execution": {} + }, + "source": [ + "Here is a diagram of this network:\n", + "\n", + "![Recurrent convolutional neural network](https://github.com/neuromatch/NeuroAI_Course/blob/main/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/rcnn_tutorial.png?raw=true)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2f0443d3", "metadata": { "cellView": "form", "execution": {} }, "outputs": [], "source": [ - "# @title Video 1: Dynamical Similarity Analysis\n", + "# @title Grab a recurrent model\n", "\n", - "from ipywidgets import widgets\n", - "from IPython.display import YouTubeVideo\n", - "from IPython.display import IFrame\n", - "from IPython.display import display\n", + "args = build_args()\n", + "train_loader, test_loader = fetch_dataloaders(args)\n", + "path = \"recurrent_model.pth\"\n", + "model_recurrent = torch.load(path, map_location=args.device, weights_only=False)" + ] + }, + { + "cell_type": "markdown", + "id": "d463c3a9", + "metadata": { + "execution": {} + }, + "source": [ + "
We can first look at the computational steps in this network. As we see below, the `conv2` operation is repeated for 5 times." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6bfabacd", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "train_nodes, _ = get_graph_node_names(model_recurrent)\n", + "print('The computational steps in the network are: \\n', train_nodes)" + ] + }, + { + "cell_type": "markdown", + "id": "1d410c3a", + "metadata": { + "execution": {} + }, + "source": [ + "Plotting the RDMs after each application of the `conv2` operation shows the same progressive emergence of the blockwise structure around the diagonal, mediating the correct classification in this task." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "30249608", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=20)\n", + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", "\n", - "class PlayVideo(IFrame):\n", - " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", - " self.id = id\n", - " if source == 'Bilibili':\n", - " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", - " elif source == 'Osf':\n", - " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", - " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "rdms, rdms_dict = calc_rdms(model_features)\n", + "plot_rdms(rdms_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "248329c3", + "metadata": { + "execution": {} + }, + "source": [ + "We can also look at how the different dimensionality reduction techniques capture the dynamics of changing geometry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b0e2cdf", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", "\n", - "def display_videos(video_ids, W=400, H=300, fs=1):\n", - " tab_contents = []\n", - " for i, video_id in enumerate(video_ids):\n", - " out = widgets.Output()\n", - " with out:\n", - " if video_ids[i][0] == 'Youtube':\n", - " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", - " height=H, fs=fs, rel=0)\n", - " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", - " else:\n", - " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", - " height=H, fs=fs, autoplay=False)\n", - " if video_ids[i][0] == 'Bilibili':\n", - " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", - " elif video_ids[i][0] == 'Osf':\n", - " print(f'Video available at https://osf.io/{video.id}')\n", - " display(video)\n", - " tab_contents.append(out)\n", - " return tab_contents\n", + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", "\n", - "video_ids = [('Youtube', 'FHikIsQFQvM'), ('Bilibili', 'BV1qm421g7hV')]\n", - "tab_contents = display_videos(video_ids, W=854, H=480)\n", - "tabs = widgets.Tab()\n", - "tabs.children = tab_contents\n", - "for i in range(len(tab_contents)):\n", - " tabs.set_title(i, video_ids[i][0])\n", - "display(tabs)" + "plot_dim_reduction(model_features, labels, transformer_funcs =['PCA', 'MDS', 't-SNE'])" + ] + }, + { + "cell_type": "markdown", + "id": "1aaf5f4a", + "metadata": { + "execution": {} + }, + "source": [ + "## Representational geometry paths for recurrent models\n", + "\n", + "We can look at the model's recurrent computational steps as a path in the representational geometry space." ] }, { "cell_type": "code", "execution_count": null, - "id": "d2ce83bc-7e86-44d3-a40a-4ad46fd5a6df", + "id": "7f88274a", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features_recurrent = extract_features(model_recurrent, imgs.to(device), return_layers='all')\n", + "\n", + "#rdms, rdms_dict = calc_rdms(model_features)\n", + "features = {'recurrent model': model_features_recurrent}\n", + "model_colors = {'recurrent model': 'y'}\n", + "\n", + "rep_path(features, model_colors, labels)" + ] + }, + { + "cell_type": "markdown", + "id": "5c3fbd44", + "metadata": { + "execution": {} + }, + "source": [ + "We can also look at the paths taken by the feedforward and the recurrent models and compare them." + ] + }, + { + "cell_type": "markdown", + "id": "b25a8cc6", + "metadata": { + "execution": {} + }, + "source": [ + "If you recall back to Tutorial 2, we compared a standard feedward model's representations. We can extend our analysis of the analysis of the recurrent model's representations by making a side-by-side comparison. We can also look at the paths taken by the feedforward and the recurrent models and compare them. What we see above in the case of the recurrent model is the fast-shifting path through the geometric space from inputs to labels. This illustration serves to show that models take many different paths and can have very diverse underlying mechanisms but still arrive at a superficially similar output at the end of training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c904e840", "metadata": { "cellView": "form", "execution": {} @@ -160,7 +2699,19 @@ "outputs": [], "source": [ "# @title Submit your feedback\n", - "content_review(f\"{feedback_prefix}_DSA_video\")" + "content_review(f\"{feedback_prefix}_recurrent_models\")" + ] + }, + { + "cell_type": "markdown", + "id": "3ed56061", + "metadata": { + "execution": {} + }, + "source": [ + "# The Big Picture\n", + "\n", + "Today, you've looked at what it means to measure representations from different systems. These systems can be of the same type (multiple brain systems, multiple artificial models) as well as with representations between these systems. In NeuroAI, we're especially interested in such comparisons, comparing representational systems in deep learning networks, for instance, to brain recordings recorded while those biological systems experienced / perceived the same set of stimuli. Comparisons can be geometric / spatial or they can be temporal. Today, we looked at Dynamic Similarity Analysis, a method used to be able to capture the dependencies among trajectories, not just capturing the similarity of the full temporal sequence upon completion of the temporal sequence. It's often important to take into account multiple dimensions of representational similarity. A combination of tools is definitely required in the NeuroAI researcher's toolkit. We hope you have many chances to use these tools in your future work as NeuroAI researchers." ] } ], @@ -191,7 +2742,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4, diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/solutions/W1D3_Tutorial5_Solution_0467919d.py b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/solutions/W1D3_Tutorial5_Solution_0467919d.py new file mode 100644 index 000000000..88c68e281 --- /dev/null +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/solutions/W1D3_Tutorial5_Solution_0467919d.py @@ -0,0 +1,13 @@ + +n_delays = 20 +delay_interval = 10 + +models = [trajectory_circle, trajectory_oval, trajectory_walk] +dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval) +similarities = dsa.fit_score() + +labels = ['Circle', 'Oval', 'Walk'] +ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels) +cbar = ax.collections[0].colorbar +cbar.ax.set_ylabel('DSA Score'); +plt.title("Dynamic Similarity Analysis Score among Trajectories"); \ No newline at end of file diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/W1D3_Tutorial4_Solution_1ac2083f_0.png b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/W1D3_Tutorial4_Solution_1ac2083f_0.png index b3f67e4b814a48b747b4e96c45d298296681f807..8cc3e6f975e99d9d207f7883625ff25b7ae3f6ec 100644 GIT binary patch literal 131259 zcmeFZd0fuv`#zj8#>mJRV$d=uT2w?SZMLLBdr2GG7w!9QhC!*gwMbM{TJ9D}`)aAQ ziRzX%McOx|eSMDWPG&ye&-Z!$c>a1`ujkiaZTEY*mh(K$<2cUiy5eb>HH=#r85kJW z$euW+%)qc}83V)efS-TDZ)_NYL>cg-?6JeD&iDFToSXv<3l@f}qc_zozr2$*nsK-E zUFE>-*Y+#b|L~`dsy$z>@P=o$_RRJgnveNjG0HJBD_;9GI%?Ga$8Ap!F-B*Gwz6lB z%nHgH37+3s_Dh~#eTPGMS9Zl$)?ML!l@snmGs2p?H~Qkn|9m;`PU&U(=SO_!ZO8S) z5C8s+IE(b5fB)=#HR|V||NV%V`|_VNEb#Nv4T|7av4{=E_(7hg#24v?CEGuV>p3%UAZkx+w1VW+u>kRrYbrxpPYkZlk7pKg zOGvXBq{5F9UAJ!*Iii(j3THRHK3TBdPYHBRT&1<5sfACuHXmM=1 z$b6TbfgzH+fH@f1HI@rlne@WeZDhpT)X2H#YCvZ(74~ zb*9y!UT1owLZG9ot4nd=-aVEUx1s!LJS1zcm0;-T$E)mSpI%;PD#h+o)GjU$`e&Q2 zJz(`APkF_mfu_v*KUb`NYFM#x?#qj{TSRT6)n@ATypnc>J3s&EBtAdc9>eH;Jb*2U z-E;D=R))pf*{QFzWF4`nDooJzUEy!=Gw=K*Jm;~0?crucMa7%0W4$r4;jVIaot1C# zMxQAz0y;U-_)U4%fzXq#b7S@C7n_9rHt##Glj}(1%WQrq?Yq(RvNz+9jl4^2=4p)eH=WZ2lDzi}>oO8Vck#XEQDNtgMFQTuQqlyezd>u3Xud zUay^ZM@q34w!YndEnSOJl&VVD7Npj~+I;6ZO=I<;k<5*QJYGE}&r+v+iK z|Mh*b*r!kTOG`_~D@7gOa=`LlGWM5kmvzbCYqkak1#y;5?>oDDIc*MoE$+B9$Ry=a&sk03x?bPE&9UOA zpAw9#!;d%3&&`Z|t(G#c2P4R?=ryDyN01u88UWv}*gdqGi=Wb-EUv; zh^`_#)_d296DJ;*uOMq1L)+o+hNYEm*>bSm^Tpq5%-%mfAL??g`(wa1I4^d zOyUN!+L@vzyD;qs-Nq9yT)0rq+IMZA@oSUT0*?}piNO}X*0Hj(va<#5*(CCnvU|O% zsF2^gckh=gqap>C;5yiFe z#BIVdCN`EA(`r>P^{6UO*VD5gSU@kiqatMMR8e=giI7>{3H?(4Bt^+Rh{jEsRtE;f z9Q#dJR6=gudO#MFjg8Gg>o#xP+U#9o{IkDS_PrJxN>pT$>XTqwCDsUP7(ZYrKnSpUBr@?z2*l3UIQuJB} zjl17QUG1Qz6!0AQP(jPG(QnuBn(Nc!+vr zwGXmML}0MK(vW7XH77jx`L+U&cq!s@Qs*T00?U?7?Ce%TgQ=x#X$=z(uChJp-OA2B z^hS)tR;mrg-rnTij?T(ZeW3{~^4&&h4T1=0@_t){Eav9}u3t~hn;FdAnAV{EV1iY| zx=0WyNdDNd%Pq4<7Ut*pRN;G4!n}0C-T>+9&@mcQ&vC@Z#}VJ|iocySQWA(W^|Z9YGp_ zH*cOhFzC1SKySF);M>RNUj*K|bv`qNmqpxJG2eAeSzG&Fizl#w^3aF;0ZSX5W(+os zsv$tnyhuU=;}QtDA6&Xv9eQe{-NyTEM8*>+ckaY_&P|*4)WjHOgjIDaPbi2FU;b$= zTO#&Q#f8-O_r$PqYGV~N5O2~i6#FoiI*#{iwm7%@odvLH@x-Q7E%cmARQCzu6cyDx z@WDCCsOoM^bF=o<3$Mz{YcSR<{+r3{GdL-_$8M4}t9|5mv^_bya|@N#M*g*aAM3Mc z&vH8NnzOjdl%46%f1!zFhD?(e?%KF@G<*luHqN1V{ z#`UW(IXrgFomF8$p`oT@l|UyUdY*5w#qrN0C!S#VgBk?3Z9D8SJ*s4D`((HhA6+Xs z8HrT(=<(w{$f3#zw@>fdb(9AI4+MJ74$HW>sGZJtR92=~7EBr4m-J|y{Ke^mOKdNV z8jFm;r&pkjHMz&>jF8^OI36E-K2(b!o;5tYVIp&KrOG?cuE*GG>!sJ|<{o zCfnNDJ~ge~S#?iTWq5d)5{;c=TI9W~G2bl{kx`R7BKskd)PDC#YyGhPJ9mC>@miQ8 zu`e$_-vN7wRl>E-tP&|N#cN@vRlT|~(<;R{+~qi*R)(shW5&BAt=OL0SYE6p<@4LB zuy~(34xFX7m-d&ve*F&PmhQFSDXSd9uVat&9*^i>UR6cAD`Y;3393KQW@unw(wt?} zh=kzoUOq3iAT`;xoO&d2Bj4F>OqnV^SQn?X1E~)SS>7igpdRSqscnZ`=I&;C%@`@p z8PIFil9PV4Jufe>?$N2O?eB+Ma-Dej_)cxxwoQFbSw$tuzlAK$7oy>m)W93Gpbu_C z5+2h%v6N_J670LvK%eEnUSwwIc3R?QdJGq^_()?%yuuEAsfE7gmv#sCC<+(76CsF(cDA zk(boGw1%~IUx~>6oT|B2V)Q2GjvY~z^D9=Y2zF}qs83h`=ss^ZpCHMU*d5{R%}{~l zcOKv*bte4%`}fN28+V^M$IZpXMLQrQL>p=QYwg)FtLCg!PCmY>&e`IHN51pqCWgRG zhT~F*mNAIE>T>`@ibk!#o7AN2l}s#W2D{SF><&h;6H9-F@$p(i)^t&edQUM-^sf zW*P!d8;&RWw-@=$Z{BmxyW`8p7mlsxg~RyfrpMB@9B}?}t^(VyqQ4LmpgdKCW%{af zmNQ^E?{pX9W`=f-eSGEK2(S5Uwbn4poU+bYtNgKGwbu9V&*VnbBf|*_3hrK*s`lEA zvFsRM!Jx|OzkE3ZgT}p2GWHigvJbZzcq?K7YkCOM@M5~uodQI0~YbCjye07b1dGu0KQ**`}%_}U(e9Vufcxg&# zrJDs+O=VcMS~^&~%d%3Vc~DIg>~$oIuRpD)u?c7f8e$0Rnp~t;wq7 z#%A>i@;3NJrL#8)L6pQb%%xCxacQY}!xKZMQnje#Ymzk6j$>1MgbuuY`}XcWV}Dv& z5U)mXgHwBnpE_?Ha^-znsT2-CB?nS}zSxU*M#0*GLRnQ&@m%w!f#U;ll(~w7y$PT9 z8GpDH5~2}bT2eUG8Rj}a)zvXHq?b@-AlO6G&{IQw;AhOMk^D3)#dRRPJ}@L?mn;Hm z4U$asR?DnsMhjRnra%mgYuARTX6^ghoNXuYwN}-iUni$Kcc?(+YiHHKcvG5j039P^ zO_{!ZD4l?~5;|v<_3mmOi%YFWqzQP8SotN-=yn&7@G$vQwvEzCtE+E z{M3P#=qAhDq-IaxB=r@_SUefo>9*?x1Oz-r-^!?8561!!V`rT$Rg>P&$XZ){z>`}A%t=rFNV(fbX|S5W|_A!ouJgVIp{Q)2qvc8 zbF7yZGySr-c(bv;6F?dGA$4Bu^2RlBll$| zvv4y|q9Z(~Wg8kAO6Gt;Sb4n7>f&zYI1ZRKr)Xy@hlYkS<=a)>*~>J>s~eYr;z{tf z38(Kf9_1T!rqp{EclK(zLx&DsDG^hSRAC;nzJdyT^XjyI?(Cmmt9zQ1Q8-KZ5AD1y zOMv8y)LCq0p=3}_3Kv@oT0W)*aUVQ*R>F1EI?ifXT-U%LdU~{{{Y}=}C>gC5vAZGJ zJ&(dz8ZX5f=+?ZO&=8jz1yYfMAK31!B+?zi4I#%#yiqxeTq`vcBww%RNX1D;uq0ZZBft8g(Q*R#MurQC!k>nHP zGdkGP)N3{MR)(Dx$gLtb@ay8}#PWtl?G3A9-l8|lwKsmxN;Ostkd{?G?SW^p-D=9f ztm5#B3s_csPAl6#kqSR(yb7d@d~fF3HjG2L_Qxw(|ljQQMonLt1U$-G^<^gW@VdGwsn^ z-Mq-yxXD<9bvKJI>~*v+tEy7d*4B>S%Zc1x-TUFKnw0X#^2clvuD@UR2ZpV%5Ecux zh`WXo`(SgXGo|PqzYrkv;~bKh39GqT_Ro=dvSth8IPc9ztsC7KiB4DJ;p83mXe{mzffZ| zzhQcQFuRK_ANMLbt&Z1JM7hYketp!kNRkp*RCqkmqJ|Zd>NZ zCedgXw8#{PM;aVHB^}Vn)6jZrdt|xuRbG+1de$77H$h$l=jZ`zx0(zsEgi6@fmUT7Nu3$Rkrz(~_@g#?@w^&q(Ivh}X?`CDjR= zTYoAe-{s}sF+#NP#l|@Gq_ZU+!}q;Br#d#Z=YS0>aeM*9laOVM5NPN>F)@+p!sc8? z+s`a9k^i|jfztIVHwu(pj(vfT#hri9RTUQN{wH4fukhJ^d%W15F%F&n3o3Abno8w!4!)b8Dh zjho~|pTAqD5>mlxdz8{X^?;X?UBu+gAFK!MPheF&*}h{(MZaBd9k<8i$EyUSI3P@P z1KX)>z%lY+SR;qqHS;l;X zS!Y1aJ;qNpmDMyszAQ+V6z!_2b%xa%v+sqC)ueC9fp&WxF2P%tR%nGhux7L1F~Uir zw0$6C{>}vVneGVDv&SP*3ji!?L7X~_e3s3J3P9Vim!9&N+dDd9`ceu%yjst>YgZr8 zM2ur^ddy_1!r{Y*&9FD)CGmcWhYoq$Bx;mJJ?t07f~EOapeCBQ78N4b`S)>umUzJD z;U`^oJ5yAH%U`e6k)0|6o!JeZ>W!~JLEM_H!t_!WTu3?=c}GWw=9o_Eg)0aik5Kr< zAu-U_?_r4CKt#17W~$f1{PA^)?(TV?+S>`MMg21z$tk|>&UW+qWHnvga8iUq<`A3- z+^}xlc?Z3>v^{_kCFv0ndhtPr2DuI%Z0wZ;+vhb^p~o}e!K}GPUGTc{W*@IE6gG;7 z4_`tx%DP2sc7|V6!@;caX*@7M!c{g`!jE5NVy!6;;vGZ8?g5v1ybuM0#`;~SRWX{> zy=Uoz#!`>!m|A?nZDPIwwQ5hfVjpUUt{t72aPn=5Aa!HysRNs{Yq6mS zV~e?Y2ke>j=bt~wDqpzp@Pqr;Q>ZGb2VbmU*zm&U90Nm}kJ^(n#`xVq&)F=cm%pxC z_kfA@;F+;_jg)Yka0iG9;2VWbCUv3&@N6+X*%EOHroAh z9W`HWXEfmrLJFfM)mx^frb^6mMp#g^cccc^<-0kxPor*DZ$0ngIX`7gb#pp77bCz;cJLj>D|hzV9(Hzi>Jc89)o##vq1>zuDN!b+apK0yY8>o$?%hiSu&K&& zsvpV&49@8Y*4pnhs15SFsW(&XR zjEu$=ae9TGhJ){w^IpY%zJnN+%Nx3V`}R6;?Cp~L{HHZFHJOUHZry60?qmA$+Bzmb zswIGkBKV{mGbkr2dAK_nIjgrh?LVAp@^*VjiBd47wHEmN^OyI)a-=-v;j)zh0C9!# zkw~#*=5}4=7IMJOj*W~=FDwkDLX3L$z=m^1^z4uJ*+XB($JJ4rv2N*ND)DDE*ETjV z=*Gl5PKo)jukSRmG(jx~9AfRVyBHOJZ^>4x{e!55S#*msjHf>ah>Rj$`TP&d#%}PM`bws3aU2EdXnr=?E~Y4%e|vZo_}5yHK%mp&z{;{M|_q z1QXKD?-PXDpuKKPRuq@ouxV4*n}^R9=7zlZ{hkGXa>?uOx9|W^VDg*!7a~3Zcjzbn+>~V# zg-S}T(>Pu~+c?^Fqu&Cub7I-)BL;LNtQgWqSYr+t`oTltTWyw*d^CRa&7KsVwj8)xR-ApdhL68^?0Q z5d>!@hB6x@K?k%Sl#tNDo=a+(61I3B3BvjP#|g10gW=bN!@Ippp^(-U{Jf^JrL&nFOFqI^xOaMdmwiXt8BDX4EISF0Jwd=kU5rwE-5Os{Q zhirBs&(uMtHE2dqi$|)Mo+w;MijL+8Ylm(FbP>HP!dH6U!`5opYN>*5dEe!kXHfF)i71mIVwPa<2P16lkG+8@Ca zHzKjcdjX@H(S|UrTxohPt8YW5zL>sM5fnXmLZ#ko@p$?cyJ`t_Uynck{4>$^kBq#L za$Z!>`v+T{+t;Az@rW7Wz}{9uhGE{c$ymG(lTh0Kt4_0dO8z4h)>|hQKyuqdqcMYU z=jeNhLZOVJ1foJ9i`jUWSwmcdOEy4a{Em9!86~5Ff!hj_Z;LK14?-&CoO*y2+Xu?< zz+m-#G2;X5lMv=}Pf5PHkQxWskkFaI{5ob6T>D@yAd#^*D-(q+&%uK^JV#bAh!o9h z!kxfCGlBZ+3fy($$UQ7WGYk&aoiZQ=f=l#sR*%l=2)#}g1PJ9^$ALsfexH&Jegg7< zMd=@MSdG0z3sZYF@_e;G zH!4}jZiztyR161!=u#OmH1c;vY#u_=jV8z)D+DBp1_Vd!(sZjRLWc?HcnT~ueV`5F22ld>OKktq~tr;jY$k!==PC?~bm7t~_01XdhkWz%qJGiqO#we4a zZBy-=E@qMv9Jq>jRcMx_=g*%f7jDT5%1zKKEZ8WZn|S!IpSVGcQ%2hZq&N0e8tL^8-!-TNL_aSF=~)(C&uZ${LdMn;buhdv|$0Z}iAV{EVbd}3fwySf;UP`Ew9{yRhV z0L@<xz@fY8+sA1$FD?Dv{?2bX;o?3)nf0G7=0=cTtqElg<%K}N6d#jjew z^LO4j5V;lNO)U^}AV4Ys#t}NNnanC6y&t>O2QJle2xR>YX&3#W4P>h)YsZ@>XFp1K zzPp~NW(UhJzPh>X?q2=l%*PhVg}!vt+T%bBwa7^J$kMXPz|1tJUBy$Aligrk>ku~v zzK8*7@ZK(qhG}7|3+3g@V`15U0=1h!JtFIncha;Oy0yephdy?_*gBTVZn}2;Vp;YG&YNw+I?sgRsQy<+XtF zSN)knnf`M*Bn6H52@%^3XKcWDks4>i)~!@j84vy}*iB=6H+j7?0JQurD_e7Ln3gS*rvXdoz9r*VoICF|S3u6I5ObEeSdQ~k5 z%&OL^RjXVfn9v@dKN>bSJw8CavU(G#kZ9pt*HF-c}Sm}qA|eUNGNOFh^v`y26YHt=?JEb%nZ!gFjtY`&UImwl9G~``h0sD z&q_mvVk~^oq79VYnCp~6giJ))v#bwvgNup|t@(Y=AwW|?5F2D#LCwa%3b~9J&{=dz zqU(GvyFG>(*JLeIXMXcoJ%U-N&bfq;lvkW(X>@yK6^BMrt6=>7jZr zd9cD^&6fRdK$|`!^MQ4y_Ukr=$n7VCDJzu#Y9p2JAx;tFNw!^AAXtiTUJ6v9@gtXy zsS(zL_}7rebn={)py0w5p$Ioe9Sm5Q#Gxmheh}d>@G0O3%8^%GzINt({CjI%wCws! zu4o=Et~Z!$2bf@xc!^d@vIHcqI)tcRrj(OJ7KhkJs4j$m9t0yW!;Gs}*TO6l`yt;g zBrJ>0#TTuQdGv^w52gn)TLUSShghDVZPii6YM`*w8S%%*5ra-aN{SHNw+~VH0Rjxk zlDL{(nOL*Z@#B9ZM?N4~2E{vt?y)g>bEsa|EeUy6V#~)MXeAqf%HS*NLGV$8oQp42NDaj=UlK~m3>#hR^9Ke7 z!~gQC()S2S1B+k($BO#@@sa2vKDWsQ)DJOaiNbOPW`&W7DIm&=d;xMGFc|IH^Y zW`3c6%GzQ3zn|=U@h1AOi)h#L=&t7?O;l2r9AaWkmTu?Honw|pXuy|u!ZBUC(YkFn zn4JU4AAAHT67;_?XzW~i1;|NRw?-%uP}kKqd>e>_-(0rQFOY^xzWIbpljO=De^mY7 zA96j)6;uO`Gmi)M(XU+;boJX)F_TXpJAK+8aI=4W)tWU)4^0+tuc-EI6tpjG|D))X zfuUhg(^Y1@Z_mc|?b$RUF?{R&IoyD~6aM&Z@vo|!|MAzv$;EfgI(Tan{kyCmFnM9G zs;dPup^z&uAN!wU!AQpHxrJ~@Q)z|of&qOp7?SsYo3i900hyvxH~SZV{jaMQvMkev zb?IS$-VdQw!M#lMIvU{mn)o{MU|B1fC_gn{-a*Y~(PUQFr0lazx9H-&=%?^5TxC zG!ApTI~`-xGZ(>RJ~dcdUL_#b5O|Xkb2)H#(t9^`t0a*)t)_a~zU61*be+#J%fLHI zF5=QGB%u16%XadEwH5u1TU!g8C82bw{VKZ+^+HN!cwLwjo1&|KyX>Q8_q2_3;gM=H zuX|&*CD|K?`~(+q(C^&j;|$)$cA0sKvY)S$ABy%t;K^DAub}OWs@jv-B8#{3^5Z7+ z8Mi|uWL!@d%%?gBINm==d%AAm^MLf9bOMSYa`#~1CZ6kyBjwCQ7UMd^VatdcH*UxU zytrT^nX01wR@}u71WvhdZh0|BO_-e8Wtgl{-P{80-v9DDvZ?5$eiBySKi%48t5!OzF zH~)DkHu6yC1w%Ld{<}%0f}z#S?7YI&pzzyz+;*{DpZhTU8s*K4Hwv$09XY~zYZv`8 z)kn!TK62zpe@1Lfj9O#vZOpuTeUDt29wm54GAZyzf=)W=)xe~zO4#>(QofudlMmYK)=7MtO>O@c1j3^ae*Nb~;J$7< z4tz3@5h3Q3to*@TSPKhHjoU87UQkrtC@n_r!{VB6<4)lH=RLIP?}6y**rcMO!rPG_ zh?yGmk zd+Vy7e}3E)e*eA>_t4mgV7%wCb1{`V%748&cm>CeG{w9kwvE$^I{M#;g|<+Mk_tRAOpMb#OuM z%h%zWFHB=RW31mEDDJ<9^?AS5C3losFi5GE=kEAx&fa_`CrHrg7B;WivB!Eb1&jA9 zTCetB)A!d;Ysv?pYsAaBEs%ZwccB7?;YF)V^)Dt$+;)o}bk6C0{m<%OOA;7r{nObV zsI7Si=Ej~cAN$0Y#A7%Z^7m8a{4Ygvp<5Scs8-GCpBZwY&(N`B$Fe;jg7FIdotq>4 z=Dtq3o{RcACr6v_TkSWeSO46MY_+Yc)~*%ERDkLgQ*ur)bpOR;Vj-ORgZ(v!YL7p? zU1nfg=Ds*M8r%Qtqr)Iy;a+Ny1ks!@E0TB8)!aZVU+77TVZ@y7i`LRRFPeN`GSNTo zEip*>XD|3y%+P*{ths#oayKXf;U?JT5;&MpQsw68Mr}P?S2rWY`fj8pHYm)B-|1dN zK#1id0wVnifZ)1{e_Q7BRqW5uwC%OoDZX8e8#1`k}pM6?j zjcJUbn1X53u)%<+gVL_sX*VR-Y+p`G*1eOn-=_}KzhPa!M$F#D(4xJnneY1!Sq#5B zp@~?!K{#QSzQ8B=USVRwB5}q|IG7`}YGcR7I~`pEoB|m(i-E%WZf@uIoiKPE8M5l~ z-+wn=tbX+>?%|34uOouoMHRU@FBV@S>w>e~KQqWi50&qJ_~D2C3}a*C{{@rKUlU5J zV^i9bC;b`U*4@5u=~Lt{a=RZ_u57xfATLjScw+G;ia|@T8@{uQ1jNJYPX zN75uE8p(s_8a-NoO;|!gn@Kp@xo6MgW=U8l)PDV+h-M#I;Wr8ky|O)E0;B#HsxY2Z zbz)#hba8VFY1$AJAL{Ojf-P7!%w07o*7^+)doE3nt3-=akl zRz+15NmMR+;%2ZGBBc~sP5Q@8!5vA?jjBMWskSTxN7_R?>!AP6Ch&pX2iJ8k9D+1B z4$<{bbRdk;U_0kv9?V+5VFU3+>ey8Y0rUogf(GlSG6aC6At&c^Izlq(^JfDnQ&cF^ zAbZLyDsDGesGm4-?I$?sI-QrM?eTg{JB#M98v1fp+@dcEZ>lD^JC1n7{+z*_K2>zb z;P%`QD&W?alb2V4f=DPOxYHmWQ!8Cghl|HU>*xlRLWzbD-xsW9`IfkO!F&@Jr9l2D z1+>v47^1~=uIX2x-zf@4UE!fU2!wDlXdrLTmOhMYNzE1c&OYW;dp zue3zXH03EDEWrHXlC7j0WMpJy6wD@!92u1^;qD`{Q4HM?3$w8xG7VU^jaTlg?uLc5 z26F0XgK-35`rt)U?C{*P72HzL&6|onL#Y}ZOIZAu(zX3RwF!$9FQ!P<4AAQdD7y_e zKAN1GG6h{qddxU_i1`>yo7@=W!3dil-=lC&hE=zO3(g>-?n0$eKX)#Y!xXO{^hLJW zH|>h#w@s2kHrZyG-$WA`7q>SJ_BK;Ajg-tGDaeh%Iqzp9Ve{#wU>AP&2kXrz+nDE) zR#RK6!xHb78dAUQTfITwk=#Sg&a$xAEDlsc1$dVmeJF@R%vv9x|10{r_Hk%u7hl&# z$*h%ChPsP3p59MftFz^YZ{&+C#f@hI$BU{qY}gRkR9adZyR@wy50EZ9;0=y=SeQUG zg{tMk0OR8goh8}Zn<2~&EWx`h{w;PfPyY2(g}*O5aQpUc6-#z((WM{|Y3+2EP_8_? zl|}73Z@f#^p%U)_H#axd37J(nzg+8&f7%>718#1hAnH_@aLE{&O9r2Jw*(Xx7FG!_ zEp~d0f<$aP^Il7I5m&XzZQ|Ef9DturV(l-#)DfE_9I>EKQdDQKg#G%!V>WuoK=%Rf zn8@g#tO~M4xas@b+nnvAm!P&lH{BgN~7O?`^}y%P}SDf`t^-4YG8NX8sd^1 zlryl}#r_`;gbUlwOioX$ALdw`31zTNkW!rH=4XZ)a~!CI5|&|eIfj^YHs3X!$l%1} zT$lA}$`>V+ZR>5Q3TF3sLr*@E3(b7k?;-gg*YEG|x1F&b2hzq@91k(5ASl6{69;dj z36$7r^n*Ng8qy_1JY^I=^um~?-cHOL+96JrYMA=e)u)GTGzhyBcJV(VHoXX5zI=H{ zN3qx9j*E1q!B494=;STcEZ=I)Q;=@{R+9Bh=Kly-Vvk7e@G=2mTISM?jys(bgg~N8YrJh*XYyt<1 zb=Up+AN$|TETLtb*F)Zd{?=`x7=D|9fdQR_d^CtrqCuR&Km58=yx7;si1rw|4o{H1qL;AkQkOYfMb(O-z3nj3CnJkOYm;Z?XqY}R7P;u*%Y}IVLh$k z__yNx7Xs;T2L~Hvw&*^jS1&MO;1zblB$%lHYD(8@&H;K$zc|$8nCV|YIHEz15}ohz6tactS`3g6|dJ=~=9h33?2BJr~P7D&mv7wO>kAQ%J^2b3D zOqnve8Y?~t#N+Pt7K6FfjEsae->2f^Y8DoW&|T%zWu&D|(Ymok(mjhrV5kjYcF;sg z%L3{3Bs~YjfC6YsiAE?SOgZVdA#EO}5nZ;Nb4wFXO0fX{xe`8%15s-vLi=`NE z0sYp83=#0cke+tRqB~U?X&K-iDL?krs}taJLnxD8 z#19OSJpTbY4UCPOE+V)l{_mjkc%{+4CdtKorhR4Uf#~(#Z|m0smz>K~FfcG+mR|f9 z?}Q^uxZsU+d4mfK))Y5g;(rYq&B#;j&VpG@qaEt z?=QcAU?a^^#778KGC|c%vy_+kw&Z*iCcYyvOX#gwiJT;km{HD21BBl14+)f+9l z$w#`BkN`MjVNRzezVT+1J$Z6>rox#sXTIek@0opJRir(4=jnTA$R{9n3MQAWex|d8 zRx8?ly*Kh}#}Pc1H3W+sp+=!U$-oaB-9Y z#$f*vSH;_nzy5m8$pR|%7?2TSk^DjDFOg_T){0O4p@ZA1sS8C0!jk&{8+{X;h8?MX zw5OI+c7A@I^!EU=TRfeae6Qd(26d5Ob_hM#>?M2U;~bZA%}BSNpP18N9fTxKS<6 z`92-E0I)lsEHMwGD;)VMk68~@U5NwKR`b7>^S=N99>|b6eq8+Y4%2SBN4EsBqOvk?;5ov(r5l-2CMwNa`P)Bpf66M)O{LDHxmj~p$2BaUsPHwCEKO%Yl{T}Y_ z?xdsQ1cixBJPyJz62`N^JFpE_g&k0EYPPvLY=Hz2v}IRY9K@;I0n{%`g!>?$prEpT z$&FhaM6!jcBoi)?et($2X;`sibt(#*tKkAGkn>w_z2RT8c7g&I?_L6I8#uz2|KQF?N2R*TzK2emz~Rh zNsrhG zNry-3r1e}KC8oG%VddDuY~@OFnRz{wKb5FUfnc@_>^ zBqlO;!~{p86K3@@>}CKoX!Spef`^9wh0pDF#nCUVul(_?8QRtMBd&L2N}HKU5pcJj-4Qq}L=c^nd5^2?u>l`d9_Zw+?RgSSQw_7N;Xy6_?q zd75XxFnpA+R+F~RpbgvFM3&y}OZWCjoCTl)kNr&!m^XAWa!dg*z(n2pV;VI)I2Rty z<3&mwZuIL~H=}vi9u^td8UR|}(J&W4Dbi=dVLI5H9kh;3aoe{{klcoNS2#Q3Ya@Zb zGX}6Da&YuucanLn`1Aum+VS}_*Oo0N8%7+Tu%({4YBeIB>E~CsDA6!Da`Ew1#iTA{ zmj0I37}<1wB;}Os6)A5q6q6r^v3&O>PxLpmKgsegzaSWzAhx(Gn~x~zlD&xb49hE} zn>wz*PshX}(pK}{yZq4N)fCwR_5V*{{>iay}uTup1!qJ21FL&xx(w9l#nl1GxO_CW`XYKaC*Z10IQOsdv?zjTP zOFV5Khd*CwJN<1dYd=F4gWxX^|7GPDU>IKUdBCGR$?${pU<4{Yo_Mk(3T{9Kxi96V z3#&uim&u2DH_Hr^0cLczY*keH_YC1+mt0u5Ic0&EY_4QjN`eT0yXBAg#|&{G#e#m@ z(Z&VJz;@=bgeZMDuF(Cy)pzzr9Vz{DpgCIw+!ZA{%&I^Wq0@=v>~AqX>ra_Y%dY(W zcP~76Asnn|<|lC)Tce!SX?oO@c>hTsdGty)XLMD?E9Q3^u2ySSq1fr{%-9oMfiK_(ZBQ*UGh}s zb3j-=zP`2ls>R9i7G|aXtdwZfGSndbLk`1ZV|n@)tX#udSXbA)83yT4cp$F-M7*)pL zY&Q!)-t&J4vk84c`D#gD^;G*VSnG&4E%AtRQ!rVfc~Vj1$19D$k4A(7`?5> z;C*}26doL`{;iB^+YHWvoE$~knWoWLezDUH{5-Md5)*l2s=iFjMh297=ICnfZOP4q zbL_7Q9){<%f7J`84@o13I>49PhY`_0p97xYA=nR8-dq4DWDX!BCU68Jvzem$QLhFa zg%c$_z;FYVi|$`5b4f=BHvayd_DFLHd+41#4B|4N%*I}>SD>Htga$HVkw&8nBGe(+ z#9-gbN2@%QBUnS%MH)>kS7`g-`vtA0ux!~frmia@jQzd;)KCdO$7PXv8!Lr2VhIr& zg-ub`P+k0OQY?=RG-;q3ru}w%A2KygTuH=JlO7VR1yvApuvx}{b=DBs78D+yOlGY# zfIS2z0icWr#M4+`pGlFt)OGE~jXH3$7IabY{GtHi{0K~T9Q>BFkwGviD8Fj6;7%+g zockD>OF;I~ep@fTd?y2$O@s=VxD;VPM9irJLyJ5cC@PE-Uk00(lAsk% zZc)Yl;vhf?CXIeMzG>8EAAW{0m1=1=I7b_HUR<{8*W zfD-B2u*iVn1Z_lTVbvjyHDCou>>FS*Mp1_ZM}@N#4ux`v7_c%Tw~(_fNPd7__8>w& z@?(_~iJLIl@ucx_aJ;mf1q12tp-7K9ihMp@SHivKT^POF_AGHWNK$)4d6wq3QTkF;M~>h6eo495`#TD&IrLo z`Y1ViASUDxOCgCaa2>xRUD9x@YLs>Ha^o~P6!}5ZOm3^fM%3NWqM3bgu?f2Eh=#K6 z&uYF)P+FIOQWhysZ9)Q&vqp%ky!4>xbO0}@o3mb5S3C7ItM%Y;uXWM#=sARVhU4 zQ12{FaKu6sZbEHqU}Ti@?cu~fKvo*XSwSj@M`;A__4V~xpC&D2{C*vBWvd_bey?%C z7+@aA&LsH{HoGix<`ayrR5cB?Bvb?=!`#gV-2D8>ji0R5QBZ?b*U$AJyAcOG$y3O* zB4jCh2wQzfB&TiFkfAm@vI8&?*g1#;^h0coQ+xdBamOEccrHJ|!8Q%FkMtRCONG=* z$}wV;Cnvv<(v%Z|rTH-Y;T|{46<+_3M}2fM^dWewpzP(R97r24Ia@=v#%um7yXNj- zGygbSG%dc#;y0JB!5rM^0~%|%7wfDwyf^uCuTBKN4C;fb3?c%sx^-4=*0@TCOx~rg z=$9CSDx~K0yE+Nu2te{Ym8+NeFKM2^fED0IyWzRrGuV`F9!pM;YAp0Bgmz?@dWL1b zlqNady#uw&A#{b*qrbvldeaXCe<+{7bPC=8(9F!ElarIg>n_|y_@MZ`8V`}|8!T{y zm48br)|tj=>tYS>f6(UBWpV^n^8{F1?811`)lm0MR zPgBpf+ol8c_o9Bh#nEE@VhvKg0L>W0B!w}6=_}5I5Y>R$7xv6^LY;E~}n19+tk z?p7UN=_M;BmKgWlgb~~fn7KDu&tr^cCNRqlGB1j>sMS-*i8rJSN24gR$?;Tq7vlIv zkFdts#Mh8b>oHR%iHFGP5OQNUq(qS(InjOS*!30UG$=YaORXke?N%^O`I(D{C!u*5 z1b0XHI=bFh6qM|&YXW9yWW+Wg-IAPIhf~7-=viP7Dn! zpx~353s>Bh6uEEf`)xFJU){QY-nwI{`X~)g(oPmPK z8Cje68L2v*pfGAgk_M@q7jYZNz9Gv?R+*f(Wk%z#N#5#s~f9O(vzJ~>1749fGn*7q+2yin32ja^bz^;GoAN$>0pNS!2}K4t|)J* z=wGL#qzo~KZ93BNPtagr**P?n0&u8I*^eD@5yzIfF3)*B2l|GF<~I&m(y{|H{lFXH zVnWe!^x#l9w8CUG41sArPKh$948j3_sg+$==cuY3OLZ|if=B3tT~XA_moH7wd$%Yu zVUZ?%@MhTcJ+h;0<1qcw_KftSAjIPzqqtQb-5ibo5!)$OBBjylJhlxR&khdyV~Ur<)-Z-6o*+5j%Yw8_8q~<)Ohh|cXw$LeHUAp{ z2R%xSy+3n1t>3gM7SLyFF`vH(nzK<*r_fuY#79nW9@i)Ui~@O~M+poIqXK`YH19>n z(pxMUc?qx2w`9JvdXw@I_zq1n-7qT<{)L4uW76CVIfT&hxKEB8s@>~EoB0Gaq76n=zc5_8bs56#k4wet;xTHhsc#_~; z6Ric7o;VUEA5>5(Ks~}90_UIhpsGl*@gNm70kJ0m4&jjmaMi?e$H+@=lc*W&K`u2~ zOpJ_>b~msC343_xVY2NSz&qNZH1Ju{xrookB4k`G*@*)$8qHH^#Ht0Z2Ni3GoA`$0 z-Am8WR)2x>nx7vH9wiQfW9+jS0DG=_qECv#fxS#ZfbxaQGsI2NWGNh$ zJA@q7J%@`ovGOxe7@b^P(2&L<3%f)OltSJe$FlY=)qRSn`~2CZEVXfVUB>fv8JyP) z`-*H0RLGu$$8FnKG(X}LTBRk^n-5R;CQ{Y$h!;5XU(b8d)WsxJ$o9}$^J z%MEr5n>MtKye^mJaP)(XBD1@1ReUta)sQ+ zMG2gAo=2z;yt`Us?a~-JyRTD(nF*FbQg5j_as7>Ty`_F(woA$Z<-EGdEZW}V5Ef*W zvBk8vY;BG}<4v{)*7lMGvsnv~*mb-=9-^u$ERl~nhBa*BX9yglmut%C)FB7o5Lv(#C-Nn#D~{oG1Q8K#$&ehUE9G1xL{^e5i^MQyV955PE^3#N|U}*V3`zyyq!<%*wNi$Il#DGF%yn@*g zdHutrUL8dPvI)fO={~)M1gvQNe#ByOg5F%0)Pjxa$;@V3Uvhw|Yz@q3OQwwXarS(ptBF4xAiWr44vm9pvBoVIV)SW$tA;VHgyfT4(f~em( zmLUnt!UShkV)fUeBqdmZu*n^%Bsi9gp(G#0&hG;p5}t{D^r#m-!s^xG53As1iRc=DoWn3x~RlfqulXE6;LJnUG#hhxu8TFny?CLeMu5?Tg;CPUH08 z_6e!-m#uWouV`a{B-3tCDaNnw-QxJaSbOid9{V?Z_$nbqgoY?-&`ybVlGQ>|Nm6OZ zjE1B^MxvplBt%g(B+4iik#hrm-_xl{j zah%7g+24sGF{`z?`N~6>3sP}LM* zkrgJYtuui1XrkP?c?Bgu*jVygUt2Ax*;}?iE_lMlLsgaoVJUn!EYL{KBbI|2j@pe( zJF@LWAWcGdPaa9c5u#z=fw@b$;I30Eg(_jR6APL{3OwpR_~m^KIrw`if*`c7IYH4$ z&oJDLrWFP2CbQ-`I8Hzql&Lh=W8BC+apSHC;E@jMz^rU!?KO|htDJEiU(nM<5Zr+a zPk4c#nrq!!G}ONU#qPx%_%_sD1-;Zg4;XbUg8U9TvUM~!$vauYKjmR<6VjsRv_ZDz zf>}5paIEJj#Y#HuI+VYo@ZSro{2h3Aq-HG*^o3IrbuA5^9Vvt~^mq>yV-2?`P2g^% zoRrI}Ixzh>H6fLLSHd-zvt8RLBxgBte$oP-S*1yMhdOLqp$v`2nKjG8sbP z%}7VwlQIBo{|y88+weV-V69Wye8_RQYdf~`6PrnR5d>| zYQ`DW_=f9n`Zoz(fpm*5v?hQLf(lW5Ud2t5gbxowE`g1h<|Y4J^e>NFKl#6h=u7e* z^q~hwv@+nL(C|fUmoDMt5frtC(@8Aw7ya5(s;EoJ-`~o+XE~mvA>Psp)N_DiMWBpA z!9!&-O851EZ7lyD@EE3D9X-IMtqq$8lT?$9poqkI{_Z&$60#nj1bw@jvQ-$akhEmU zzf?3qo+$OC{%C&QdGh)Xil0LnV~_R87C!=W92^~q|1-IiRL)6CO;jmv@ejve9IeNF zW|`xeit7QExbM+(*>oeF-VRVqk*a^eM=ZJ#Xm*^XD#c^!bKbi?1O2mQ7qSg;L;%)J zp}&BndIw&I1DqJy_KgUe!9YqKxcCcDs8iLe+uPRJ&3sQB*~=N9(Iyb}6x5HIZWG%$ z-{zX{176d8*Gzu#1RG$_#TZeNU-gG?u>_ob(2EqqUjvJ}*lLaJlgglyj5dM*&a_mXRq+@E!L7n?b*oaKR3w9b|t}1ciMC=hJ~!@CBmxz<);9xE|*ShB7nso&(%POR&ko zdCd1=6-b`FWTthR{|pGBC~hO;c0jOa0t3%Vcl;ficgA6Ie{X)%<`7Ar!y{Q9TG88DtV2?STI+h~4T!%t0vTW?L`DpPqzg8?=MEbed+qZ8^4WTuCcDKx_ z2_(;71UodnSuL&K{#9quLIND?J6Y)gIPj>P9b4RH?M0UZXzNwRZombYgBR8#ERUQ= zVt?+Oq2;ZT^}y0$SZC1)N=>DMJT?$OuvX!pWj1>P&L48?B_fyb8?s*3;9(ILz=J9) zj`Sh0##QPDV6aOp2!}J}^@UqNzXN=4D2j*NkS6$PNTHeQGbD9{G%DQ0+u=u! z!w*Z*Xu?*gw0!vkcg6?k?ZA(Ryl!Lb0b*RPuz!FuRj!m1t}%y1Vv;8gVMq%!9X5u`MG*gzC1dNk{`2dKxXDnizfZaR@IF7ZN6Ds}5YZRh zfElah-6m-fk;me_xvoIaU+hODTHl5C5dJG_bJJ?0J^rzL;)mvs7T|_TgyDJshaCq9cglvY-XI?_f zAIA6C;lKSQS(5`0ju0bH1D?O!v(PD8AN*q)_7%QN_cgTMxnKgfcGtYxLC~5+2|iFi zL2LVpQ$8CsCY7@|o1m7;fzWD$(5m5WI%v%5pg|4 z%z=d|N`GPJL;R{3$mO-9eoJ0@bS7$-J9R>{W?3#gG?3eUIjEXJnoK)@CH*Pnt)2e;;5>4yj+`%Qwm`E>W6by zhRFv^QY!5q?DqAM0C0*U%e`>6L{ndY4dVEB@*nIlz7AkgG9s7RKa3(hSd8{&0BOfj)`sKu#1{rb@_@oCw0E?I*{Xj%KrUY$n8j4=})=-u_E_ zuWMlN0`153i=Z_Gs-gNYh0ry#7xLl`oCwl5qh=S?Bl$Ye8+r9b9v3m&WhZ`F_T+EP zfnu2H2b(|+go&`~iPS7aZIIB+Do8#vM8Jl#fs!sXN# zt%tEJ9dV+q&?rg3!3&HE{W_WwtEi}OdH(~c1=}6sUC0EB?1OKx*NG0l`%=3AU&t^> zLujFCuzK|(vy}~QFIM5mO6(Ilc<>-O&>frUQ~~Xn3EYvdHIW3Vq2&p*Gaj|BlV>Ql zq!m715Df~sUJKP|>TrQDn?wqU*+x1#hf$D8o-S_L!zDg{z7b$05cz*zK0J8v8lN>$ zAAmA$)pnPaS5-yx(K4Aga8xE89FP#%^pMolR0P6UpawM!PpO=Lff}mB=NkU^swTDd zb?``8dmE7=hiL(9wn>)h~a6&DvSQs`mB1cbxo*8PWjhc11 zyTD+?6ZdTV09e9g<<^MH2R2Q zemn?BE6qcuVFWKCyzjeHaFbe{vkG|`2&KpH)~+GrP`D&dFTfZNezwir)o|aGOlJ|- zlNFl2{9om$Z*Q+E*)lSclCE);JwXPjb-`q7dHK}sCt`vBe8i5Kss(^MP-GiOStjz% zn6aT-1DYF>C;S=kmS>L?Mkz_NOK|2-(x1N#)}s=p!+ zeMA(7Y3I4WP;l2prvi(DYGs@FN}r!YBX5bHjt2y!jm)^npf;raP-qr5r6}NA>*I;u z^gMI=wB4*#-w*!lJGf|lA0rODF0o!~NS6>w&ZeXl+A$HW&N_0byGU{ha(o%k=V9=2 z)833>38-D$dqw$xZIQ<#rr&bBxsOe7n0Bh! zel_eze}C!Qx0f2cjpm@xOs^Ay3BaA9WpWvO7+-GjtKY1AsdT`4Xsq^fXc1`tg`QMB zOFMbz94^w>2q(*ysQXle@&X{sE>) zk;Tz14<*Ok2vLA9U_D-f*`Rqx{FOvpyNEA05U?#x_s8Ps^H)Q?RmHFOvp!ZL8KURW zF|*d5<=Uv9UjVz31XvBKJEzR`9UfCM0I`OTmS{8&Y{Wp*E96w@SpB7f_;03U@4D0_ z9x!lgGR}dd0(E$B9b^^g5{cZ1*wAtdmXw4-05UH~2O9SgI5HVxl{R#Z`%dnk8BSne z8yP2hhUu^sR04#8C@Gcz>)g<*~ZM%j9P4_ix1 zHblgGXo1+(g#<}1xU>p2Z#=}nz%^JBk<;A!Q{mo3*0$F#B2UqZk17BXN$`%m_1tzZ z9|Vy9c_F4KqU}IzBbh^xI%C1rGA-aXN*un(FngiM{;>OC|M^=kZOBZt&u)`Y9V)IE zbkf+Fwhc5WS4b(;OOS7)KaNKo^|aObQBH*Ebu&vhH@7Ob0lLojd|+VUf*ZS9{r(bG zHdY(c7y}m-x+}0fCSh*S*OCvvTu2w~Isr?Mcwu*E!QTBCKQu|-vI@Vf9jo~E?6w#yQ~W5%1m8k-m0H&~+j zT6iN1o@$LfcvE3$Ma4QAPVQ+n(AQ6^bJ|_Hbs+5An0o`2^wTZ0dceV1^*o=HR)Juz z4hqW`ZY9$Dzfk%6NcH z1^RcY6W0}H$bBm73Qz%^>%0lcgd=1!JNx|dA8jf(lMU9gzBp42G5&v3RUB0Ecd0kt1RqKGIbC=WpSnzBg9`tvg&bM8QPf6k;QUBKK7+V#n&Uf`( ze-7wj&CcN#jGhw!BRxWK;3l8)nnD?mHBIh8|JaaenJKuZ*!)6cx#PpUZv(@1u0vC} zet=z;uYAO}4-$2f{DhHjTUJ)8Zg&J9=zxMd>aBiM$df%DQNJ^l%|QK{&WT75;9H!T zCQ;f@Q`$8QPf*OIZECtnMjE$hKcI=n zZ^S)dxIc|u8%xeaN2dU0%<|D4w6ZcvkdUW;E4<{Pj8%3`hTMT!-p&!xe5mk<`hb`l z(#!a^Bq!XyD$Lj=0@~gL7XlF7IoBIfVWaUe+*wdasQYBxPeL(L9VJ}KnN?8wpe8n5sY1WyRfuhY3PUEeSJVJ{hHnOWSry*M& z-g?UwD^_IssPqrmviL&tu0vW{)Fk63p0ARn@4ftsSVWBE} zU_gsXsWYlt7+J`WyBOn4+(A%t;e)@wb|gnQv=*0^zJN2Xh#oP0Gt_Trw5EIy<2XGy zPaJd8C=S}9usHM};C{BiSs)ZJ@fD0{4yaG6fSD-LNC0BvOE@D&<}b7_nu4<1AdIvt z(5m_uTHSXmmS9bSs13z5o|s%AFlZ&Lp+Y9a{*e_xNPx6&^J|l^xb651YLOHiB>ga& zkVR>=$t=6O95ef)-?`>*Yx^Af=~#%f1Z*(6q38IC*I!?QhR)a5*SHzh8h5})=ozCJ zF-3lf%tvcPpb-Tb#7_Y=w*(Rco;}N-)r?VYC2Vma2rEnpxQOVsqtez+PEMIVXc>lm zzv?lGa)}Hu2(%_t0ApbmE9AN!qRM41aqb?hb7Cu%4hMYlwLY^fd zD!LT)C2VTK@SSH}tV4vsyQLRH2KYEVuehHVj_f>Dn%pE*4a>GuoD98jeaj=F&3G~i>}d`!F%4S7Q1B&vtW zX_C}^_>v*jgz|J9ctq!akTK=dD}b0=k|=@&5SMwR&G_Hqk%XYgkA;RKxnVbYe&T1) zAlz>o6CV42Bx!iqSu-+=GHAzgH2DG$8d_5T3luW@ZqrW{foP@RRR#5bpUnzRnp5eB zt`ta7@_Td?^!_mX=Y}GL+^Gcshkp84>-cZBSD&EJ^4dR>erGMr@n@SVZ1Z7?{)e2t zxu&6ohXa1FyK`TcNJ#K)UHj-{gj~)9wos`hI{rpW^>lodjFs!!{)m1mm!IRXueSD# z_u}gOR(r>8uholAFZB9emT4WKG1p}i2;41on`SRQFbnjcixF&_2L5!5E?mfix6VFk z(sFbqQkw2iSz^Z{D}xJBxU;XVzXub1c3{yRgeFgzz^!f!;myWuy_7gSWgB#SVTJa= zsK;J^ocXdun)Mg7Sf7FUHi1!TJ&Eaq$AEG^e|@p$3Z{|zX5Of?vpPQbC(bB7z}K=8 z`mWEAz?|6U5(Qm`A5us&Z47m_Z=oe5N06^fJj-x>%Y>E=ec~INL3*6k*sqAhD5It& zCdLM=X(pcW&i6&Lmo$Kb=Qc(D1s!|*#0lq-mn9YtJV{hRfg8OQ zhML$XOjv?~f*UJ!IxC8b&+-j@KuP-fSX*q?@$G4%?U~yTUbVY%9X~qs;5+BY$V|jZ zEay66Ii*);@H2^xjg)6a2|Qct_vfWO9Z1P`c(+VIb;$vep{PDXnJ6CR|KWgSMB72Z zZ=N}0MCJ_sHkV1Ddgj6DeSA*`Qcf(MKI02AVKXX$MVCbJyJ!&ZY44QU!~2K+O4#92 zJ{DXMPISA|9xf91^lgg2qu=eycUV~aN+kxO9>L$FwlOr&D7k7&NaglBBOktKp_sUM zM9Ciwu+~lAfpLfzbD|xo)$m_vxPMNjCD-K1ll?EAx8xiBUYY4*61cv4YPxsU9725% zgBBP+7CRisho*EE`o0ZVjLV8yv0UI^(Nknq0}(lD?U0oOr%ZsmJ^j&Jex=d|pZKiO zwyA&tV~rEvb#(=UjL9gw1(^hQuMXbG>f>iF`2FWQ?qh{8Y-3{lebjeiPRcu8m9<9p z;0^O25*>p|&)hBhkxeen8N;A>mZ1#qb!?^F6JYk83WndY8U%mV?p4GoybXEe@B8Q{ zo8XO|gD;Iy!CH-#``MP{A_JrR_Q-4M>jl0uwO0pA7AAml-yB&}^lkp|kuwgRa9CJH z{dm%LSe1F;m6mG2e?gd?6>>&rcn~S}qxcR+7c8u#G#x+Zux0tNRaJf=A(IK{uKx#S z#|CX6SJ2QQI(}W8ot1_nyz#RfzDIr0K?78PFuh^>rOK_#0KkWahnsAEb393I_|g@b zU6;Fcns`AImWt3hOK+mpX=!PB2V3<-H|Q(vSQwNCDp-r6nJ31Fc`l3gzc}vd5>tnc z^?hq_byB8`l5^9towaGWS!4MZY%%m}ZgI)X_=g`rG@aM=7PXO!u<;gq(Q19UW_(8L z&A2K2>h3Zq6Ut7~FhIs1aF4IWwOm%AbW3nseEQ&UfcXR=3>zaco#Xn(dt4|;^TDLc zT6mU)^G#)Afk56p__QVG@Cmb*=4Gi};_0adCE^5zUuo+U%6wHnU{Ii)Xu`@Msf&k% zSVm^rKSS3uYx?k!Y6wAU+>EB3@x`&U!Q7`=F!pS5-;3f)vihHIz=hIqLW&tD^8@4V z8@D{fCIq)^S$c-gVwl$X372u2 z%9n8toG9*7DdFW@xfg&>zr*3V)4v~jP>EVg@lPdypSpaW;&d~NK=6vNvqhR`4Zhds zcvX-5R8?AodlO=btc#)&670LNQPRE_Z(k}R$z=U(gD4jrrdOwG(&)oMSOIM;Ld96t zrIjTj!Y_=i7@PV0C7sIq3L~vj`96Imr!g5h7a!n`5#90<5_^oqhTcSZAH5s0=k-|82%LKLWt!?zz{r6&~RO>#yq&0UwDtF-|UD%HiWy!N6I zKHHZvp&S+fk`r8pH8GAtz1MatKg!sszwx{(w+J!M)QQDur+_Zyap2uIgpc z>Ug(eZ$<$; zP*9)>DTIi>5<^w`y{r2$*E0^-WC3u=&nR9SqZ}_T0q`p;kcB}_K!f9%`P*Mg6my0b z-5JTdcU}5hb1~O+5-?3HS`roSAM*z`n{oR3HnBz{ijG0R@z;sX1#sX`I-)NbSjWH- zlVLz>Hf{mx`S8t?X?@!hU%B-rT9%0dO1c`|pnf7Way<|&Bmf@(9c^8=NOdxOL7;j*N>~O~R=tK)p385U53uHlh|4vLNY9oa(3@o;C zaRe$*SCw7VoEa%^JQ6oa%khm0{Ou+;cksm=)l?$HVM>;7`jrO2rt3nxm#B0K`w`UfyblSE<$^K!T?* zdjN|SN7_FtD|-T|p7ZB-OOgwK#r{V#nvt>bR7O}sqy8h_aJ<_1 zO*x*XNjH~Wym-+(eaj@GhHlj)D5AxD1av(Zil;Ot;ar;7Bgc=tLm$T^8i5oI;hp;o z`^|$^5-KYEs7@Y1{ab-vJ-rZ*lOJIJ6MSZK#$?s#u{KsyXEq|0G7Na)*y@g06N|pC zxcK;pD1+^&W|3#|p=)-=W4LaaITeW>q~JIufk{9C=qDSVQ<{L!bIh{))!q&Eq6V+E z;00Q@c*;=P<#Q})4KMp-p2xj^PK3T6YP`*$M3JXepanUEV-0Fk=h_QmpP(>QB{>EZ zz>NHc(VD^4(N=0kbEMa~PmP85hX%(~=XA9?B0 z>=qZ*pcQt0v}H)^jX%Fy|3tMXXc=4Y=s17Jj+!-WLn|Gtf` zGL%(Mx-cO#8HE{dh^A}XL<(aJ94Q#I3Nrh>m%YM*+RtCUBz+~@mGR4m+KZO^a5a)} zdUkFh$+mbyf&&Bdzz(r6=(M-OyR)T-dqd!Gk(U;+k&gxL4YqI}VSKR&I0mqF%>y5= z_NBupGP#_8>YUw1g@iE`<>xLgo}YM>wFUU44KZY z{w;T0f27UZ+4z0AHZY@_GzU(&CjQ}j*S&6owuy8y8^23-25pxaOfeoekB)O@7cIWMh!MZd6s;=o& zu}xc}F-LCvnXs_Kz;$`Bie^W=ow|GHabQ(C@x~i`-T-w#D{;cw|Jbq1NR^E5<;xH5hd0g3aTJS(d&sJ3Q2Fb`^kR4#3rUlVErHXaqeK?s3yp{vi`Et7?-QJ4 z%oRX3d@4Qfjs=b_x{Y9*;ASu(h44WN#=qp~zuf&D5bpNB&_Rq}KGZ+92w|0R2t8}h z93g;WPk@v;<6}Wbk(ZY@23}Ji>N6Jr5jzG-QbpY|nJOGSB6{;_&;KJod0sj*u zSOX>rEuOXm0FUSGw<-$fBxS-Jc2)wscv(!> zj=s2-axPz)D`v?G7%dN)y;z9ZgN_y7SY25nEfDyrexGaCzQK!o`6GlpzTMS1Zm^oVux|eZ?h?ID- z2+#NH*RRI?IXO8BxOYJU3pw!U7xr(tl(fiz<1zRk8M#?Qkx%43;PpJz^SAKkcfQG< zk(rqp|9$+pakca;#zlKI%5nZ1@`YKH4x_>1^4k@RVmnm*_k%(}Zsjy8+Km+@@JsSP^a(`X_h_6xF zsQ_eT1UWh0U422-b^iV@EiRnC(>a!BDj$T6z-GW)<{0G^)e+-+GWt7~YrDIkTF}6> zWK%WTGLJ9C^KN0mdp$54+;58%Uid%eL!|u~b7Hh?v8e~ahzA{Tdr(Djv58bft(}`% z^aw_Dxu2Vl&mTqg`rr3aCY;ssQKR>QT=2n74Yv&z12$+PsJ{(_EVAbZR!{NZ>x4`b znddl~N^pGDH)MM#AfYia=tfiy{?#DhiLM0zEg)j}=W}0QArnU>VE({s%s0 zG>?`l_i4?c%C9@cezha2N6MHK4=L5*VA>8;T`&p}2Jmcm&LY#ROP=4|%kfets(TtJG@TU7aMv>Dmy zO}69)%N8LJHMMp%xq(9&Y#|C*ippQC&X6=C1LNS|^8V*o#Z+z2-!*x1NdLJ!r5gze zy1*EG%)>f|fI+J9I3=fZB;RME!r?0TKRpj+19bFxAovup;17`-GGHORao&HYjWH(o zqze#~h2~#WYZf@kpitKVXk z@M||cjRdhY>=8~XJm0L_%%fNPV-V{d zdhkb(edXV2#@rT1{PV8!3r|G^t+-Na*Wz;TZq5)fdGRYi3n*thOnT>ZRGB8jq0w6cPyECeE`bvVM z@gHaW{oB8r$;I>?AwIC&y@Hw>#L8Hzu*bt~1|@oY!DJx~el)!MJnCD#M=x6aefAIN zGMm2Rm^CYo)zZI$4|uv-|M2}kXW@H?l8(VS^MMxgakmOW#(}m4cc78v%VM^w*wr+j*V+J zaV`4jQ_6&zfgkw}7X+4(lJe691%q)R)m6`>VeFR*s-5*2-0((ahcHW8V-e71!X2=c z@FbS8jX?=}k~7?rWAroK>t=&PG)s&}%HAl$wwA3@uKA zyIDZJK2`qA#sK(Q;1vWN=SUiI*>p8KRC+ISKhkh+9(bww7$Zgk>TJm313L!N>#*V6 zrD&K4t^q8iljzo=B&c_EG|_K#urL52{m@Rc zANED^i9y%SFsOGyklnXe-r*uRdR8p^%p^I3_!V?47j*z~adlDp=LqiH1D;Fs9ZIHK zW_whEJvpxcfEBHh!M+DWk;vsJ4By!hG|{~MzXGpEP_+vlP4+)Q(aa#XzdxR^ZQg5ZE)_uG%+BC+=Aq~Nj7gJs zkd@GGR&F67t_XdHN=r~M;W9T+);g;r-@@}fbf;6&ud+Kn-+OyqN52!So4CLojA1|p z3TtW#VrEN(1`?EZznKN8wvoj2E$^|=1_Ac>oDG(ktKqpfZR%7bNXClWt|o>2=k$Q| z%JJeuVcOcQj?x9&T|d1$9sv2p6|}4>9-oEb$sdFlBIJ@;r@TpIyh3Cr*4?=O=cCBV zrqpML3bU)gDkQ%bHrjeKc!7nSfUHGSRdogmmIAQj679cD{)2kjI0Qt?nBSye0N)8# zDoP#wmpUEY=RL>=aA(T<=^Vn>Vg*#e#%11wj3(0jf~IxufzQ7-%_Hh^4uwr)PXWns~uszT;HKN4Uv}?VkexZ^C8C zUErL|)9fc)^!zBP0hIJOU>1t@Ze6ddn&LYWJ#9z&twtb6D9!j)_Qc?KFHNo*L|f2D z3WEtH4i?|OvAz8S5?kKew{I`+ZKr-;33{A2b53)@7dOT2Ao%$sBOv=Gd~Z}Nd))aQ zp!*&}0f-?4l&TA;fsf-&a--_q`aEs|*diGVwZE9mvaejZaI}_roD%4386nJG11V2t zKd>~~4c%ke_U6aBs&&|U(thB^pVdRx{XRk?T$~^pY(f>q`z-!NdmKk#J=^GI<7HVJ zf7?Qq2ydWU1hnS^W(~xQ`_k0vF$}?$NaX3gcse{7Hd^F$0VKrc%~&A3z6C%JRBE%B zT(sG5-n=0mA0Q|Ij4^jCCPf);d7%pO6TMUDxpM(jaDVgUpE!=nZZ|cQfk3d?At7%) z`4A*_d~Yw8_E4<2oOEL31r#Wucw?~fY|szMb1JK;EnbB}=fwz3j} z2Oxp1lqCV3WbE&q3-~YI8Q0s)oGly{we2;~l%~I(H*{cC$tX~~loR#0p5o-p$)g8V zhBa+WrsYYn2de55i@}Or!G+A&+i}L!;9SH@Jj84WQF*%W-Qf&L!CQl>xXY6Ze}M)q z`l2`jrSjAt4E3{?aQj;mQO0@d&uEd9WV~D(h!~1`jHZY>)n`!IeXgs>HVvXa^_A~0 z;#3|*c|*SyKL$Iq^DqzCoy9Jpp)m`3u~RDgeN(4QL0mWOegH*sdh=2poj9%`^s1-} zjWXeXHJG|biG2dSIQwB6cS}^aH!&$bewnU`>d1FBGri2Nn_Sg|T>b>-GYB_9RsDig zJn$y!IMVYHVBYU7RUZl2yt_9n4Z|CU`uWT|0AaHs*EWCPtqN#x*S8g2;`kyb9c zn{|gUC@3@_Q)Yn}Y(D=)_K6dh5#WaGO~)5tqDHHisW8UdSU*yCiUF#&@JG%nH!9V!Vxj(i#{lm~ZWY zoVRK5zn_!AKXeH>;;>S)g_P2S{;8UUP zlpDCBsF;>wf|g+*D`Obca6vPbxfn{@RA`ysvK>n^i>UOb0C4_+>99`TW)!Ik(EVPw zxWj=H-zqc>bt-bQAnG)bA9(=rDz9%?kd@zJNZ23_ODKpECbYEmLqnD_m1>ZhnguR{ zi?%hOsy8`1+`adDoE&-{Wtm=mas4a-NcqYQu^1L66MR~gl1c%VYaI3$H35(^ijMiw%p zkvbp+z*a6VV7;Zlb{J9taC7=fT``d&Y29yBCrGS6Q2Bla-B=_30EH2B#}o0BUH!&t zi?@ICqzWD+Pyv3CQ{>qTvM@=MVOSY-(AXVr7^OHCFR1`%Z8U$hoyn`awe@?Er^fi{ z%-O3u7?_tiIU(*4A46jxg-FOS8NppBbZ zjP!smq1RtV(i=WL$H=q)Los%&)<%EEb8l6{fo=ySfl{ArRB|LdAhrZu+iZ+j-;UdT z8N6l%W;u*^>|coj0OaQfsF9#8nG;z`FeC=QMNxRbbktv%9jvbh*tWG8;KB)8pLX_w zUZRODj0X7U+b}4~j{z|s8-sm67 zOeRn?gq619Mw(<6nuPNSEZ8y;8+?xAcQ=C=EQClyELaxz4!7+vsQTGE@DQ*|I}rWa zaRWcpmd!v~kOJpH1~V0AJ>!|AZ3^e)fOg#pD|tK+;rNz#*SC1}JY6I@Y9{F$C2xzEDLqC0@Ju7@RMZ zD{!?4hIst=+dy?(s*RZ^0`T8#aDWG3B+B8qEN7K>U-Vdc@ug*B=0r};CV|4-IV|$PNEpj=OVYb(%i68jYJ`eoI-Fmxu zhG!Xu9xjPK0WDlkNMI)cVno;URjs3lRqLzLLs3&zLEJk8s3RIP)8`$XIBscXv%#BH zdw+g$C~m@cFIrwS z!;CNPLi)?F&8EQv_qr_QkJSdgq3?A0jyw)dyteu>8HyhEtqn zj8d&M;nbS?m^CI0=1)QBDzD)8GQJ@68%aZ*zx>8%?MggXR2a0zk-P3MSaGFNSW+^v zzDK;;PPsvIt{-^#fD;LPRvs)En7`|BJu;FHf@#g34%jN`@|Ln&tno z7r2aAV5tc{GiVXBkjP^DrVOB-Coa4Ah<~c@{qiNPRJD84qVgG1{ZU)6{8a63TKtpY z>@{tj74|Yr!%dgIlZv*%DA9!Rh4;;r6<2P({EL!;U}Gl6M5+?v|KcXaRzDm&yAQ1ND{ z0vKPE*^wW0?2eXBr!zwK^r9Jkm>uJ8Lm-SY9Nw-|~y(?8Y;F;sN z{0F~55%$>BuB(5bEDygfy1$GBOen#6V+3dR&HcSwh|R=g$s=4;76z!&vaMSa?46@r zR1sd9RIfYZ*4)~fsMhLte55{&#_9a|>(}Lj|Nb(*lo+85i~o+vqK(=87|5T1>PX`l zmtX*)$`B#&hIqkkM!ajv3l#6r2JtbUi|1W3*Zd0rei;f5u%Cb|=54UN(TNX3B!X{~ z+-lkh=CbTN5RSEt54_aP8cMAM!-n2U{VGJr3Oq0-7ZKiii{4GN5$4{L4?s{(nFu6B z9?FGNXU>dckk&jfyCojH1545%LxdzE`nAPXOAMwzK8+=!udf@R2l`)N5%w zms+k18_OVm`e78!2BKUQ%(h0|Lf3V6@j)$0)p~Y_=s0IAP<@2p1ls~{ME1G9+%9DH zVmY~W-{HN55?|+e4Jr7P1Xd*$0;$hPsoZL0ZZ2^k#Ym#$_Wc7$^W3;~{z9JY5M9K7 z2fYBtF&2!V%|bW9yo17*x6aC77sB$^SJ!8t>Vn$#)P)O_c0@tppcVw20?R2aV3&T_ zy=naRyeiHXZWsIH$t$7p!DpBvAP@kiS|UOp{{UfoiFEi_wJbU{jdi#Q`M7NE=|(^t zOKojM1_UedXB>7{CGn6(gKQFY_4TC|Xw9EL1>lkBefKV@ZI(n=z>c5?La5DfSLfIC zR6>m1Qdxs2)r97HoY95eFJHfg|F0g({&CJdJ<}zJg@qA#5oDHHSP0%`6gV;3r?X@Ui_n1|n}36IXmW8Zw^b4V zMj)g0H`ZV1G2C(tG{3`F@g2^|ct!((3v4Sly3Ek?=L2!tg<)W;;SN(f^cQwfIwFn= z{1c0wKFtT>Opg~09MEL;!;GH}ON9Pdl#d_;VqrkEXr3KDZi(A%sdl(36hdC(@IrMY zW*m9si>Aa5eIX>0D{;{1HemdqE+stxp$+wK2>N(|?RNy=nR$g5TN%#}yeq(NR;Yf*}WHFsZ^_*2|A+FFwW=IVu*UibrwE{|@DZISn zfG{9HQ*)DX;Lm*z!)43Zu*YoIqDu5*pLFkU@!20I!RiSMHvD64q9%vdfOIQ>i2>H^ zwZ3&<-QZ#eiOs-Rag7x7*AX~my|EF2@kO=_w+MRxE+6)1oCrB65g>;l*$!wc1C6VY zQGI|3du^{bHoUoa#2KC(`TTl^z<9xWza`ZDDFQ^nn8_M8)vf^IaAqvL(L1To++JG` zCPHRudGiyxgyIs|_hih?qL{P+Ahi{`_&aMXd3brxSd`z;z$kqon1NXUF9-i@AU^>> z3MU$#+0Exq@{x;|8iD-d-qqi^TGR?_{J?Il1=m2RC2Y!C1Z|Y)5);$YR;kKfLFtkH z!z86h`$$7-oDlYD7W^6x11^{er3(yjSE3kFKx^b5Z5?a$4vYr`p3qiw9pG}Am14Hz zJO3Svt`Uf~{xq;8Xk*~DdX&zVk^t#OaM<>lPlPIVkFHfwz2cXaQ9Y`pNt8$oglgW1 zpyrS=1NN}k{uaMw6IPai0VzP0!NLF=klZ&Rxj)`Ss^__ZyxyYOrjv&S&D;%_UcNLL zXm;@TQ8*fOtxK+Q2=JxCyYubC*#q!Arn8r(nd5Uu@rfk==j%@8^q(gtMs5*jfkj8L z#so|R2@FlHuC8t_Cbn$fv!5t?(eAb-WNo6!2>Ds$v-9l(K^WJbRo~|{@R#7)Qbt2u z_2t}CqejocBebe?m#OOolG|hlBPjK8)4={9AKH{J_)}# z9}~AA$xRwn(CWwuK7?^&;G5IaYNrjuO=4s3$+4D{luX^Y=-!TME;K94F)=J7wEH8N zJu+6)&=T1JXehm~sMYC@w83DrBo_7L$+FHjIWx6;f~_DuT!~;JE3_M?R=m6-l9I)8 z3R5?fG<1R7y7flXte5>^Z=xEH%n^UxVqOH*s#(yuhPtz|%;}cj% zj1PWeVy1E0aETB_q^>in-dcw@3pSqTBZKbSEK~K$qV~s@dCR_jayqd;MZ(HpAS-eECW&Sl6d2Y+TGnJ%riI?R27 zYtC0nXz!S>lM=dAZeL2xU`iNJ;{j70D4GEWN=y$h?thx|NU1?#90Q=``FLIXu?AC5 zdgiq14GooCPC~P&zh;g6zQw&dcQXOe5er6>6BziA?K=y_$z~XpV+X>q+U#SiujSz~ z9>!+fcZihn-(inNSV2DU&n}DkF32gQKJY^5aMwYIHQ6RQ3HzHC%x%>!- zRu5M2*#bKv(oI7{qa?a?tI_~Zc+W|Y$*rrKiBI5%hXYcF5-W*Ouxc(BaxL3?pSEEM z1_da;eoakHbfX-2G&oJ9RQ}xCyA;Yrkb(H$10bATGWYb7)M`19 zbNrrFlek;V>#OOZ;Px3E2BS078s6R=K;*+9nkk0t-)K;Sn zm`Y;u_}laoK|>hGg#GADe^%1vyUyJFIKe**a)}=j$ceGq%#@ik15rahuC7K80&9rc z-Z~el6tj(uTI-8Aki_ixm0CWI#^BngeXp+Qs* z{_DgQOPbgRG33rt^mEL+%=Z_k-GkHVHp-6RxsD36J9N_?>T%SV<0!vF6k&V;szW>T zCRSluL?K45Iu+13jV4V^dPhuv`gIb=pv9bz9B1idMTW-s?&BaK2!mP3YW3$S3*@ncI(S<*pt0O$n8@S8{J`LZkA=+L6JCFrB^Q7|Y>gqZb{QSN@ zv>PfUi#CzUfQGP`TwGYp0)BXzI1SYYNf?=2sQCi1h7UC{8%(RU&CGZiVPRng7sO8| zc8}Q)Qyr`}%ZLUHv*jPY^OG(Vs=aA0oQ*AWX-89u`-C#(;Q?rPHATPa8F?%BOOEMAMTe=3L(jVqFjJ03GPlO@yrt# zFg?NeM+LsSf1!bi5IO+}TH6Q2UIE93n(S{VC8>D_{AHDBdl3cOt&S3OgL@ESjGHl! zb`)KHECfYd!S|#T_nNW>sQ;=jHDQZ0l92Lwu<`cRwOb5LP1ZUU*6xq}R=6%^Vfd!< zRr(e=+#M&c)i`t~-5d%cr>09CbZnoC&B4v}_CQSo`K0%m({ z!FfRot`376^Y+(oTYuvAR|2S1U%v820k%gyx1St2W~c^{vNkc9j-kCn@X30I@2P8O zxC|P7$Wln747o&81GIjTS~h%ZH_QoH(5rLJxcl`kC$CF|YY{^h-j+P{y zl`>^BF$VOR$9X^>6K}8}egtZ6^L+jbyyG(?OS+Gbv8ie}Jd`f0Owr8)>WX$hfLDvB zLjAg{#YEVGY}|cflkY|EFM2RZj47kw(sUPAJhiH;3Qby97e&anw_XRfCLyd^s-S$a zp-AQ^Y8AOiy+zkVBNfwE@(n#4Nz+;pVMiQF?l&+RR1E3w@6W?flVX&^Znvi?V^^C7 zW{Q6V{xHHdb2p@-FwUuGje%a-!^6Y;0$p*OSFc+22VHn!sq z72Mfyp9hI(JcAty<3OY=s4I_TXUN#9{VxKd+UVfqY_x}2<{zjnq%7rSVP z4@6I>jwxCKP%^QoK>#3yEn0QNY!kJ6k8Gy&3PI%%R?wxdxIntwq8V?@Qf_INu)M#_ z1haWH2G8Sj>wP2_4BpwqOmFTAa4rg%Kgzn~2%SLK(}?hJF{g*vrzyjT1fsw-y}x+g zyuy(YJk_^o8A~^r8R+9;qOw>lc0LtoH3*g8H zuISSJ;J6xCfd9aQ4|4>=ypYpRnKo@*w=?=E$%O^bN50>;C-Hh{AvzbXZ%1x@b3512 zro7Es-u(H`|2(yxO@V@Bb5{SrD{{y3)2(IV6=(|nbDl$z&hS~FrUaSC0L%fIW`kL; z1T34#Z#LllYVzttR*RMw4@>6Z5{~^BqWA=A)uwKb_gR;}J$}~YyLYd9lo?-*FPpXG zV|>|QhUk3Ms?LJFvU#1xn~;=g4)D~ua~uqzc_?T7@qk4z-wAmRdXdM-?lk1yGs$)5 zPMJ3c|M@^Ysb|-X965wsAt}-5OIZ0DQw+MUK|qAxTw`nN*_LsGwb!Wy08F@m>n46Z zi&WeMy6+KS1_WeUH{c1e%?H*r-M>1bKyp*Ip{X%9!L& zEDS!+64=?9}R$qq}V+=P3R2KKs8|AL2y2UpHfE&9-!!YDvsZd@&f}}I|r{Y?Yv7C|Fa2O3&Glrp^_Aht_5gW~#05Zf2BS3O1K5e`mjcVlP z<|>99gq)LJJqrVO`9~yuLCwU%5QB$?eHL;X8$**BC>A2VdMTitn+STJ6PFQ6QL%aR znsL_%azaH!3KNWx0!^4BDmuo`?+o7=yeC{GjKs}Cvc<8}#>lA7(xtH`QuL2;B4EK| zu(66%_fPNDk*xLO|50nz!qYB(sjhnIiHBAn4;5**ALKhTOZTzY%xC&qpK`WL-hHfi zQ;PKA!NlOAg#I4+i9S13MBu=m6c$Eio=y+}sDPDzQB2rBf2t3~=aO@xI?#V;)J}&-w z+SO`^IjKa1MwMi;8nj}n0RDtWKuy}>bpdExOrsp&Tn#+Ayw0Osru_(2XHkPSJsLk- zrZ)r80>db`(AzV<7_$dPJq{m`a6HHbn9i^(>3)ba7y$)g9|+))rtFfdbf&OjE*)Sw z$!LHo0oJ|ubM8{K3$mKtg1fJ4ivP^+T~TObJH9mXRQif74|t>7@|Pqlzjinf;~|-z zm)dv7VBTKkM&K>ka!K z-!O;@K9Lga*DNhxG5Gmh#-bCpw{U?GI)lLsvw1s6j|{AVm7ysG7}6LBToPs{fiIP4 zFJHdwKi3SJiH4!Yg99xiAKA?Js&2yPo{z6JBI<8BXK3Jr%?(rG4|CIVfQod8!bv*; z)={r{0slmE)FW@adsl(AtgoJhD7<>3t(Lm_Ob}gW!GJUmVe3VFAuw68A;5|DgXoKN zFR*OF8QAzZWvZn(CgM3fTvL?%w5PqUel-4yZA@t0y^PXLk&1&aIqaN+X4s~e!87xB zFFSJN$fpjclrX(;uTR=(rkgjUz&Nqo${^s}y2=1pVFi1x)<N*rdXpIk4gfw3CRZ6+i0uJ8QXo#7ovn1^jrDl>GRFfnK%->XX~d@~P?dYF zPTr=u*ktRs73s|{^;M7E3(74WBcPhSXN#@Bno86K*`LN+@RU#O)EYcDUapYLQ^1sP z63xcq_U!MkcbElVT72*TK6q$U)Ct7twZDInVgVvBP(71PI`%#-R%{EX?Y`%yGME_G zni>-m4+%113MkVRAZ`yVgWq$ zz3>>4$v!h3$h_#gW}nV_HJ;%QX1zU5^d*OCi%WH}*Zt;^!Z+>4nzX%Q4^pG*x1iyFG8RZoG+z#BmxBhzF^;ej!Z9jBhSBbE$?12`MBj2ECNY{o9cbA1)@6 z8;NK$WN;`RBQ-8ULw{nQXVj~!^U}hTM-I1QDhhq_b-I67Yh2wmmv;72b4w zCIkiY{w3VL5O0U5s*Ay z7uHTurPmquF0&eHEV7M4*ZB^45d-m=4bSvU^H9n5fao;ISPZ8D2tZtHhF;vVZc)@r z@O68nU}DVnKam| zgB#HlFN2Q(O~lcR1#IIcp*U?0WfgPTJ)+!`O|s_>O@{)HOhFjPWY!bxi||ZBNMN%; zsAV8z%e$P3Xi$=spELvZ)cSoplmtf9{L5aUSSZ9-A}tOH35in0WY!cRA=2)~g2m;1 zE4B!tM@-Sw{qMkAQj3dCxtWOx8i*zJUo!?4nKG^AO2w^|d301$#?} zw6&H`Y3MFaZ<<~k)hW<}i@Bci8ucfbv(KGbW?})BAijUnxG@_SPaBNS+@?!LzqsWA zRVwGnfUKz+7ik6FQ*2bHfcV?%z4bfjiVV@d?JZ#LkB#@xcu%y>QQ;q4D4(E<)v``lbmA z%JQ;cMWRkvHc0RAxaHco3T2#?KBL05N3FTXErTpl7I_7kCGQ;>E?U;2`=6)rV44w_ zL;bz&3VT^teWAiPc>Tz~h(`hzVY2^Y5q1OD60Ti9h7IeTDlhwq?T-aAbN;<>*ncK{ z25*?~<$tb-4!6MvJ-35=%OCk$Wn>mu#D#rCQphX!c0z(wad~+GBF`TD9q4vZi3vju z0^J$JHKImY?z*~Ujm~NoY*kVl;V{BVkb#I-6Pr6%1os%y6lk$feHHP|Vk_WBle4k%#9w?4fj9upKd^hCh(eewd6>0mZ z(j5=`=+A$$E#=VDY0k$a8V#`V@*Fn$Hay#F>HzLE*zm4EcXte_m8g5AR3&5_4o>}3Nz z)WCHx|0Q4sDN-wm>qN~MU=bqio0!G z_ma7vUvXzubbKR^D;!;YO-vGFF$MW+E5;9);b11;)?FyV>yKU9~$9+&_NCSn(gv{Ft_P`?r-b?&kEwEbkS za*^RO%p0-_z_~k7w=I!*UB)!~;j-{<)=NP_r_pkM99>7Y!pHK_G=+;Yzz39}N&x_z zlCapPd+jK`=+=J6fxmH_;y9It@fi9qJ7`4D@>;b8l+7&vWF#pV%o9YE(n3fw+!O?_ z^ipbD1J!a11<41z$z$+9)rnw9wiUBZW7nmJW7k@qXs}$cg$>B=cv?LkBoh0Y@R&2uN^&`-sa7B-#^^uCWS5*9!x{#y0nlPg5dKRBg&?=;x!> z%&kAsvTweh8niTQ;KiKB)`EpdfQ;`mTkye1C=K2{+C4jDG6}Crn*aCL5M7B$RrXKL zu*KPX`;7keCVeLy)Qyxr&IIiP{9e@LyN#ey1oT*l3Txc|jr=m#LyaEy<^be`Xo~)E zRZ0wD;PD8!J`)2-MOG2!XDi^v`o|Liv{rEDUrR~{|Kr{#if#-{VX!f938zYL**)bt1?l#pp+*=IES0=DH><=?gU5R%(meuLE4;F)6tf(&V4n){PK?@G(FXNc2@ILgI ziF0$WsuE!pcuB4xFjNV3%JR`g-~n+XA8poZ1KMvu2Cvkcwx`0YOAhUayK>wsL687Z?y^^s+|$e zl1}f9odQtT_MS8R{AjTlaPU}40&ch*iq%TN@C08`2PXQc5?)YVx7|)ZaPYP}1TZW9 zM~1Ec23-<=>Xswn*?oVfbDjRA5{I+V>`W{Y+x|gNH+@x=gj!=@*>M_wJs_hvku)=KmvoPP`0uhurlo86EO+*R@o4<`2C}tYE?}7=_FO+#nVP zQMY2lrmWlyy*4se=N5n|a)Ipi_Q1Y5cKGa>Gyhp^?55CWSUoa4P$JYN+A4t#PT{8v zPG=@wdHY+&=7N>~_FmbJfZL!+eC+3&7;^RVbv*$Y2y+z>5b%9btxSsV<{KB&2633W zk=%_@X$1yxWr?EhPg?IUUwH+FL(x?Xp>s-A*Qhw_9S?YH2gZ`3~^ksY0BWKE&2POH- z2P8S5P)9iob=DFTjbKf12PnYROiz>x-x!w|mo`w0U)JOiY&66vnGW?a2E!t#ECS(< z1H$mt_3qrzrn3-xbQPEqV6O4BxPXQy1Us%9*$Vj)48cF`ooN1JG9duv*vsGlyh&=N z%Qs;6MY*q`5)};S1dwpS$mFWiCm0{lL;=P=#J~ham6OCw*ey}>Gm}vSvO2tRu8$9) z8StI~n5ntwe>Xw-P@SQdGrOD??Ye!;iSdAl0>CmEjDYn&=V2lyzUULEyVOoX4fhPS zmk*qtw!iJa-|;@-ENB=Oa8rb40%sKziwn?d?Rzr_!znh{QU;4JvR@1;c4$}2Leq)$ zShU)O<_C=L@J0rAXZJcMNV~}-llmpxF-4 zSrZ)_uMs1zZXq!6wtfhiKAey_t}$<*pI z9?@8~?X3{F+Vj`B(2V^es>a>xgu#HM&j1BkDBwcP*=$R4G(P+9^f5+5RsSHvbqOYX z(iYDc42TCj0L!Xduz1|Q?luime-#vk5{urUucSr}0QKF*p1^-KZ{nQf#|E+GEj}bY zhxq}+r#sLZ7V{Mi3Xbi9wl|P#b1zE+QI|6a52oe`n-f|c%)B`U$}DAQt>7>drO(Bh zp?mJ+?Kys-Fckcs3x&J_Ls`%stomKXtrp;5yW?$@7AZ-v^p~QlAr&cT1fr62Cv);F zEdoD*dSu9!h!#hhP|39az3_}DX+k3YiZ{gUJDMKB1E0Y_0Sn0Tf!lG6M~gbK)S#?@ z%6k)Z9VF8xu^R^~6P`O-YjlcIp(9IVj4QeJ{^gvtj(y$$+y9pQXLaD4A*@@_u2xex zBpNz+5IffsJ_1S{nrmg)P+>UtwyCZ3pDs|}`+p~5Ngt&qP0n(i9cYp|#bbC|*d@N( zZw?50m%i=mouQyFC?=Nlb`+?l{eSIonQ@;oY>q%Oo~f`n`>&s6e4@Aa^rKs5YikMC z9b&P~=QDslivb04txZx*F9>1!H8bIyDJm{-g%f+G;wGFUN2rQE*_zpt(H1^KAd(Xj2 zsqgl#sYA&IKQR3^X>zZ_=xgC$UTx4JWKNO6;!Fm4WB{-=>v?O|QhsAIS`!8Xg)gxs z?$9LjXvrLN43P%%YUXGEJ2w;EteU2dLe-lfx-@Q)MK48r2$xdA>8VZ zqP`%)H>$|<8nKDTj4<(7gvJfE6$^tV+(~$)zn}vmjIm3>Nplp1AaIxD+VV{)OAy-p zP1`O+QDMz z9y}3$aB1x}4%VxRCwQTlQ-mn&E~udl4~TWZ-T3x8!do;rC}{7`2p|~$HAd9_ zb>|H%0@CS;lVd8>y&{fRVe)(Y_&REMQ9qLqTd4=Maf@ebR_%-N`RmA`uWuQ}^LgIe z_mU0;^7lmqSn|HfWGdK0A_pl$Gk)(`kaSIE(6K<$0@(1&)2@P-8C!Q2IEIKP*YFE1 zwKRBm4CyEWcN9~h8HS}pb;A$06WN{7T+=3TPMdV!@Nnb_aD32@Fw7h;2-qKq>(UFI ziqyV;jcZCb0(oglM7EubM>NpHZ(q`6Po*s&94vEXc!pyzY;1i8fHoksDP%QSo&@gx zbM}K7AE!BNIK!`$D&;=q7C2nT%$INcw}PQ*ARl?$3~UF>(@iDANeUUU#!dh=u_r)+oFaR; zWDvsv#mziI(VM|{hMtrTmz<(4DHr~Bq%$!R1Of5;<7SX@ML#X6pwVU zL+U5^S>3)$X`O@6Y%n+rTr-^(4#4t@T=G@X*^nVCNlun(*>40<8lDV@`TMU**p4NN z*#Ko#U$e(lc)(XP2o=8jPOcujrMZYnASE&620j3!mw>8m5*_UOS3r>%Qlca!&Q~{$ zc+bExjy75oJ-o>8g>p~C+Q%kd-qV$a-5$H#brd{vyh^F)s^}ipMlC=o^*>Rn!`BMN z0Php^*;gO=TnLiKQ-LV!LO^yPg#2^p^y(_ce_>%zs}Ce4dC>;5M;9So{(^{! zH_aBy3OgqUlp1%iFi`Oso}F#&$!9O0(_n~x$O(AAsI3^M&$#zhqmf=e4A?-1;phJk zizPp5`$qer6Mo@kO|Z7&U_I>`ZGHZ)(hcOO$GRs^IdZesn97|IMYxQFZTF|l^7bB* zwS}xh%asU)Zwua=V%7<}B9(!nOJySB;wlIjsc3-t5Zgu$jhCtwEPwR?mSXVOIflsL zTnc4{{Rsj>o0&XNgnEA>&H`|xI}e9DfP2C4N_-`c{s{~yL%a5WnlQMEF-IjzZ{gwV z%P{Mw;7``sTw;-Bt~fOaV@rBgfXCPLN>zXH|9mDi2!QTTT3%hKHXkq=-PIm+Y9um@ zkUPvv{x0zq?Ha{kxnq|96<{$F152GHWZ8Bnm6ApQT>MOM}4j~X2GO1A%YW7J7xx4FPUgG1CvIffKDP>qW!vy zku2j8-eCj=>|hkt8qs0I#te#EPcS8Q*k=# zDMsu&o(D>4^^=Q0n8A^p+xuLsiTl^3&A4meqmx?lrQ~8Yecx@jR}QZ84c)}df3L=|9%VpC@rWPXwY5}O+5?q?mawrFt6OIEAF);i=)sXX#h>`dZw1)FoRunc#Q47WoRqB$^&0_dWEFMT-Asry+ zU6c?xH?I6=$1ON1Hl5{Y9F+s;h-FcareSUdV&|i-7QF;LQc9rK?b@7Fae4iD&BBS{ ziFyO69s!}+p@25Zb(xK6;k{vbA)d-#17k&#rfuXXY|#@A(NC(> z+Psq2uiim|?aW}7ktEMp&&Q0bZDWpchYUDa!KzZc;3NQ%c*}<$SFc3;e#W0@{XFM` z=I`*tFzeA>uhDb!+iq4<`#5_15AX*5BX=jETeh)6_Fx zzLVNq1wJFIc>Rhjz?X_-j~9;MVJ1(1^tGsZFsS#v4wj*GV8T)5n-oJ=T=)ZwEiRvfC%{86RVT0&vY6;72;-$B=8NhArkCDz?i&K>^BU`$lhVB?bt(^ z1}#+)boGgXl{G+h%vSLN7+j)Q5I5%L>I(nm%_A+Ho%B;Pb3aI-?1bc`$92{ERB$h2 zue-nNOM2HPetDN<_@-DL*!Q#Gx=@r`UDtdk+tt)0bn-B7Lq|skcM&TjK6q@%WK;tB z4-j&Pwl)LX9PkUg6>t;xu1sljd~*N95ory03DUcDrQso59RagV!KV-Y`sK4>$~%X{ zZplacTPm-@!5i|ot;Z#ni;Hufnhd5fXlWM!v!}N7`mF303NHiYtmx2(AGwhx0h%W} z@jE~}i;2w>1auv*skpH#&tw)p|H%{O#-^qv%a+}K(Qoax6+UUWL=IDSjr`AH17kU; z$2I3Pz>5H@sIZd6pFRCgul?SzEY>|$Y>&@LkRAy#f|8`?^3VQ$RY-O=d=d(O%A9!* zHi&~`Ym;}fU)weX?-ovpH43Y)=$Gws0c#opbv_hqtPH4$_F^XQB)}sj zG~5|KCE6w^N1byPOuSt?9B;lk@EwvP0YE4Wz`}kYuT?i0fqeQH&=lDm@*z7-D{~6b z#8GFOhgZd0SHsMYv`BkK?pElvnv42~3^|rc&WKAJPZAP$U=_Z(c`g~i&`R?jt1${fK-@*ez>z@6cmx3qI&<@$Ffhst z1T96+^k69~M=$R$Kjv5G?wr%McJ+xBM}BFWB=6Mh-B=DZfONfATC&3v9+iD8=hDzw z&RO3l2UPaTsg$^mjNC(2ddqc8dNzTKkZ|TcoNIb`AH*qrnDc>i_RePPM^%td#1=>R zux%|8kE@Rgo@5voBmQnftd0oxlZX^!-JU(J3pKRL)7vsqESH#~0XqC!5eCkMuvwiiEIR3|qGE)9aOu#_r1N7pKP*+a{ac zxWP}&qQB1Z6(UrgO*7q~<#{H~TrR^<;`hqYv2wk2dc7^$y$>aZGK=yiO^!QMd2HeE z(2#ulC#QK(pkdxeTmi^yPNQkZpz<*@|LT3ePDQq5T)A5QW8ezPvp(l*kIk0%JA1YD z*Zkl;{A*9)kJXMi-C;pSjT=@bu%b&tSE6y52r~*^nXj)-t`+F^u)$crfp;xG^!qf9 z2Xqe&nNXK|W6$8VhYIiDP6!&mRs>o?FjaZufJ{ltXCkzA?am|PuWvDMgx z%5;p!!$en+8EK>OGSn>ihu)w^Yy8|OsDJoDvnTx>9K0fs`J#;#5@EK_=#q+?bxISL z8hneb6$OGCGPPg!RfXuODt$P)#kM`JJHF*M2E!3>@_5+?qQn*szY~D20LuwLHy`|J zF`UO&?U~nLqcd7i8C-v>*F=o34e3aCht67K*YJ)B@&~7K=gyr8<-NCDMl0Hm{i$1v z7h)xdnIAeGEsHY47wbcz(dZQi8Z$)mtWlUA?4grzU-KoSb_0%x3`63M!P6&I3Sm+a?$ zjU7@$CP1kCbh`d+g(tCDX%+9So`$OH=YD+-zjV=1jo`{@&V#4uw&~>MSL?!%0Q4W5 z%TKv{J)Y}cGdUakC7yY)1B}Z} z)>|B`IXX7S{pf{xvGpa}lwJv|&JIjas;)S-e>bb;!d)Wf3y^M^zKK6ZGzR@7nIzyo zesh}1&6Mo{UOsv8*EhiRn-r5Yr3Ucf8yxlE2+p~FjXYY4b+o<__Tjdt&fKdDi0)v8Cz%T?J5-D@5p-6FBVct!ovL* zHinYDDxg19*0T`Vy65-^0#!(r57U8z6*|Y!^*=?4NV-8(;UAY;hG!rn7i@>Um@OlL zg?sTB(8i$PV4Y_9xZ#z@TD749oeRa7Q>3YnZA8i=NCGk2zJP+huzK&M9ru2HLqA}~ z#Id_DSV7w8Yk=HD`LT<*`0&Hre_@AOX)gHG={5Cd_u4D&=~LF>x?MBNiTU1j1zp` z5(g`Mmp(}Nv~;RQYn!219>k2h^X=o5#8r(Txa_ozZyYn|9#vhI=!GMs+QP_c#;(7J z2)fQ=8*E;(r_Bwt(7Qb=DoJx0KaP8xoqg-Nw9TGwQ~KcGDKO39mMAl2X<9CFx4%%g za##JdveU8$-xelP1kqjGak~sI`$ahOsR_kt^=f8O=g{ylun%6jg7YZOO<~)YBzP(_ zD+kMO6~dw=9d?BZ@T0qKOvD>ibojSGaICnxfLO0fcr0~*ggtgy*EWWd+Wq? zaADGPCF=wva#YcKzX{B_D=XFbVwPI<)DpKYZC1geDg0eJY`Z~+m+dCd(~UrbaKLp1bxjI7ZMII`u1diGxiiaq_WDaK!}dk%+(#7d9g4gFAq$ znu=0&mx=2nL_KIFSV)=Xiwh56;ws)Y6uxD||f?7wyVA`I3w2(hu0~)q{M>@E}f3`}ZC2 zvR(J9^(Mmu99fb=%v){ChhGjnD9N;WwisQLAuj+UPavum7-^HtB+_9NkP%tI5K;nc z`O?Q_Y|IEr^Cyc_@|14?zDC?s`1V*kF7Z>l`55@aS-^Xtq6TR6$Cg*w159J;dxqO@#q|vCWQ7#E$l$2>*g{cD)5UDHv#SVcwy??>L466oLZ$ zCG=Mgx~-dJBLgJ}>OM_BnDank3zZj@Mc#M|iSt?Sf>Hya7L$;6p7!g=J+w%qTN4se zJuVJ*6LzyZM=!QrUbx}pI)D#?ir2%#FJNWTeYNsQg&!!X2uQbtq8`Lwo`x}CYnlt8 zVEWAiZwB-vWXS~{l|l1*Rp`)gB%KGN>XLW)k73G+L_tLMiS@(eYYJ){+2uC~0cjw`v_Mj^8LuFqT8dZLIhywto>tIpSJgL2 z>|Mo+@7nr2b*n_J*{^|N^gLT|@NFiWA;M$Hi>2Va0M1lycUd}ve3uJ0*8R`~#jxzTrG5ug`_%co4F*2omju~+P8 z0K2h`xBL9U9a~L#&}brU^!7VIMM_B(y2igYr&g!019)T-ty%YpH#ZwA^n z>Q&G=9p45(sIX;cf*!(zyo*tC0SV}FEf`?AGE6qNkQsTEr?u_h`|Aj{m4s6|T($;D zjZN~vJvgJ-} zG3@;)o5$U8?4eHO&9;vym+L$3tKHm)mK{ZUVMnGw$}e7V>{)1l@XzULfDjxvmR)}X zcb~4Rw|CCsoB_wE#zUt$@y5_b(j!akU&ZEN$K3t+qqkv}CaqU(a5hK3_RrIE-ZNN+2I zB;>lmJ$4mG0Ie&$M|7j`hMog!9E<^KSMHeP@dAXvYTA@w7uOvv5La-jD-@&{kUX}2 zlY5Wq4E^{Lv_q)wB?d-yTt(gdK?s7G#Vm;5p1ylG7r3=YYN`~G8Ud<>@3AzuM_)>@ z1DNLn>rw`0y8{w3?4w^{L6haw$qU0sZ=He;sQtkAnT&*X55&|F=^JW3OlIg;z>DaQ zxl@?PBANBh*m;W3H&KWl>!e9v(VLL!l_Ej006>%d7}LVSNPz_u`R8L_1Rb9Id`S9+ z_l{l{E|a-U>%&pSL;O#lNkVn7Z^(`W`65shSUG^*C<;$x4TfJJ_2CfF;EIp< zJ(f~oZHLnYdh{58hHxS|i=7&i6poVUbTXB+nr#nJpB(GHeq!3L~9 zAc)*z2!|wqx}?7+85F4r!KHLTgG_b|nFhUHSD%LEYQqh}pwax7M}?jeG^tp-Vk39s z1rcg-GTv6>Ot`r9iW7r}G=klsOwY(wvH~Gd^F&F@bhVp1M(%$6ahhH+IOpL8KCd}= zvY=};a0@nu#goNCHLy-lhp>YP9IJS20Pj5H^n+j1Fp$ow!*WD|R;=P(aNe4n%$DP~ z02)R}A}l1Z9HhAOBNVgbMl&s!Tud;CiW+8e;=EN4MNs(ITQCMimgzI5><8x`Z z02Ef5QrC-3q$dMQJ^lTm3-BG2lIr;-n|D4U#Z;HcQ|yG301RH>3}`<9Bo)(A!p~Y- zd{YZh0GfaGZR@Z4aoOgf695iS=*FSPYZXOD;wq=U+mNt!keCC)!fL_UHhUl1I@sIO zMJjGbf>g?+192S{$C5wqDcSF~RdQ%MezfZPl&Y`EczjmVGyx~zhY^B_!9%;7`0l$_ zq8DY2TY$5?cieODb&?FS3AmkE+A-!t1qjW_Ih5q-sR?+&sP2Mwi9~hZDJ)n%;^i(@A*K`&di>6~7R+E2X0bYzE9PXNZ0F>z>*jI6e);-FUzg!7_9@9*0z zVnB|3OAG(aNnYt)5A0xagpk1ZYs6uF+(!)BidDo)9Cx-A=_M ze(gzvZXctBPwM*SqKW3kvcvTb;u6!(ToPw6F&OG98};0IG_()KF-r!YUuo*RK+a}) z+i~0rl4ax0U&guJC5@(%I?to<-t}$HJ=Qh{yDusi?XPvNkmXUqfC9waVRm5K`Yqrs zb@r}&gD_HzW`xKUeCHPghMfFq*;#1S~g&Z%1( z8D^J!hh@8Kks*qBRL|zc(wZi7QDLz$h+Ox4bPux9e!)eTyWkW%JNsEgEI`>sttg5i z!+ZV_7%f5Tutihz?9L>KjO`q(00wTNFR*&1|3kq_^t(uP=zBfRCb$4ux8*-K*U-{p z)$WlcERL)W@D2yd9Z5*3`6cCx;^V)7c6d$1{=AY@BQa#|22!e1&63;fY;5tzPN7x* zkH&JCyx&7!BU{NfGWK(FDf_hzr}CfVZ1(2!{6pd>yHx2~$PJk}=%~0-bg|KcSqYnV zmCkg)UFUvzQ1pEE@gW(QY>dh{gw;Ga_Ubv1J|^I_0yfu^*5O>kKg4a}lI+`QvU&r? zcie#x*lsyK^0{k&oUM+KuLpJJCr91V*lw>w<;B&Nw0xWH#s{&g3IHz{&&Pgy)GyG+ z=sxiJb=x<`(Sws&0ficX2DBF@rfh6%O4wORD+h91N&8GIl#YPqC{EIVU;ny0?s-?| zH%oKTV9_C?$gt2P;JKm(GE6!3Ai-s`TUsNX#HbLxtOxk>6w!ZSz- zN9Pydlww+L((Y&MVbg-!80T0EfZ+LwjdBCN#=U-yR)?(V%=lDu&oA+|PontNe~zR( zq+t|}jC4pwm#5e59?#ur_H4}uH4|u3>Xv{U&$X63t${(i&f$MGa?Ou=Ge0mWRn_S< zH}5qj-fOe7c^BHRa>E@idD9y8hF3dkf^24D z>^SF?u;3@Fu3UljinS-1*o!mom5l&IeIVn=m0%;XR5QY76qW4r+8dWH={2W)`C>+B zF(|W-sfPoaa>bnCLDGk@H0QdnJ3`yW2J3WJwjE{`Typv~i(oX*4z~~i{Ygb<4sCQk z+*+)fmh97dufEb}+E3;uZ{25K+Sba!#=(E?82|bZ^|)(wi-%rTpA!s`InjI4m?!*# z>nqn+>l;b~vs)Ag(t%jRQdISuzhY_6aI<+=BFqA|4QWH3F#d%-(2{W5AP zCj=pACs)p7V5iz`6n19`IcH;-K|4H*xRQI2D5nGZ{JiPs4L7h{3BTd%-bOG=Us5S6e*S1Keu zT~w-kI__r|6Q10weJ0Z?Ym9FtywE%%BqUT3(ekGmO}4wJA}~A06-jSXz`6X8J!xZn zdRD6wHca!)VVFSsYA&zccKe;a*w3E>1DlT?T_GweiW9QQdYBhb8|+U$xWcE26#x-L z{-|+dKAdQ8_kXXAXMo^(I8i_A%LX2RX5Xa4uDS9js!B&E(K)QqhN~AkoIZ{LvkQyY z?t=$KAXIyMtK~H(rXij>2`byRZS$+gS_XkRU+pD`0y-^Y?0pV-d3&~1+z9v>k<*C- zuhQ!X_;h|gsfyrL;t5S+0F=1-p}gS?|9gXJGvjle+JDV|ILaFp=K!7=bTLoR-%bZD z4Ym4eHvqS=CsVSrs)dU7+iZayHzVpEb(oilZk?E^ILqNJzk@287?Y^SfcM2uUVKO< z^6v2=dq$|~98A_eiD|`5wlIlz-E&HG?pljZ)0rQAUYR|4s2drve)+SMmV^FU!c1## zPSuFy)JbR(&=U|yG{$}lW1yoA&t{05<{N!u%$+w6=;3+idmVuMP>#T5=q_S2DQQe6 zffF>^N7MPY$wVOM``o;fk7K3TQn*^DbVTjy8;9)@JSkpl+!UZjUYYAhNF9`^HsbEX2D_CeEQhQgR+uUMaUa2$tGv);FeV+xSo zV2(dd%tIU$o&*H9v#{9!Ii!*jMQ{Y{_|BS^eVW0yh68p=D)6=T*HL6 zh-z+Vp&(9BMg9{JK6Mr)cX|qpQPEu2?oIP3*at_I)vBD+3@Z>SjYbCs`Vw@ z6wU+?L8DosLX6!9_j6Zg1f0RYfCdLyrYtd{aW&p5XAs+fq+3d~Bz&KsShPaXn0(M{ zY(7qG2B2q@ydWim_-29~ z&}%~y>|If@Y}vL=4VH`0DA=U0xgrLyn1^V1Ny-M_oG&5U6(6I^ORSZ)X!B16zGsFG zl;HvN%%?*z*dn5i0jvolqxO+0N!bve++A}$B!q?7I}@3;m-8qSF*WZeFKXJ~kWE-X zIbNYCwdXlI%m$t77Y38X)`tBdnr2wanp(-#ZJ(ifF55pla7N>yV+L5;-U&l%uIxT{ z@ywXx#RCh!`DK+T{qUXo_|~EzwjTYp(mDbH14}|Rq~SmEttNXefIlS5FllH1(6;dL zz)raW5f0?=1BU$^g-lq}P}Bpc!5Q34NOxZM) zL0+6(Y_b4T3;HkvSQQhy53iY>IKmJ}K!Op2HQ%zJEr^e{1kz2*e?&J_IhsZ^i3?Yv zLL-SgCeA2(<`$DNET$x15x{u2eQ1>$96;flcozr&ND~YQ;ThW1X)WMr7J*{2z=#!` zRTeC95Gxz72Ua}0r6WQq*$rx;Ib9FgQNrSWB{*pTr31n7Eb>w3qI`xnSrl^$L4S95 z_lEr8E+j<6Z1n}kJ@P%XU~~p8dkl5baLcS4Fa$u^R&l)&`&s~TT;a&{i250nC)ULX zpTK@nwoh7a|D4B{Pg+ZU!StjUF6s;hNT$y)n-k0*@`qD1b(>gk-V~LfWVIlHyu6?l z=SpI6{#NxSKBtHmY^Egut*L6nhXaJH`qD2R4tur-I~}<3E8;njHQO4XDQIYxKXCb! z=bl+*a%TxWFcb*p*tqIHdJ!N6)S(AJQDx50>Jxk&=|$_l9?ZJ>~%hF&)p7 z7-ar22(y?XTT#40h3I&D0L?U1>@uCCJrp_m=}18lBoJZ`b{Al`z@B88Ij`SsJ-O3> z62dPl%Y|n&!}e(b$Y5?cPXk-%P08tdT3azq{hN?z;euh4`L+XnE{QLcCAhn@&iU=X z7q#+zUADSNxViap)@J{%b0z#`n!jhyelv!h6Wnjg*KlP&FU`lTtf;?UE+%$XPn6v7(BfXP zE?Sw9w@Clcj6}sLeGUb9i2(IOh*5R8m=86~(Tt&v;YjO}a!qyhsgM=1`l+$NU>KAq zylYAcck(zs9#~aRQ_`_1G=2bs9{Dho!EJKFdU>W4ar)JeW)O%?HtL?T4PyU5){>$8 zpu$2G617m8!gs{55Gqsgp?v`&$K;NDwh3o{jvtH>=W4~_Zz}=gtUl-uWF;ER9y$Lz zUs0i+gNiNVRihp_tN_(`S5@N0Jh%V4Yuu~;GPDC!7EE7kTQDp7NdAy88O5$G))(c% zyMis2rb}qB9@o_^#tDZ87x$756&&yYyfrj6iM5FhlXx?jOyhB;I2dndkk>dG)SVqw zWN8YKkS~0=g(d-SJ5V)O_)jU+YeuNw3Nw$0Ll})bg;u}^2y8vzJ2mhZes_=ko`ned z$2daCFl!0oA=do{2wIFHvvbZKS66xX4g`QNkYDO-^!cG9JGON>|BFW8g}e+e6^yDq zw1>|rKdC#QAz0rJu6AL|;u=OJN|z@esG5=FUS72*SJE7rY|WOjqfxd$RguTue#bYmh3Xp(7F zE?X;x=ksJNy$oJv!u0&bWm$HB6{H)-lVC-eh}Y-pTF2+@PRAS*QrSvoV?#>HxRcZl z9JLjNW>LdHcU=HDhSf_c>4&~DsUe}THymw#Yi-dMRb|=TEXonTk{A=i3>Hmd+HoZd z=xJQ{n#2!@RxWtctq#t634ifJBbbS}`~7-s>`V#vss zJ+P@h12_A+c?g&tbOAsAE<~BA&+FPWPZKAPw2u#LYkh#H^CN%v%{4brX@FS54;kt$ z;-Th#f5L9Q>l!w2pN>u3TwGsa%yaMQO1p*4Df$@)OV-Aq5PUHGZ@NWtEfyyo0SMvw z7&Qu4z8-j`jVBF?FqkRJOIw_Q5T`6i8q#3aLgtbT58QAPag!`KvDtf9gT=@1e zgB_6Kk)s1vz0=X-m}*2?y!!!Zw8C80PY8j3?WW>IyDNZG7rV_1WHM??|1t2(Ca-Y*6mJ0vU5zDc77yherx_I$5^I=kY9 zuq=LzHSISxTrfr$+;~(bgyf{7umc+-r+D`b2!G1k<%t)A0UU$z`)6hWI1JIx6v@;E zX%K|l(TtJ?DtJ-vC}xViZ94|?c4`LWC0|Hb04rxbI`;9VQ|pcxpq>N z%(7m1RDZ=%cR_>@dHClJH4`;>HMo$q^@5d85e$gPz1M527R;Yym~%r+wD5}NY1t>e zg~zUNF(Q-KT=qW~&;Q~Z&*68j-bSINf=c^WZ4?RV9mTUQVOkf*R-%=>vw=)4Q8!Fw ztccXYcVOAV0X<*@RD>}eT@ukPWRChp$-b0w51b7|J_^n-gQD{1^ok{r*Yn%%5f!}* z4^hyDCIJY>+-!2)S|+jk(7Fpx=CLZq{F=Lww6?UdONw6f`ct6{y)yky6_tLp_= z-be|vhPbH+6z*y(+=2>xT}bi3=p~KVnG#s+=-{7P$}llt6lU0R68`PaEFM38{Kt}( zG|c3`Kg`;u1YR#^X3~Sqd}AXcqvT>20Zq*_IFJq2{lpIU;9s`fQWVS=fa`~r6sB)bwFS;$fdRTUfVmk7$CaR_E&J9}Vjsz)xeD zL2knw4YBD?kvaFDcoZo}R$JK$>@O96UmqKKTiWdF;=#&ieN~q)>E6tezNzG!tS(P~ zFz@d7S-JOnSIMw#VhA3d$NZnVs3M!Fkl1N zBUDV{)gH*yz2v_mBd@>Hxu8NVLGQzwS7C{g#m!o{+x=Xcp`oGGYx%~{P62^?L5Z%> zpA?2_un`1mh`qCOZ1Tj-7PieXHz_e&K^!Saa#o+LfbWiP@%w?>*?o=li@cYTDOr%} z=bKSh(^p(9AXD;5(!o6@JyLa%<01ifgXCF7;T$>MXK)%He2a6k@rd`g)skKFL z4t_t9@ZqZKBUA()1C&U!t`J2dGHVb_H-DTMz70TS(U@Nz900l?zshM(Uw=PWCd5kZ zwRJ`#y8UamTZ-(}ty^e%{j(4@7Wyi=h7()2MiB`#88J~fGkbcfK+1Fy_8S*B4#}`*xSZ0wXWVRZSAPCk1<$mjL0(?*-sHf9wz0J$iH9hb z=r6vaeM^>8M8{6K(P&u6JrjV-xFr~f*Px8kY?DdvWNQHcm8(6RHdrP2Eug-t$o_Cb zzxjkw&$iXbBek&P9&K9A2B6t}Zx7JVIx;;6C3<8@0OS;t}<(MLIukt(3b{7AaAFJNP-D+ctqi z5})l7N{wsVbq)7Es$UrX{mqG~PLVgTsNSWqm24wt+X+${-~)!b4iGsgWjY$k z5I8p*@PAXJi_xI1k6F46WF~qVKqWEdZJF|lN=PZG?dzU}?AEAqao>|->pYNAH*r_j zMM9#6a)4yEbXS3Wp>gFYHbpMp2qjt?o^?DGrgXLs0$gak)1hp$3}y@^4aR9_lN)W5B9&x z!*X^!zqRND!ryR`SGlY$OKGWc5oS*f2vEJ7`ZlEn(Zand*yKY9NfNir&I{Zu8fp%~ z$M_3NB_y`JS$1%5rqmg~naW+(V3I&&x&~K`VWWR*>&Na_>KYn}P8Zi_J)dH7D2!#L zROE!=y-G?Ol^sw%Fh!|-E(W@IYNTfh`p2{ANGjLJ2@0|jG9@-0(|jo*RcJeV>r$Xr>87hDJ5PIxXL`3si~HF5UkdM+ zH|c0y40u%Y=GwZHHP3h4(wS;F#a+Qt{cb$R5y?qvw)=62O)4h??uT#I z`g%}SV&}Z=`~GZNI28cZ1eOgoOi)c4eb7X8yg0=eXCM@Evorif^mV`tZ>4 z5O}+F)qATdaKESG^~2Nwt5fTWT=Fx;$!D2}E5+akxe0Zg^RbAT`l!xY43w0V@`~a1 z$~w81MPNctqtF2r!e`f(h5IkOvB}c7(As7c`utzn|(o>uTfR~?k-!&Od?3aB4QI%A-1)=AfnhgZP~_! z><5Ch04_X3Rn!CW($~dfgBK!P-gNX2wrm*l2CB})FmY=W0*^SSPzIW6+M$F)rUGfk z8$={S@J0Xd-$;#$xeaQ#-9|S@o#uG6;x{>f3~+K12B#1(D8BK)@eP3hexmQxYrnb% znMFUTB;pPCTz~T=nWkne0j@r8mp1-v>yG}1D z+i;MEkhXef$R;5(oq#RkHEz)xHSfxX>PQyh$vIuQQL;!&`0*0sdoY4BWe-s|i(ZQ7 zwGPLpkH5Z0WBLL$j7F2_+w~Q zNU!pP80;x{1!&rU0ih}em_EL~)FV?rPB!W!UqSgmpgq(R$DB%>tVB{9)LoZ2`SLsS z%y0Cod{#si^4^n>JV8lVzZima1`3p8E9G`ZWh{Go4<}O|EH#`q1V}(; z^0|o!gfJqk+UKAE$7K$nl>5_%0LGE$4_pJQZMJP+8uUHfaH2eoy2n=X*)KYF60Uwbjs(tEmYn0|^-KAiM>q*Wt3Iyc&&@NxHBcZ59dFK!v)FFt_Jjm2Bv*DY zVptdmLN2VI;vVx!w$bgY>_mUoco{P;VJ8;DdzRiIH|=I*02!e8wcgTYWF5z#*0M~wNWmiiCzMyU~mkd5?`%>yt>@G zvWq3{q+q^J+-JgP07}-Q?#e84E$Le&7_>C>-brj_^tG-IpvjISCh>scDCrqmEQTI; z=#TEY#bk309Xt5DG<*eS0b;99vEMvtX6IT*9%h!q?%~-+lEn~ekop7hh?9u1jylH1 zZRYl+p&C6XU1*fxiHJ#T%hCuK*?+-m^#{l_bxpi#1xn_Y+?pePmG=UW6!EI77vE0X zbhG7qU$6zvu@|iiikSKB-GLvNfBnGzdj7Y}Z#VyKg?Kz6*${=s#y&)IEsZf6_A;`; z?9T0*iTz}1hM{rCZ&FW^0UG%~(%c7*Gx3)b|VojAl~r=Xdi(v{WVzqS}oP-JHUiqjIz*Jz>LQV+oH41zB9P<-BYvQoDj20*#( zV&QM`CdYLWQtyGD>R^G-*5iP$*?Zk z0Lt2Fit4~xi+`!8bVW#v8;0Dw;43(XWa-Jkta}Y@SYeM4|Jz=TQF&D7B1QmvP!VpLS#4Y1%18e zyJM6+N6D(P1w#2%?FQZ|0)LeKAn#l$-aNw@3J^-1V<5_8S&eJUwQc?-`QltXF6Jf& zx>~n=@4Hu?7ypfWV~Vn9s8VcT8pG>yD0;RPH;-J}+H&g z=^BlO=}I9=#l7jdC}D1FZ9XUpp(n@?ozlraPicOJxE>ia+edPA5+D^il`)EyWrccx z6z#HMxl$q`k1_D0RCDh#v-SP&b)EJ+I0RM$g8js_gy*7uB>EZ%EEr>AD2urz1Jxz8 zNhQ#a{xs@9KY|LIf+ERw4O=fpQRne(BzXbfAa%pE6!ME}eYi!~9SJW5JaR?^Ho;t* z-z8r^GTy(=X^|d%p_O>Tx&o%o$zx2@W5iRop<7Y8ra%O zQfgu@bqd>%N#^F}m|j{rUc7#p=W?~wwEZ&E4)+SB&CgNgm@R)pOZ4%R1iLL> z)1I7piHX;WRc!unhWdC~oQm#3#p4EA--%EM+BYVU2VSSavrP>oALtUmS|ErN^V^RrjbbJb zp%cbAd1q}g{75jM4B?x2+wLTTW&yN?Rl7Q9AC#OkB>mU8eCJ1_3DJSSzZ$He_Hgg4 zx`G%S9ByX^b|qC1x89GAtH6i|FJM>ThHbj}h74D*G0%er^`Ue3t98X=z6{zBx3`ZTT@WCXqUytA3o@a(Ka+()9wfzm?S$U^UsiguMX63@vUI zhG`6bP(~;=1UQd&%Ww2nR1K^wUw0ODIOadF{vtcJqg!%S`iE#I2QmJG{32xH!NOtuTib93^Jr}_rSGx!oH(1eN5fJS!L z9x>R=G~gr1*=FUFWr<%gD(bi2X8d`wf%MG&-?R75$%j zi6-6ks76)n?9#H1;ECBmcjy9KX6>;UL)lEiP_Y}}t2(Tvn#tf0Abzq7qnI(kTnr_k zTVzxT4#Vzsau5&W&9-9V$iDQetJ%6W*!j=0P`Pf4Q5{*NUUzMt-!nN4coQJXN_QLJ zPpFTdp@b3v9Ku$v{)=pIR}2sSH7YWm&3p^S6Eu#~bMaA9lcpB?IQU6P>3 z9IA|K7%%2ygzbm7DeXg#G>9`Cpc&DEh<6Vrkurc&u(d!O@K36+{^x7hC`=0kKxZ@%T#U}1p^FHDo(d8)++ZpL!P_eYp7bVGlV^*d_zM0? z1Rec(Zo8j?di4T)!t=8h`(sroC~(FICv^L#qWAmn1NIR)3tDVk$>Zv14%w;y+zV$s zc)G-18kj0~@up3i{>$l{EkEbKzr)QmJ39hT(YFDR*uVSc<;Cay*Us_Y`C^GF@Mb0H zI!U&1M3A-;vq?{2)vv3gXIE%)u!0P+r{==p8xt_0_5Ky}s0@d*6B}7*Nl6vMF_Acp z`!A^JNloMdc$F9~(j5=J7#nHXFs^2N@#2NG1?Zp4ncdKgr;VHWL4`@oz$-;SBw&pI zlmb-r?$^hW}uX4I9G=7KeKeQS$wd00@;j&4OLq5dbLoEmS+Q~(7Oq~r=KOtbMblwoCe zam(f_gFFIi4^uo+rLaWa>-|g7$si#cQmJUj2TG zFv|ksWt$#>G4B$ZuT|Ta^d3h|(cfhu;*EhLeZ9X*Cp5M)hR3OqhQ;KEgOaHSDg`K_ zqFyziy+&HrECxXqfHYp3qrSN8^KU7J?crf+YHl`%rjOwPZd^U;*&}+IHGsB#KLMY} z+-(FVAWtR?xev=px#r^<4f#lcT2N4MHU`)*ec2n+Cj?DP5b;2Y~Ibhv4< z-Lxv`#y2%K8t~y<>o=+V^Lwkj{C}jqc{r8*8a8}uQc2N3Ndu+fDKsE+DwPsRs1Tw- z#t>3SgOCs^QW{8!5He+|gk&gXNai7mOo>wBJFivIe&6FezCXT>j{WTY?6TIn*KfG4 z^E!v8fP5FT5{AX)MP-(Wia8G)1N(|8;Mmdl*NIxO%o<^R1OSGcMae=>#@E>Ev?Tht z{q8MWJPI59wub#4I^%%d2qs}+^P*RFm~MDB;w&8AyDt(j-XpBU%*@Qo3PI68Q)r06 zT;8aFHZ#7Ws;OewV$u{4+(oH%)3~^p`;WS}Z$vp%*(~$O>;r*qpwn?cDuGQ)#7PAB z+-;fvYN!62l}nbqSb6HtDU~AGbLit;XRX9TdLM}yb$8p6O0j&9WE)j;2FPg`0ZF>B_4<@D52Dc!-<8y#1?(N8hhY7#J z0riiVwZ-kx1+)yaqWxaEgGD26U|bX|%1w-h-wFm1$X?mgk`pE^2QuJ4@@%PMxK zQI)zLc?l4;hp7W4mAs@uI51f_@)e^HqHH?FM^Aa{rNeb@+)9(+XXfZMd=Yo z4^-bKlWa`v>%i!t+QPa2122kxhC$Izl;vwVchr`2;oVmpRW+I0s@#XuT5>pYynGEn z28tGCTp%84#Q60&2qr#IiI%_Mz(F#Fun|C>;C8YxWUqx9cK_Qz@awPghAjH)>+JqO zh}%McLq4YyLK#c76|O;(0$L|t35HOyUAYBd2mz9SBHsR&cny2qQPn#;sv*Mxqzo0OE3B9wz7Lscia zlS8Ynf=|7?qY=&5E#p7?py~|TL}DMpzIFgyUMlUUPu%GB3+@6G1WD9t$s=_P?(<+S z;T`S6t3am<*Rjb?ZEYez;uoA!$>s(}4ET&T+S_ySJT%uNci7-~pPU$!u1NzVqP0gBquIf1C2~ zH|P{D(Aq^cL{M8Weh8is3CvNK?8dqip%WXY21vK3z~QVZ;`UoMyjjeiq7#d#`PE>O zcLUwjg*h@jL}i%Q27PDE@it*7z8(}cg12x4yJ<87S=x?gr^fNa32`y)g+A+$>Pq*%&bH&lDRURZ44e94>Bz zlQD)W?O4m_mE+-C!m^bDL6mU;DB6I2inb7@C+){8cWe(_v978n)jno#g^B<0IbU3f!7v6jO7z4rjy( z#*GC?zy}}HbASbzxi&h z{;_#DK>>IPNZr?E9p0oo)!2dELJEI|RBlj=v!S2il)vF{yLqIxADcATB#E*QWIcu< zQuE@?J!|o$PD^FvkPC;Y^?eBuU!~=9=fP$Y=?zro1v#Bwn50pg0N9^?mFQy9$_WZ2 zqq|J>(eYjr*7-{V%y5ke6BICsYg!&N&?ZX%U5erIh_YVjpVx(s6&AevpaG+njPGdq zg>;~-TdHozJ016LUct+D)qj&BLz$)4wt z(O}uf#zH9pDCr`+|CWpqJ}4R4E?2vNzfI)=m7l=IWx8_zY#7jXhfSrJG$(HVx4^-zMYzy*Z03TBSP#|=k!?C>pY zn8?XFBMHn)h=(FK&AEwY5!-|T0GEg;wD)}K{5diie~#1tqJV3{vIEukJp|LyxDACL zTln$_NCBe0_1J;yUvxvxG(H&%jPnKO`nf z=Z;-REX2_ZSU0!T2~38dQ`?LxO}-=sQF2vtLAh2z*jM)O8{vcGL8pKND{`Fhy#sAR zNfxg-@O6T|brf|Cp$S5ZHDtfMSioxWL2BuSXn>EEXaV69L;nTx{mGu!u6yRq z{k;rA&RvstlBU5!WXoYC_A5D%gp#e!L(CLt{)o}$kQ)wQkCyw(M32H95*d9H(ETzl zSf%goTKSp|cXq5(5I%uWdm~vOic-x6+VqFk!v{wzTGs<71+fCy@$YZ*{nUeOjz|o2 zti#Nmae*EioRh|n5A!_u)VqMO&*7Tzr}=FIh;|x&JNT@nz?`~JGlit3@dKjDq1F#z zRwFb`QL&xifzt_3oSlY33-qgj;-*(ZvlcBL?t9GE39|A}WJ}R$5fEjZ@ZQ8{+4PAh zCjNjI=&J(YQg+PRg-1fHwyU zXk3@j3q=%vcn`xM_+RD19Fl*Qh}5iMgn}cn-F{o?Bq$Tn)oy}*JyOmh!}K1e6Pwu7 zFb+ml4QleiZ~=SKD**vA!-sKP8A?KI@{>3@Lf$tv3k5?#VLF`w1Dm|6mop~kMiz+y z_r>6B*sKC~CH>U2?Xo-Cz8pYVi!UM@mw_)l7NMb(5>U66AQVYnI=E;!$kbPus$25b zg&46L0qRdc49}5=uNx`ausTV#^R=z~)dXL^5r84_$%iNW+rVSM6$40R1IlTbNOD6` z42A++x2;zJn7FJRov#yXIJP5m$a&bdUor6xLb8!bgW84kG*Ifm`2;itYdhnn{r`LF z2%{4<0Q(6lh6T{e!R1TCsJizt+^j039&ZK44U?0}QoJd=pv_#~dLT9-ep9H`a6#HZ zh8HBYBV_@^ofr&;3Vh2*0G$Ohxg!c^X(eKc`{5!|ivPLc7o2y&7`6*sHv`bSIk$EY z*t(Q^MNb3puz7v}Z3m6_#F`<~yx1U&5Im*9y2giRKw>=#z+s2VpjRFhc$|kad8Kbet z6zv7kKhPsd{P+%nrF-s8NJIMA{PMpbl1;K`{~uG`;vr%*fyQ8L&2~`^9izKLFkD^M z{P+&T`K_|@YuDCJ=Jtym-X$!IhTtnQ38G^feuHVJ5dYDD7J4*8I6dZp1VLu}XkCVB zq4wv3)qrfS%h(s~Q$EFAnp<02+i(of>zTj&4QnSL`@~>j=0oiYP1Jh0k%8Gak%8D{ zJh`Z0SyE_WGJvD(B5Ax>4hHIT5~t7H;WWIcO;Zp^4t@bz&T*_15ujc$DCA@)xSPT7 zm~!M91=3NJ?A$JZrBBNh4~r0gMf>vLzKoe z4wu%qa3_Pq)jU!t!IgrH8{e%i*tB}>Vg_aw2v6Wwfs~NH+$7;2tm79nfDLcDFx#Gv zA7N9z*M1*ryo*W;X~n}-&+yfX*D8C#u0|pUY)wDDg%U4lj>Z96(v88?ccwU zR3B_iMt}3{O926u@=q2$cI=N!Ht#T2EZpE^nz_2-Praj&$6|fY6gSa``dMG-+)K%J<1)QnFsqalp_sJnN#?%1Z%h(+de-BfQVB`|NJE`m(9z{7{~*v7)jQug zx@kcm=XK{LR5sb|q+mWi0KUv~bDM@26;PC)#)M=p;4CUvuFsh)71*zu{TSuKKY^*J zu&~pf8H;Jno{jn_*$Jrx+CKK#X=*5MSHdctU&d|{5+<$fB%2I|59x%e$J5mCS+mZd^y zn1K)Niqj%@+O%oig==b=@a1XHni=>TqUqQ5V!2d2*L{r*oz3qngu4xOcyIIA7b6!M z3%exEJ@5^zkllzOPN4W2^k7{vCx0pHmOBA-BeD@(#7Qz{<<&>hX;^EhoPUbBvo@Tm zV47YAl$@_o(e{34Q5IZ}|9Qm3V6upTkB>I@MUZ*uen)H7fTc^E?YT&=IqT-u_`SRF zU7Ew{G3ywHn5b|e#E*wz+NSzVrDpr`2rT6mTIiOi3yLAK zF)*};$v&$0Kz}=;_GgVMMz;a)e@o+*fPmsTtTGu|{av9rFO9_bVvp3XU#^?Xg*k8HCqTnMi`%O2J$h6!T;HT*&?g~O z1m8Z~btApN{A&aXKylatm&>-Dsej z-WkD-BmqZ*gtW5r;6T>nouL4F26F6cco$xKj^nRl;&2Tq49%}eILc>37!*ici)q#F zMp-U#Eg&cNv%0*rt@AKL z#A-~96ZnY3|1L-5wR{2&LQp98s+3db9{d`vOMGn*$2yTZ8DMi617DPptwb@&;78#EZ*T@* zU_DG%{9mm`I}csD&e?4i=!AcFE^${B>z;j3Nd#XT5rW0hEd>9<>dw5}=3*N}+P#U+ zdhrK2f!#NZRO=F$N2RC)%#19=WyA%$ar(ozUi8HI{boH0W+DHI~i?uQ1assko1cx9Mj z*BVl4GBSNFT@3cIXvTLzs*{)}aA2U(FY_d+*4YySi9_c$TMQ@Z#ht?k2Z2-UyLZf& zFV?8q_n`s8FqNGlni2`ReUtpdcZ_@xg21+In}pV*B;1Jf8<6kTqj$^!MnNU`J~&rNcq!vKp+IA25b{!a z)Z&X{uh2ZW5)6Mh-!?#;yl4nY!;Jvsci|qXgUa|B#&g|1CkIB8>OrM(;-pE_d)6;s z5t1`fMAu`)rcw{09z8{LkXh&hyKs($;%g^`cVW?O7=oZ*y;iunP#O7K*;X~(f2ge` zV&pOcG=`cdAgv8#+5x51^AD+JWcPvzoBv3;aS@@phf+{DuLR){#f@xF!D_BCPb&nt z*&x0iSnfSs=km3Lx4>6D8~Yw> z{Tb~CO*_$oY_*(xzCZZ%FFxb-|k)qtvzouS?bS0{UlZjp#W zAtrAw*;c=;))A*ReCo;OW8C);Nv4bfR9!zHT7TQ0`J*)wFM^7R)b3WGg%6XIvi2ut!nC!-67gV>XPogi<9yAifZENo0J+|b~YNY3ge&L1MwvSjlBuSw=lX$N=I@2fNlmgf%7DYF3Y zQ?x4_aNb5p;URQmURSXsWGjG|-|SH&F$AtX>6|Wn{DZX=uAI}b2|%C+L-ZJ&z+FRw zwL|fT&6%H$|&c8 zGzXZDJH2vAPD3pKOSJ+HdpZ4(p1!_l`*}UX3TPEp_X%%G3H|duLaNp(!v&0Z z)#LHuprJ@27nCs6$kOwbAdHmeU@bmx|NQwgUn4`}$62#Huo`tAAI)<<<)`eI-!>Fm z($+hS+rF^%nzwc?)QR?XGsOdY*4Nb3h(CUe|ALzf;82w97K+V6pNX+4eu&5aah{~} z2c~1D>Rt(@b!iT;vg~$vYe}~dnFLr`kih*%gY)tqzdtwa_kqHaIb&obE}yxpkYjK1 zk=tdBSpMh>o}{VSjIgZEKrBagIu?lJ*)_WjYXMym8v~BWgWm6GMNw`t#_y!-n)ZP9 zk^XbVUmTz*2(UjZer1oj8JfKYyS)y#sexx<-NR_ugQpNe1D2kawEeo|4)bjy0Y=yW zj0-mRWEMGT?7Am_hnEbdqs*|oaOrn}0>V)61$yy8~r4GLxS`3waur35GZ(X@vV5VIRAJM(>TI z76*WhoU(AL$8ZfCl0(-xE<x$M{-O(hYy!G-Mm&cd-!(ZOo<=;FbB2QN@$BxZ&2(xR=<~g2kyM} zN=oBhUN)(g3gVV=1q;9k^fB@G>#xaSB(OW}nAO|Ui;@nC@u5o+Xv_Mu*c{@}Ka*{> zogAhRa!0;2+9G|)B>yte5 zk>Y};F}rxCO*87Yh{Ao;fY~KW5N1}f3Ps>LWR*5w#^A{akn@{|qz0>i+_Leb7HQL-6M zX5h34pdAIR03Zrr%>4;d-r$G?yXZl-4REgai;rp4REWb{6DW9zMa%qeMk{8vR&N%u zZ=1LGPk|CipgW!-E_}f2>w$Gj^1w3@16D!!V8gQq)-x0(1{U%+SS3!uA2sUc?^~YZ zyEQBsC|%H_NpcP&lWabVUmmdlJ!tBK?{)_H%>%0k`AU*N0J=-4?;XUs@(!V86se4m^gVZNU2}q3 z(>hpkr~d>Oja~6i8h6uU%!=I5ZB1pm09A;EGvmW`&$WRPtpE@VQ4ypps93o*BxmH9 zqY9qPP{5W2pOP4TfXq?1j%Hx~a2Dlpg|uwQBHZUO_VeH`_Rnh(XNRHq7ryk8RNy+O z0T>NCJ=kN%Iyfq-JpG^20@x8717ZP!yG*}!Yj$$2ezmd545c1VE~<$;>Pdsef4o)W zf}{eb&7RGHSKKtuvWNo--B>dftHdG@bvusGYl7TxR1{6l{fFpsQAUO_4fG!aJ;dqx z3IIJ@k9lok@${F8h+s~Tn*Qg6#>hG7w8QNm__-ir)B$#)rW&+N;)6dUzZh6;LVHwO z*N-x<3urcJ>>7-qEkKjYni9a1L1(QiU&|U^A6x|pu4P9z4?GM|2s~8HjyiYSq0+<2 zL~K#e?ls+)9o^BJlen*w2Y=tR7j)xPUWJPY?}+WRM!1pRvrO z0MDscNObh*(IdS{+o~38a^WJx6$Rmr+|{BQnWPQk;8`ASqsw)6+^b$P*#qsD*QXor z3dRu|H;$!2Bryxt4s?wM^X0H@mF*tv#sA-aW;~vETE75DFW)m}x*|N=!TWTsv$WzV;@O&w6wGYD*XAHbl;JI69V4R>!tOU z>3qj$nO&7F1dA68hek7`qQzkf6C?9kUYo2cQ83}&QG_fR9H6pr2LiKM%NVA4u0qOJ z3^04f?jiL6ojo8s)WPduBS`GfC|Km*D>lHL_Tt5hhd&)aYHH&jDK()DR04y94IL#z z85Lw*h2rl1H>3;{?L*Z^MprCK-KK5ZF6=cDGJ()h(6jLmp`9;l|5UZ;HP7~&d=Gxe z`npSw(vb&&+5~2IhNBcl!aFMP7L5Utn-a#s=^&fb5CFj@aWB|JPCziaI)~U1q#wWsSpJU<7;oo(1mjS{9^5bH9sx$X z=%Pg)x*K5Z(|d1~Z(xuQyjfl>d41j}>P$K}?<{xcl>&h!3#LM1y?7H$e4sYHiZhQe z^|bE|o{S5;^u`eti;7%~smW>r=rrIUY@25kc|xH>AY5>_knt7{-j)aV>b`99nQPw5 z4_I?OpeUD@2Oh%M!1VK5z?j=^$j*YWv0`c$>%5UiVqf>OHS#ZA<9z z-*Ic>rc*pnPCh)+HujIx1J|sk_Bvx$6)#&!Y(&tS>OnwfV?cW&yDjipHPds?ojYd; zNP&lA)r;ep#d0N~Ue#@JCKo@@%Qhi>vt_R2XC-b0q)gW7ukTVQ;6QqWu!?1_Sv z1Ld9ZYmzvk<9P}Ih1CJ9kqp`N25HE-DSa}&HZa1e_14V|^!aAk>p^Xi#l6*m4}U0* z%W+KO&7T&tfh$>;B{%Vn0-a?bbLiU;HPM)UW~|GF`2bkAqp{020$gb47tjIj~QKQf5##B(u*do!bd~qd^=*1=8~4RM_^Mdk3vf;C?UpF2_yq(UvXYgR zty<@M%-hjxa@+O;f-A-69a`;qamTG0xv=?IAhL?e7unzD<#7}%3#%1@@-%+qX^D+!de8I(ihGqFj&}p zwnYr!-@hxND_L;9wF*HE6Q2FsmqaUAdCh_8 z#=bjlWtZLIhw%g3u&UQVMpD~aGk@A!b1i0nBFvzshMb0i{gFz`?+e*lNIObPcWMPj zF*U!y8f6$viR`@cE0)`ai~xb?W31h2-at$b`%FMZsXS~b%5>B#q7CdDOTYP*0SWPX2?TtScXSr;TC58|@yvvL8!<&+$4Ozv0~Q z&RK*V4pD{>FWQ+LZvMYajOs41?iAT8h@nq9K+RzOhcA4&6@JSca{`nWVgM0gdttZ@ z(u!X?3NC@@#ik*tNgyB}sxE^I$SD$%A6T;(QxNb(4+S^&>6JSI5h8G`jepmD4ItMb znXLn}gmDI`TZtx-L6Vjf@UZx^}x(4SL3uUgCMo`3m3 z;)8>P66i2-Hb8MeXmfnlmt&f#Ll}^kr)H)wIj>&5vREml1ElQI%l83-hlU4HBwV4^ z3n6w?4MH?=uWU*&{Ij!RE4i!v%vU%Akt={B5@|t(x8MJw@IATZKq4zCb3yttCh$(x z4$AE@fKuD!G63}?PRb1(f4Ubs=Bq7xB*neQo+(uO*9xxd$M@Z1ewN)}7SYBFd5p5` zBzTzgsPHTS38!nXO$9tU5qIXXgjZDrp^&tJ^$xt7SEcubs4bz0K^Wtrc-m97i))P4 zO;8@OXQmHBA#$az2&2L?seGQkkJirz+F|*%S55Xp9lLjw(tSjU$_(- zAcp4-SdfX#3;Ce|o1g5J7dYW2ukb-#AJbFY4ovS;_;TL`Ej!wO;P0aOoQFye1G)tk zQ|8Jh=On|50fT9wK_`^WRNs)Ki?}&xnb^sikmvOLcWVu-rFSXyg_pKU=K^2Y)7h>2 zGu;lC?E1s5Pep73v}Qg$mMfH6pc38o*PMsmH9#lKg+Zsb_v!}6fq+ehfvXDadK$@c zj91_B6!aqStQZY3l^<=fv;gzPN0-$6~$bH*c2(e4H?OaDFUWOt;rFbvBjK5;uccSHv;3~CBjAP|fwf>4nxBxW2S)cBe0L8s%^)b~YP+Dtb1K)H>XtiS6hucGLd`Q`goO?a{7$hL3ySf*<{AuMA*@hc#R zVECyNCcaY_9GDF#)Zy2-4&`RUZV*8!nUJ~M9~vUa(^8pFR!NBGWGEoUbnjGfyeIwk zNfHzsg_6(C_=5Lv30~n(vd^r6r|%qF%G!wlWOd@@rKRFIKqVJh%)^)&fIx!P*`uid z_h|lJI#&l;3`{Y*n*6snq<^b914g2Mh|MHr^P6qTeGfm_v_rOgwrKZi4`XHNGZWfW z50Jl7>J2yDGwCt+-WUBbJENU!f5%>p&zq23HgGh%gmEu2HqL6jVcV@D^S3_QT1C99hYI5F#ZIYf)4wgci?-EeSfX z*Cw-_ZQKVctut$^*$OO@}kTVXzF;@r`KpPu+3dAGUq30lQvO{hr zYcLMA^)9tDMaxDo2EuFG8e+%n8u`^gHVE;^GfHgnK=n* zY!LGISh6Cgt*(>&rr5$E>jov59w8}O)s^1@j&R|a5 z!rdC~YwoBI&ZeVNv1B*&R1`bHa+3P=4bL2ff+v_p8Zzs+mfK~eI62s->*=>SV@pz> zLZg)dEPsl0P*?Ypj8?zjRj~S19j2B?@6U!=%`E6)xKL}N1$_5wL@xl&&iHW9J(>Te zO`;nOI8b&h;86E@*8!aAw!QCKV^PJGBy4*~pp7uY1bGo${+KpW|AQeCrfc6J;pze^ zS1H8SGI5KX@t87wCJe6RzEnrreVCwqRK-0 zLFuNz^i}zFQthIW_loOlz>>Sy6bJhiQT>{_Uq+Cl^Lt*n@En((JIcd-7>*HVJz2JF z?O^v5q@}(W+@JKn-yqh4kVPg2q&Obe{2>Sd^wr@{CBB^8AV?n#cBP8kQhFdmk!uWD zXHb^ewr!MG8a(I9UciE7Qd;_Pxs{cbW%bE&!NpBoJ@@ada=YPwJyd_&_{2q}X`%1R z{`jWRMIu`qz9aM@s3^=6O#eM-907+el7|w|{VR~yM0)Mowdf+U?%L$!QA7bQ8vp#?`(*@GSNL~Sy(}=r4e?U)T=vlebrvJ{~Xdm6oB=Vns!zX zS~8t{b|3{~W6(n_1M}@H&ihd8K_nEhtEEhThhxc9uI4%~9I9+ysBc&79RbTe^wl0b ziPEQmy$(%=A-Ws1s#9~ph-ugE^eg^!)X?-J(7@XulY37HY}H5zv)Z`Pb=rbh(RuT3 zO6OugzkKfOsn5IxlKfdlR)I40A?wpprte-o74rG#-^ZukFhXd-X{XC}a_p?*;vVMl z&p;Uz@Yl36O-JY*8H(fTv$_dtHmfF2;;ew`}Glx$dyW7wqEqqXfvPS zEa$lY_2l+@I8Lejx#9fdleQ00lyRbsWuO_nu)$C$U-gFUQi~*$(w8DF;lkZ5#;il_ z@|7#vtgS2}vg&nc(S~TFKjtbguR7~l7tE>76R}~z>EgUN=yL(Hu7ozy#0NHujj%)$ zg)msZ%?f4;a8Vlrf=|jnXK-Kt!M9Hl8}bk%XJmE@3JY_BY=|L1!tO2#JezlNfSbjQ z-0Zs57GVQmZ4EHY_(_u<#A_xk9zW?p>cu+`A3n^>l{kc^>!JA4+>Vw{l~?Sm#Mk() zt2t(_yIfD_L|L=Vt9kQ0U(C-v8cSD`;FtZFs^ow(NbMCQZ6qVZjD*MM6CBP0c@b57 zX~DCF5xm2NnXu22QL9WLKgI1*amxvu9g|uErAJ6d!GLfIIJQPz*FYZh<)Y zVzy9xRd85pVry!nXWotXDy`fet~#O$hgMXy^TSdJPC)O?ovru&c4ieI)A@?)fN-cuwYBcatG-0qncEJ@3Hx{fnX!U2yCjEb1r`V@4 zHr`MGW1pBFkuBWf9y1k9CBTJq)kWkG^)zE^YT@>1@-l(>81rz8z-w5mkR_SWmBqi! zi3H0>P-4%G3Qb(V;C5#^%az6G>$k|0a}&V-V(^_)vx~u@#W*eM>Cf3-n}FWk0z(1> zDhx{9K+~H)9x)NneNen0+F;BDM*}}zl#6-K4qaU{tD6_OL^f*%z&T~#&lG+%k7n{o zHv{->HrvPbr!~p%SD(P6>a}6L1{aT+s>K`I<`$t`csT_YuQ@+ff%R=DYH$v(29D*U z1XEAo7@+sWk1`4u;09GyZ%H2T0x7l`MHWT1UfNpT`Un}ubQi(<0V5g&Q=zr&?Z}eJ zXyy68J?_3F;?3v*!#mjLnNz2_kNm=sCT8+_>d_G&F1_E&Ls4Vda4EyHdyJx8d8)l7 zAT*tU)TajU?sf$OpvhYA^ugol88uEU(;_SVzfQnc%vN0+wgR?3iLLVeW5$>QY!V=O zR$sN{iw^P#$PJy$XW!&{j1Uc{%5w(RK1w1&e_ottcr7MAnK;Q>5*hYrc@MeOTVwdY zulUjdpi&%npR>8BOgshmyJxT%(tYes-5?`bou^_O$QxA&*#=V5l7*%CH2{LZRFZtI$+ zqVOh%(&r^FRrkyqJJfcmR603srxsj4w_z7@*}Y`F%rJ6cc6y_Nq6jz_mGH~^U;i@> zIFi>iUV?>r&SnFgdG3?G*eTqBJ&#bTlBcMh{?l`-ufA@*@Ori@-_QO3f-9#wQ(22< zxZN<@AUJ+~q2(d>7xg3qKt2XTJGFYi&IbMlzx7~a=a-VaSh@4JVvoeaMHOS0a1Q=$t7w#m+V)6r1>)>>OJqT zN>^za{%b4Ck9^^<5wzA^b$`q{`K@F6kK~E6o|p2-*J8lUH4K=I8(uY6L?d)516_A)vMn?0{aWE~*%XbYx z9XlGGW76`^kT0xK-?=lpym;WT7IZ%++6!Z1^kOVur(sr#HU{+--WRE_y0wF1tT2=7 z?rb)~!Cw^;_yl(9p~bh*6r!6V9wc;m;HAOM2xmHkui2*M1M>T7GSX)r0Lo9deyLS8 zVrftY0=zl}#?JSlrvuv8YM1H$>W$itzYQSh`L|IBf*o8-SwFZVVBB&{u3YQ7gA8y} zm^13dHh59GMvngbyb=h^hqVfZplVRyuG4Od2TT7iI8j zIAun)*QSyy>7tc4z-^!Eq3Ae-A=-@EwN;;LrFM~f4W_DqZL%<96V`ro#Ir5H{lAmW zg+pbCivFW#6BK*~GaK&GmoS<7y{Puf9USz~RoqZ!o0+boqzMog7X&^accz&*@8dvE z!-&CjlYntH<#Xr@N;27}WWJHJS@d-N%acpDMadE4_=hL3C zy;L;3F#A8S#DUwUYW9p$K4OcD%810gvTaKG{n)G9wAy|8`PS&yCh4~{ zi#8YPzg%{C>ovz>8_1OSWskjXb+9-j(goqU@fZG8SGPG<-#FGVA#x})D5m^<^gf&0 zO6J*9pgGlEr;W<<7$+Kdof&W2%`Wr5Dk{psY@kdVJg(!l2=t2-g@G+RwR6W91=!t?Kxpxx)w`;XK1h83{c=6 z^!&=wU1BV9a3`p#^LjE*HaFak7@8NMm!x?zX@&i#)vNVWF~$`MfX-N|^{~vPKOs{g z8#xZSt#67g7`hvHwO0SaCZmMf4F#Po&EJfBIt80+?@+BnL_+@_Oe#By zV43r4x`+FNB91=?0UK^{>sg#BvumLwlClHw9_y@f)tzxL48Emu-bFX+r)PhDZ3L!) z=IUd&8S?kUJ6B3U)`ixs`H z1E+)CuV=siJ6z;`A~;_#puW9*Z>Ri~j0<+nyrGFd{HjW&5iX(?6E8D;$GZj&E}a>M z@BE|u!V^CR?YHdU3XPJCn|3k2B4j)mlPTFhr+T{Z2zL-O$p8=p(fm}h$JuwOqaHS~#B}lc_TjOfk%=shMv?>D31=MShUK4U z!r!h_(1q^&P=UQO#rRZ$jfw`WGo@@t#bv8+Rp?nd%DPuXL7}I0%>Kxk)%8E%=F*7K zSklLF&joMgSh=<(#pb)V_8vzG+QlKlj)3py#8d@SsJ$E3Xr$MN+n7m-BLLoau&)63 z)gpgI^)mpuu3(asEa_XBnd2oK{y5m-@OJwpaf$a(8q#>dqNg%s_9E^bJ9l6EJA)d9 zx~^#zDyrD8u7OeKcLtpfcHw(9=ll}0?y>GN)7dH_rd`B8#P6vpBsNUkIK4M^BR5yw zT;x{?B+au}H|0*{Il*A>ZFL4K@^k!6poiT#0&|U*+iJ@nX-v ztEeYpoO?JQZeRpc#CAPX&z*Y6uDIAFVa3%kA1^FgasX*RD$9;oxv44?e&YDp{AL28 zJa8HA*?YEm@>q$_V@i%it10f7r!#}C|HqF8&^08k=Id3}w-j7Xeinc%sC`9~PX!s{ z7NRkN=cskN0KQ0G;Fp?;p0j9jb|}$#U|EyYpN4%ZcqmrlFK2Gd!}0Df!l0CZzFXO- zX!$Lvqf2MM%1qoiaH7q?^7+PbAwVh?^|~v%TV`cEvZXh8Sn46(;P~)ucOo?Yi)=oE ze@CnC;rQ_Gs(R1CW)1hl9MMO9jbmq*kr`tmNcZqD{Uhwl2V;HE9`6I+KtewQ{fT1J zwz4Ky!aeYdAq%$YVd{sioa}eCk6E} zn3yO2h3yV2x3+SeDbcUDgvnnStTvux|2+=Cp;zO?@))>;p7~Y{)C(lJ$qc35qsd(0 zjT9fl7Z(K>N>mbDyqF!jSJL$MAr}o)PMmxDfVmJu4~HMtrWf1jaipitNg>WVM3?iC za)p~rJVzO^0=vQC2J3+M_pYETXXd*SBgbpzSBbarQo zO$#yRN!@^U@BYdg@J+x~vH*iI)L=MTXUZNslHP+ts1D!|n5<$nOGZxxZ~z<{2yGwd zWUj<{MIJUNo8TnF5v9ceuroY-3Dk&l&Kd=rF-81$;pf-kCr8+I3o1YQ|JVl!wCf5-$K26XDaarbnP+U7Z0L)?l`b z(&8zEyu>h%O+F4fMd|sL&0c&o1q6Eq*X4X6p|Rk~E&1=i(`c(fR3vvn(5H``C?9aD z8t8eGiFx6eF$Nf`Q}ieb2k_5WhD*R}qIznF=%53$5{@gdcvuRE6O=L{HxrHM`ibq>>_BM)Ss(kf<5+C4{t=YmmzT@?TaH$z=;z_CsW!7#7&Q}R+(mAXe>|&<* z5#!%iSgw{vgX#GN1=9oWz~V0(c0<{*5uIMh><+#!nZ;~)>q}~ zK2g`L8baFia4aCF>z5l``y%o+pwINiHepExKuAV; zOm39$XB{_#?W+C)ID3bVH-?iL`AECGwAcmKgQeI?b;wUJ34_)bOuz3 zbz$lP7#BlcA!*w>otS6fI||6krqz@`djS7g%+}@bpMLv9{0x#FD|{IGhNp+w z@PgH=r{O1(hu^_hH@u!adBWEX{xus4?qnTJpmzzI$e(WV#|l4-1l#--$wn0Y%7{jx_hqi|d87Qj(-1oSCMZvb|acz9sJ z0~BczqN7mJO6>XXihKCd2YJI44HarjL4<)o=luCr{>b)dz0JN?TSa-6My;qT6+dJp zoz4ZYVcnvr{;)clMuTLs4pBq5)u@16pG3fo%r+RCHeWvj^-dQ zqA5#`fxR-?GyQM4%p;AuzkSPvhTwVmyMdu+1Nz`0w);d)T*H??5Ed(r-H7vbwCj3- zj4~wGJ319sf?7zWeDHx)v!Kg3=5;N`92nghR=!iP>SDdD=JGcK(lySU*S@Xv*>xb& zVq@QOL$sr4l{SJu4X_`%=9CwuuU~7@$=Pg9U4-rBtzpP4l<40xV+6}x@!8wnx=LMMAfmFqvT8%D|+)R<_ z_!Pp^E?5G;zw?neZ?rRZ@19BUbX2P9jvd^Xpg#y&iz|W}Mi5%T49K2LMy%vXfU6ZK zHhOzd!0Az*n>6b*q`>EgL=GH0Xb656`AuOj?~MAmL`-ZtQIEkx$oH+#E{eSj4U<8dr79GbO@E9G>v)s$_d^$U_>fH*$4mIv$$Nr;{rRC zFkP?KkllFxm{;SeHzM)xe%a+0U;={rZ@4%LQkYN}xRWiDffyevwh#N41mJ9RVL6SRe-xIy)${(1RPh>XaU4i6y_oeH*_jC{=&PTpo3Ac3)T}5=j{s5Uorc^Jc;N>f*(V z;|f{G&_LY#0oSAG%CQH~Ax1deHuS2znF=q}%t=?Ul>qXffP(HR(&-b-%0?5el5Yh5 zn3ZmrI!=?vCA>V?87L<#UT2^+jVz^DXi%Y`GBm9B?DsV7IT^|GEYR~nfh#-b(uJ#d z*ImuLlSF~`m@ifC7tvXXz7S>#SUdPoh0(R|(d~4nqrufY(Ya!6b9=k6US{(*uAW{~ zky}T?EH(l{MaXElZGo;Cp*P@T3T9#3#dc*6IGZ_dm(Wkekp*mlg*{~yXInOZo6s{) z$F|GE({qEa?sb*sxO4rFg+)ZpAqGc(@tLOv<)g%UdQCF4EJd4EXD;6LD^%mZ?2x&; zK6e3c&*a;?2i#;Zl<|O|HH+Nqqsft2;2PW9hFdu8y{%5d>wG`dA*sJA=faQ$Zn7So zjnDzM8s>tK4P!q;jQA-c=6f<~9DMs^bpwJjrv2ma2OikVuj3+$EAs@UN-@%(thU`& zGXP(ZidsTNu}N;@2++#D84%!x)sFMK4&lKMio1odnc>45+scoO4ThvQILf%?L)0GCAg-v8?hvF(8-dVn_R3bs*O>< zl}fL1IB~)VB{S_q*b7b!Eq-@P=rFFEBmG|vpw~dO@LtEx%i!!Gwt|SV5Mq?d`0PR5 z@|)1DW3&fH_&LzXC60-##^RwjDL(fb#D>I7gp&|%=-8qbem=hQh_Cm^RL0p)?H4t+ zFcb!98ArA}u3pCAxjQ!sQ-y>SvRFL1lhl(anrPAl_VM(2psbN^0*!jKw9et6S(8i} zm2GKQ7HIsZZS+7ST5dJ~cZ^(C_{IP`pn;w3kqAu19!(?4ti|rNEZa;ZM5s z`4a}>!TNq02Rhz>iuhdghR{@x?_-^J`qlEfnm+kt!&!3@2JQsHAI$u?>1KWn?yB!X zr-Y^JOFqCbaeAtpSg-9jF@SiVCEmR5n3xp#-s5@I%81s1GToLArA7Wo$kzY7&|n$P zyA6|-9Z$GGcZCZfSi1AQVZ?V=#Lj{PHR_^VLMeZh�wH zb1)i3V6ApbN7qn-eM`N$#<#)TwQ>6$eC=iI%nk`G+kb{jY5eQ;6{Cd|){CBSQ8+v? ztX=rXiKR#Kg)eHR6 zsPfBh>zbIB;%Qr8AXNPETIki7SJ!9BXf0j3^nKg`&rzoo+*VhQ8O7!xJJ%*B7=ybl z?Up{JW!M~59qCn7x;TaMT6*%@?KYSCSb|OUrsmYQIh+|7pBws&V{>J1b_PokkfD3U z1&Mv<_JzmBt~9iJfV~W18D(|Osaa$RdVIU{D|3l!7*cEK=|#7;n%HouZP;)o(Xv+L zTH#%EPn$|_bQl^y4ff;h&A~gUSwCG^UF062pIF)c_|x5MkA>M8j}TD@0fv4p9LL6H z6_&bIK}MV>TE;bxa^nbc=fW2jKAawO} zdGFY+&to5MT7J;;X(s3X%*x{V=Q6@0BGhec;y>G^lAojdfphy_GnkrLMD2WkB687EiKK6^$_TiVK3em~1}CTV0q0Br+0t_q5o7ky?u?7J zj*h0Dp2*?O&ntm!iUlOh8JIbD`X{^8!XH^lhD6GMUgy5Sigy;*<`l|KFWV!tS0upU zTCOo_!oZs`3xMss%**qTmOw==4#e0Ap6AB_F>3DK4L-!C!#GJWMAnEo$%~$uD0cTj z)OuUF(JnC)mz#{^TG@~H+cPHSY6BD3Sp6p9*H8}_J)4$u&R5rY+oQ&? z8LWsE6dgm4b7s|d&uLtnpY2+0e>YxnP$7RyXaD^9KAD>KqE!#p2HPmDeKek7CfUW# zm2LUZ;y+zK~u`xQ4!pE7@)E$7<49#5ZBHmlh>Mf!UG-Fwd}+hU$htKV|~Rf%+P zSXk&fyL!7VZ%~5BM6Gqy$w-Xf>V(gujg}leuWj8ab`!u2xEB1g>ec3Ry~Z=I{A4~O zD_$gL{V`_St~29Lta|eLVEm`Fo}#y4eLaA_`+h{kR!vP!a#;FsRJs=$BmK>DY&Cqw zbpn)b!$Z+rF|!0Zm!ltVyZt98w^#MvPKj$O(5Kvwh>Vn})( z-^tte@8?x%Z9bFE7@WKq{(G4z3>d&2Z1ZE~zOykmyw;>&ouK5Wr@t%unzQ7(6p@43 z9^xlW)jFEB+PlAN)=pR3sjMa(wa)2e6<3P-#~*CZuk4$7+{rv;aIju0;oN%bg+fcS zeeV=#xV_oQ|8kqvlEu~ckKWsRST!|Hea=f;sms0%YF2hZ=l0)OvY=Gp#G64YWs&Eb z#Lmb_&;I18f%4;Yl2pVijecTA^v{K>gD zUoFmBAw4izI_SXN_PW5xm<;LGQw!v8Zbv2xjwL^2Y`(U*8tp=vc7ox zyO9UxSV{6rHwPLWEm)aL8vtMNThz)?Ac2^|S1hiwT^i1aq0yaSi>c%3Zdn|BC)8Uy zqWTN?@O|MeuJz@0c5nOSC3%a&WTcwMUJPtl`ul`X+O|K( zMRL{t7Z*hWI&rSwR(s*M?9j>XuO{i~`|L9sQFqiZTX`su#}ydv$Sx_+nYsFifaasoYvbLuYB`zsmZVCaqZ#1 zzI~pV)$>LyOibtPloxOJ>{PzWIhor~+`ka@Ljr7F%xg3Df}VKndEB{k<7^%(>P&Xq z0EI>w{E+VgC2K!EH-#@&DIS9_& zoai;++X|!IK-+@zgeX>-;T-7>Co9C-E?sS=yf7a z_?JOqhy6a=ozSTCm-c4QTJZ>hK6;MlYv7lBeIZWeZJeB(FN=!2cV__U4Kz1J?dUh? zlppmF{!X}V9nxziTgsK$oott>e1n%UI_@cqJr01!7^tvziglQ(Wc^u@{7N&4Io2-( zj>G#n!Li@kYp&emYN^80i?u%KuaytUHJ9qTn^Zb=Qr?Qb$s)hLu35Y!_$CiC;lCSS z*3-^&dH#IX0vJiUw*=Pw|ptpeQEt@&%<%&t>6>%GhquO-jsH5<6>P1oSn^z_8Aj2Qr zd<8nzW6!LED4-4eGx(9@hnTowb5C*Z8olKW1iI1Ck8djAdHd!~2pag07_pT=HC^zn z-7XbUodsKt_Y|%v)H#`4Z2<^MX|{yf$0H{IX-R*5yMo>EZodjygkStAD*soUUYTZR}e91_Q|La^%&cIafS z(Ac{-3W9}C;E{@z&ysgc#@$GN%Ly0w!D?Gu+vEKo0*s{NRqS;|0^olUVz43{*4^4T z4*VW{`#u0}V=0(=GG3A1FuK@enbv&{wbR^{lJ_z)^k|~D@YAg`lh5y7EE`|qC!|d5 zF)Zi%j~~Au@XoRtg55@5W4F@#*+z5)!je0@sp(x&a$kC<({|j8Pz?K$P}60jcKGw)>*67=6N+*MyT*#sekNt$4iCd;^Co_LbkkgR~ znZnFeZP^d5wCWJ2a@EB}@=dY%Mwn&66vR&K84$H3yh}2B7&(ApCo={%!ZS#ehw}C=z-unusL1GX$ zz!h1YBH77Ke9MWUBNEWJ%{f+gN;|{RA+;At3*zVnyILY{xEJ$NN}$WehJ82 zA|$lYIa41e)Y>1#K3_p2KAN2VR#ip?O9t!hHRh}jaKY{cgl%GCf-6;H-MUlWG9Ag` zm6SUeXA5hx?YrlR10}|d2LlyE!%S_I1KwFqx@#rJyn@@ zvC@)~a6pnEK|X|6(1&k7OR;qp_{geb07CVBL^Mth4D_}_R_Rx55s?2u9h)O4C}?4O zFy}|C8SmXK?hCwMW0o9>!_x#fQ}a)VH4;#j+M=eQL+w3KWHK)Vto5>`XDQcsW9&s7 z$DfYgy<0O=%K8rEhmjaztK4+_;z6@SWGIFwBxn{E79PC1A|U_3;L!Uthj{pQKfq}q z6dQSPoZ9L%QWnh7-&=d-t?wg!&cIR3BzbXhar$Ou z=lV>_QiBtMl+r=;e0cN4!17Yl&^gTf0Sfmc_^*_fm*;=5rXBg_w%gnY2gYtOEm?>|AEI7FF>V5u@wwYvGU(d1UKnkefCK#Hm@Fk9URE5wEJYS3u?k6Yj z#ucdrBk?^JXpi5=AKIS!D+Lot6h1FH_aN88i?L z?!o1eEnWABhh>9vj=gZ%{ zx%tRX`rP~YEh-aL@X4g>Tt?%whTG-;a?Ygl0WB&Dy8oo4_JHEbCu8?uGWY;?H?vku zLeqN~D+ABj2fg6s8#gSR`(VGZ*)~;j%W)&~lxXMUo$cKgH)S@qyqd{fdYs{kX%PBT zXX#EFHLBqSoEF3#kCi96P>+-W2Ze8@GV294;9ic@Pb%J3ucKSx-{*5m!B3;jyg1U- z?o<5#)83nh)x5s#!^>u8iiE95g(5^nnkPdfL+y|#m5N5Kie^nV4TeN0N+LsOqInLb zP$|u|8dMrrt2C^p)$lv-mGAf2&+{C=_kH#s?{U2E@%}pOP19PR&;7aYYdEj-I-9aXO05l7+mwL7-R6vylR#mtusXWV)=xw$sH*Ei78GE!GCz44`-#@9049ZzeVI8bmy zFrw^X;w9<=@(b4!eRj)Vtb22b%0bCHg&TJjbrT)DM;mVZlF8Uoo_l_z@rzsjOT{p` zq+>4iUsjXfF}_KBm?XO9#Dc4GGQv`_O7r*|1QC%U+WF+=-11pQR2{ zI3k@tFJxB4x`xNPQ)5gUvrpt)oIFSZ!`BSai3_s zCO}?csn*1_w_r^P$bt@OnNd;ijW?CFRrFjwxTW^EAxTmk8iUA{laHO@d6*Zd>todU zN;8Fi_hL}@M{&p3&!v`Ex{`Eh^rJ@D`H~|_S?4o2FVnoqli9gNV%|JI$%Y12*4ww= zB7*i;KU6eWxvAgN+Zsz97hNk=TFicFO6MJ{wSx%n_My<=D!NZ_bs_w&Hot&pl*eG1 z>ty?L<^mtyd2G;jZ4U#O_ z9!#m_qdQ4)+Ao{Aym8^if!$#)u5CB(FQP2C*q%^RJ)eAs#YX_47a11ee5<1QdyErm z?P(7;*{LisSBueH`@(-_;pT6_Z}Zn`_mvAaj=whS&Ib80F{-gz++j9eQ~hT(*4uY_q{JD%uiDCn;u zKTFwvsy~EhK@>B&lbO^uneJa*eJ|=Ott^g4zUlS;TF(91b{iV|CbQ4$#kBOt>LES4 zyFBjc9rA4oC1OdR%M+dIp6-zc_w{u5xSlPJ^=M$cxFLpFng7oJ!^yN5 z$9KsM_clGR{r7z+uN60Z~b8Bn$x=u*3tVeW{*^1)8D0^Up#ZND3tLl znWgWnxyWm9%qCAczpKYX0?d-@zi+iDu?WZ!6B7FP#7aXz<==Z)a$xuD6eaQ-yqvOk zum*~T{rB#kDSuO+IIGstj5^Ug=U#iBPAK^a3dQwuW~l*L)VpUA&ghY^NZe;XKF5qK z+V=OSOaBV8KX<@l*vtB&VaH*?*ufWEf}N)1=O~oCIWB9hQrLHRc@eoClZ1~W+8Quk zi|m&PGx#a&ZJ3iY^V1Q3^tmjK8rWUV%UwZoZb zvSin#yZ0`yeXwF?mvFB8;Vm2~q;7V2E2G+)yxzOh>;tHJr{}7vW#?2NjPvmO^+7h( zUHi*w%YC{U4Edi9n%(eYiMGzXz3a8iTLj~1OMf_-zKDE{H+uDKOxm~&3^?U1pt!B( z>K(CiE}4xDsd{8vQ_cRfE1AEB{Ynxq8Q63;miyvo-HPbHWyT`3wvM2h(jrB~O}Z@@V?^m*0uLsv~;_t^lzqQ|9 zZ^KsW?{R+dzcSVJW^8X){9s+E|9choo1UIlHSn8>42OPCnLWGZ5X4GT&SDM%?tY|4 zUAjTp>DuCzkDkJXyw(}{s{}D+zMw`aF5yn+%>HLd{mQJf2|j41tE-!Ss@Gn=b@L`- zc&2V}%bQLtX~?ooAYfro)t@PJIewBLUB!ASj=0WcU&oSP7-Q3nY1Z%mgKNaSZ@+~` z$dvupwdwu7%Z9o`^v9#4kHY}MjFoJWR1`ma^gx2tq?cltRMN3Du?|WyFZ;FDWZ&W; z$Con1ehTFf0dE^{PVKv>GjK|ySjF{jyP(p6z~u{8TF&?Rk%&RUn>=EV&74a=nO`A* z`+IWoofRL61s4>$H$BCHIcgc`{y22zo|lY#ngR00|6*IoqM{14oK)1;ukJTT*aZX~ zP49Elud=U)(mD5EuWBU8nakU`4>NuXT5mV?>3Dcd?UW%q1|Jw#&9upWQ^+-pt;Zzl(3=Y1ny5(XYUddE#U98adAuI@EN z5w*UWCVZx*6gJy_>as`7CfCwzFMl?LHn;Pruyn2j`Kk7HQvhba|27MMl9H2OBkMp! z>SKf!+y`{w_3Jkw(nAIYA$c7PD+c0!_imnSdGhWjx6(92Lqkcjv~$hlpr92%C06>Z zwXkthSiSi9vuETDkq83SN75zmU2}=&U+*6yYa(mM9d&8vWBnV+3~{7t_>&fXwX!oY zIr&R#|0g;#Wb(qAB@!kn);X&itHpv3;@xN04~&-A*KY|gF&m!M?6HQIGx@rHMuS*J z19f;VzTZBz-tnmkug`CnyfO{qY7IvL0XiM7%ZST$--{tlV;Bx_yV#WPyXGxfTm{yr zs5RC0Mv9WpypffGKWt_3r^r1#Ffd|h$X+tPcI2BR=N>?t1FXq*yHh%>Z0qTzjU&y< z8>FP}xeI#lT(brj@+f16l#~UC6Fmx)&e^cN0hxbJuN7Z@@rQTxUW1K}a2D4yZ#jHg zxM=mj?yBn}nM;VND8rJ2tn(o3+$d(>zCehMsU&%aJKlV6ou* zv77Us5jq!HBF!FfV?azY>utsG7L1u7U=@0i>#IrK>C2Zgq?B#8dDCm}8Sw%t#X8)r zrW0gz6(1j#;Fyk%S-ypA74kZ5eSPcU)-3;KFq1DB;u=MPEnu7Cbdf718SBA=AfjSc z+!kbO8&y;c(KvDd)b)OYPK8{FA57QEnwpo$KJh?9b6kL!kyRRrD(R49)sQda4)M_- zxXm?+kqrny>W&rg7EH6PK-_;9ONDAB)|zC$Xrag5k!%j>U#qgwa!yb)d{NnE)V$q; z^>24tGR44ieH|Tr43i@A`iz{XM}iutI{Wtdz*@AOPb(s0h5#Bbv2(*@pI1o8aY3)Z z%2#~kH!}6Uwci}HgLY*Hz{hcM!8Aq?mgVf(=iS}i$Mp30xM|HM*M7=-#%JT>u8jJ$qf90fYh3~U zYq4zHt5@5m#ZlV~VJW#i?C5tkPM(HWy^BmzAjLbJ=Z`-u$aW|y+Tp1nxqW+)#|A*# zz%F5%WLU0|E?8_RnGjV>9i(-37j9eJyDmi@kFu8r8{w@T#4{Cp&<@sr0?TzHzYzpT~ zTsnr6I;Jab+N6&Ag%%{9xTI(O2YxF&2*5tY8gh!872Uq`2z*{Cu>R5AyTZ8=qIsU+Mr_|#$Vn6*EE3%B?Rn*ci1B5Ns0&Fm#B@G96yT2*0>_YK+@E`)St}kZ z+Foh4*f)3eVZO$Wz#^^5n3$OMPv$vI;yFz=k9J<+(ruuo0HJ}sDE@I+R;&D5K+$Oe z4nkL9aE&Xm1D`*C!-FpRu@9FBuU{UvD0qv-KgTmAU!;(aBXSlt1x9$(&rr8B^NiYUTO! zM+%y=&rAGIw1*{AO;n8M(z>orm5p8TttE@dL=w(gIqt^ZV7q4pd(ps*svCO$uH^Z=BN$=9o0xuf13 z6`8BECy62WEbHneTfzEd&v|oxYrkqa`a|mBInEKD<{wmAw#ys9T-&F2mJO%GVq|Pj zLT=L@*=E6;eaWosgekeqX+1r?;?~rkcB(X0SDH!3&wF2*RXDsoMe*D@T3@B1q=sf$ zvh`HM3fs0XbHaMGv#h7N62pq$H{0H4jF-36;j z7R11yQM}+AHYTjtb(bv4Fb+Zsm&=n&y9_oC<9LVvvNm8w7XC6#`J3i#Y>&OQSa(Gj^E1w`I*@sZ4Fm1TvSoY>)2RTte_nc_7Q zmI}W>4o)&+A>oVU7uIb1crb!{`TgTD za|;WtHA_(2{}YHYViTNDODn6knIi$$>yl+qSylCg&IBm@DlP3Kimb4QFxPfcd-bBv zm?FV}z5DVd5N`m3jX|WCE@Z0wp-CN=gu??%0muQqgb?MOJ ziE0T>aAFbJ;jwESO!d zK}3Xo`{hBWi@$yasqcuX=?i%5V)yWe4+$6zv6V0E@bU{aMRb!zUur#!I}r1v#Kf1+ zpYMh7g*$D3ikAoePH=osM^>#`RWaiN2a}rm1ixpM%fRQal0J7IUJR!DIm{NqA^@bD zu+%gDEubMZf$eo60>cxuJmriYUl zb3aV`j9Ac`6kws1KTJ|n4b9EXC0LCiA*)D*3igbeq6d?PRaxM@dv;c+F@(!#^^qiC zK75$$IkYx{8uJVA@>yj;Fue^xB$L-Y6?v6F+;;RL_mJ6 z*KJiEc23&pHMjmn;+#MJ*d!uSp!jIvNLI6Z2to&xcjUq}5TD-mm=K)g8_54P|A<0^wN93o0@@cBoDWxGD-DxC)s}GUA<`MKi`jwt_cl@6RKl$elY+ zfmb7Hdr-jB+}7Om`KMidHcNSRU-Q&+hSQQ>y-Wj}ytls9^m{bMaR-MaVrF!7c=~Cz zc=}aURkIq8zAcx|CeH>W9|-H*yL=a3rZ*7r{fZ}VJR z4-<5q9c7cwL;zjFRC{F8t%N z($b6I?mc<33u5FF)6-zFgm~7>D~LkKz`FI5ZrcKrkpFBeLN7&AfNw|fEQynC^|n>H zg{MG_Btqpw5#G}d@?qO7u4^MN#&Ila^&|UaG!6M0EIRhv(V3XJsC5Anpq*=$lqZ8K z8_H5bs54?kn{zfb?kgfHBoDRK=z_doP}lDo&27r-gHjKy<+>@qRT}fWS$BU5zOI{U z7N78a{j=XbIyPuQU&&8FKh(}^13;mz=AwjA3Jswu+*^BqhN56%{?k7tZXb{Vn({K;(^Shq{_@yTQkK$Hjw*G{$c4=-(~gA z%aqL*7TD697bEOZUS#-SX}0Cohq4DXHAOr!jCg%qDlKF;2{xv%1cE@3xS-;T(9f_d zC@}CtcJI!O%L&G^vY=V?`3KgCIu?J;IcZ4`+LWD%1b^pP^MGR9asz zO*8eB7_1xT&qzN*>**SvUe8>X)Wogj-{z(;1WCl(I$u>B^+5!c{q@o>K;X{H4nAX$ zR3)8#^vIE-DdBHR&-Q}dz=TK?lmw!eT(NB*AH*w>$`T62`92|lG{IOyM97&N`udePC*n|NHY!Ts?vGU#EPaQZc?bpD*9bV^e_$sORm0a|lin%mxC2WKka=t# z$K?h15X1Qos20R43Kdg|Yx%n;JGBR2u1;qL~W@zi^hy)b; zGQesoW1ls5fDLttD!=p{cqqlKDakl&$a+Y5Q<(&0p`lT*fP<-DQTPVcp#lsxj>k+) zOkX2@WeTzHam25pw|PG7T!+t94K~`@Ug3N<^hGqT6PjniO?|;eyjicGrlrZ{$9y>b z2a?1NT%LkcFNrlS9xDI=ruYwTzXLR9`qrh_{Z{mIh-P5~PdbF*W}vsyDEY5)9S?a^ z7F^ago75ZS`tS|W-`d*Q6uWDgOQKbA1ucsF$JaHUhWC84)a6Ou`?z_OqZ^lZUx?qZ zVs2`U3P3()+TxZ)-^kCjEIN+ot`&a9Nqg>+MdLCkX_AM!TIbz?JichvO!9nBuOCmz z`!-^;;N4^NBh+`uyDF6NvIOy~zg}VHFA9guxXX#F&X@1`$^6G(2UjDAKYeX)M{$-V z_tt;%J78kE$6o%*_pj^x_nT2p+dd8V{ZH?L_e{wHv`g9GoBORp9__TtE(e|V+==V| zbsOxzX1)8qPlZdA+vd$uEbPDkx_TK0_z*(UfBt?i&}FDxt%cdu|M>j~j&Pt`FXbmA zr78cV6mQSpmyIkUo<(Pe5qU)azI8mXt#*LYy*?vjNli`FbO2B0pZ@aBE8tf%Ehdre zk4;YTCUIQm|NCl?wV_<*n^a^!{=YBZdJdQ^KMBwym1AspeZVtY@wDpn8=;wf^zV!O z-#wpqcs|@yBX%_|UE0I-KfL8_xaFN!vNc*asKbnGGoL>^_wUz59x{cKIyCs7 zA1gi!;)Ug1vm0m9eks*=AS%Jge&8x)PNLo>i#u3>z{0-?~WO)Q)-~P{NvwzgRVQ}nDC&W z4H*pr|9A@m%N$QcKeES-e|p8GfYBqFXlXL8_-C8f>$3zWBK$~AhtyNC|K!9i#)*?4 zxizeXrUPQWe|p%xz}^{7TP$y*di{U6edl8P-UsWlu|xiT{=dD%|Ly4}*7rqrV}TZ7 zCYk($!_Bs51OYE`!03i8|77p7FBQmcIQ{f0ng8((`V!n{jmM$;61!v??C%JFHcfCC zwb3aAc>x?L1p-+!xR3}2!Ov|OPvs?f2U;j>`WwdtsTb%#M7fmy zZZ=m12C+x&SuFQVjj_w^6}W|%;fDs0@%<2pya$7UqC_m3QM#iOzLt@W5DevhNvCDI zXY^(cYQ%tOiO?F^bo!6LgZ3CUG_tV~qn9F}ysiVW;jq<i0!QaHOY_txZ_Y~5os~h`Aw^O40P&F3?r(5^Q^3| zkH&R(<_Jnull9u0H*5<8pc`|ur z7A5Q_Ycd}^bt|E*TZ?a1^peK!R&ag!J57x#ATt)GBKE(5%lZW7KEwD2sHoS{#Za-G z=+?>E4wP2@M#`q6FL6=WSq0s` zd$*#?N%S)TfgXXGOr-;5-0lH6eLjuIgj`psPFT8O|1?x3VCG3HrUv%CS&VwhDckqI z?*lZp#c|daM~vEXTxr!@0CFpJ)Gb?#P7o-ujGpbe)D))^1zKwSXEC_)HBB1HN||hn zpOAbszQd<;%FbKM&`>Z+-%Iug6;c|B)r-}9-)_g)wpI*9!M$K& z0B;kkaYQ~^Xw#<8sY7K4{Q*Xta65FOC#~br_1B(v&P=q4dv?XgVrR8p6OHXm^bD1| z@S-0~QwQ1$beOLs-DHuIw6nqbrWKWyAI$-!N9;A2*RWQ6P82e{%1;Z|3f$t^?YTxg z(GxTfFO5lz&XQKB{qSLTD>&Q|0vCJU`prHK`~_XYZjVE+%^JXk+q-xzv5pr36$4Wz zpQIT8)v>GAcUp`TY4xjHhvlWcD;>(V+Bs<>b^{mC;N6rF*zmf^nW> zhX#YtK0(JCBcE!ze3|K8!HaZwSeNGWg z3;B3@{I_|hA>XYK6B!*{M1};PoT@mgDTM)`+^sgAtk7*96V8K^CIL!Ss7YL9s;Q}1 zh4!ojF&l^HiiV)3V!vu66AX-138ux@Uec$h`lfGSplgLJJvyOi1XLk^v}!r&mO%X4 z3X*Q2ZRo(VCrSz12;-j@BhCCr!xqi?ZSw6w3MCAwZ!b2BHZn)cAKntBpB6sJxy4hb zLZWtG^iWLt2@b%9`n5q;mgq7rQYWKkWZ%?ly7zzN$wvx(i9N8<^Y27oa$> zKj4lMF&AlHVQtS;R8{TU;z^H5xd9MFQft`)Z~5pBGG>iVrWyO6mM|9^gpnxmoVsvf zOmONa-qiL)Y(&HdOn{T5W-*VW4LJ;roV(YyVj6~2P7`>_PIclWv*s3&U>U3(?2d{H z5gEW_@}3=yV%ej{36C!Jriny6M3FQ$bKt;*G1%@(U(J9^Zy+!)c1Ilgdppo;WV-)i zTrUg1EWc$0yzNjsSOFeY;gCH~ZuP@7gcd^NsiAAQ?$Mu004ccp+xwl@Exkv^aEL2! zYbnOK+>3hxw^`TiFu_D6t@RgSaW+XVMt6W!YD2GITZu9&+QoYq`>WJK3DP%GbQiRd zu4{<_xgr&=rl<`^#j{CykP7>RJ^*7~TsTqfIT?Ji4_>dDIz~$T%Z4yi zH9L3N5conDp1C4RR{t`mWrXDq9dl}7QdY#11dlGXR7gL|(plmty?&55rVyt6j>$ow zRz=HRoJ=rvsB8auPZq=n?$2tRi>;T*<%|{E$ZuKJdjUXkU6`}4KGxh+iC^Cv3!r;p z3?ua9qo2Z>XBr!Wt&=tVr{6-jBMI=10Cf7p=snQ!a0u?19}KC=Zu@KXa8Ms^MBa>z zn8GamIx? zhI!=GM*IQ+1sk_)*&)0V9U*jwi?b+7dkp4f!I(~7f#Vbis|0;6 zLN^Y$IAFEg!eKa7sM~kFIklw)gsa2o;CRb!A*4Zw>$N|xA{=EB{Hl7{?>wiV!5Y1| z@Z-8hKyYKMDFi_n3Kvn$;Gn4b&Olw?V0oMYd65-fI?LtBD0saT21NM{P;3iCKhzD4 zLQ-QwZMUp(ne}5C5)p#_rmkN=gM!7q&`{A=tHCz;bY(n2f>CmkaJ=t|s zp@*t!En~3O(a=e-L>qfL27o^`utC#S`(SJf0}ZgpF~8(f3`)|j}*soxP1hLbK=feq#KRyK^cHV;VjiK{rDvt8B-|2D9@KLANZodxZZM#zp= zh4Ia8q+O|`gI?`uWNXQ1gHD*<#D~SElEI#EjOb(GueyUN*j?J!j+{Ak;f=(oeYNLa zS`I?7iJ(B+!cYhmH&*q0xcQ4b_jNS{YWs%23V`-4z2CUMy*5jSRcHj}tq&of=~ zCTZV)!VGs|4rB#pAq^QKt{g8=nR+rlK2EK^F5L(hA%!s|j`SYA68PYZ%=pl$MHH~? z2)cUZ1Q1rlCL|`A6Lp(sRVE;w$6?7wpJSTiTe1?Dc6x0({7p3LY6@z_)(z=Nxvm=y z-9LSEOYeTP`JkRdv*x>M1F*yu##C4xU~?t0u(5}1T4IQ-!-1o@5=E{XG#=J8FK;!% ziU3=urdSIHAziQlMnYQW$UvUSu@cWJ(y@tLLk(OgPsH0&q<4nyopS+l53{)x-xOM3 zX~nNuw`*6IHNJAo)d_OoNoZa+noQf*GS7Q=I`C@}(W8ftPD06V<5^uzz{iDN%(J0V zr6V`@#ok1t?0ucur8ZaT>Gargc(=Jqqu(#%-x>C`XMI1m&B0@G=|ki=Yp;0CHqiZx~DG*%86_3mi0iiq??Zvn4fBfi5qDV&S@ zoktvFmnjsIgU`~$Y%&Cv#GJ^}1E0a=qmh%tl|mpOOA;>c-?+RCj}ELFA@y`dGEF{n z`-T<$?IWz>S5Gsda!nI90j&ogEl=+^)@Mx*46;~}ba5I|7p2xI4OLy0ubcZ6!Quf$ zSD*USdJB@Y!7sj#^3H))f$};Urc^W6KSuMRA)2a=;-U58b_L=Dr@n$8CZEOSRFgfa zhc7?NdRwja;3i@7n3+YTr-wKnM6;y&Dix1Rq#k1GHw}tn!o8+_zpHJ$?W(loim_L+SSgF*eY(3>-G-cS8o6lEJ)G!J6`S zp6!*l^Vh+FwY-FXzunFLd?uiG{BU$seS4f`=OG%)My*_ZHtX8nOy;wj$xJ-vT(r(b zHcj?A1(F7~&bI{u^dVJW=RG!x(-VWhaM3gQoHQw_3Am_YhE4=_8;Q2u`z|)oe=fLU z8$BefA>3l^394>vYRdxDg^D2{INdKe{VmGXnA^rWe3{MVYRJ6Ve_kJq0*_3KJI)RI z>FEA4jis9+AFi{oSvP=O&st~hB!PHIG2(0aH=Q5erA>%3Rc^CW6ow$%1pGX9=g5fA zX4=h)b7b2#=Ien(Go}bu@?u}1*Yb5$dx|z>Bg#Fd!m2G1R!3{3A6pEirbcJWxD3j} z4*b^iwT(+j%^1>T5y){!cRcZ}q?MS@CQ~ag5iv#s6jT^h3q3oK^ZOXtr}=*!Pr=^Z zxO{t@*ZRh#1qihW=XVn}IC_U|Qgo~wHcW)g1>ZU`(f(v|!c>fPj{}d5k5OsPq9d>y z1cVwJhas`h!OyYWvKPnAvx-V^rmIiy%bewvG*e|O<-yJWJ`+JKu>m7%G0@`Dudt8B z9eT;~f#^kS zqPmPKptBM!je5Zj-mP1OgoIi)CnQhmBaXp{CloXMtY=fkxn?}2>-Eef9PpZX3-F)N z!0xMTk12-?s<;E;dr$m1=P?WyFBY8hSfnnWhG;&03!i~9#E)SO>`#>7zQN5Xx8M(z zw%)G!sgBBbwGH=OHZeAMme&VROeAuZC`?YFFSLp+YNg z_|aou%~*x9VVO{@9uQ41*@X8Ry!~T^btb4`7sepfQ7k(2Jj+v&y!m>Dk}}=0+YE)j zH!FTE+hs(nv#?}R5yj;)To^aCY&X7G#OQ0FCWDQ z-p!+ZuO2>=%1}>O?w*;X^s0ArBd9uPd!=dMOb*+8%cr42TAX)Ar@sm;Fu*v7Jo=?m z!Bw$@*i_(hJMGga68quoq$N*QXlLTA-S7B=e{Cb>oi4E&I5?Y-?T7$p5(lUe^l9OG zWnk;(BH*47OCa2Z8oM?!U*9$ZJhuzrsV;MTEP`mkp>)?=^*{sHuJynFc4={87D}-s z3XfrMO=F&Jb=UBRLKFduqBh==HE?eJUEjIc8_#eRAPpiPut-o@v6~6O+}FrC4+a{z z^?%gp)1gWGj4!ppkm^if(r`h*vkHc)F~*|>Kr~obo%B$j64S{WfX>;2qan%=CJ1Ys zR$47`q_4GDg7d;jl=?{{u7KN8&SyAJ-n*ZO2{yOJlhSk+puTkxkMMS(I!!7t)M5F0 z(Ja`v=NMux`$hA*YCq7lLdChG&0L-})Uiu*1a-{PTyRk{o6^-?FO-ctk)fwf1R~*E+AE%j5fmU0;85weIKHN zX2%TLz4;uM^O1YtD{@x8^9IVFCI~O|gFrak261gOj{@i- zX!^E!KoT+Jtbpf4zhfbU%cvM?$QO76@%}2LEQ~K`Hfe<*j)d?^v;gEot*-9g7@}7r zY6pUp>%EuP86AQw^Q|+`qqjfVE0Z(kzj7`~(=*y&$o4_|VC`pKC}T7aj^WOM9_^aT ze`Y_Z1>5C&tis)gPhL>a(Ra}9Q{mmBgVqBeM`?xIp&}xU6dNmqkRD<-bw+83|GwV1 zjGdF(FhF=#Od@~|sFd*~5f~B>8mW>^IFq=z>YS;w)DA#XXYB=-Q%cR?JoWIOA^gf_ z90&N4wa3x6s6w3R9+2qrF*>aHCbRxRi4@+-(mBhZmFRh35P5Ce=cU7fB+m6mirDA| zV^VO#A4&S&Eg7(IKrulBR~ZVtfm6Sd0N>3_SAS?DRPuuBnbwod>ZK=c#hOb(hs?!m zJSAtcn6M;~^!YbU4X2n!tbuM1ZJAtNFo9{o1~FYd(W$0#%`fJrKncKFUx%~?stIg~ zseZ75>#Mggw7ISQyad~R(AgzsG=|`g4*{U*p$4E%`@oIU($+p(@7(HdO57740&#kz z(&ee2P|jNEn9(2$nBAY~aGgjrzx|flAm0itfE_r2XX=s7C*wtLoEUs-ZovVq&C##TTC#>(cbJ#C*KMpU#T;;D1=>M+mpV(R3$1BJ&Ei z1mC%1q6k~P4@MRQJ&RKp8A+y1mbWZ#mXAZ{9ltSM$dN z?Iv6KFz%e&VW?G9wPC+?mh6H&ve25|%T+px=SKzyto zx3&+uHT{K#Q)N9(DCzlwK0i`Gokg(+ftz5&$A4uJnQM`gp}%&TjSnN=s>EdKhqYsl zL!-4N9;R1zKeFaV=%oSX2_gi)!QiYYkl&n+whd&7g@Zu>ch^!six8%~Nm%gJmTYMch*uD#XFOeItBq zYjU{)?v;l4Zcl~v0>fF~h99q$lq?@**mzo1R9BBMg-pnxSm|!rg(~R^56}f8-3uuj z$GNY74r-)`^jV`Fr#yAwi>Oe`R&Yc|8~Qw|yR>)u*-%qx&XAM4A$OtuzK&#~JM0)Q zf$<5;iWlZVs}k%@f9SUewJeCkq*yWzSSfv-EzWg5u>W%gcFU+GfJ@IhXEhpl80TeR zNo($-?63_G(916}Qc^3iqygJNBk@xSIR$_!ANzb6(-C;w4f256D=V5+Lh2D^^J`oU z7}Y^9R{ij44{vQEvGYA5g(BEhXD?L{`%lkWl!~XoWm}} z08|gvX$Y8`)dQd#vY|+LA!SgQiK4-bTARm*fZ&gT-7Kz9@DeGH`OpXnYAzwIqhJAp z%#p6nn+Jv`G9Ab-96NPtOK@Rjlq)!zs6~MIF9n(^3U&pl&v`ikd*N`KEVY65J&~PZ z_m{~U*B{}$i{cAv|7xJ4!5$w${~W>vS8W~S)Oca^M@EDG4kU&78lrZXRd^o%CRsfi zvvruKZ5MJ}CYcsmJrO*ChSnJ6ie^1XFUErZ2aobCXVFq{T7mt6)VG~;4~PPrH*UOD zY)aR52QM1nElAJ^>bs6kLmTA@G`WENEw!O65t+#=*rCy4fJ&PYA}w%v^4|ZyR;%d^ ze`^1|o=veMe5t>EfbjT#3fjNco*HU(-AM54tNvNy%mD8Pkb3n^@2A*Zl_ zG)&+#go_Qx&4LoSCyBrorbtDNjl=I(A{yg(v-hg#l6jqgP-F}tr`09jo7yVBB?AHW zB#ext;q*|XN@c2D>v)zw``#15h@IZY?u4C18U6^HMv+*PJ=V_G02HB}4v5Psqz5O_ zX$Xd7f8f3Vfy41;OWd!%(K8!@F*0Um+NSvuoDPJctv3gek&e2=Vc`dB<2sc=chKhBMj2 zV{SA3Mik2MY;afC*r5jr5(imW3xD2iy8}Uu-~KLU*A{?Eg9`T`S|sHaQ|k#IquMh9 z@N3q&nU?JaA@Lofw|G|Z;x$fB6KWe}$q1RY(}1L4b^~V~fT*hocrq?g^F2CH8#2qm z^FUt|IbyyLdw}<(uza6gVoPa|@323-ve97muTRL|x4x(Hqy60vN~$k`mTM7lFN72% z1HH)~1!2kq31%h7Q6>(c*S|3$y#QO;C4NLW4HmWQ4+f#JTRjkR%TJ*Fb1)FhX(p0TJ7t_d*yEe%9fhZ+8_Zz!oJR=MW0MHN_n@1XQZOcJ65b%8T zh6p_o6e@eXTNLF2pm^$ti{yuEOdL@D+KB$Z*RJTxhTk*FeuPRtcHMe)NQBvBF@uyR ze(s3@Ko4sOGgXQTM;u8JMMa|u7-FKLw4=o%VjLC>TN=tLqku8Hm?v7*k3r1ci@y9TPK z#SbJS)fjzO(YfJ1HP+EihnDvy(ni>T>BQ(hM{AZk5p^Q6Xv&i+?45D~)cqmS2ZIhg0c?7|S@`lA4m?4B z5J$@HOBnsN_EoMLAX9d29a@#hJbBgyVRf_1(>svR1o0`k+6OZr(l1$PaG_WV(p~uj zZH4|wWgP2>`)Ea#a~mAoNlk!P1i2TIkx40MVj$|dExL&>3s_yC#Yhs>G{k8Rx=twe z*dY~Lg+WXXDGt=S#7)KxWOlX$XRb4nGJg!Ba7bT-Iy<{U54c?^NTRR5F>>C`a2|9Myh&L#afjqkuv3N~y_q}8U z3m(iU!!F2e?P$tiR_QVCgiqvU4m=YgtRF7V(Of^#;PjS$XZ!XCX59oK$F{6&Q-)nk zpPphlpb$i4X|*bws*Y+ZJx6|^54Lzu;*Dy0CE~d#plW;GpMFy~P5_F@?{ig3L!_)C zAQWd!LnH1IhcO~aq(@ELTd?}VjYmi?(I)SxTSaGf{XU0CM;k;u_ff!&`p0K_^W3UR3WWc^ptBp-|i%a~H{UBHOv< z+;?S}Mya?*|3d#&QU_4Kv&kNNLD&Wms_sM`sv1w5m8DN4-FBB^W_vSfvevCnuF1Apd+#+-& zRtI=an8fI%#FlTeTmK4Oj9^8S;h+?c;7r#&0i{{Z3iw^8}=uPSou)YQ@mm z1DXdNX;w8d=BMNJU>JQPYlV(TgjuB4iDwZ6`$c-8kW`(*xNWIIap-VDkuIEYikwb_ z01a_kiKOofgZsS=eGc>t)U^HR@rDift+!B&Yd@Q zt@&M~%8EfzXgFu0-ruR|IetgZ`vRg%5LDKon9&J7*9}Vr#DXYoC(HQ{!`bfeoO1IQ zR59=|GOvw4>xy$&Bzziryudb-EXp_Fx9s&iVOfaL=Xd#WJ4Dtl0#F!QPa%g8L+l&z zLk}dZEiAqfqe-Z7MA+BWkm(I^Jw*TOEo@Ff$XcO29s5RFuX26#yB5#LW5Obh1X^I#Vd!$6*^u)wO-CRt7)QL zMk{6n;g4Fh6Qp3(lGxqUr@xb>t>7Nmef!fwhX8UO+3GYq!%xAiaf_J4RUZ#-b5{=E z1`)!-KqX%R*O$#BEQPIt^X8CY*@;#q*l6WqDGIjTJs`TXej zg)3C=64maEXazhz5DQ%L{aY`GsWBc}ID#NjA|M_Pr!F?x^Q)Jv8qnkcJuzkHg0 zSon(E>czx5w$p)@DZjq*0TjGg#0Cy}y+mJi{}ov0^xX59*6ar+X>#hwZH{~`Mf9aee13$FW05BWG1V4Ht(4YS5l{z}ux%-GrIJb^P%!#+V z+2u?FCsIz>TcLvrnl&rJdOABCvj<+~ZYLVSBwBj);>EtQM$ueMNsB@16=q+iF5i4* znSF(pn#G4J^BXjz8jcc@T{!e)=zs=qxHbS`jxA4@zk`-QP)S~P_jR)R4uFWTP(1T7 z5z{5Y)d5Ro(SWNJ?9Zbz)gRyh%3#RRM=BFPa|uhK?w($f%+T;xjYJnDUBG+6&PejXKSrn7w2;}p>BGk$PpCIrVsbEjD za;Br87vp!HOKIrV7(@Z+2yPM*!^kuBYFx|qnt$4p?BO?xC_BR)(xqA4e0uoS73Pm; z^0ytN@13_fxvlcN+D-ZHjU>h4QvB&f3OoDi1C*j+Cddm5xOT23`I?2kADIt7X7i&Kbs;eQ=GRHL` zt0i>|T%JGZsYSA8fFz)1deM;nKANE%IIQ8W;Lu%RP%Q(|VCCeOBcx(SVoE}mK?XO3Mk}7BGMm9 z&jt*xnn}Dm4DsP%;6UbJ*v`g5L)S>D_{vEg}u^b&!0@+z@+e zVnP7s^?q6px^87Fccm=%ddmR-U z_$%*xT+J4{rEqY#Jn1AZefcjBSDM2%@8dsmU_ynKt|Zic;w3OMMC~p-DoNpxx?^0@ zYd_W#E=TO*aRl<`vZ=^)$TvtT4e^Lhd!iJMNgWL+6s|(>XW-7%UZ_iJgoeISm~+cj zR|IxK8l@uu>PTjegYMpb5xcPWLVO2L&ZsvkYQ@z=gr1OPq3oTdU(2LdTOs)v|C!5G zJVMmEky++wT6aRkCDG3s)4p^`(aTroJ(2LMK6SyLn%c~vzUn9IF;bMnL;;FUxLUfY zwolxY4Fl+q2j$jrPx5~XDgYf;x<}81eMDye_3Mb>VJ7kbS?V;r$%Y~KpF`<^-57fCaGIT&#w-QH` z>rm54pk|Mm68}Nw?Vwm03FSy2T?AJg2ey-Oh~uOMunj^CA-)6S3sCkwjB5blG6;Dw ziv1W28}F#&J4uY)U^3Lwm>B6wW@bsmwGd-S(2H&|FbS|=C7?I3;F1se!NhbS7d|I5 z8GjFhfF2`z`IK2kCoNP2L_tr7C1MOqDwQoEkF@wWppC$1$lVRFqcJy6;Ekim`t=qV zJ>-f~n6NqsR^JF64O%atRr{T(AlM<0Li>|VK(r$2P5atw7X6honE5P1_NhCx<`;2&uP(YRY;qE7;9`|}~Kbszz zo(d;CBLLjG06g1Szz_T_S9LneuBB3`H(zH#l?njY%f$$A?w=!6p-SXovD2dg(-Ltj zK>BrUxhf*B9|`Q5u7=b<9wNm)#Xa+*I7_6~h3By(vP{PFhB=#MjhDy(Wy5mE$J@d#) zcpFZ%@PKrAr@bUx-1lj#T|N+wO1KV=9_9M}@95;vL9LlY8}8^h>q~Y+z=I!+1Bk0fC``C1-3^K;cc_wd;9Zs@+pDSnvDECNRpTfj=;cpgXaiTW$3dc#rUL zz(x+A*81ipF3JK1H+cY*$=5egOV=nw?9@q`dB|qQ%%YPu>0b##gdI?Y5d4WDfG0Bb z2H^(MG*|5=bQP>KD0v{MhOtI(%{P_K84LViM%n>;cfGSx9&ayB=6l z0=Tu)i)QL5olcNrheNNzbqTA3V^Rr=#LrXyfE}_?9#0d2UV3Q)d6iBl((SWMDbOB3 zN#HL{;TkUXPe?2yH4ZNi=wFbL8J$@6AYpLJN%JD=m$U1)lLZEBC_>(RtIrAc3(5cq z2K6nfNHsvX0qLBu>&#Da$dR8?f>J&{UviYxs((#a00$a^J|TsuHN}9+xmilefuhHj zr=xgHp$bpo2qL|}_&9A#o*AL4^iDz)mUQd<`{ltKqe#4Amtk7q}28nE(I) literal 147893 zcmeFZc{r78_%^J0H}7_#L7FHL%9yEHS`?BYnHo%)WuEOy?FhAy%*i~H%wx5sjFBO; z$~+b!gzvnUwD*3$@A&=oy??#0qhs${)_T_S+{1O9*Lj}TeNR?eV(qG}tLW(H)>6)# zlB1(rxrmN#iRW+2@rvmxA0ayYlXB{~ymjC}i0#Uu2$-~Ro|7n4nk zaC`s!FnjFY`Om-T=n7w*U%K?)FIXRTb^iCi>DEhZ`2F86w0~q<{GZzpWfDL3?>9K# z4g0@0K^u_&gGS=M!uEgGz~yttkN>%jSu}ddignS+$?tyHe`$Z=djGz;_3ZQKe4Lk_ zp9v5!tPU2l^Y-zHc=t}$qUYW5JlnDAOw-RD!^6YxL}q8FjX9N{oeB^%uB!@@P>_{< z;?J)itCC@~l~4D<^0ix8ZoO##qJx#~D|qXzCXa#zS)z<3m6i3b^c-aq37693y!M`p zJ-B=RrZO*Gxe~f=yKwkmQ-*%A$M(j2haAh{)`Wq90Xc^ZgNCO@E!k?{a?bAcnjH<9 z(|vt^izJ1T(wKyoZD+>nElN2vGGb%$F^ivsp1j+4cZcJ<9Zo+&S*0FDOZjp!3|1bQjz99feMwGNC5GnB!#$j< zoO5$?`EbWGBc>*-9PH;8^Kg%Z-o#J~%D_m2)WWytXARPeRky?wi9wDKq!e)GY!`dEDZR$lG<0>-WJ z%Bk9Ki;Ih!C%zW%P`Y~cYS!@go-ohn7X6B&4x?S=X=~=q$PjYU z){fLJdweB-Vq&6xq_b3N@Bj;C!-fr0-`>mACTScUO)U=))GuI zbm}bearN{RY?BEVRX!IWkk;5%S}K*QUH;4(<0Tq!2+!(Wbg>ryqw{|5H0M}URJ4E3 zo>!kHKfby6nBQh48ol(6!N?T1Z{MahH+wAIHc z$>7E_gtT!ll2TG$EDA~LYD#JrKIOAiZ(rXitX=7&b3al)eg5o&efZ|h8QkWXjr*kK zOCNET2EE$QU!SCejm#2*ZIC@u!of<}>qXfnVDPRnO+U@BNdP0rwtM#pa>>2t{KIX> z`h>o^( zB_~7eGnGnxu3yhJ>JTLzw27J7h`%m8oQ2Yzv1qt4EiTItEAmo~Wy0+D&+9P~Z!uOU z@2si1AS0$V)*|7-RF!KpYTof?1MA+s>^pZJ4|6g{2oeOxD&m zJG{EUtSQ5|zOObe4XZCAJe-Y>PafguT{y+7>}a>^85x=Ssj>bv?eeYL#cbufD}ubL zbme7bAN1A4guQm(7H8I7kv-KHNB{fpvL7eO^w331B1WZW7W60PlyC}+jE&XaUb(SM z;xAmh=UwPY#Ze)Ix@VaoeAnf&Ow}xE75d(V-m)BOz8?HalSe4w_r5P5UkRIjx*a|$ zE-^DbA(*lKsH5u9+3#VFb2C;&uV0fV8tbclpODcSo9{3!yy@hP8_&8bLy{25ICkwi zZewE;tDG7sY~J(a-o|}cUL}b(6+t3Vb2GzEUDM-(O-*^cHO2)qkBi`osj&&#$k4|MHP%E64cJ6^!)s*XtA2E~RR_levwEjATE0RO8P- z|3s>&!Hx>W?Z-)YGRIzsl6L7@PA9*3BORUdHot>cZc^l2W}D65pX>U1A@b~@qn@s= zJoNPR4fYzi9%olyTR}nj?6PIcBENhwF`GkJ{GdIDi}*##gc!NHx;~rSf@#*7>IyJE zi+7FBX(=2#b?Ww{+C+<02-1n)I;O*y`>G?{Y9Aw7p4}ti!54Jc^y!J=lCm;Bhmp7Y zA~D2GKgz1Asv^6C?JB$49i)fAnLagBU^ChkiQFRp-JxJ^Hi!20larImK}`tn=C9TY zK1TRb9kNHfN!>jC+4E@QcBz(Zi}KGy+5KE`Gx?KU+uW=MTJxIZFJ9EOo$T~Iw`kEK zEyp|0Mzoz~#(aA<=B9l2A2?8^-8z`vpU9h_p4XVz<|$;>b@BdDv#$Fw(a~?d^~ELm z|CpIc$d5u$ikxV3YD+ALdir$t`xQIvFO!|1mbHr#dHmM$27W=hCnF|Io9XB>!`Ey+ zSZKd}$BrayF_po9M~@;6o6?m?*zKYoj7q*ZIl^%H%P>)PR*)C zp>KHn1mbX=6n9=YZnd|kDvaK>fG1GYM$TdCTUD$=T!}$}Nz?F;ANJ=051$V^?Ou(z zWLV*Vpqcme^)|hx^aKH;=8GRcce!2RqthvCIkjD#myiIShP7z^dfF>TUFd;wmpMir--a)3u+VFM%S)>c-mapZp-!HzmJ}G3)@>`Ruzb6&U#$vL3g^aM{ zBJwYHcXjE#k36^au*qe=Ym&?qZN9Au%u*KLzg$_n`QR}>Kff*=gqrM$Ry%%!4+mXt z-)7sp_q178S;~jqwr{DdBO@c#PsQC-Zw%$J1PNP6-des^PEL-B@4B;Q%hPQ}na9)g zYOAx%y1xOa==D}dbS-`N^>Z%W8n5n60Uu#}Y(O12XA3tstGzuii4=_urp`oEM z`N}P_S4Y39WB4+buV;>L$+6<#|`wW<1MytZozGlh%y7jg{T{( zmJ$&ar4R7d^}+UAPpFDfmiSnIy^^%Fhhm~?%(B%CkG(GJ-@iY5v_d4!W>h<_;6hUV zRIK=Mnf|&23+(G?#Lifyq+mRJb#?W(%A<3-$cEATYA!C(k+=`(hD`?;K3pu7^67k;Pw>BCMvL}@wVmJDy)72? zY5+$eEJ1SyRxK93m(uiDxw*Lsw%OtE?O{cr(6u3_;k=1hr@2{doJZwZ`(%RS5!L>- zH$J$zrohRm#rm_mUe!|KsRn1 ztr|(V%t5;{m{UVTBWIO`s~E07_5HIaD}}6wAgk6G1aA}~G6chtpj5G)QvI#>J-_{T zleFvaBg-Z&2R?)%s68G&dE$h5e?msA-NcYy!?Oqsnh!8Iw)J33juM9cJ(AR*2P2*A zy;#dVsAQa@dcnOQxTaLm3>P!W9J<}|0KJ%$&UKuPb(Qv#}i<8*Xd6IxE3!^EcqZ1?@RRt@mb?Vmeh$BOPxQyORb! z7rM^fzJ2>K1Lq}8JblJwFw$CnO(r0KZ&uJ~L#oa>!4`8b@i~A&<(V8FzXMKDZgYHw zjWs}ii9FLG!WJbp@yfhzv(pp9+^NPu8by!l0IfMEM*Yw3eUGx{d}u-j4Re`zhZX-h z*qrtL>W}F2VeEYQC-1If_vyC#I-x3w1ROS-Lqt|7KdMkfYqJ6XlnZCi-Y>7z=LxFZkL?@P z>`3Qaf^VdwYo}fq`73elHi3&`tv&S`PW&1&0RjnS-K?D*9X&{W+@oTwhOM~`pV;D& zVXl9C=@N@P5`#N9erv`0y6Oli&R{I=oU*K-2&}wIdA4a=`Ss+r?SSw3&AQHxIso0* z7BEr>x~@x9)6d5&#A;Ur?mv38fM4nS)9DG_63e!QhKxz&?u9(UI7FtREmp@dF8&D(pE+cdVGg6EC`^-_ssi~Z<;#_hUf&_ zytZ#c03v0sUgPta+Cxej4r!0L)buJsSXf!FPmXjlj-FT5etCQ4)UeZB91w5g01#YT z@89<}c1@G`4%D1pRGIqN*Vor5Z%onQ%Zn9VI;eo~x~0oiJG_}>FO;@XWmdTeDnQIx zxPcHL$Y}4OeYT5OvIz-g)jZ}@Nx!%$<|4=9siBj3IbB~>YqN-DeC?hdNy)QkNsXYLUohUNPqJaaHQ;qpIJ@bzIZh|Eq=^0a z@(T1zV)+p8^r`Mq5MD8mZX{`r>v}OSyUHJ*?2Bk zv_X4ESXDHNu?qVF<=#&INtr$v4tigx_8zwY6V*HTmd?xcrsRekO^%;2{6xg*xo_VV~Q1IHc{# z@mmKReaYsWnQoi247k0Gwkj(=4GtQbYTP*QDF&d+IaDev-Q+i%+?_e(rd7H4&|992 zojyStYzGe1P+6tQCU7rtvR^N}ZI(4MN=!FsxH@rBiI8yveHnVS>~3Lo+ya6uJ{7#m zyjTDJEoZ7>bd-1V8V0wI`E#>&pv2UX%hUQng-(3Ro*PECda?>=v(Wz3`$wEA-hO`9 z)vK-|_pP#Zb#q$2X7iKY_fb;pd3LgwF8LRwT&?q1+*IIHP?RTDOrgiz_%~)4Ct5j=SCoxP!<$S1Hy%FWzr+42RmQ*odzS$w8LP`~qgo|BZMBL3u8($Wi8u3U-H)U>wFG|KMX z>j3JES}~V+^y}Xn3BJ#^8cqa`C&f6*Vkwu5gDc4376vU{u|lQUS9H9E`CmK!1sIz1 z=g(V?ar&t|D5+RK17c7Z1>EEjAnf;1G8rO*<{yVB_^~YN(j)Yd+6>d%6#$5epxwZ{ z?l_1dS8%+@sxe1A#9X(r8|@A}hSC+|edgdFL^GH@ADr*n=yPBe;4 z7J}@#bHCgSA1zu>N>m#iU;IyHpLE~@^I5iV%fUtp6?APlIA;v2G_rjit`A;Jj8c7P z?Y0AoSCu&yvuEw`2w&xNC=p1rNL5^1oFQk4&`I%CLIe9>Q>3NSGYJ%BXSzTl3o^01 zP0GEzVn52Gqh=D0t<ipx!kAVt7wX;Sf>1XI? zT+`<&OH%N`L*z^eQP&^fdc$6^vcYQ`b)a&6L%=q|skI_ioC^}Jy*d#aCgItK;TQsdq`I%@%~D;yOYt=+0((L3fo9?! z&0|cAjOoc5X1ylhca~nAQ;{BJK%lUOc*a{x7uQXh-aq_AMn)e=CfUaw2n`J-gbp%o zEVG!ccf&qGK_zVLDum3M(Lcql?uYCuM`L-(L*;+*9?-3BF`*d-? zOoK(IT-jR6c=z2vzxDC4%@0@z;=+yy8OrKrjtlvSddL=kl`n8~AO(c0o~Z57KvlKN}FT$NKgp?X?OnExRF(1 zfNgwWz+eWsrE41EgtuQt777#2A&wD`)zgqCT82^-dSBXq@RB(a13Fdxn|+apnTT85 z=*fz*6^dF$XQ-E+>xQmB`a>3bBU`}W!xL0>gm6QO^%XAwMhE{ZBVsi~l6%)PwKzID zdYYJ*sY!y8>eAeO0f zmUXMEJRlAMh}7CzJbr}HCVV4m?ehi(@mXyECr_{qbIvu>2I<39)O2}vR=@{mshdFB z{QdW-S^)j#lojilpR?^vN%t~Ba74z|Zr!DYBw|@?bwxve*hG-A>kdl{?vu+wM||WY zJ!eTw^s|va3B#^$pspT_Nzt-MjQwN7hEU+->&P?Ia9le`@qmE(Z}o*T6Y4>qJqAaqf~Aw%J=PBUD)lx}eSNaYmfTG;x_ z6Y;8dE~Dzvh{g;PtX4bOE3pm^r2jSUeX7#MH1T4}%idwOEPpoe`9t)o2&fje2N>D2 zUUCZvq@QTAPf|{O{i$`kQx`ydf=WgmfNdh#!Q)NFf*GLW^j_Rtrm%TlYoj;&Xk~0? z)t8cStQaADw~`5}@y_mUjT#YAt$j}r&@M5lje~9xa+nBOZRMYmk-Tr$t{0sWifWnD z7sN;Y1Ur~=Sh{Sx*-5H=*y*(-e_>#&Q4SfT&<&Ag_wo&7wWmbR&16Nv>t<)?#mWji zU-8LcT*Gm)=+ng{)SHhQQ;IGhj0P%IEUt;UP~62O2+kVmCFsCwMh1pP`eB!4y8wRj z)IP^HKG=0~6Ft3F{v1T0z~G)3q`67qefx^9P8+HqYV_3{F1hisGNwb1O$8CKs4|!0 zwHNif&xLDd0!NP?%>zVbm@`U;@Zz$8Ru0M@Vq{SM0kOu?JtsLN;i@~;5Uivny-P=T zw>~(Ru4xpd7duWi`=J&iNPct%p<%3GZl*G3n9c9A*Ff8q-kjllGd(7)-m0*>MMiRr zUEd9|4tjR1oGb+CqW2;BO4l@yQA5z&Qo2lLqCypNg4!t6{W0eyRT(IB;iC|U;u<8^ z(HZPTz4Py`8A^muC2srhN9KkFOFAmcP$*T z_ejO)P}>esYgyF&W_==DXXDi}MJ^~4Av9`VU%p`DNhLrb^d~Sk z0+f}n2-m)S5*iL;z>eDG{t=e+HTC^M{)SlAqvq@Va|Ls@ppAp_UG4yl$OF__^5Gs~sjJJIQ@K`PV4cTAW&k|yHQeVZMK9@m&-l~g|2RS^^iZcmZzjvK%2H(j@Btiq;G zkCUE>j+XJd-n)0GO#^fo$ahPMeLBm&eJTg!Ib@(37cI`?NX&d%lQ;zEbCN(G6BX9g z97IIPo1N+xwi(fM>B>yj-N(%>L#>HCx6N%&yd^$l8~BUbJagMabFQcw0oO$!)%i}DT?u`O*(n)NQlITfDQ3PvIDuKvFYg~8x+Yof z1Zof|QicG5Nt6!dlAL$4AAnf}H>my90JMfJ;0dZz{jc^29C~f}c_e2z3YdO66;NE2 z6Gw=p-Hc`^{rg+(hHDY}3_L^}e`J!?k0jR*UgCLUlI8~4^XDI)_GC^V@~V8n_wUx{ zLXN0`+X|WXvy*(;6sK=e?ep7W%xip z9Vle>4dGD+`1ldaU|WF(rnQ=A^T6F~EG(zspm_gda@6P1Bke=uSmcTsMlA!jo+5r? zcIjkyA1OGpe?JA0B!kFMYG&p9*^shGi5(w*0D$8<*vZC}Yj>gh4*1Vup5CLVC>Nn~ z05Ntc`SPDAKEtpeIKWg8T#~eS^_n#`pc|w1h1gf~@5sE3QbDgSUfC9uOnIfc!aarTz{81!E;C zTySs*JS(`n9CPIUQ6?~xsls!Ge=ey_xh66AA>|~9wE}3~Nr?K7J*yxSKXCl-dUV~E zT|>;Gp->YQ6c_bCO=BOfc`7r_n12)+Os_B$+`MM%!OPA^rU!W4x;qDZco7yHzJI(+ zzx9F~%2PNw1d%sSAc*oFZF{EWh5#eiwHa~&Df{d8c#2+~uxq~O>FG%xRE!j`Se04U z9EgQrxbsk`Pv7YNGV(0{ycT&v{kj8$-0ISSA_Pv{F(&`Tix-ztu8POhS|oleT(bTR zvh#}-JNWkbSHFKh);2d=-}jiTKNv!~mOd8wfPx)lr-o4h(WwwgZ?K-Cg(c@v$bc4? zvdv?Ra=)H7o88IJuMlECLh+l~gT(^~5QC*@S#YhZ%#UZQ`_x3ioT1}5K$0@23E3^r zLDhghQfT+e^j*1eZ#1Z=?B2)#NX@#A`8t)%fT2;%U(;`#MATi)a6lHT>wu*p(UuUD zWnchtWgfQyMW~v4R+%eSerEP5m)ZI(yKHU0T2qi)V0**f(JIRk%M-Sl)ihcZ- zE$U!)rGv(9h7qMIX7*!sI0YfAO5)O$F7AKN-N?iQ<)VE#|*oztH z=y|ugV?942yOi8CJQ-|OXWbY$$%YwhOe@XREA<$303Xt;gQ~GjKn|n@$Qc2X5G@jC^vkFp@?jIR7EVt!S8)ivk|J5!)GPEesJHOFss(PG4O@02HgPFF!!M6S@X88WH#qz}R4c z0w!0$b+~(nn_!wFf+RiaMVN=+ig3AeC-%y#J46*knY_!`cz9K1z{7{3*aYf85+!fm z$jZxm+l~*!kl9S!)5QL2c4j)w^z&&}78W+J79@lbrNpGY@CM^G1(%mEPe6E=y}NEZ zq1?fM5EB(h7?NdRUQ5pUvX)ME?0+@K22T4t{*PTY`uG29u%+!by8kU!2WCB%rx5fm+e6}1RBBu2nj6PuhY3-22Q@>}L`F~y+{;n}<< zah7fVKIi}I%k;F45S8Jjk5j)pD-f1w7nTlmH-GJlVG^5hr_31Nq+=kO5{(<_!7 z>Uj>hq5v-xp!WxUo!44^-WW^D#@|a!zqAvopdwTV%JLO1IwE&XjSt3Q1HOR{P52)I zP*CXrK&lg)CEx<_+1~#mHHoy&f9>^}xIsik1abMxm!AOJ$j2#i5=X>=17{JX$mIOfNUg>oq#$NP8iLx?Etv{wVxlR7eAJ~|KLGc5-`99z!WH9 zG4SC$VOu~%UY}r6S63%ljTllbhMKc#q($H+t--ZPcqfuO(pL>c_1*!`tG+OeV6w8m zZHI4zD`>~RCP&qgZSl=>87-Cpv?@^tm!q;Zm>{bdo=Q) zU!-v7$&;ONhEB7SQ3UE zJ=2oHbSv784^*KBfUiq(aI<+#a@xjsfY{;0m853}<#i0k-K^aQ<}V|^Yvcs&ZxRXy zQjCQzu5OO@4i0LHbRFwl#M3Pomz0!b_oU3f_ls(W3+-Rvau+7+vUnMr)Z@3A$g{rTXk zgwc-7DY7%QN;keQS=BYjdH&KSzaEiXdgHcjq%hnsB%};s{L(Ycdx*mytF5-KS?*h5 zydIv4I71;JA%WuMOP3xNVVb{0=AwV!o$1uK>#!bZfJgTQd6PfY#{nD@ANwkH)2>~+ z6q`y^zKTDiNfC7I)u;X$1=(JhT1h>*UFgOsFlmsb>rny53dc)+eq(w$ebry{-`1Y> zuLoo&+yfD1kasB5Ba66_^w{!QQ7fDz>IqrhEJ`KQhkUySZs7&t#MMv1mS> z%HUdn?ldr)lxmnhh!;~)4>DS)T$>Vueq1_86doQP0}rhx9ojcLORfL8sCZ?U-j;kA z7#NR*q=&kQ8V+xWE?gD(;@aOY{~TqVgNhb(CdLpsB34-v zxho&!uP^DYr%#{OWSU+RmyqCqK>6_D zL%;(35%A(4&*gy3HDvvBinjcmqQDoV0Y+i_(q<)8*SUFlgG0nZ0jT)qlQIE>K*+1| zCLrI#bD-|>I`aew3P5&LsgdN4_U$7j0AK|6HkgR>y1LP*A!ERj!ml#?v~3LAHV#~) zz_dYAI>gt3$4}wR14dd3KfFX_QK%}+$0<3+P~IiQ`kS*-v57gpNjo{sts1kU zH#RnQflWbFRCVoEK9k;@RjXE&3|`|Kkzt@+sa@{m&qcb%-t6`{gsDqF;U~>-&7CCq7j1RX(fQoVr8N%}Zu#EzTjolt80|WAzWWv?usvXpvzwf}rku26 zsH-c+hcqhThke8R0yB|eTCqHHrJbT#Ln?cj0+o04xxvK#8ScHxb}|wB1z4M!6$N)6 zGwiz^BlSuzJJDgg!s?NPXNPIqo=$Y_KdW?U=E|^?A5I>ivV4BO7jYq`!B&6l3TK+VMbUAsj>Ztp5)QWN>Y$WMX><33x3r!Rv`pNGhL4xvYng3dQ{L35W~_QvzI$%oneVg@(|Nlrj3}U5^X-BH zrz{5^US2ikyKg(JpGJj$vQgjGa>wWAPSM)=bDu9KZ5M?$NYZ!0P+ZJ0D>hQr_g zAy&j-Xf_)-e{J7Y3)AxCx8F=Q4kZtc#?%<(Wq*1i)#cb3bxYW%Z%UJ%_Fm_xT?!NHMIoYW;cdbXfnHTLv=)*oBLMZ>7^Lw)wOX9JGr?}M&!AkdcwGYRGA-S!veLtnihYHUM-oC;mw~bb^0u-Kv)=24zQUc&IwrzqE@hW|ZD% zbpE~VTfs0)b%!`@LSBR$W|e1UY3X~A-#(8Yvw6MkxcQsd#-HmtjC4~u&ndOKax3k% zOVrnpNSd)fgK6TjyPH%c71mg9+VF#R1BLv^O=FGmt3tJ1{t;Mub#VHI zeun>9+t9N(GAcnq5R;;Bu2W;XHi=F=H&&Mns~&&hc>3J70sl zaAA{vD81@%ir%{?e-d{u9LR*9Y$V+QLc~-pHuLR0H5`m^Ke1zHC^nhO{kdwOKl4NG!I(y$cHc^zfe_Y>tD?XGWOqJNAtui*rmUjBcPcH_?PsjgJ;DQP?xvw2}H&>B?d zxOeY<+I$3UN?h(ABe}-RuXT?o1uOjR9u#zRAMJ)r5(HbVPtsIX?)>jh3-%U+9&1S=uK_cB#hjZ818k%p*6UcVIIm0znQ+WAkn9Vw4 z?my-;wl($D-Sz$8*{R`}P1SFrMdE}2B5R`75&Z_7{9STd2@mf6%uTe*?}gh@+fGta zGU9Kf2lETQQmGVzd&-TBp?}<3Hy-%<9Rh|WY7ypfhQ3{vLK%wdC2g!39 z18NIFpe90!5jZOj9?ggjzcOii1R_m|$Q6EIy9$M8hJ z;$_XGtKQbye2?8X?K${Q&yTn&I#3R3=;-LgL5p;Cb4%?p!xrbfn0$<^rDs~D#LD3| z4!LBsN{OiUlljy^&^u-aZ6~zjevQorGW2`VDx+<8wpEe} z3{=8#fSQ!yoBN?|3;AMaDM>xO*vY9n6=MZ-%_L^`v7htf5htn&+5UuGYXpf)lz-?O zJj0cK}ABI)X!YvtgGX>{&Pt<9WSSfp+iJlWd5RI zlFa>t`geQ!gh~v&5D(5!32h0FJWqFW+O)g*Y)>+5fO5E8`YM}+AOsgyyLO4nAnY{i zl!x=zDpdedE~^JYk=~G@X{V|hIGGK?(DnX(Zx+(W(#3?*_Mk=v7Uzi++pQ`hix$ww zO`wnR+i*}C$Vf`YhR+v1g`EW5cj==$Db@I`C3n)Cj35e1_}$g0L;^wOs}gh#rjV|F z*pK87tJ|qs2f{}9=1p?r>7TipHjWo^PgMBV)YO!$oJZQK>B3e+L@T3IgB~#$Z^^UE zK>NTk5HD%E99%tUYO6;Kyl&xv-RhTw28lkml?~Sd#q8?z7VzjElKRe~uqd}ocUVq) z{=_cKS8I_Dh+AA=)o%uD30hG1QmR2V$tPUmcN-_tOcJCBl`bau?NK8?gctaM_^DHe znF>UtNFVaA8^5Nog07Ho8s+1}eAd;4(j|$yd!AqP4FfxVi8O7ZQ-(Aa!^c7#3g%Z) zK#~Sc*i9+bAgk5&Jh;@dqv7Xl%rp%nv2^Vgo>LGA1u}HBwfm544-K1)B8ub;jNaca zq6lS79wiq*GFa$1(7;;S+U%^X;;8P>5GRj41tHHnD9CW48Oaq*ai|03p!d>m-=2V^ z3!_kpE+6>=J1>eaLPOX7S~FJ;`dCE^I!mV6j_Khx{?-l&4-c2)=%t}wLG$SG+ zdP#nQ_^^kvRlwl!BW?{Yu)&g3=IQhd4D1jYG3>7H?h5Fv#3l!u1O1zG$BrF@L58vm z`$y=7Rc8NL`^o02H>UdzpN(X>THanoZW!1`+DZ-UR zf+6HTcqg400hEix$_&kqed2Ju)b^igleUh^5SWgl$GlcM1g0xB(;|AGjkHpbE^`2? zFws9r|28r?$;hCt`#nU4WYO_7=pP39llT${he3Ks!PcOGs)`tTunsc6Z?s;%0ExX@ zLl}?wJku%d$97(*I@@yzV}`aYE8TSv*4(iMp!{3N$^YUFB&-!I=;#hKcnzRoFaGvV zQbN}6KZA5;e0~yDga7mLn&l@I?#}b}@LCPkNd zJg;i|pUYvfl?;CFEIO)2yQZ_$Le08|py*}|Bsl_ai4z3NgZKvl%UCFIFGawzp;s>o zTtdWXo8ZsIo>El6uqgC6W*(TO{?B}^HNOIYFnYqwffK+k#DTqqYJnX|n&og;1arbP z>noIxKFemS&F(+m&sR5@HNP^5@%`lMRZqqJ(b9^gS7K0EQIW16Od7O6d>%;U6cUZSwnf zHsVbiW!byeKWXjqTf`@T<**-~t?MbmC#9vOU5+*tfvS4e_{hgcVaOWHp3a^5+K*1~ z*95P7aOWJX2USGJ*D8IqUqC<+k`FvyjydzbnxB8?Z3{K|QW4xJ{0q8hF<4hQ`eu*k;a&bxccJ~(reho4ef_hb<^7dX zf}Bm14{ctfu>B?rILvo?0b&y2rsMNzuP|Z(eG!DAP^h>;7Q|I~3N)>ekxwC9wB=P8 zapxyQ+hBifX{>X??kDhY23!-GnD^xULqj6VIJ3(l!p9k6g@NT>-@UUT(YqL zWS1_L|1&xHV4PtmtDOP;@AHpXX!h%lckfhHr0tlUZ^vU3K;j2nGyMO2+mlrv*a?aJ zOh^Dv*6?xun+n(dT5G#^K3RnxofbV61iJDR5U3UFeonoS^TI>TKl#hZ+|;%`d-en| z&G(biH81)nG`-Y<*-zf$H>$yo6^|Z1ykd5MBY*zOxXvxqOrSra7pMOr0J#rOzHFJ$ z|C`?>Priso0;2&zpt!M7{I~h51iLNV4AF9;&tk190H`aV+ZMtgiDwWRQA-nMQBu;C zq@{V$4;*KcSjPWIFM{}c2#5q(QQsmMdP2bP%HMuV7TG)h&DQJwH7x*|NMMtMMw7tJw9<#9El#JPq)nuCUe z?n@3%&T}a#DWrTxKmr{M0<-h;=P>ApweiXqz-oJkgoK>@3Sgcx$Vx=lKUb z*114i*RU!qMY#P2;Ms?gw+?n2@UvuqTW*wc+J1HyJTL_SRimm~9jm|rB0y36jZq~e z@U(jM)Ep;}|0VXVDAHGzn|&NQ>Feh0?R{e!Jqk)ESZH5m{RkTnpidfyptuugjj9xQ zr4I!h@$#0uw-Cyq#pKmmEe1MeV)z0lElYh4 z1B{Z(a``i$-m)CPE3ohSgZ8-18o*efk|SZD3oTo`Q=i%K`xS~DP!j{U`atX#=T!@A zpLa3iB6741V8Ow*33zLO2=Z|pLmcR9IabR71i&uH?6B26B&Vfsb zAYV!q?fcqW$!L(~HYy8{3L$&QsKNZl9q^fS7a(v`s-bYIGVi;*eqnHJewbbc)}Wd@ za&)6bxDVKhCxaZ4!6X8$!~u3BVAMczP7b^TIedywL^}QuK-m@mx1D*4Lz6{du z!X@?VTVVAjg$*?vJkkkZbIysw5eRC;%IlBbU>vhT4|9*HO}w+%0*dUcf<=jFTI8G? zYB(Of-<2cr9lSo#u*vR+v{pUI8SDhaPdGVDR5i+pSU4Fe-Bug>V2f37S%2{`HV&%3 zeYAwyK1cXgs9DsC;Xl+ejMPA8oTGK05C#$EV0OY|!4*&k&)5|sJG5?Y!eZ?;U840X zOuV3h;ol@kFF29*V6JAr~VPMb-sJaVm@ofZ}(FJ_vA4rP8` z7tj_}%A-}XDGzQp0nfEd*nd&WqK^A^QgAQ=EhrP~HK>N>SY+b zSu3R~9BEHeSdC2<4}WbOn%gYLj8P7eL!(wXw=4b}!oU|etmv>5>@*nua3DvlhO=&r z+;uSiAgi!=y$wel2#=#WChi^7{m$WGjoMvH7zUJ6dOm-?uyUR&Xjdk^Sr{wQgG&x) zAudYN9)qm;sGy(#_CliRo_l=oIJ!{1ed}3MxYqgp{&RimWU_cH;qZK`c~y(CSslmy zoH%*Xa9la%ste+>B4xf;puOSE$``-L{a0Ek1NSu7iyjAt&2j)0z7Wp2)^^G!4wsWtHdz9`8&yNO3E4$y?5y>i{#kj(gYUd?m^HJ;A3T_kn=*`Ut7ZN`ZG#z!G6|-iLk352Xo0=|3Cs~rI3SBU_=aL zU^F3)*^5X7X7(aqwtnE$JiAfEVv9$YIlV$S@$9cpW z@fT=`Wj};y(9ZPC8mK)+$FSpYgYE3&@vCU5ulAS1h9Tg-1GC%*9goSn|EJ-eTL zN`1y{S=vSYj2bTRkBuhi3@n-J)~%fULfe(9>)x#XD7T;#twGw_v}MbFim=^yJp8d7 zqO+tuuOEe{2*-i;rytA`MScS6!%2s^<#`G&Z0K-lm;SZa5ak=w4LN#$Ft!u2{N?AP zGv6+ZTEJ8jM8;p|JF#Z;z%iKp7g!MEKOlflwvoVpEFe0>!XP3v)3!8*%XaSRfrMg69}UjrB{~@EMPY>xGWUF zwUFH03a9(zAUx1@gam&M&#c?uGAP-}5g%Y1W5DbYKNt%bjI)SK(bNxrQjyV12EC9U z7edl3`g(|&;_)y}pfH6BBt`HVPOjmc(gs&XPUgcoCXr->ktnF7Zw(Ho6rNYhOBOiR zHOsxkq{5q<@!kZ4|3|j!C<@70Y+wr{&zzAxefo}gPL*y=i4O%~qRz=W{IMc`iwqm)Q z=o+YVPCvfhMSqYQbJSZ+Ih1~IE)KMo7r_~VO2*7o(}Zzhnl8up(FJ}Cy~`|z2jxc> z2|CIBLiWKqf#V3u4&lZ3pNiZppf~35r32FS!^BmE+fIXQw8|5ts|ZIm*G6n1w5%w? zeNlUa)u~{9A2uWS5vGGQCGf#>?Us&nPH>u=k5D4gH93z(^%5N^NZ>Wr$KebFS;BLo z82S~eGCZVK=S)mQ7VfkCYw@4-Igm1N*v@{^ZP^KLXA|5Uo>A^Wt{xpU-{!EZQ%%d@lIj;AEHZt;c3TxPkhyFDM`5_u1%ZhC~{cbxcolK6wY!%J8taZ z-PW!^9UlxJuE+h9Sxk&koMYA9)Q9<5la|`vTXhnI))ZQByy2?;*#aWI*kb%+mRrmHIn-l(F9JznYVYQY7(LCRam zG~UD$4=-hbYgmfr;C&-tasbYp?kg@II$1?UbDNJKj*U!NjN7c6FB1a)G9NHYRD(&1v*(Tix5h<69c zRO9TEq}W#`exF7MSfT4By`=2NaG+=*N5EmNA*ComD^UAArhDb8Rn&R$H+~$(%(JYg zX%v979Wl?4FcBnVcAA{mNqB`=_(tL4GNE+4vyOf>G@h_(TjvP8XoFk zCsG&a6V{-)nU%jxJI2z_4`){={trHKI6cvE7=82BEh%^Y&1-0IKaI(GFV(7nKPFxPHE1}9x8FiWGo1NMRo zVBg9QqoxT*8S2}hh|?tk-Ub79OdpxIWcy~4aE%lBY|&v?1Bf67A3q|YHzC?;Ll-gr z5=4M1NgsjOw|6J0;dloG5k*pW1v|~!pn5w?PCE%%F^Buk=BbfGG)G+3@!zPnewAupI= zk%l;FF=+=2Z9+P6FhNj%))?&V-QoIxwzMkR1?VqWJ+9DkITtB6r!1epP^}=bDSFGpZtTT1Y44>qc>(^%p35bccyq#xm>6E^M zZmY#{RpiujLW2?V7Zeqx+wB(0L2|xx&+PQD5**0n3`gje@p`vT`z#RmwsEK8qmyV#ye^q;Xta#+GC}VW7WaJ z1Tnki3_SCOgpMPQnwJdaH2y1d>r?466# zk;#EEtfnvW7F4R2s7V1{alh@c>%z|JR0LMx`Z#PHnK>R&Dh|hWl2C(HM^pwxT3sY& zGTs2H*ef*ljqDSYgY=H8y4koEBae{Qty)Js=(EkA;ls4LqF<|c=FA&$PnwL{exnNK zyP<6-I`$RYdjfvX)8WL2DAIF-P9EAh%^Nn5oP{#uIubo`xnUEc@CIB+#HqU5kJx86 zhlTzFk#8faWu<}{XNOutC8xLPqI(AidD{6opw}a&nAwiD!1hW9Mx$CT!BGyN0<&3M zonemIP5F89Wx}{VI&e@)LeVj1Hj8*ODToMep#c1R(qDRl_YCsrZHoQ zxm1=yDD8z5CF@K=8<8Y?DwQ=tgzxjb>gv9KzkhzeKfd?lethoF%!9hF_xtram*Y6j z<5c4@+>IUUD9+tIx~guRSTbEcAk2*_RJ2oTzMTC=a}JQ~w4H{TAoosJx$41&X42xE zO8@ZjV+DtP=khZ?B^>YN$}It5kC)9)TG5~EW$+dJK9AL(waxrC4m7EiQIe+O4jpd%P33AkDH8bvl5TOsVh_A zYT$-sxdD7_c$!V?v3iC`)vi3-FQf)SkY#q`pP#cmc5&aFNw4l>YR!L45L!&4+02Ku zks)8NiD4jof?Nc*84Ye_ZeDY+O}BtdUGG)SU$q`xC!6?l+`+FTkxFqd6VqZ-9re#- zB)ruhxPd^-aaZ5pxM9t^_urH8gN#Ko)&;)66+K#Tg123hoMHCm&h>(xe{vOjd98t+ ztRvdb&Bg(!0?P}L=Fwf)K756saN^c*lbs&)-H3p0+uWF(OQ$Sd*!&(-45?Yr-h%#J1*HqKVKW?7peE=9DG?hl!dI^p2`fu`q`cTCr^~9*tuAT2}Km2cd}9LN}WO?_YG4B4%5s#M=b8wP6=c z3e(*FdDdKS$7PfSeQ$SYDR1^%@gs3}d10=DqaXt&VvjF_rMgPN^#N3jASh*4yzz$+ADl6>M;LAbdvY-A#Z63Oahg8$wQKi=tF|Ljw~V&8`4 z{aQ50{E-j0)Ov0CFU;%i3e+6(?FW7)@{}@URJDGC6<&=Ua*+@V9#1mU%N{Ps}LEkVA-KbZ5R%oX@e*=>^Tv5@dN=WqIsO-hdX>dBOchH2yF zJ8M}7fZHzMTdG2l9Lcnx{KCb_#cf`9Jp#{GhV$-KQF04b10X96qC!b=AwyU+jm0-k zqT`jt8)H>}K0CZHJjt?>FopdYEcWIh`P zU39C0&Zew*76n-)#$Axy;;jei8cu$9*zy8HY>=wwH+DVHOit-Mq*a&M`|4FdlXKtp zu$yKz1*y1^_%!+@p*azBso&6{SVq7$&hc==qWn0ZzrY|1 zUOC3_%wW~+%rh=u?mNA^o}T@kW~lB>Ol}=q4+JUnw2b77wJC#$D};!OLWvOiI{L2s zGlHpcDygn^#?J3RiGzn$Bsofyh1Ay2_@XRC-2uZVBV}lU#i$I8>hokINWhs0CBjGt zy#@`kXLwBHq}S%le-ZysW%N$!e0F={{Q?y-$yGn}Q9pPcSX=oBw9r#sI=kLK+8KyX zW*^aIsO=XfTR#3qvZnyC+ykJtv4i%{E7(J{WuQ&(Jv=uJ{THAgPAE7N?4?$y`591Nur?64v;k-Tp%^mgmsLKcr1Gi z9q~0q(-h>~PoGbRHYEumf)v4(pvoB;XlGT3V5m?;qDH=vIeA{Zhsi~N4EK+yd`)U< zX&!UCWzG@^!jJ{g7U2uAjc>;7)AoziF%bjJ%vfPC>3WWfA}TX!hfbDZy>gR82wSN}pQ# zrrkgP{8M$?HNFg$so4L>g^=M9J`rP<@`)%yjNBiOp)>lY?8U}X4f-mjW$W(Vv?k*c zTX&yG#{=SQwJ^Pdk{8l{dUu85r2dsJQ|Z3A<7ve&mK}cbe|*4DmGk?B_AE2}I#Yka-C zOl{SmHD0By@9k|@KRq~FsIOOjZDdXp$;7nkyo7m^URRN0GYlGvjbnAnO5b({5SY>U z*-hNrpQf8k;ZYvEQ>9XlalQ3ct*~F3OjbQA;uGmstQ}5+-d19hpgU0@Gbg)g zDyfxk8Wb^6-hcWFT)Yp;RuC~4R>%;OfoIb-khi!{GlGhRzIL}JPt%if)NjWhK1|#L z;T;{0Q#R+U5d}=foPKu4JG1r%#yN2Ly8MtwzAhFv8l*xVyr{!R6;)?#p=w)v&E@BI z15gzxOSOL&|3m}`4g%ot8zZH-P|x7PwMJ5r2mEM01L)(AtQEhV|A`ULv72gD-S4Cd z2CkjhBRP7Tf?zRbs0jb4Q z{-4+95;Vy91yKN$%dLRS)KwB_+KjXSpSj7k5d))zoQbfDO+hjRPhHwzy8d||w-PV1onkvR-LNHb*Q~Q8*camYY zC`@hs>05XV$V?nla;qO5Id{&2ixh<^5U)-URbJxo7yA18mL&O!`-g}=P^QtSUiqdX zq6WPaqb?GHzv;d@V&^-6I9WBcmT;&FUH_?Kdh5^Mr-JW)0x?mAGXHS9>=h~}^(;%B zQb7bI3qY16^)nSQ<>4!lehaKcAy9$n*z=d)_w_L|b3m*5Z4M@%%hNl0%_q*5%mdbw zIYCsw;)SkWDv{*28?@;0%a1Yt7p*h;`$0WsE&T33|5L|5p}?*!iW#trdAaie-p4l5 zP=Pi)7(zGcmc*H;7e2*vN?oaVdAS86TQ|tsYc`e3YK<4ae#4%lxd~N7%EG6!E8ZXb zQd2s#q+tkEi7UQc<=l6AO9k>{6(Z4A*aq?N(3z zNZIpTY*VP%0~9S#qcKi3()VI#vlZh}#`-Hnkw{!zy5i5Jvqk<~_#67Z?zJ$!OyWXxRHF#7(lXqA0$?0t!gyJsg0AYs zg`#&)3ubp(&lpMD5np+&!1FQaSwRS>2E?NKZH}aQrL^S@A)Cwca%7CBq=I{9)_v)~xvpKh)!+ETL}p^5 za6L>kIKI F?@5kOp|3nGe~MkkTP46cLDw{&1az`BTWjheJI3wS6c3Vm5x zB<6x#GePMoeRD}{iPRwC`GZ~{1^ZngRB`JmnB{i)*TBlpo#?j^^u-4U2fJ&}G~az~ z{_&J2+cZ>sC#%LcLVjt4W#5HIb{uB!{9x+lE7-~1@uu655jX~^95`OKWUG#w*pxOL z7sF zo{W}7S0Xenm^UoXboE=ewZgoU%mUvSrk^xLUdV164_J z@!V`}r*AiHd#SQ6nP+&fhpd7#-qskVcY0*sY`PISPsEb+`Z>}=hCxfVCMPEY5A2o! zZ7?i22v>{^4XZ0(l$K6?Zdz(#C*d^VG^eFLf%swFxDd(S2NEF?&L|kOD59gGcRs_W zhS8~4Aq!@lSR#Ri(^t$ppC+EUU~YC@~aGK!xT{G>H;og1RuE^XyNfn-Dbg*nix0jWD8mpa8#W2egfmJ zc(&l|?YG&7Zyy#mZYLI~+LjoNx;9+VC}~K&a0vRJ6*_iMC#e)S09_~hY5TB8>=0qX znR+DbworF4Q?4`x!c1As1_-dB(gQcG!jS5gks+>Y#9)EjMC@RNg0f3BC+erelpnMDmfwTdv zJ6ZkWG1a1^$ISg73+C@wDxPQ3Bqq5HFFm;*bWm5D44?UNrxeF(p(u zx65_@N74{Nx{l^^s$}n@e?cj?_~*7-0uUk42RqG$T34fXba`oQr{7?zetP)Dw=B1O z0>U1+{LHNg=DJpF%>Y~hSQPQnhA-*&){Y)+lenbwS-sP3ziK>e;F$0JHp~rEj<&G2 z{7!ph6N70-x@2uS81dDnCJh35{u=VrgcKj2H(h_~ec+Eb+l_a4e|eBqb7{v!7rp+M z-z5aP?%jR*ooU<3_c2EAQgZI?9VB$KZB~=?K&yn?TY7eP1P2s0F6;AnPM#IMddY~!Hf^75Wu zUS2fO5o@X|W$4;`@Zin<+fvj;ct}KW%GXb*#R4mcFfKphZj;Y(np<7N4H0*9Roqz{ zY5QT6h5_SWM-gL8k}WwPl&B-<6$x_AjboNw3VGE5vkEX#22>{Q%C47TvZc1%V?bF8o1c zlp?%P<^pzVCuVry`U|qwO*YDi$LprVuk}6h+w(Fg7iG&yK}qArjh~Qc^H#q&y|dKN zh*Tyi#!<|^b^g~O*?%nrJ7Ggzlvc3{rQYo4qKYTbjK6URs?7NaQ0P; zBQ@D~47YHA?BbRjwk#t5nihS;>Do4f6KQuY6M%K@)A|!R)k)}3lJU;vr0{%TmvH`qm1MiUU!y_ z-uLcUiFeVLpfK#qa${EXF*(CdMDz554z(j6@}75(K7W5cr$~LBxkGyQU3AKaEp2?m zS6#J_EWR?m@Ks(Ja}-9X{*}$=cHkRp=faDy6zL*N$>oDotmxh=8*48c<-GUdEw;I zz!6RnZ$b>!H}pu;@Wl-U5hz{-W>3kzWBJ;OJEVMvi}cL*bn@B9kmK&eu8`wH>bQ;q zFQDi{X3F->n>R;UN6t!g-0gXQ91C$0akT=^jJe5&4XsyM*51Wwzi}5uH!r}fLbmNN zwImG-5~5fnfMZoanc=LF)lUZui9g|Z|JF$>Yo6h8ouLposRl~|i@Wo#QP6Eyc&}Xf zcuz;&N$ZxZR;o^&I@M97e*6#K!>#J6T|mTF8uUSWECg+TQOD~k0W)_B=&&fZz^&G7 zQNo$Ixp_=r1+sZnP*9MfjM%jE@|>P8Z)0UTp9`%DrG+Y8ojQ2zx3yY57Tjrvyh#4u=K@`GC(zRJC`vkofM-znRGE$0%Qi(m65#r-G4ujjsf`|MV1 zI&%$V;1B?ldX>=3Do1{@zk=8_j-JSh$=fp(fP>-SGWL7^pf6mQ=eX3Z?xfy)@MeOA z&}HNgfija%*4*zLPE!0pFGdJ?sgo})Jz{iKD0U;$2Io`P#ekk6c*O zxQ}#ez4MbIw57{*|HMy;`}RO%QR07*m|aF=4+cw)2B!)uelj208u>+Cw@%J`s|=K5;46{_5H zaiJ?Kn^0e<5qhuZIxQ?N**KqaPzM05bj+*XAn#MxLU8Bc3IRpt0^zaGOZ@R!88Noe zf80mb{*LXPS2R49z=(lB$6Sp7Sa`lYHbRSRVx}EEdFa3{60LxP86MemW;Q6(&2-gm zsn*TqtY?*qg;VG5n1!ri6`MmhRI?^H`g1T+&dyRs;pkC>2daOY@gLp%wGyo@LI{);pWq8|&Wr zflG37@t6IF06>P`Ex{blK5I%C7;r6ZCPM>~%O({EgZgr_eEsq4k&5PxjfHqeG zd-obRa3F2K6R4mpV=ux;Uwi^u!fsWkOs zj`oD7*<{a>e!C^AD0BPgm>)v2e!OQ%UP2!>TM!ZZkiNSSiH5GM8$=lajhn@uN;l&n|KueT>>=x+El@ z9PV$;l=?L;um7{vkbc}rqPHthNx?Gj zAECb9AxDThQv2a`QaorLi$eDFn>#k}1>lc0=nkdW1WGNnZm}X_=}(&UKho>3t#0?v z3?A3WQbrActwe-H2bJlu2$Ne~C7gsT59&Sp!g*?o$ENG5E9ce~S6Txr*fx|jKr8vw zDOYw@Np%glnuln>F`ofQSv0MlZ-y}~(CwqssP?N{Oup85PPc)biYUDYFh;hd{QlXq zRPC>w3egN_REZsys-<5fkt8{!(QgRryREkm}PNH(u>i$7l(zl6k$V#(PUx;733$2Y6H|Xq`Claw@=?1E9932P!{(yu6LTHGce-?EORIX|9pp z;aDh6Ae11&WPnDgif$uW)WFfk8UA0x?{uWBTdTR8X<5(1Vbr(ZhV3qfDC$ zrrNM!LlNL_YTb*OHHP$iqP3@p&!w#JEWXlO_B)Vsn3Wq9m(z}+L}u~!7foB2hUore z!%wE_bfEt=jP*epn%i&MQ@;tt6|>@OW{M?mxbkj+Zui|rT_PXw6%6;+>pMOy5=WK}>9tBVVhH)U{Hd$&*%&wNVRJCgAp8#Lgp>9W^@ zF430STSO^s$K7Q9h1TiToA%IK$$p3&N|WExq}xtZzX_ir`?Kp#pLzdCJ7~xuuB+{$ zhF4;=i>-f9RmN*Oomo@h?(}{6FnyEy!`(v;H`Ph+d|3x@AU_^tjHs&$CCJ(Lp1Q1B zmN@6jux?<}pOnB-GE7OA9<^o57Dunkut+MEdq>;`lgkooL&$p?BT;@uL_`dW{{n7e zkgt2d@b_O2)do`io&?e-bVirFcJ70PkP(wWkE3_*-pvCjGFo!a5vU3R^C~#maqrhI zVkO{`V3%8UC`C}w=y{7-aK&L2@I_Xc$Ky-)75YL+)QE5Puy`Ix6gx_ zo&f}Aq)18`758LbOv*7iTfi&A0rO`oNSJ}1l3vATicH{OI>76az8jtx!?gF9hs(@!g%8H&o$nFZ{^@rmc&sw$DpDb zOj4v3k*1l;A)0)dbhpjY;gGXvY!W!fbrkvs0RYO8_ZeKh&ABayCcugx;ZI?lX?6(! zPv8lEscA~@2&GDJK#7n~aV6_cqf0NxH^}Hz+4B}EztUu5bV8#J_09O(L;ZsNtr@2Jd0FOZVpwadXe4268^tPS_43zs|cH9 zba>7B7xM&-G}tL-e__P5vG}#2GI>$TtC%yNPDRI&zAxtT{jS87y8F>cs5R_S>p~^uFOrINs^9 z_$Eg*0!5N&(o%S<2R02>35=HUw-uG1gzT6-eRz$efE#F@wG-pO@X;!{Vd2n ze{fodfac}08;+^}Zp4@9k_|=mfiL{t*H1`XDmFP^9X3!)dF7ag<+@Wh?tFuUM%%hxswZaL)rPwC$D=*I^z6bt#}RM~nXiOMm( za?lk+E9dmT)Dg_me$Gwb<_Z1ZZ0(m%4oAd7EI-V*dV=usCQBuFC1Gny2QJ7R!L7&z zmLJ{7vPWGnV`-h-S@paz@(W>>-4QPU+K2y~j!x*$}zJI8u@?H3)S_Zz+Lf6{kz)t}ZQGw#+!{ zMvHHJnO(YW?)NWTiasCtQ{j;Gix=(t){5z~dn5#U%w8M*^V7*~nQ_kJs>Ji*r-HCv z|Fwy>978;VGu(`$srzK5Ow*Q~VgPIk6pezLY04>6&mSzA*$Aze(3-rqgJ^JVSS*^8 zRY|K0rT`-tu`VQNX!X#_FOpCt~)MUKS-3LL9eQ-UW(Y$I%39-EfzEDdOkJAEJ2?UgzX9di}f=2WE=0i zx3sVyBY6mP1IY!4)UtcbexgAiPt7sRA~}X*H}(rSuO);~1I@St za?@t^ltNr&FMC2Cch5;=gyW`PZGm&WJ%BYorMpt722u7?AC)BQQMt>kwBS9}aRZ*; z{w7^>oLfy?tFucXCl6lB3gwo#lLPKZ@hd(8GH5^@O{S(mvJ*)Uox!o1Y1m+02Ns*Y zX1&i8TwuF9OrFg<1V7gn<*r^1lc>hYOJ<^NC>hiSJOv5S<3aoK+o|7om@hT8-uPW1 z;NqiHqP>u1c)azgQpzFLk}XFRT2k1u-##7|#w^264R%u{Z?4#;HKz`~uxvH)eq-4> zB62b5a$v629hE({N%SPUN^g_M%Opa95jX#IhinY(n^|efN+3k6JC=WPtOh`7{b8S} zy!fgr8@@&{2y?*dK?eb_T-5^))Djtad5ODzD6n;I5(LW!BGEN}S)HBIgA}ppMIdSC z*T~GfG87T z;=D1Kee*$CStMP7+<^{#XN@@H(8z2DWt@Z`#6Dmt`An@i-uc_jF#gvT^UkUdkmXXT zr-|hQfGv@|49Y1|Az_iFF|wf%$A+sDUbIW4Qj#@{fjP8^*Rc{%D2XvvWcMZIXOh20 zhrnCPJG*|pc$CdtG3bvf4fitgulWsCp=?dAppUr>e z7(iskca~}QW>Q7AI(?=H|AfVXbis_pZ!)vpP*rw)G~kMaO>k91SWj4jGVyR}JAKUo zh^sLwNx8(892ZmR-3u3=ngH)2=5shY$PHt-L-gKydSb3lVWf(4J{|S=@#BW~3upXX zw=YYMCQ%WFQMxETB-r5PyLX-7m_(-)_wH3yRn}`$$#>am*xh2mRcb}aTSR`DiWC|d z7LdWk%V05culP?!hbi7N0o;s}MQ!ZqE2S8OWPSp$bT!55U;|M<)CjshF2fDs?@&O8;8lgw+ABrI~A5>;hK;7gg6}Slnbm#6}^kgBt zzG&usDmi8D7i;{&i=NhQ)p17{5eBqizqb3nFRdZh!rt!iZnaB+`qM7f(4WGnmXYoO zcBLAC`b4PD6m$SFmEZgqQP5vD!Q$-{U@NxN4o&advgJ0%^u@Mx_tN1CP{&)6!6SyL ziUurMD+cVygOBoi5A*HEHIn78`&h~;qbedd>l>b#otKBP?=Qjz(nLJ1%;X!Cnyw>l zoqDW}M_Q%`xVE9S+)*?Qm@dY-Eh4pRay&VRN)_tsaX(moT^}+BG%#fbxTVZ6h^dXt zCJ-rl5oGDk(mKXkFjW>SDHXav6*olPHNeT55Ak?PS!Q=P|zK>_2_VG*t+%cNCH=ktwl?9 zlw@>44Y`J`y;|M+N>dnUz%)HXJ~EzziVyQ4dT-N(*@;*;0SJRF?a6Vfq0Dplf9gl7IyN``Cl|ivx!-Qf?vo$^jzZC`~;2* zM-}V9WuO*dixX%oY(Rik21Wb#8Z&&_!i7Dz&Q?D{lMT2(2#cVer=g;8OE(pARD_}6 zM&WcI4kY|E|5koJb%nM5t*?%}q%V)9si^}ugiub-c+iFHk!^o-w-`=g-?OpPH^ zPF~9D{c+TI0)l$n288CX&oQ}|3k%Keo09A_c=h5xK<fs?qOMwdOFV#S4k2SwB|Dv5`ir_CT>Z2!*w~nq zdlu@^HUxxetvjdvsd=vR8y@?cMdSFzC{@Df&^DBbr<4o^a_urenTV<+Ur~MuAqRPg z*!YI~Hc|GRd~~SYajli5`eEC3yEVLC} z=FBN32ZR0o`Z#^mp=;mPR;U&;=_@CEeD`9fLWF=|E0NeeRXRI@=I;vr%6FQ8;J-^RMEFnwaSAuh=D@Hf-m6vtBer_q z*|q1?xZq-Rj(2^X*F)e446AV)qn25POF8_Dz14>`*Y%| z!<%&s=mi?ly(Fv}Vq%=MTs`s=GK0BNnfd`C9YM4{EczV^q}}5W&&u#Gu$F`c65DFw zBBpf27e>u0Gha0G*~+C$H5X6{HSnAplpIbmX?wre&@ii|qAViKHkdHD+KvoSYgG&8 zVDRpjvdP=O#J;pu7s4UO0i~r6A-CDynSAUC1uQj&LMg6(gu$sHo}hbD{(AARIws(MzX;CUdszMY^4%?4o+3_-ib+9ixagqs#zoKq-63qKNJ$P?g+H_6WxY=vb!f zjflTRKI`M7wff>8jo#Ds%5+kRX(^4QxDr0~SF2vM()-W9`jeE8s!%bB0fI3s8S$k% z)22KIS%P?MO=lVvvbk8HK%uqm<18xby|JeA*@q3cuV;3~|9>8SVJ0cyANNlv9!fTxVcgnd&kKD*1Go(n|1IR8%b@xrgY`Xl^U|muUCShOh_;; z61(gA9Gk=A+siYMovs`TgZpCH$ielFTZ^Ua03skiq=zQJEG<7hyZ^SEuz<3Ad9&$+ z6{NWjw2Klf(t zve%V>Dz=`pX3fHdMKZrkr;^HRd^Re2Py-`!Big11{|at9khIr_S^sVgLfPIdk^i!%Lh!SKQXC&K{LL{0Sk`SY}nO4ad2 zg~rm6QQpXQ%rNKTXU}4#Blj3&GN}Ho+?g8IJVZk5>;y&-?J^g9&fx}#W?IH-AU^s$ zH(A;bqJus&t#E_fF`p?*IAf|%m?>F9pB7y5WjvGNr1fjK9O$gt+0{g^{^ysE{hhPL zLDf}?G9!^3J8RK?+RgfxMh2XtR0k|vK`k?z-FrlAKPsm z<$5s}AFXggOHb-wL`qjhkD2%J(46`tKFG@L@u6))R5UFJTc!g9C!Z`j_t(pdwax#Y4V#8c$4!_D zb8f|@r9WW`;O)1wx5LeewN=(qN1nBSfBdwkNR|Y%QbeM4744-A8IX?!nBq@?)MUMT zvJ%aS7zw}q^P6V%T@LET#}QEfD0+g5rqNJfa^>`3b@97@I_u^D>Q(BL)^)5p`Lw(q zvD(O_i%(r2r-$g@_2REP>ZEh}n^LEnHk{$bh}30|;6?k@WvSNK`g=H|`%gbE1xBI% zk9ChHtl1Y2S=x!7i|^ED`n@5)sFSC~0qTo?dwL%DRj1oF)ke*uqfTxBsL3{<#FMkn zeNBZeP?_n*fwhY_mO8>k-0)p$r8%5YBa9kIy#(PW7KX_4TD5H(4$s&y)`|9a>-7fJ zrF+4q>!cys59up}q>&*pko95l+cs@F_Dj!@mzvLcr7Xxt6bdY; z<2R%2(Hd#H{X{EPSU%le+YF|AVKpPs$P-Ng@H)2R5KY=!;N-#!Kx=#NZVX z(H7`MAaWE$&u$$Z4kDM3`cMeb(=RV*(+U{p?B25{O0Jz)qYx+9$Jq-Q17#ZdZX60Y zYeHP>cJ1cBHdyLN@lc(OSm@Z=+S$e6bKSO?&sv}b=zDW`%fX>UeVO{c&b5`IMcp8} z7jF`M7^kHdB#TNWO)$(F$vZO6X-|L1!!1VuCl;&nyG5i8ie;DJA}AJZP%q$upm`fK zsxJMWzZJL{xAponh-T7SC>979vaxWE&sk=!%ssiRNFcbpNUPB}>K%YdF`arg90wCbup znL5`Yb*wSQR*Gg zK3aT88C}Me_ln1224xr;j1bZRgzKoCo{6C*X}QCtk30%xt>&8Eu(;H|gHPonxImfF zl6hlCCnqE3OoXpBzI=k%B~}P>8ecC$m;L3>={y{K(IPspdw*VU_k&xNAMgJaAK%Bf z*49FSGBL+}FpKXwKF&a@Z=iP}PKQjPdY1*9$VqQ2?t>}BQZg(=zE^DwNrVdoQ+>U7 zv6H0pQ|g%**MH*jqH{QbE=WEDl;RP*gOa%D9iBcr1xAz>8}=R?=AW1k-3DQzKxuNt zJ=pV%|Alk*@%$j8KmrJ!9GCgsgdwOfKlq|raRNW#f#Eei0 zq<@_nOjk)MYYiAB&GsoBo2KcSL~Kp0T#npGnSWy3Vh&Z{(afUL{c^l>E=!>Y$0=`% z9gmjWx!AyGh#;ZXY+~(gC<>k8S`5bhaHm4Fds3({&a8{L3dk%HMBUgd>KngOC@`X7 zuc_IZ2=7KG2^?`btGDZ4z0Rg=N82JU%$!8o(;-;n#*G_o+zEAJZ5Ws%$xD13utG6P zuNM0Sg`oMuFD*x)5258sIk$?!BUbfw_J+Ff8WH@# z=4G`?y1)t|nbheD*-Ltx)?>qpH_6YE3gt!JU6$Ta6-ryMkRn(_=0xh9r7JPPsnaMu zxJ7eHG8(mkC zsSkqMnu$W4*?|3XV}RTOD`*soPf8UQSdlsK-A##De{Ygi@eUXEIF@cgT} zox1D{3cj~k9pOU;&`bzvTsRHU7*~VSm$~e8Y{f9R+sPcm=BGbjwEy1oK(257>q_}r z1OkB-#OgWiwJ}(HdhRvTAULoaWvpaNjJRxPl+qDF)w(p=rjWXB)oEOuV|rN~-V!k7 ze0w$(m720FcuD4;Nx~Mra2ZYr0cW?$aRn6H`Ne9zdyE4*iu+!zUnte0bS7 zs)&U35gC1+``wkajK3<4NUT7BU0$NL>)KZ}O7etw3uVum)X3M)g;z@xGz zjtV7EK>(fr;*=+P1uE_*;wFVpm_hoVgI}I&ME`TbJ}%Qfn>IYbsC(;wxjx74YKPU! z|6p;qvwRBqiGfn4YLM$E8WhP;U+Q3_c|8U*<09*BoL80UG8CEHZwamJz$zEp#Zp63 zPL36~QO74$0z{C`>BkOI1l~uA{{FL9dej&Co<;Q!THqRY^IT%<-np4Geiq}+z*RZC ztO0b~lpEtj`ywV@PWSDK!JZQq%0ToO75{1gSlzJroETj^sHQS+vC$^s{L%V?faw5b zEM9)s#N(B4wcIeu!xv5AAC9}#rQ_%-}X$2~@y}zhC zcPGdE^%jMA4hvY&JQZ(&0=!jYLo3n?9bShS{a35fZ`YT1mT$hl!QucQ4i`U)s82q2Eg+z4Gry9$5XnB6dND)GT`HU&Vx)@N&^sBls$G6 zy>l6lIu9~iO>m{$i;-$eo-+`Q!9Aa}%>4X;g?_dJ)v|rMPBT3M98N` zAk!L66qJP2?8ecjLJuEah|#F1w^53U$!~?&mN5=+EXxRoM1&uroD+8~Sg#Lk>Jpo# zpgF(Q7d25+P@OITIDtb1OvP!R9x`zYI=)eaQd{tOk;Z|ri7PV5UScMmMpjG{SFSB& zTJoMO_MY(ho11|Yb21k2{7$lW_UgxfkCv5Cx1MGlmZ7A3=U%^gBbH#`WKuaHg1z$3 zKZhNseZCv-vcL3aO zY#`(Gc06fG<({;KfwbZOlTFKVbr%&J8%bp!Xe$3&-e_rmqMt0>cKGz2Mkc}HhC%xFQV+RwTJLY|s<`7(>$(KfI1XkNHRDh(JIjkUi}tG40r@$|nm-4D6;e%agsdDF8;4<&%XWKiB`Z`U9V z5`f8g{X{r>x6j|WPNUjkj9j7Q3XTtK8ZurskC?iL0zDr>@+lpl*jhj9m{$Tv>Cus8 zVHsX)7q=Ku+ky29$`ZIx49#F5Q7cM^EK&MyzZ-E*g(aR{)~{b5bg%QQqOBol{+m-F9ea11-!DqLRbfhSb4`zCMTEqwsk&o`%|zJG;VPUv}1Di zsbxInutRd&g|UFmU? zKI)Xq@tvvnZav!CNBumS&tr~C>~X^yT!NkzHVCL7xHN@nuQ7`k=ilGTK^$$Nneag6 zAqBlY^Ej&I$V@A!Pj6!qEo~ue=i}rg&b2CTNqdCX-a+$v$HLAC`IZ4oE|aLz5|PX7 z1k(ivts-|vvr9~V1Sh6S{$xf*^vT(UBV0E0?LO+z(J>Y3c3*!202n8|&t*5`aVjvh zFy8%q9ss4d`##s-e)}yCOiGzcE$-H_^`$yhx{r}bRiAVsolJ(P_S$Wvd!powt9jZA2C2LLN+~C$=}lc>h~%RS$GDG3k8ddQ+rB+XA2MlU;3Y zX&rt$zrFMWnWp=EOhjhG^b>@AXyF(glBI7DAaT z9X@GtJ~3D}2@t^|neHDU@)AHog>nzN99y()!GABhIHzCnAi#^{O3m$2*|lXhe2rhx z4c`aj|NDG{e54Xq0!*8J{wh2E){g3nyA%PyS}6}_yy9p%#C0FPYy0fU|D>gjgv>dz z28beQ@WBG7iK7q9?K$Rr-}ox^o$%MAIYJq(lOCykes}%+kOQ}H6Y$blmGl_E*wH+6 zU;Ckq)MdzysMtL%zRaW5Q-#Vqy>B-W1MLHJ3>9!=w~x);eELy=jdNOufhTIdc!bt+ zZ|6KIp>E&@FUBvl1HFFwDde0hUVvh7qMX9{mhRLc&ETk^_o~aPG_Jcz!<# zp(nx+(V-UknMGv$Ra?WK8zj@`C^6LvIrP4iYlaR#SUN9R13o3g-vJ6hpPo4-ty)cR z`9=Tiz?Y%FE?Vo@s>$UU<3`^78j$_&o@MICuvBffBUWT}^-6~!YgVEUU#JBqp9}J( zP>MR(d+@=~YB63vyMBE}si4|YA?|afT&=>lqJnb&8dp7G3k9!|43(4>v_T+ejHbt^ z7@QXq1e9FT%`)-Bkc+>wnG*n z6w5**!A&(g8;&^>fCrGtzdt3pYFvK~qm zD#~wKo>iZ#OY>+{+?G5K2J9CPTxsm_0~x2H+>XaU=G^n-aXaYQ;0`~l)XT$IfjS}S z?2n{LKCa!07LqaUdfXn)c}8=xCV%>=6aRLgS)zl#9xOZ)Er0>KhR{UwWp3v9X%1^iDOpe0qcD5%urmY|COK7u=l zbV+JB`O^tC-mtKfk&|UTpH!EyT$_5hw2X#QmWnfcgP!~y<;VgqE-}jsKH{mxKZRgRppRxpYn6XzCwUv4l1B-H-}F=@Q$JeQz&It zgaO4FJ8EErZ1*7x8V&@2pFqM|>(B@$3)V6{l+|Gel_szBhBh?Mvq6<5X%TcFZf% z!xMm(qyY!dtzgDUb_0u45Gel|m6t(_p4bS6o}K|<16xMTj`MIY?5E!F<&%quVFIYN zD9PVIpxU3wann+TqWPHTke}-9fhh3zrXj=rWEugIN>|Bn;^g`B=WnPSk6OAC^o{fH-ygn z&DkIw;0g(SbWOIHnh@@0=Ld6>aTW2s?R1#0Ps}?qv**Y8oMWO_C=I9FQ&CqE_ChO~ zpisOm{elp%K*Lir7kt?U1q8+jToMY2Q&8i5@7_I@qh0c>xmaSXj@~~Z0ue7gg7!(7 zG8Lp$ZQ~&$q%xg`Nr)^HK@WAyXddhgQhJdRgj91jkw_d{7WM0DFkq2`^Qpv0`8vJ4 z_VLA}IK(c`le_18OSwo!WFK275Txud8<|~m_Vc?xPd?f{)oHs1RvJ6v2g}1ua*KAg z+o&y;NM(iiqY27pRWlu#euU2eP zxyTv``as#h8}j|NCv#drUVFD$-Tb}qQQP9c&C;pej)c?f%V0<#4Wv-V5iWZ&^+IUPa8%LH_a6BC zPO4{xZ4f%XUsF4H*BygR!VTSxV8S`=Ly|w57@ULn=!y)#f$xVR2yjc@`&*6Q5C0<) z3>NDFfQ&d)E2nSuQ{d?H_vlX7Fk{}-`?o~?aUq#Z6-tQe$sVb5u6SqxU35`idH15| z;wubY5nW6k`1j3M#h#({gO~xg^gYqEj}6XtD<>*x zTsrQ=K-ro*G}_LxpQTHeZtxoZwyG*|!7pfHxX8(oG1(6&b4Z+R)%ZrM)*bbXw-AvWcf(DVfc>wNe{eRZ&HT`{RNOA zF0X|sKVJR`^c)UNCjM+*Eb0RTbA5E@wG`p_XrRnnTs}l-B**89lcSk3xzwBc`srQZ3%pIO8NpFEYn5wCwiXeOPdx8@X~W~ZOB$(( z^|xn#up}|R9vsz+zLZifDRr{iV1&+cA;VsD1+q$j+IImLU+S>pW*!8FDgt^2w&PK| zUm&a}jrYRrXin!Sw}a*1Dwqc4YRBF9(gMrK52=6x(g z;eC^0`So6GjEsPJvfS4mf~F~xt`N`y=6HL>%I!gGgq5MuwmdjaQz zlXK#B0CcCyw1CJtL;=F2*N)Qf9J#T;AE)S&4^!_&Q}M{tpa_rMBUbVXa9wzDG=`7@6`?9gOgIr(m=^_R^}155BdClHl=vUprOcgWFCRE zE7Cm~mSY{j2AlpMdqR&%izl5UH^jws(7pCzMu(wprdKvMM_0+Hj*OA&I;}U2LoO8P zEuu9161ytJvkxib-l9+Z-&jsPP{BB|sO|a%lTg?0~G zM?e60cq;Cl%b71Aq_=&4@Bg#t_P*ITQsaV!_=-WzfyyaN=X0Z{KB*FJ6_79lYC=SL z;#g$!n(|3j9@pvRMDGXT8)^CO76!WL;$~s%BeUYw*?(=`oS^Y8ad2^p;HN}U&>Xng z;hUx++LXryiz3eGl(*3(uNi@&@RR9s;=lyL(lw4_^VEBcWTDrw&9ZhB3k9(w1K!WW zu|Pe)#c7fGT6i)4#Dvp>Mzfa~$9#k_nz+dOjSkWj+bi)TRp}+vF7bQCQ zg9R6BDs02!vNF5SHTB62297d#hXNL%_aI^FCmN=4n3UY4S35Dg;=8@)C5RdE#ei{md18T>k{PpkH&X zHit8k-O8e#ZSFZ_@YwL0Jq48M^`s@3iIGgF%F=Gq`3)8oHjJOZ4qMPjy|NG2W5m6HpC@O_evgpvQSFSW^VWGdhX2p69#w-@z zSKL2I*=}eUXHh{lC8JKJ9X@Yp)=B^BMB^s1_%)-73D84~T^{(?NbCFxO;If>NLt0i z6eQNURd?CdIsX7`tR-2`G<#rB)*#0FW!nvHWqRc&l1}r~)*ipyd0#_CceUoXVKNcs z5j;zcyKCjjhH2U6qLKtaaXx}Hn3LFEq@~Q~+0Y^z za;2uY^|k8UOiw8iNtx#0uS?MopIW&|=H*2(Dbf^@sO;V&97)^*#>zQ%1{IQ!Be8qR zE}lq_@y={Fb|Z!m7uu z$RD`c@l}T@^PpcKiej9-hFu<|5t}mMI_)sfsMv)Fqr*;vedgQV3J4=pDTHc~p74)H zR_zm``eTi*64heXt@n2e{-kCJEzT3(CQvwD1F1$rx`S3E&e0CvisQ1un?mQZ-sqDp z+x-mRRg?}IxTNZTMV_@{udiIkA%&-5p&024Qz?{#>9{@BOM zW~9k9DebszGMd0sm4kCS>1Hs*^m@(gYw%ETWvPQ}H|{j5BS-VQ-cPT~ePbHWt5=yC zrghY^kbysMKR~0Tb?W|KOMgRs{CK#nkr^?9q0U*=cK|#@ZLShGGfJ2Ao;gQL-d>@( zl*vDv=YqCJ-)WZJ?QFvRunLEvZRgag+OpkvC_+*dzve+71_)59ZDvVWZK~Vzo$*5Z zZPS|lazGrqDwDagfqEhQKKn33Aq%68)ny;j)a#mW`fJfS zt!7r+2Y$k&P$TlxdaE%TZ6#e2;<#f^b9W?bZ&%pFITl2jz1Q+PNBQ-|VC$$u9cz7` zEK|jm1kGkfX{HmH9N*41QX4A%(4}`;F&e3m{x`Lb7+N*^>4hS}mc6r0dCVYAv-ex5 zTTHDZ+V!XV{a&B3D#`)b$3uH*MSK9;B9AYJYLDV*OD(i)^zV(9q73kwNMBg@X|!Ac ziPf%3G;^|Oi-#oxq=Y|#+GKdrYlMlG<`*%fd6D!YQywy6f{)JHxWXzwa3>KA$i^#K zKF_gRF|ql)3r(HXAJpVb-D0^9&=`qeqmHF`wMFm?1-F*J1cYaH`1gc5eBn|H2H9xE zLQv0$=|-QcHS}2vsgFdmB9dI9+>hOV@5X#L!wk!o9jz<;mL~Y=m>wLm_QMm3pDeG% zfSNJ~B-$34ljl11l14N1_d^;aL#E6Ar3ny%gUAv1V)*dkYI`5hYni-*9btz>yg`!# z(j34?@egO&J+|=06MdKDXKg&Yp@FR{*0l!Gj1S|J(ud3me8CPW*#|O?tHb9M7D1$^ zkP6dEYN(iy0A8njUhr1mS98EFJPh&?rUMAIu*OvKoXv|9xIG%Qun@0o$-;T@(}$+_ z-pgu$STs=;kZw~68Z!0~X%()fQZbV`qaMC7u{zql+dnY_to|@}bb9>g!io}k89u+7 zL2od;H0Sy}F&q}{Z291!?;kW`q!?=c-pZmDGL9tvV@b(KNyM^EdQUORt6eHdD^;Ew zw%Nwx{HuWDb*$K#Z(N!U9kGhZ{S;KBi$3ePesO)may7d1K1vYGz)q{59&H#~cy#tg zY2i5!vdxpnwQt`(+pMPPNtEGiD>lwCrS71TOt3n5;jf8fuOB=8_igW<_RYE$wfb&{ zTl;98r?DRb8hXy#+ArnAOx=K2Rr(i0)n{zi-f>5}gMI*b zCyg9oas)`0`6h4gf-cI3BQEMyr1^eap1-GkA6%gN<5J= zGzbZyfe0mOpvgQX6;ecnj8Phth@#LSX_7KTGS8JMiUz3+m9YpRX?E8BQoX-(uJgw^ zuCDi~UVVLr``-87YpuODwB4s9S3Hp1*;X8UKILahkI(Ll)>c^?!cr@;8w6Mv+%bSm zTo$=a_-B-B!dYyq!%6i_m0kE<7^y*6SSmgBT~>7Z$Yt#r&v!t~9F$1ne&eX5UwHG8 zT@XYI`uu{$Ks;q^^s*rTn704x)}5t?54&YJUwUMp*50jg!}Gw>D-WLktWK&vgXWCt zeG;F`E(iNuUA1Y#4pXVJtNpACx=x{>X5HfRl9TL;Yq7W zvl4n$aNpXg_W6C4Y>$S2G^~3wYhG=9U8Gn|fm^E2jop6PIW^p@qd%($$~MRn@{Iwu zNu?=}iw}Ikan~W~)TMcHLG6h9aOkm?BBvruiEtI4K7I51K_D?S+)xr~H0s4HZ=HS; z#OFcv#X$+bRrRH%g*9CVP`$31guX!S5Bx0IEL7pk#V#oC?odlmX11Ll&K&GzxvKbU zZhPU19e0gRv%eRfHkvPl9*33+JmYD=m)I(c-5n!aiu-LBije54vI8=a^3HE>Um`A? z??69h%Ej)zXO#Gorjs~|)?oJQk82yTs3+Mi#4beVT`QMDm5vtCUlUxSZNA095*D@f)RQ8{W_e#pJ@A0EhjzaQCu}^b*N^7 z>?WNb2t?{8yZi$36j-@^hs-7^#u-kTkzFv_zloMLA7}(fn9~JLi0D?JCz!ISK$Bo| zpe-WUi@L|cTal8&gZ^G86sKaN5onJSt8a|#7qRKqgp+>}IE%?%&Hzp+6j6c;LFp5S zqZ55%JisR1rCEC;Wla=PPi#;Koo=xw5z1Q&;afu6!GYCl*H*dZWxe@onX$;wLAt9a zX0Q;Q50*KL3{Pg>+WM@6b&t_?fQ-m_UDpSlm^>zR-sp~`wFo0pM^O2+%}vrM7*H=U znU3fh!tMb6!Z$#ywdKA5Xh1B41%xMb20<8a#l*1mnJ~7h|}%KE&+n36ms-5q3{d80u16?d9{{_z$vR59)utOt;=Kv^{DKwGw(KUzb|7vhmu8k(J?6up{L#f0e(V~P3pNMKXApsj zCeRQKLW8+3*d25$zx}d>CC*k;=RY^}>ht3-V)R)3-JlgGt_M6~fb}H{VNyyLtWs)B zM2ug0+sI5EkkWy-jahf6NKMseq~3e`lC=Lgj`V0Y zGFoem-oyOFG@V$4CdcgGQMg7(*D&ozlhR^n1If1?T2X{4vg8uxHI4jM*wrAoos}j8 zp0&%>=(5^-Xa4*uSC`bLd48;EEDX|m`) zf{iY)t;)K(x?5$>?tfEr9lgS@K#ia@(R;d1c+D;=%-yt!Sj*+N%i(+RZ!{MQy|4Nq zO8Npy0&CO6;F*ldGfa}2>tP*3d*6d#^)}rt}Ia=KF+!q!!nC5ZImLIh?P-4@YYH!d-P=WMR&-VZ%4_|8a+U5 zm{Joz2#4zot0-7=o`16EFyR4s1p$@=pch_AR)W?CwlQKiDzCog|EJYx)1&!-2p*-6 zdyit%J9+NM;m1i8_iGDsO<(=+rYmMT4B(@3yB*w)m`pU@++#SJ{bj=98g!b{~y};qwsf}g*F`50? z+xMeV&K5WL7?R=>+~y%tD!~W%J&}G1JBvg4AUJq|j8;aM#1UWxjpDbX(^f|9)t3-{ z)DB#~7pUEWj~9zo?%=!0t zRldK+4TS{?^f2yy0#7+QpQu_#Z)>?{-8tw@NM!*vO!zgLbFH8dpziS=x)nkacP-wx zj2H)mi_~v!LG=VRX72|w6+G{FH)It{>Qb?5+j9%DGGz+WR!&XQID6pn6-U#HNiSK4 zcy=z%2M8yLLE4GS$N06a2OW7JHhhXASpE`zWBGpjFsd&f`0VoJCo2(Zb?VfXXWNTX zRYyzWw=a3>zBT*D?EmgiIfyGrKxm_90ZPCs(i_VG6c>3pBmvyo3DORd(r-VJ;nyoK zxI?z7tPzKVK_Xk#h}7cCE2@Fy>N zbkh^+D!euac-|3nCP_1EAc8$Zpd4xn%nZ;W`vUmwP{BopBjntI(}U-i6%4S!g8Ozr zEe{ab?ne_q5A1!lUEp5 zU_=g2M2W?VqeqU=$SQcGp%H!K!A@=LJxE<%6%0Y-kNgW=g(`5BGIg8#(mRL_oDI)?r6nj0L)Cpyz4*Rt=b1e$`R1i>6?~ z$Rd+m3(30~Xb{#b8Q+o$7bX*!V=;XI2?ZHQ;;HU6`W9V*F0}303KMeYx0xg!KY6m+ zEeQ?y4Glqsp1VJ5rnh!DToBtIvrEKp!`urEtid4qBnl@r&8yCFz?8_8gZjtOglNB$ zp}~PP0TM*po9Aw0Cy1Qcy=e!!9U_GfqOQ7%TXO_U3=i`2z%q%;B*h*0&==1(EOmaR zmW8sGb@sly7jWU>n2t92Kfy&-V4W7;Hu>cs!MOp}YBe3qE-`4^B+?54$@Q=wFhh#w z9QgtJ5`B>AWRrjl$&3t{sW**&Pqcg9wst5a0kg7{l+dzJu|or;K>HA2`KVtlhz zznqaIf)#M3Vw0A~sXCV}!?~Nj4ZEJn%$Mcz?i%cIcXsVv%l0>Tk zA~>2ML8C$7n%W({?dzv^)V(h6?9y>u?K0)j76`X?Q_pi22 z3uf6T4Z?xR;)L$U8}M()4n=<=Jc4hJk=Q5dwOy$DWMD07D~s+Jj`Jo@{OwH=Jy@GY z>%$Y|y_5z~R}fb6bpmt&5!cn3K~7?X)rdsKc0dd(mBy^0k!s$1bj--szN;7yi12&@5ipjW6MnJSW%B)ZOIO&(D9^o#T!mssz zM&BIGJ=RJ!t17KZ16xw*4r_k248j##AJ!PqLT2uUmV8wj)LJW6uG9;s!X6`5gxn;4 zl<(|n|M__TL}2+I(Go81^WNw09~`tU{N6v1mmQUF#Yt$`ne$|WJ$I+X3D5^f)Xyago#>>D z)-?n_MA)d!db_3G`RNgFGO9y;zwE;fTjw2nu<5;lS>>0NoxRVVEasnj!|h2$q?*Y? zuY~>AwUiUASNe&V8m8P{HBWjZE%-ivr`ox}izz87+DUKFR*z1_y?fgKl=slpgNQu# z`5v#Ld7Cxp`HY8tk~dk38$@102wh#~e683?bSXIUJBKjxUq3g$^D}A3ce^7oevam(v1?MfJhIx0gu?9hp zsgn-^>8D{=Nvh0;_hbriFDsMD7;Kjz+WR%O`3YR~`SD zAX9cw%{CQgJTnxdF4VJ&(^2Q&013%MI6J^OLGIeOrFbLHh)P7-zYa6B!DT-MXdl@d ztcz2P}jBuYu#{6_)efv8Q2OK;!x=>U%05?<(H{+wM^M$ zh&HKRBpgOOpE0(N#`RGL^j>t0tZ7`8Q!r4CXg`QH8**c%#;f zWfJ*zQ2`F7z3o*~DfD14lNoSxqKjZ}lCHnXU5I+k>9p{Jn3$z}?ZPBaI1XZ91)?G_ z3=_N`bRecBff~$3X{wgc;mS#h@V3`6s}AZKLWo0iqRA-|A&>mfftE!;?THK%agmhH z2ntVe;y@&2^>&}=_%RpPj>;-B<~k8N(T}~IjMBDE;3xK!9f%xGP7n~)!KHD}2ZYtru!Ty#NG4jLGwE6)Lz5V;Oka@$lSbF?C!r)}SW zpSp&yLW{^|6>SZG*!i(QJ6@#~0Q|qy%BI1^S5V`_<&-tJiIWF4&D|d}W;R z+y|=`R1=mI{`~UtvCs;>ZnYrQ6D?8nQ^11ohE~Y5bEDPbX)D%6#@f+TKsff&p~H24 zr^*EiKY0I??-Jm$%?W}Q1lcs70-DESTYAC(-Nv=OA>n0Plv>qi&zg0A)H`yIFMJxJ zASgDB&BopZFZUXB56XeaHmM|%sC2#}Bmf=Noj8B)e$O|U|G#}kFqX1}&Og$_RGjZSXw1PF0%*^V1J1`3Tk5aBL2}4T&N!o|?%!8-{ z%d=YV`0r};Pr5lVm|S}1#*OUD(a|1K0K64%H0bk0F+u|hIvnm>f1mVAFJ!C0zy z;Se?jY))P$2}^jTMjhzUI*H>OBN|!kk5>VxHv}PS`AmQioU1A1Mv4bcJ8-|c;k&Zi zCX=WG=9? z-Mvt|ViY34M6^auV-$`oGcH>@VA#Xv$(L|QiTLZ!;PP&P1yIS~R|~M(7$>8< zJdNq`quchaJ(WY@Fef5MRv|YL)lrU|*-3`WByEAN1npU?Ncl$nJWxtqaBjl+=Z<-# z;+j6sJ9j^UiroSwY@{Sjg)_}xq^AI56!dkps?DA^@5y=x*3Q|ZfJIJ%bY!Q`o;T@X5Qbp1MZd0a(OaB;a;fm(1(}KyB83wOV7T|yd zNcB0AlN;1Phkg4Bu*uTRi0%ZvrGphr=s1VQO-kXwYP=_(n(x01=Hsn*sKo#R`eXSp z7N{(#P|1h{YmZt&sk8ss#OW`P%id95KV|$FQs2`jL$lzTC@K^Hu^KLL7OcflV8T~) zooW73y}!OjRC1HBf~3KS4wJneLC*%Q-(21AkdP+D%#oL0OZftfMh0=V$D{N7J322Q z@v|S?e!2A%vb4V{v0<8JNyaMV*A2tn%SBx*`&Teg&vD=+#dIN6!a*KPm^?rLYC5mM zLY@nsSNgC&5<1z;W4Nd~c zy)31SAPsP^a6O`-EK5Dn}lp;wk zUNDdEZP*1Q*n#(iV~()!_CM<)BMTFmv8HkMejG$`N8%GM%dYnJIMQw-K#;s4@w^;B ztGb*TtY~skV{e>YD8i=quN47?QG8;mu^`S@hl44k&P;Q3mu<_&Jel1`;2Zfsdg{S@ z+~dbrxoOaT88y9g!bO9Ns*}{k;Ed`QjB!nlJ|K4px}CxWWXAt*_Pt9Z&5yWe>&$~!yAQ3hpCs0 zm|=ch0c+`bZ_hjFKQ7!f9+_2EqnAGc=`I?nG5<;QfceIOwa(ykHL5R;b#1}0r4H5I zzARvN&nC!5`}9ei?ycqvDi3r~l|k9zxQIgjGrxx)YMz@aF%9#2cey zeslmHv{V+|vj70)fLI~=SVr|`ETbnf5TLQ6+DXXeYS5c1tLs>wkp!_Nn&)NY|J|Q- zEa(<$@C?6(D%O}LLX-~pwCu!Up+;;;@|~cgbd!FpKj7GYu|luDE2x=2o?#W@Oc~8- zt{jAVIxVEbMELvL!xPVhhfC;^U0Bj|zN~Sh=gNvU(+0Mndr*l|LvNFJhd2o7)r84| z=IGaW4WH94LZ?uE+JAfeq9g`hx^z%N+e{s}WKnnZiFs@~x@$wgS2;iP-6M8x4<9tW9{3`&ad zGiT4ngB0CJ3?Q|UYzuymiX)z^#VYJEN)leUerPEM>~(Kl<1Fl_hd1Ar!${473KN}? ziIR*iG(!Y;l4(2CPF>i> z7S@wC+EA!>^X+@82g%T^S`7+n19Y|j^(g)VlRXCz8(5(MmP26GJ^&-F9C(>>0n;(A zV#h)}WV^_3BHD>j7Mjl!5)&VR5VQ=k1wEOaNY+xx=Y&99R0iK34*^jHY@C#Y@qE|7 zGehfj;W}G95@?zf&KiH}N%oK^E~#%W0YGPDFJ5{aL9jSN#JEf!wz{9$M665wUl&N| zv#DtSgI8MEA5DY|6#d}NFBnN(>bQfWM}7T9yeC=*pNlvqt_~+y8%VSxT9HX}j!_-E zETDN>1yCmJYSBN(bXfs)wnj4fptb|>OP>+e$cP1X6-i}*n-k{`g=uyBMp^hslEd9; zN$QgUgHA0au`)-?GEq$O6B*ZIWa_aHDFW$p%Gu^Zzm(;)gk*%P3?BcoYJ)4fX4A&sxF4)lt0pwrhMoxTjSR(jGIz3WE74ybnyH8bkmnh7o(UzbV{V z#o&z7)mdhoWXi!ni{9bSU2(B?A(z#+f#ATAuZH06ef#X}B+*|YmzdTTaO>eXwjw+< zE@t)m^`+QU;(}7ya_s>l-yJ{GoU@fdifo0B*QAu>pVcPmy|s?%=)#s_sGs%e*wwq{ z%E-jO9?Xbr9(^dS`<=WmZ*A}gW8_gu4B1xlHlClv{yR!oRz^k( z^=sp=N?~s$#pj>@y^msv8DnJ*l$iqZ3vfs7xDvD|8-tWNBX<}#7Zwr0#UPD!(?8Ri z{IPM4xwWqD@xO25O3t0O0Gx1C-EVCKj$G1qt6v)kzHrnzVMm{{!XsC5<>(fHQykC< zEst*bg{Uj^0wy%$r!Ns>K0yuGjH#MJFGNFDR>7_9=Qj`mdz?~QJsfjdf}JlNAAJen zU-KDeaxfR{_Jm%5Wfa0%W&9~FJoWqu#K9L-GxN}ObzAV!c!w?4qc2b7&30;TO8cm^0}Cp&=n9wD!T8@qvNSh872cwz07v*9=3#9xbGjJub;~WB2!s zy%Lgt9_Tzc!TQdE4UzB8cgG9-Vdf zG3n?aa^Sd!gK(roXR@2k=&I2%7Dj7fDMFBdh)58E)v~Kk0=w`wYB1{0Zu@q#>_Y9! zMIhi_j*A~>^Y@8pf9PaH5+J@}N4m?%zA2s+RhbH(7#hxup&49N_HF@#mPwQ0&MZQl z>5Y;#Jy+v0s(tdCbi+m$qk*3hx>s{(VI7grOta7i^Ciu zntwEu38?7-p?v~>3$tU(b{+8^jVO-XliAdY0GyzpJD|nS2?ZGsUYwiz>egisGr4c; z>FSOsSYEzKT+7xj)%-yl-dm0We?KXwly^ZFd69qI z=;Nr3n5%frSoR6@ZNjUT#~l54BbmLN)PazQQQ+^&ys;uBMH6taHQ%XqjBv7Y6cOZE z5mJ9faUlfT^yQd{Gc5>fXz}MMNFYj=Pt=rr<&}|lY6f?Bi0=PJ(P;*ppV5@2{!`U( zGFwFNZ#Mw5-It60C@L*&XV1XK0X;Pt7%%g%=j?}9X|s`O1_FyUPclV$3{OU}A{0|4)ZrTf z6kkoU2g+f7fXuV8dyXNQ7epqKw>-3^0s!MtFurdg?EBQ$w+v;iz@$kWaCS92)^3=t z+gje1_-5qsT`Ap-&nCmg^m-U(yakYS?8y#J1_`k()E|Mzj~~2rY5Mpv9Q2WZRhsdU z&HDi`EML2JJQ()54pl(5yuhD+LbEq0wf#ll1CQaDTH`a`G4EgOJT-3G{;ojl+hs@| zLx3R9R8rzc&{DuesSxnPx%H5zP5`YWgj$^2b4KrsuRH-BJUSlD9yQxTJ;PmjYd9OSUefWgU$#$T|MD0%VDMD4(D z4JZ>DVpT`uxJDzY3?*~}jUB(^qN0l%MdF&FH-S;850Fs@GhujzcA%=6AmW_=B((Z#Ofo3A#8FQEoolx)V zGZpboM%(z3WpP%PfFM5=7Ix6zpBXby46n<>lr6wT?+Dp}^5v zAb|0x6JNTYK)582VSJbJ`GG?tTPl9ZXCF?j9g15Z zJJC<5RrUR<1eirCm>SY-%(@_}d5f^toCE0sdXCsElQK%L2#2gZj6!1dWV20;dES%Y zqj;NtI0+X>9s$0n-l&~0y#{mgjfGZ9^ zL7jJq7OTQjt-8T+JwP!qfqCDMy?|i021sV`SP#}ZmSD{b;FV&TZ^5hPJAOG%V#f9l zIJ7Zw=40{?Y~*=%<)<8nHmqO25=hM~?a_GI&kcpI9MD1zM%Cx@j*x&!ZD20wX<#l# z8QShsK#5f}N0OdLQRF->(0=&u=*x%?o$qLn26CAy)lU8z05s?t$t z%U$!!hUGJ(_O0Kpz>4cvUZ0BVELj1DoMRb(tZ72ubE6Hl-ox;69|zoZmA$<*a4dhM z8u!@@{$rdVYm{fuVypq{1>OifspP22wjDU!PG<+-{OML0QM@JI<-pCLs?mt75s|cK zJBk8#S%&>#LDW+4s9qngN1^N<^M250e_cU{XH;I-!-4(Ro0W^*|&x zynV7l3#Kw``W~5KrU^*{zY#};qE3g0OUkhu%0`&Ou0ql_;^dRlFj<jc= zQ37srQMPW?iWMYr+R>P!j4I?4RQRYomc$#!-hlx;Tpy-uSwy|Zo4I-KJK{|XV4Y&5 z(EtciAZ_nkd8ahUduGS#O2v{g_5V^o>|K1q{zv0E!vVdp^JYQL+vlfx)pM;qF`=yC ziMW73yo>AIkv!$;r}q-EhYqnbTI<*I$xjaU8f)N;9cpCtxVkz@ULipHAYfQMm6#hC z<6MHD%SO>~1z7b;8;AFKJ%%?GULoZWy}NWYFEQFu$@^+?)$@p}{o3d(S|_b|t=4fK z^jbebpP@s(1l4~HMCm;Bw3@8;Lqh*C)D7GHDhe4@kB?Iz?-CP9Q{Q4;p31qb=+3M; z=644*f+n;Ter@H#L1-N zSLGWMJJH~4h{8JApxvss)dC`b3^aC74Du+we6vb()&i^-A!IzUJm_8 zPn2uP&k7FBP*Ri(3lIN~*|w+zqZ&;ucjwN7d|*5Xfm^Q8kQ1}XT(GY)FmlaavkZk~ zsBJ2bm!C6xd(xVCBQ|~Sk}(UcH(&6ttCKFWakf-j!&+edzSA~wE`&Lf%Qv9^EAfbu zhVp&C&4&|@8{>~KfMaNux3951imT>+FefQcP>vT_$W5md>T95dHE4wP1rR9Ib8lxA zgt-Y;+t~6 z=+PGPLdGA)#}e>;yyt_e0688zgZE?FZQfQl3~cnjRh`ynggZAlM?=aSr)3~rom<{tD`5`!DXRPwP1YkrW3dG#FdkIczWU$4s=|}0u!+61$ z4>L^{94o6g23R(Z;s5WyL(uSyhaYDA=ia?=ZOwz38$h=bEIccLI3qZ`n*gy;y#3nm z*RNlV?T?j5cB+J#H$Lk@Z(ZT9FQO3dgq}Imsrb8V&_^EC5}H2*rtcegwCI~=Zs@Lt ztb@b9hYF{tIDyvTfGc7jLW$|?9taICkzZ*>q(zn8e>id3_&f*eA$+*ck-jV4&@nkyA`jbCtvy4DyyY-})qk zIJPhnz{U7`8BNhw;S$PSUI$-y+vdHSfHy?n(Yvg_1!91&b%nnJFqTR0M}*>zn+^}t z`@43^9_ia##jC40w@@%RBgV7*C2wYQSxx98iW%i!M|PWG|5u6FgB)XRz#E-a-Z>e^ z7#oAc;wuvWKCEif+}%Jyz-u3c<=XZmukBv_w>+Z|U{WxWq@XV3*n!CHn!)>)LQWHa z$<`QhI|g}nN^5FXi*1@NvwUNTXFRUN7_^zaFp!hbr+g$J_2O+I3>zZVBAanjy)uEr9)U}WPK(HVD1?#<1IF&Gni^Zp+kYBlPA6dfw0wLBrF z_MhIiM?<;py-nhnfW(E7M2`*yS)yr(11GH#FY_%G+fJUpju*1yf;47zOlm5q(OzLu8srXr%+{)c}Vl;Hl5t(nt4Z&&v&Jrv& z4j?D=l4{U6aO*?pZOq9$e)Ugv zF9rxe2@)s_N4sBpdEq$77RHii=+A*m!#YTXs04=7&haix?|#rmYBzOLSJyj@zeHxW z2V8TPoKrWaOwDI*)Cw)ux^4aPlJA&U5B7v`RF`WVggGa%yw}usbjEB%`)B$O)4KWR zM|XDlIckq`@kTBM$O>?_yd}EM*QU3e*}N1EuOHa=F2|;no0$}AAbyPXZwNiq;qwG5 ze!MaxRO<_Ji=V)>8GmSNj|sA62Y3Fb0?k!zPeQHZb-s~*6Brn8{oFA<%Sq*eT;l1< zwkaB3O3YgztNfgoAG$cL(3*d)YU>Iuq}fvjw4bJ0a$ReWc#(IR)6jLz)^t-pzL8Cm zeR$;}@v}&dF)kqhRm05Lv&SyJ$Oi&Pot&4)5!Gvt!{3Eu@CD#VG7zSec3WY;NuA60x`iQ7~Z>m@Q}_m;z0lUFz9K_F(9> z`8j0-d73VByrpUchcX&H{Gr%;hq_?}ZmVPvC}=6zl?jZ#j3A#JJItdO192c={o?v# z(0SNx+qUe+vzdyD6S3`p!h$augDg*W?FQ3H=hhxYyZ(`YfXhG!xhGCswR$x-q%N_> zvX_lz-?g=IMd+DqyBC+0HN_{t;B^ZY*YmbHBkw{$*#~;Mdk}LTMP6mvd2K$TAwaf6 zGqzw8s=Bq0hkM>0KrlNNid5KOuDy~-NQz|=xYFdBIc29Iii;<59L!sc z^H_9g0hFWi<|zzvZkQ(*T6UoZjeO0 z4$rsD3@Znvra$< zlmw1#FZ7C*!S>o{ZaJVf`-62az*}W)A)dnfE(hZa+jHb zyQD_2`!T208^qdsdpEaDsp74@Xr-_`VMd5i9B1(O7g`FQC+D<{1dLnFeJe;ueOAMJ zKCnwrw8l0l0khx`OFrYQs(uJFN9V$~?hpzT4m_>^V0DjpV=i4%R5%Gcitu@&hr#j9 zl$IVt5E0aRsc4Ah>4G3G5OCL6=uuq_jzr+V;T$tY0!Rb`smIqoN2Br==#xHHwru2Z zZXLHD0<=6hOg2{|MLoONT^5N*AMVjK`^{a4XP=PvFTY)93w|zqGTWgoso_4a@ zK8cf!o;B!0=1kjaB(erb5J5+8tw#`lOY^H;6Dc!9r(r3Sp{Xbv>14s)TyAPQ`Ocj? z3kkiN#W6%e_s!jJFY_2%2#&WNgXff{)$3S{$qA7#Rge*W}5 za^%Ps0D5@;DRu$oF`Ym=ly?~!$?Y0ROs~WhIjPtp*K4kZPVoWY%#sh8x8&HN5xfsg zP$~qRAL}zWJk4_QMnJedMmQkrDPGz{L<(@F#XA<>vq^=ktd>=8cXtRvs4Z{=FLX=K zh%(b$7jBwR0@U<4;H_zp8%+aT-~#EqE$fvA<~%sAj|#WIyVl@f{Z`rMRFh}COrEM9YQfM zCKoh%z2CbS$f0gCNT3rdXWdCllWUM~*QGHV&Ei=sRg71#k*<;V=Jj7~PX_fJ4vqAt;g)5*oF)5{l zI{O^fI5;-UUyx9^bB?6swF=LX+D$@&-JhPj)~khv!(Ypm$|gls*wNYftFj{i5$4HF zquIL66#Jn{o5*}cg8K>Z5(dn%4txH>ycqZB2 zM`3&Ge(1b;Dq_nlBkq^ZX04=fy;?sXO#&?a@65>r?6t|kZ{6k~R6!+ecup0~;gxO8 zzMmmSzmF15QuKmICVE9oa|-|ytq(df`eyR6>7cMkQC|_B(60@d;sc_!m`i?Nrea>O z6#msBJipyN??XQgdQ|;NE@;xL5^sDm({?l`esne1-)Wo(`h6Tb#BT?!zcDGwd7xws z&2PGo+>{rEp~pvaLfSyiKeg3wF*15OPf4jp?cgW{l&x0p`TqNUGY>oK_mJ7#gYPq{ z>O-iX-$DFU>menkkdq`LFsnm{>y@vv|LY9|Gk35C*t2T_q23)|vr$_;- z$M{yzT#KDKEnO%9B1*##mDm)6+&c+U=_+J3z}sX9@5WCOK{=|8lgSPFW20{a5?A&vp;x1RlmvI2;MU=AX;R50W^El^a%fA-qHe+fVgV%uwd5|JaP zamSCF1MfNyKe!M6B~@K_ql7MoEo%zeZySKCqH;=af;X=%n#GuhalM!^6JX~8^!)KI zYr;t3dS;ST21mMW0{WBg=jq=^1tT9V!oh?KvOnq+J|19X>nJZXLSnkko~9}O64_yi z$0MYA?$w&bjm=k!9_6-7Ca#$lNJ$_3jQ^=u8ZxA)V^zyQI&mw3)jq#JkuRt z@k>Zh&^R~7(|6>+7l@du>?nwIfS_>b6K3Gp!|T22QV7|kIEkme4j=|AI#}WR!&?KK z78MvgL;koB5Vif#$$O}PsaSMEqI>m~Wy^lq>tfio^E~&i(&2QB=ASe2jz0s6=4Mzm z19XB)!1BSY5#0H+W%_UtwwArv_z9Jw&EX2vcZ?P1N&0L4ISWsUfHIWiO zeg5eAa7`Sj|Fe%P+FYp+_JVEQ7NseB>Na_9g=H@NaSzG>lX6}Q`N$=49v~B4;Sliy?_gCL5Pq~oRNta;AWo0eqyqmF^^hnbkB&;LS$-o0>`O)(Dt4{ zr9T4!`d&*$ZVtMAzR)7&W zveL-A$sUuRH86+|ZmYuRmI%fXmNUap;s>4#h5@A4gW^-4BQkpD)kr7$z~If))D#EN zQi5baZY`JnGvINAc~QC!Gx$Ihr5$6}7<2t$I9Bt7*eq7s2b9GH1ISQJ72BlZt~nv~ z17LW&(k92NPlC`Eoa(VDDf}>q^1{?-tH~3yg)go*Yx$N9&VFsQ>Ju6Y!TR!!OH7o9 zk2mphNTDe9L5VN7m)BJq1Sz5Fz-TDiLOn$=1m+PvlIlD4<1Yht?d|!hV>%R}5L-|k zRmEtE;*8<1Luc0&kQE9il25F^asR;Yf$53nsqDz$l2ejlvKA5KkEHz=8gu!WPpFJr zpxt%1Ol-Bx%#hA3vpxpTQu?|QN9h_+*rO;j0s0&UmE^Js`8?WjH%-}e4hn{M$PbIr zau601W4-mFWm2*+C$&s%L8mq2kJwEG!usF{tKdJJclaS3BN_XD`LMPgezw50X+52L zWZ@^zgKoKCWKL$X%&mub$y-z|gh73P%)GQ%AY2_v$eS*r8Ga+jk%a63a0?c_2iwAi z`HD0O{`1FIlOR@bz(Tr(+F>l}OG+ga3x77hg-qx#6>Ugpd%tHPoNgWmgc-x#{}>># zW@x&bZK+^}L@EGm%hiX!KwY|^_lIH7m3;!I`eF@J*NLeAN}W;BMgL*=!+t(N z@U%DYkX?+kkH!WAYh;)+dQ#HsqXSW1%|bKg!(VU&BDn`sgg0>UKe_C&|JD~EY2)7t ziWk>++*>V+QmuCnj6)eM2x;~}Bp|kUFE`;=z-NrQ3xLz|M-BD9>8(+%@H@LUP9(Fh zzu#3`a-8XwscFm0DXK2=_jX@a^)m2q%of@(PNG1aK`c`9Iq%it?Ksbcnx0M4ivUSf%E7v&fNbXZ8Gyyc(pir_P=G@OsNb^s!CO{dz)l5g!)y0$3uLc2TYd;~+(E z-`1Ev%(=uIug{ChkdVw=yjX)$_w@z)xRty+;#}~4-F%9^9`34FaPSEnn!zy8#SH2Nh${PKX13)p?E&!; ze;#|hG#>jkeHoltOSO_rZGplOpS{z)?!;9%2nSU{1b(}2N`^!%ay?y}E2YJeI-~L; z$4>YN_@jDbVrUXnLq+d0VvNy-vkjm4IbPF*bW`t} z{Q#FrwzUVGN9i!SaCq=GAg9qcH#cwdS;mhEVjmv_K7>fd!pgdEza~a8S!_l2-HEx7 z3JMcNRn|UzS?<>tsyG_4UV~b$Z4ZDXDk<)n>>KY@JQ;cVxN3|UIf>geGv91v4NiU= z!$2W2UQAV={`ue6WBBRohoS}pXs4fm++m7P=8Vs`-rVK4Mo;e(QUf-b-1?m;e)*1v zHJmiGa9fA=8IQkv&UiB!n|~D_#@YjK1;39XKsIa);MVQ4W2OH}jt~Mp+0p#D6v&g6 z_(~}}b7Ve}sO!`urKH}$oB07E7CRBUnX7I*dxxi50;LVZ5XTA|_UOrrvM2u)IHfOw zRf-0>;wDVHq4K7z>k6ulvkM*mBU%E#-HI8aULf{M22I)mlCATDAQ8N8lgh}*NneRO z{6A4ym<^EFOjXsFwOcU<=CK$Bj`E6%V)XD3W%ct{?}0L?*=3#g;q3Lr4k4N$EHZKo zLtF|9!)S4pY<*{*0thW8{Skk<9zcOA5>krK9E38&7EA^3Co0-f(CMcEA^()6`+Wnt zK`{ba=tCyf0`zav4QZHCPC`B35mZyL;@{dW=Um2bC9Y$O?>n%ywLA<&X40pus3>8` z&k(5vARENtYizD~y5iHD;oriT$LL|7)oQb7casj)!j1M|KaeqBJ3X>oaKwH84sqAv z1IP@8ra>5WsBYbb1~{Vwp$|Wf2J5Xq5os0j?~ygsvMsbnVSW#ikQ%UL zJSXdWUIW~V;PHo{X$qVoCo{^+SaPtdbzT4DV9L)SSU3slpYd-)$g{hqz%doiehgZq?g1p(<@tSnwpA#OEE_Ck7!}&)c1Y_bnbu{6 zJlV1L?|PIBfRt#)+_@8onS(RYcmS9VqA(iN!2H^mxt@yHeNWJVDXgQH{8yHzd+^yo zwR07A_b7UwKFt9h3xYoi$dGR)^MzR9ddu5t>)Pg`(jlauFIYelMkk;ZIM*>uhomsI1yC9Cg(jcC8^s)qcV}$F zw!=~!QjyvWj~5^It{-JN0?QY0AM%G1(-v^_%WSvV05VAKpWsKF7W6Hus`uuj%@}f` z+eV=)VgmlYGwrJ~>}bF)!$S;(prBY>L==Sp$)f#NUOi}Nks0;MN#*L@yMhep5{>}v zncgQ_WA0iyr&q*)fS$c_`N;9w`WvL`5yHcfuO-y|hUO)_)M&|VwBaBHFun0>&_@$X z5&@)o0|HF`-vBeg#P6iV#8VLw0q|?s3e0Sm0mFIbQFV83??lWzbwU@*;~%@xPTjB+ zEf?RJ3TKCf=N;=7yc+$%pO@kei=a2?+QhVS1K*uIusGwwgH9WW+6ZgK5HXG`vpn%O zx1{sbT#!$(Y@AiShIP@c4jKiMYH*GR0bk=_fNT>c*n%vhlu%KU)FEWwN9K%4)hFRl zXp29-Oxr#Y+9iT7fxwEOm{8b1S!bJL&i&gdDK&r9rP+%hMFK7qw8;(G!6CTTOar)h z9}sdN$&fH*>OQ59A3r(~HjA{e1W|~M*`PTnNUkG5Wz3dLkT5W+;%)0Z@sj`2(OJ6L zt5?fuIa=S$N;Ww)W}X1sdthi4nzQ&5O|q2?M|N*DX+$tR);!IA0t>oD*W@xhQ#J+% zC@a7NSVDLLLZ$Jo^)%)6#7{$?(+t1)BBTcIf<7IWy*cOAGr^v(qY>d&2qZ(iU{M)= zBqCcOC$w31vU@i~T}T%B=n0`bNks=&;>Juq(A&je^u}E`JPw4kYFqWe_`J9?!g~L! z{zzza{(n(_nEREDBobHRsz7+w=eWy3d<3Pl9txt`Ysj!(=I>hsK-L9wSSV8n9iR5t zsd^k^EZg+%>6zzyhqeOgOYa(0>YWclxIbPw{H^+cW8;(UfVi6rhI@y9Ek(UvGPAEX zYOdmgX#Ho;1pBo?C}iHRo7aB&m${on`a|~EjZ+_)CaQ%6&Aarm?9QyZncUF|BYTYn z+0kgwEDe~R=c0YjKKJA*9i6wR7`PZ;sG9&9J{@bm0q61LC};7Ic!zk0o}toHA4f5W z@I%N(goFYR!E)=3BZSW@ZDq&aW@El$^@&f~q6t{6YMioWh^C_5U)_dQYVq%0`*lGR z?DGm#*lyom=U?(rdxT`XvXKBB)I>}ym>-nv$tmj<6!$sS3Jtvs@~7fcP%t|P^$J>i z31Vinz=0D$(i~rYPGsMmQ3`IgEb;=3x}I^mLN+mBj-ur0iU)Z$sCgT0&PyFG_9u;b z;qY&DW($;|$rA#jljn5=;FS95KF)mq!WH3tP+%0}4j@k?xa*@d_9eOc z`T4OikWE3wQ3*Ci(_?^3=f2j1|1V24_1#WV2MX!{MhZ;8z^5TPhIHg-%Y_}RY(Qj7R|wr#x1 z*nJX{B~4cFa|#Lz&w{boy%vCPx7;*ZUuh^S+NJX-StjbwX+2fh-7!B(TsJI{DYpkq z_Bn9;gO9%r!b3+xMFna3pr+8Wn0;wu#D-QYC`3prcsHawG<9QC*S#GJC8Vx*DyFEZ zsUb(WIgkbUIS$Aqv$d_+fFcq$^(I$L8{JXckxFW#-5}ljGR|XEskw_R8qW*g+904$ z{Hr(YTJA1BIgSDAR)P)NnXc%qu%Y7r9toO;dEQuZIq**s@c;d!_-w-h_F_w+r~PYE zcXW9ldvIq@rT6&mmtq13YdsA{ey*b|Zh|aTn1}xI|1sr?gl%BiJ z(RvPoLTdL~>B+lu{g0= zqBjAxJ-D>-?v8`#sw2^_`Q04?q~2tK=m$_R5>GC8AbEFi8}=L$N)xZDElPo08bqn~ zAgdE8GmwqIVR5@Jbh={$9d2FSjg6~E*70*e7`2V12#taj#9=Tb11^m~NJ`p}5ZJ)k zL5N45C2eFwlAmn7$)9-(sI&-LN?DBzDC8)&VEjSNi14ZdE`J1yAofYh1zn@|Wbc{F z_)|Wv=9%oQx%r08w5zXzlxG`u`D*OGF4ko2>Q&LEU%u_so2M0PF!n|$g`0h?y_`o( zwN6Yj@1eW57Ed2Ro#IpJX<&2TtE&^n4uKHRIY43tb5_kwni?;!#l>C0ODC)n#p*Fi z5)ibN&aGwq(MK$9U@&+={dIkPuC|MSKJ-SDh#u55FmcQoP+G9y8BR2VO%{KluKtHc z6Gkx5yAU<7G2m=pVj82l7sr`4TTE0s3KEo#2&iT!5a|wz&IUmD%go)Evbv$)-szc> z@|v5QTMJ(SWf+AiqkyIr5imA`A{Wk2^Q_{vNMvuny{4%Ds>z0Zvje93S-wlsZG78u z{jH}XYZ+e|b86h78Ci=1k7mJssE?9bAVo-%xF<{ys)M|cgwaqQCAl7m&nUgA(cf-$aIR@+rB^Z@{5XTV|EX0|O1)5Pr5A|PnD8cRWRVSpAuDFYK% z_mxh88}5f)vy!|fG(-8eu~nVy>ceLPna?NrN?T(DxZ4{8S?eS`%`%A;lEk*$*f3p8 z)90@Cybapg?35{7xpGCejKEZ+^j#X_FDbpiOlM98X<+%RvN%swft9_N` znjVTDIrq-1ob2J`WZ-x+&U*j1ZOk?<%{GU0%a2X@qDc8!#4(YXeo$id0Yi9^&BRMX zC59%~F$@HKfoS*Gnc1YdV*_N>7f+wok-T{Kr=c&@Nf&^e?Z|lDD6M>35MtGr9@3M7 zGdC4Ic)16>jDWyFr|u2(EF^`HibLiSY$OTbS0-Y3jXNBXw&2H|U9@`|yfG{s-e4Z) zI~b#!`Kj)K7jstF2j`{*r%rnNF7KI;9ZdC`HSg+fyNLS4B?{|~%}6WxJ+yNqh}{xB zsmP9ba%yA`-1a>sMTrHRV#^bL2S5jG-iaym*g^4yNsgJgJ5FcUwYcmzY_Zp_9fAHu zVFovQq_}SJm81?tmpY%NBYRH3Q)SXXnP+k0{;Nt7YViq(GT$2OEt$MNKB(Gz|< z+BcWM(8aC=r-UO65z&G2tnIk>Wv%HBancJmz8d%Ehq{`LeR~sV(imiCOcCgrO=1(2 zCnHHk|IJRf^*WDda!*iXWOw6)!%r>B4PdPp4fAh&I#s-&qvt6K#)rjCI`u7bJ){~% zHmLxPink~ke}|M76y=s}ef-`^%lLsqcE`N2Pdk@=b@6|kY!EM@mB8MuJJ-vo(IUwd z$oZ6zQl5_F3OjIM@3J?p!grv}0un#pb>Bq(6EIl5@Z5SWLnGOUER547$vKrhI15YR z;2Z##H=gNA8)7uZrCV<5ZK3@PzQORwM?+Yj>R?fZ)T1!p`XEqXePFp}t$mND=bA74 z${v-W*H)V{c4 zJCprOp#TmpIYh#CSMU41Gk-$>x+UEb;*hxo1sR3PA``%0eT6&zSoE=ki`BitknR)L zu3h61?sLAq`&(Vpm+$A-##x5QMOBg_2SYOAjb#DuoB7Oj?#NnsRp3BIHx*$KdTn2f zo?Uok55gN05D1QkJ_$Bu2d!@(TYqD@gTqbp23wU2B}3!t`;!7@BXB-I+lq*onTLl8 zV9ANNDR_AY4fAExz5ZhTuIKjdbpYWJz~l>rp+pKA9)pe$5G%-jo4;9IU|*YU#dqCs zPE1-x-J>u2t$uH(5BTw8d6MaJD5p;5{^)X4xp29z3)+exl$)^4HC?>fxpy2cnOaNs zE&DdKao_M8nkp3jzzZN3IFi0!%RvM5PLKIK9G#1W2VrZL_kc90u!SNMXxXEb&s=)o zc>Ko&ce{rd*RLCcw*%plr~8fxrJi5MoL#V;4`Y$WG3EunYH)k8cDQi#_bR`Uo{JmH zY_kr-6Zk(mJX~^d%(u3-s>+A{X(gJmKWqG-KB6Sa&4H)_VJm=XuAvPLR(+KhyE*P6 z$wt!g5wKH53>Mdzl2tnCQIVU2R&1ebMvn^m7$?!1+u~Xq1n~XYV!1Pl3=K%S8A>4dmFi+Im2mI3c_wi z89n6C_A}RaYheDBG4%m&pMUh_EdQxLW8uf#q;JFjeHy&jQ{^$UHtyOrPUs6K1l|0} z<+w3b4^7iG=xZ2*q#OMRrU@oLZRPA!+W67CTf$p`H*7k7tK5F%2zZBFW80Z@hm_-AxuN&&RUhaTh$5&uIq8g8roVcvV9KPIS3(-=a`#yApi2m$#RNvfv@qUK3 zz`l}+sAGkAp3+V8bq#(WQ?gg?W==e`$j@K)Rh#PU$Zgsp$0xfvcAMP8JHL>>*~~oA z-HC7z9#` ztqP7q3u%j}VAKrm=HAmuT~^C<9ZnJA zc^&l1ST^Eo*_Yus- z)X?p+A1 zb|N+F1H=>+;}EF%lvjg7fq0y;I;NV_Ey|7xvsrMKNA%SGD)H#c%tP;^Zje6>0=9Ht z=P7kF$iLGk%!*aVsdo<^4+W0@SGV;b$MfmYiS4XAL*qzPZjN zJb#F1J933-d3ho)B9aO=@rqY~r5w>aHrpEJ4WC~t?F$#k3De3Zy$s?`cv{1L|1~!D zJv1nY?H4n{i0D#i4L@`$t!B@79NxObsj&(wz@uPzReySgY|}Gup6o~gnHatFz%(E~_mR7RIS2$%*9@HxE9K8`61Gwn@tEKoBoZ#{_jn6xKHC+yxmVPbDX`CdO@YrbyLd{tHUCXuJ=&~2)n$wJz!c)r0fSyKn6%WZY$WS^3IQskL$ZrH;zmas|7+5 zUTse_C{Xu)s%8xQLY|>h6!ctfJ+rSK9#nL7|6jWHq4>-ggZ#%q{u~-{jMV2cmD?Kfm-H)!mJHp}Cp@x-k*LhLmu;;{tao>U(;^ zd`V_ME`85QVx!t|6X!Rm-yLt{U+_ogJ6`#|OX}X)Gw##)faKC!`y-ATCA|&&0ieHv z2+Q9ptd0TAC~Mtj35iu`{L3(oehVLFl}lf40ocW~&(?iby2EA;OxdSemuZQ>JuXhb z!qRe&g|}$FX+90M2`cJJh=h+NoWoWnwN)=Nf?ncTO+a;WVC&GCv#77g#_@Z)$G_%W zZXL%>u*Gp`wTyY8bR)_IY34?Ht`^6|f|u()*kxvzdbqX=1Jnn6DM|9K=0H?|?8E4*4;M`FE4fQ~1{B(WDWrDR>2LnmURPgfX+ju2Lr5fcm|GVFyDZ zUYYwj0j@4DkP2z4vm&U%3C;>WFgNLqtJZm~0LET{jVd608IXSQ%t!8;N}EG?fUDNY z1mGtC#Yam6UHbEytje@BsZ1oZw2akh$SH=L%A!B;+F7HZ`DUj!xF-Wn(}yX{3X`@F!cLZzU5D{RI1iy4ykgKQ+99xcAUPzrPR2fD2$8qC_g2BD&Xo>d5EtG_jsl) zIG(Qh_nCBu>ty_it&gEnMbG@yuAGM>&okQ!&u+6X7~Ls@6f+O#$cXj>>~mAJ=xqR8 zQ_7t0UAT}e?Tb28uV8bKU%=WoR+&48`~RGTuHY$CiBBIq*oZIA0ui80hALFrZB&v|pSAUZV=f_6MM}Gqtzo0> z%#t6jpGpSufM5UCxM0i?|A$2Z4d#KyUNq)VJ| zoSZSZ-N%GR1oWT4U2jjFJh{mIn8lysGpk@Ac zU(OX1eJuI`Mm9Ff=}r#XML9A23h?G*TC5w2yxdV~G*4PGBI=xc1Hkl{Dq6jJvIgvr zFYET9i0(>8EhgPA{f$oYFj4iUzwO&x) z=M#?p17z-KOz6;(0x<^p!u8k27BGHABGhVv+~Pzoxi3=(@H@A;0)XXOecGEhZd}As z(GSlWz!~Y2%VBZg4U5HgQ^!4ZnB`1BLtqLIKLTeBEOP0{;Y2J?z&PgtU|6tlW}e@s&4?iJgT>92kH(~d zCGIR(;eXW-RbqL4J;3s9KQ4gZ&@qi837xwkU z-wEiV2TH^Sl>YQ1ILJcq_>RuLLhhNY%W(7ZBaaUyIrqX8NQEddj}CWc!P#lWnys{$ z8K#fF=Yyt=6ewFvD6lLwI9&*i8zG-zM(TkyEwLxt?!OKQ3R3X)F6^>tAo?(%AiCzS zpld7rd-dwhK5&6sr{u9#)^0GnpORcO=+=-r^V5yieSZAHH1KyYb5L_ftb@Lc$Wa{eFXoUgl8c6lLR7dsdb9%) z#u+IG+nROG4JSHB1c+S?q;pVX(mX`laeDEp&3l;&b^Q0u0J8|Z>N7uc3M_=j^}is? zcnT&omT$FglppK6e72yUp5mD%@<&0#-Qw%bK~UtpYt)@?2CQ&!SqYQ~bQ|nYbiV}H zSgq;iXv|29;}zf;en}ob^D7T~8f^ifHT)I*16C`JJM?LNyhTQ<5tHL4p8mN~{ctX2 z)9tz#?IcK~R;KDP0B4!lF*!BGZfpRu5q^pz`8&8X#bZ0z;#w}l(FKQ;U~y%(FGEL< zR8-na-X!{6!?h+`nsbBs=#IGwEP=pQB{SyRS#8^=kDz`h_?EE^`3}X|mgtrsxQH_5jF zg*@QeHP#*uzzrWpM#PIlF)IW=E$Hh802Xdx&*$=Nyf*K=dx))Gr6bQ~>)rC{)8J~n zd?^pSD`4o1Zs+9QV-mN4KE1#~1)_*o(Dy5Obn?fyZx?V|;|cQ?OrDFL^E%Ipi5(Lg zn_%%3{L$T$ov8;Ulhop0Dy-gniOkBu=-}LL#Rx8kKw3_#t`%sixM1#VwWLR<54v#M z2eWJxA;kFw_Z2}Jp*$i809n^U(eA+Pxqd z{lpH;1b-eV1$bE*`!i`QORnG!sKg)&_Z%_ua{1gyRAo5EtFRroynVpWGsEHsX@_&G zMZj*sCeR9qKvyiN5eYjJ=$KKuae|z`|Do(T!ncFIwI4(V5Is2-2||ZuCKUa9Tu}+& zFa(Wf71^k`1)#_vj5q;&e|SN7z&dNx&AS$CC`j@Ndz1Kl;DZHHO?Wi0PbP0fclVp* z4Ip7tkh^;o+9?dIsd3Gd-+@f<6A7)7FuK8kauIeLI6fQlU2One#saKDeX!eoEHZZ# zVHj^A`5_E@JmxYOu2i@VKFldKkpuPTEjVe(EP0LW{r|Sqb1>*XQer zENeDk9Yi~eTbmy)GvrLg;hl*Cx)Tggw(Nbd_W@ma2?fG_CGbia*RBP@jVRUo(J!-Q zzp1OkLz|-z2KX-kAqcx;gbPP0d<#g^(wt8L)lxp3pAVbD$LKqjQiYi*({*8kJg4~F zwXCcHGe3!t4nWM_nku_%WW?=_SjOG1!>llrKvmHxa`j=d_6a6UP&u=H;Xp@;kt1pa zXRBG2W#%1s4_)C+Fup(RJG2PZAMm`uw|o0$i!Gd6eC9kb(lKasOq#G+yfyiyr1T44 zSc~)Fh(L8_mU+k(IT{{XUAJgA8#kL2{Ioe3?&amxA8J$aBB|Y7`|iGC27c-~`db&W z#AIXy^54DbX?Oy(ICDA% zWwNbnetf-JkJhr_fH6yu?q3!`e65{}vUug;HO47J!|?oAg2E0EHK5TXi=p(rPI-pj zU$N8Jx}|MHPjBa%n9C}<(pGP5K;50_czD<{I#1`N@wOE#bwdNQ$l$9vGWHnKAi zx2G79Dm_h-aj{4qtFKqKva$;9hfxm-02YuVVGwTOAF>0`63l&Ai;;;;mL)E`Gvf=r zV7Ze~bw%srIrm5yMWM3GFI|#TvdG=QHYNJupFgp$K0p`&2}XO@8JtN3`vcKeoHY}t zGshtzt3Bf{$UbUv?z9Y2h0w$K0SS?I$Nl?&Vp)xy!wcQ8!ggpdX1 zryFJ8;amZYcI|El3Mik6lS4OAFO#YQn05sYQ33WI+?V|LAw<5D_3TAwt=_;ksHP$G zKj>|k(((x9%O{+U@psNYz0jTPvx}Id^=8r38udVsOuJ->DQW|3A)N?v@%Diq%E}M~ z{uU!E>m`&bYLQ!?c=w(%G+aw*`+aUX=E4l+wvf@%fX1?3;WBaKd6IDsPi}rP-FH3e zP|^)Rtd`<4QJeW&bZga<+hi?{+5(RfgZW{Zd+ zAg1PaU4Q5-K2^e<0%;NxN^t3#6CZlvXGV-z&u%?=usheLMOw>TGVc^E=n37am}IakM)H`4+7w>V9ZZNfeI`D-Wgm|iT7bQ_Dr7YtzAynW!wWBfZX8Fl*c-p zfC^{>Fi`lu3&W6qy^xyQ*v63Y-S9#(TDh0tHKN;%c>ILA_UcALjexV%20_92(pDwA zE|p&p6%4AcS~=q`3Kch`oj27irmFw@$jX5j=yWa#zr-JWqos|k$AM$x3Gy!a8ijT4{pM-Wc2C$KAF{)JCJTw@dR0;RQQL*GABlM#xzOASf2SSn6sob~v;!jAC6($?hlEme zRm8$Cr}_oO^#)9LwTI^i^z4jBJUt&eTEiHT!;G}BEx4-m>+apG`}<$$ImGCtI6^I? zW=_3uzaBwRv876K*+XxE$S(~(5&>|OF!AELrZ*X=B#IlH@L9?1dXHTp={rBibr`ikAWk*@ojcjdOc|b_i(ZjE3-dW~nR(B?l{AOsH&veK zUXgsiHG3zo9Eidoc*d^?4ESq0a-gZCf87in!>+`(84f&ydrw!@c}OGynPt* zFPUvlqx%$Ew`*d$N8=yQwJm3^uHW@{YTIhv@kmIAR#9F9wPS_8+%2qWsHZIoc=mt^f)2&*?p0JUY&qpQe*haS?zvTRy7kY_$^OMOVc_*cp zp>Vg6Sr+#tr1>V{i7F6(AN27>UAHV(X2S7(Lfsk~W?l9NZTBx05Eo!bBfzXZ^8pg- zGJ+-pk5=t>Rslm69W}M))JJI;zi_Aw9Lv6{=VkrSUXOy{{?aknn{{!Z|K z?9|-E2}UN7Q4Y4nEuCD1htyB#J~FDs5CJ%z#ZUZWdhJI>U_kH*uNUgsmB8HhL;aL& zCmh{es12JJGgQDtKL1zL>a$BIXL2H>T)&6&)&t|KU`J_MKY!G0u|KB!l6LG8IGd@h zILXSr5ln9oJGUHvvgOVb&W1g#>D?{wOW#}wNOZKWR$rL9y3f{s6~uTs9e$6Zqsa^$ z%|(i%y0=0uFFNz!N}e6$g>e~XD&~;oAfGx#L8|QL85;L09(SE%N9R41V;jci z3D2Bx13_LUI2c%Q&j%!>j@aAV2RjG(OjK~}_R6*$8wX2k{no~FLrZVV>n=>mjkZ%A z&DqDKYU8$WAX4LXy6(tO`@O|FV8__>p2d$^g<<8&R9B+|LN`j$>0|ct5`7)0q<83> z!VlIg3c>Ik*fx)~f0J)Hm+lLVaE8~kJ*kY4VX1=o@LFc!JmJID3~OGo8@_(GUNDEY z)yZXdgtq38B=J{@JfPJ1{?IGlo+Z$*=c_L}!%wMhS(Eu$$MZG!{KJ7r=ECR0E>R(2 zH)G3K6gr%eJ^1+p-_!40u~Z?R+GaoF7Y$xEW@hT>Ck(1UmIWRjMAhgpIjjq>X>$;Y zvbxi~=cd|5Zt~x#Ay)~yc#JWcJx*lH+NNu{HpRTZU!Q{Flv3Mg51~nFf)esJ{P^fT zbrtL)b@1t;)Mvrl0Cgk6m=_qYTZu}$S9ccBUq&9DO&Y?t0KEbg@5e3bc1kfhIY1b| zH#C_bpH^XlAm%y?s1Y1wx}QBIr|lBPp4u4$H-#^%+5{$goUinLdbXChRG4JHuwc@( z-oBx{wV=i~hiEr};`Jip0HfsaZzqiG&)t}um|(iS4aptF%dg<|p!3GJ?u+z%fOOV% z$ShorN577Ml$hUKl#lU)2N%fR6Ch`pqrvZ+B9f%F>~&~@iAytpDYQhOlJaBhg*yra zECP{(5Qv)^s+c2oRSyi{hz{U=1Ki`tOvuAyHz~>Sv;jZ;L83%huhO^N&PjefS0Z#H z>765!*8idNzc;sxk*uXhmpzH$EO?OxQrFaM^Dce(@Ime2FT1lb!{OzF4EIr29j9$B z_mDTLHb|1YeQW*`=f^8EBg`3sF+Lo7DbZ~vl_NGR4q}453<;HT>xbMeomMY;b+vtM z7h(hNZ*t3$aF;<@QA%)_){&H=ogOxG6o>C0)eA;88|Io!PoQA}b%F-QZ2Lpk?Hy)wb_D|z z@LVte1dxgdPoM4G5X`4Gy;{C%m9nBDE%FLT>GcC2fnL?KbLmfaok-)O0|a8Y_kq%H z;?D{>fwGy|qpqLv`44(vI#FI(X$DFq$mUGEIlt|7T(`;1wAOa(7-uW5)&7j6X>gB2 z;9y6HRnkgpDddycy_*ve!hoO5;=ks`3o)4Wbf33bSZh3VOlnix(%1@mgWw|;8B3QL zC7o+HVLk{k$yoASp#6Q`hR1WizLd2Sy#0`93_hR9jJkhebxn+B( zSezX+sM&S7kg+8=Vc@SeA>8$KR`9c*_TAeR#xM#?I}+mjLtg^3!+?2GJy9s6UG}4* zvDRH{cNpE#^;@?t6P-Bf9}aZ&j&*r2(XnvdTkR_W~3#>6Xq~( zp&TqM7lG^d0$0MUb01j4+NPeHz@Yf@<%_4gqOJ(*r^a{&=+iI}J_(2UgM|{-TR4-w z?Wm&REkZ66a0{5w|6*vQ^YfK39W~|@7$S+u8h6Z)rG+cyfXpq7TTy{QZl#+(q{T~0 zC;TzTi9RKA0t0#FXeCa^!Wxl@O&(T(8kRkY zi5%)eWpoP_u4tmm@O(yCjnz@W5nFk}K|ya(4&k|YvkFGrh_+HxQi5jmeKB4%R!$j4 zFEByD$dk^|%QuT7oISJ*GQeof_V>=38XCLr{24; zH)@fEkbJ6+!P3G4OxZwpJL(s3h^QEZk;XNy9m$RJkSwv?=2kEY7_L0>O`b4RrR;WU z>d`fC*vk0qmx`*Cgw{EMxR!JGq}^p#g{A!4Q{u~%@(yCF3im*wQAMZ4h_{Mz?^9F4 zYUGw_jyU2i_@hWCzzUHLH2{2ogA3_Gu^i2g-ZSWOtm45GAIAni!HAwC6#%IQkekA} z^nj`=VI0Efs9T?n3MV&)c{D_ej!4%GSdbBI7n(0n0CB?Xx3(4oG(`v%V+qdb`iKb8szP``Q6)Z%pq9np z$*jD`J_9VK9J&bySTW5F?OWK7RkR+;Bbyg6k$3_8Noz^i@LE}K&PA;}Cj3}6 z3KF+t(c%FohIL;#obSrN9wDAHI6%X2$wH^Rme^2eCza!r*1{!9E=c(DO!`Q;k`*Uu zUP;J8L@9?CjQn0YP*n)(P7Gc@H)TC!|ABo;{x_i%U|VC@@WN!v2RJ(r4mepbAqr~o z%mhjYHxu6eSAn$=HX$I<)q8(10?sEN5NvDvKuA+Y@kf3v-UzX`zy(6kgZGBh+?^>~ zf#l*=AOle{h9YBjKwSINTLDHaL5T-bnO)_h%T{puf!>ZlXZjwWUl;&eov*HbdjBJw zfni!ijW!;obW1*+)j99MB!gdBVLoCqhs*i-mcd}2>Gok5gkkh~V(vIba#&YwPc0dW z7v{a7aLn_X=HEi*)RPTNBE*o5m3F|0LXDW0UL`gxFXc)hWsAQ)xRb>^f(HYlLVNTZK4}cq%*CFWMPB5-Q&rbsL2S z=Fno#`(UQ93xgI6OacTC-ZDWKz~;n{+5(g2q@T{_)eKtwH~yDATNrrRu-2OG`n{w2 z-!C)68pgjxEMxZtu3*C`CS2|g0!bTB>Cd|->1rOYfBx4*dyDg97X~8oo&6LW5E`nY zrbZ9@cnGJtS5Pc@4ih_+C>nfCAP9=66o|7D4qA8|GGXbFg@W@9$~CsDljonBeSwN; z89ux+G_?+cwQQNE0Rkx;JI0CKVUWOo56}$BAs~DN4PGn)pW7Ms)u# zu|lvH%1{VWQ3z)TdyoP31rPyRTAeYEMjV}s-vYvC(7np}*ZeVJ#Hcyx^|fhY?S?oy@@ylMgZBlh?i@#OCY*=h!f%0fn*(^#YcOOp&ml6K<2Mm zie=O;1$p^l?=li1LY6ie4io1ijI4otEOs#<%plMr!f11cbHU_}F%%`}Z{T2}-|I@W zxfnrhflqFt4qML#Tsb&R=?synfG^03*nD=uSPX+wf0*iCJrWNE3yIqy+&MVxL9M3i zx~dJa0(=4jX6?06?Ze_ldu7+W>jB6UTDcIrrS>qr>FjG9MklaXZaec&?Q@ zlj+ZJWMbfGBn((C=%u*U>J(%TnQq~PkAH7}|B-&#Y-b=LB2SOX>E1Xs??r9EytVw{ z%-%j-%yR!!-1( z#?3Qr1MjhGZ6eu6G0coec3lLD0s`)ZW33Y}*;RX$(UKrFLTBhV$yNb5BNliEsfPVU zA4nMH`k^v<39-vI>I{l}B737tB zzaZ?md4;9v=Y@$+pI-p54Zx`?{e$yQK2pkeAL4&;;bPs#$h!LaG)>w#y%Z+oJrI}Q z)H-`tAhCTQe54t_9cb9delWPB`J@jz-Nm-4(iTU=u?&;*$ghE2Dpc8N^}k5YJ;}qM z@S%SLI?LKh{HQSHtcnelMcOqgC$R`M!h_yOv3}IKck}c0vae$R)tmgoybW~wiPJg!o^GD#_t;TD5O!*akZJG6uieXJ`?|INfN`b#kRj-(7km-u7k6@!$NCDaLU@ z8=N)+UTH?p1qRAW$~dlb{tf-U&c*PVjYbU$*4hQs)o6qr05j4X$DxhT@1J|GpZZR+ zlm^s#!jpr2&{&4~^&!doHuiT#Ckvy9QVw4a9^;H5NFVAe_&CF6R3r@;iW0PDy+cFG zFtH-DGpuN~DPFk5qoYpVtBiRa53@7{N*@vlLUfQI8-J4-djR_)@x-$mR!gnY(*4iz zvB?$Qi`7gJz5!eKfAr14L8Z2(XRf5vh=^14cSkIV%lY<6?E;QL(#8-LRL05IT-Ae4 zG6)CEg$x{gWI3GKL%|X#yb({G2~jzj&y%q@p@u=(Sg!QOFL5hu&iZ%a#vnqs#jomf zpvJ8P{ptlyI*3TfwLyG);5aCcH!Lnl)zs8PKuNL?(fqC@TruC@BgU&X{aBwY?q1Ju zJoUm6saT4fn!V@~z4J{1VReVb)|{{Cka%I`*7?Kz&tbmX)|+2N6_x`~`SEO>LiZ^7 z+}v5?GZOW5XIrO&swK9XOms;is3ClboEyojQrHCgEO8G~j@^&3A(`nB4z4!x{eI5O zn1a;Gb}<@qoCS?n+mOB0iP%Tug*Q2EuQaA<9x{zqM~Dzq34e-18jdxNN7)s z8&~l;xkCZ1m>Ki2-gOeP#eNXqR;{@o5Fpc29CTbnL~{L@f6Se(eE_|62^IvDfhZ6t z;|TETBg4+gV({xW62UGCJzOw6AQ6ZidL$*fDRm}942JEO4*dCm0ySfn8q<#z;qcc8Jb?EVjk<@ZT zdGKuA8Mc9VSGd$>E@_5xbi?6OPtGbJu}eG6f^9>#E}!hZ_ZNBktv@vM#DNU5GPZ%t ztVoPV{Vzmk^HRv<85N5v&pNFl$Z)uF8mxX+0xKb3>K%*r$Cwcd6G~meIT~te@@}aq zdVuCr#>5rtKh&RHpJ#YkZ-4zBw?!&qSe+86SZ545u3Xy-LY7e>e<(TRdZXI^__pu) z!3muIY`g#L(;%}=z{=mwe7AHC!B30^;^*330$o@{CBGBOFU?U;09tvSvA+M@GF-22 z)qMs5ut?vpV29Ba+4o^kFNCcVGVrRV-C&L~-G#>Rt%|1l?HwJT{5`Yhg z8Q!4RyS7iGzbnV$q@qAzq6m$VKxk;9EUqoqC%UF)wz;_a{y5f>*LW1AFM938P4ccl zH#Ml2%+4q6{c|}k6P2hNuCbHPk8w{A{mzFJf9%`ceZ=NJ>0j z!hca*qlsh9lhWKf>>f5|N7wqC6CnLpfLyD_po+i}5Xf({E*hRcs(dNg^DYwh;k~$2 zD*+5sQ7|VZ?pum}nGAB!J`<7zB(BIlA{niOvI0Z! zK2Flmqth8lPLR_H6dv%0+2@^IX$&Nk=4`*zJ&=)t1gkgD+UNq5-M(&}wwrTii`WaR zO!@fh3iY3h6T`ymEUnn$9BXfygq3p5OJESvas%-$RK&lOKo&#>NAZJZj);k2;GOJf zoTVQOaN!Y9l8{w565l~}l6>Opa9+>f+OV^tWPWlpjP?mLpF-Med~ln8qQtlE+y#Rt zv_hI;4`Q6{`Yv0qP zJ!R8B`*}0ov5anPUdw4T|K!i68RlI;?`aN4fCGmPDZ&sAZ62Jx8W5_1*%lQDHVO!2 zPf-whDlRUb(mo0F3EnH6>S^@>#G5tzX3(YQ7wFv$pVK!kIgS3(8 z$nAc3O0#VFf0We4LyHOkts6JtC`z(*fNzsw7i7|JF;+TyMM-DB`74GRxY`cVBhyq- zk@#Nhlu0_Q=RT!B&nPR?{Cta2**r)jCe+pe5V$mTmfB_X8eA#c~LF%6_f<+&!$3*7~A%ERykiCL1^hsxp z@E<67P=S}QT#v+1mC`skC;Ubi?p&Q}N}xpd!Ll&`FVqc6Uy*Pz+}Z{&Rx z|7So4gs=>%QKRtT2j%495U+jw%nRDb-Ku6mfR5zYuC?ROYiDChEQM8(!>7qGaz$SY zI(@#{MBU#G?J_4v6ZIMOFIUUbLy4OaBU56MnIaxja{BD#kTZ*@sO9%-OP4HpuRgVQ zj}t8@KomdXiHGK+FQ1@_B&87K47eztNBIzozOa6IWFq~J*T2&mtl-YqA7@ql9*VD0 zD@R*y&;K59lC3{r1&1$8*uP*hB64U^3YvMF7k?obpt9#8rYR zi_7QppR+>$)jorN`2a0iURR6CNEMj=G6T)o!u>Zo`0_!3qI&wi;2OZ1X%$cSZmCE4 zDWdPoZoLWa_^)D@JmavqS_aEgr}{BMK%%&g3W*TbGDnni^pti6h5U{{07BKU=A^&{ zplgy8C%N7To7L0rfitz^p3R#N_k9W&PK%C?ZYRmo^~*0AB1m!8*Y~jWsz2L$gYIoO zxE|_TX^j7mDZP8V9X==be@BwvL0Wwz zw{pLr@M<#&w8kvpXn|3xp0p^l)6Uom{qM@myoWcX%P!YY)G_`qnV%Tzp6vhoLVoZB zPzAs)enOE4%y(4eI-H_Xa}8yUb&%{mV86J2u^b;={swn@oa@ZDq4zp5u__%lzeG^k zmk&7Zeb)l2$v!90WqDWxw0N3terd4Uj>|7nJ>FrgBl8)&@YEIbeMF56&(hOye65te zUvJuR_m$d-`oI2O-S1sik^cc$bN!~CJyCD*JQ5u<2mK)*90zGFsOchzk;Gq_Ip;nh zbpN&^om>bo^kpgsb+-NE1DY5kGb`o*APGkc4TWq`ETk*2l6wS11voG*h4gK?upJm3 zD{=p7c#1=a^B*qo zE%$6%yIxskn*oBoHe<2x7Ej)){U2ORv7vMzN)CpG8dWfi9py0XCV>-}kC50Y3Tf~$ zxQ}T>USq)GX|g}w%w67T?|-}j;>u6Jh6x4-*sLJF;2Jb?j=#}j+f7TXflmr1##ACB z)X+~1NBRXOn~t2+jes+Wu#$WljNjfShA1eiU#t-OFqyUdPL2ds#Zu>@N)hD8&9>7L zqd<0X1`=zX=mtJ1DUm0JKz!u~510_yC~!ZGIpT7bq!?k+MxnsD)yV@LN$Eev*E?=j zaZnNaGZ^ob!_bWH_QKcC5S+6UlX%RC?A%NKvRNp8n3L=l)O&QVjX6TsaY(ut;3zO^ zJ)^?ZocV@&)sK@lWpV47R2AUKzUX&!L}1N7Hbfz$GSRUiq$I@Khhje*_3(yhg-{E? z3LYIJ7xC3P?4W}2fq!0G-ZJ-i4*G4Z}H$525ZiYvPzy9IaBfS5Hm8|vue?T|$O zYP4u*|3GF935IL(z+$Z+oNmZSNvVv9A<;MEb6oplh0GP%~Kn-I6En+3HODChPQ8x%5OTjU}c>Vfy>l|t1Lz0wY z+?NTBQc$7E{QC0D11dXAgSc7UT}fC8?kyS$3}P?_z#n1+831}q%nNkAj+okkjD`#W zl0kqC3JnXiQy4$v7yOAWA}VAaT<2IUDiGB0PDF}|X$L#$O;LeF%6&oEhT*zN!@KXU z`5$clH;?sc>GG^fTLj6V8vmRTG>j*3)7y~W`W~Lmr|Z*V6tZQIqQSn2sQmKYlB_t)q6IP#7k$x(14}hGF=oZl3!5nurwB}?qhX1agZMz%^UP$BW z>oX%Z*u+Q}(|kCzAehbsxpz%EpyNg;$&E$+^I+b*;OFDt1xhJ#3m@GH+OEdc$NYcv zH7oCZi~k=^)T#B?_x?Y{zV5{UfiMIUHzm1TS!dq#*<~YV&yxL-+Tl1zsx4tR9!NS` zD-Ku6F5RqV~oSSSpO2VEie5)KV> zP(Tq9_SF30F*ZzNwU6+$MsXy1s%-|yyLfdog$-RVkfQL5!Fmkk8k@WKpT5GtXZw=x zt%82rnDp$xLwq&NA^I8JxavM3rX431`CZL2f))y#ux|}L?EFa*uR0*nn&6)#Bnx)< zZKLBXuwR8KazKR)8Mu#zuwWvJl}$w%D_QU%qgOVm-`f!t2!cb<9pWb$R{o~E2-38U zbDbwhO+>y1jw9%BKC!@{rA8J?q6qv*5@kQLm-jB#x3lL)u}%$^Rm-z0SC6K zggZq!MaH&9I!e(0IEk0OArzq-%1k0)A484u^_@ zNr?go`RB^qywu`CHdsF?pqwz4g4SQJM5P6@bye?8h^vf2(%%{_xNO5 zPf0;~j`rG%P3cTwH~;OakvI>#v0U49t50|_6be$7VY8{TrydL`3I&DXVo_I{-0%E& z_?y(&KP3bIn=Pm*NDet4G^?>=}!r|#^kl0YY%h$DbxQ^ z<~GqZKdX5U7dU5~ml~{|IC4kSp7rO!QeEp4 z(~834oHy6HbSP+_wCfY7Ut z_Y%6aMIu|5|3iqo=N!_+NqX=<$ubE*+CH^6u3ld(c4@{d*1v>n(_9PP`)H@ojJa?5A7CA;0@09p+dMMTe zs>&&zyTtgY%(a>8%W{8O<-zxoNxN6-#jX=w%JW6k^G>kTv0%@*dy`wZe^fu`nkeTe z$UJF$?zZuW!^LGW*FPLut*H3>x<<;dd#4GwquJyD& z@Hru1$+tX+kEg7dH9=Ctm9gqWb#vpNC~J}q@+ZY8C*krcT52m>ThtLtxVX3&>1Y{O zEbX2C1y-)PMp)F7{H+CL4cDaXsIC95IK{oCL*ILm0a=OZ7Cq%>OS+GL$-?iiJRy*} z#(l``4=F(2Wb@uU{W0|gIdJebD^k)EdQc*dcG0G_)1^tB+^hY?0oHKG zA&p$2NisU5VNYuTD%a_k>D})*s{WRUiDyii87F~#(|5Xc*irnr?Q^M6ns{$+{Y01E z+L0;{6;iIYin^~L-D?ZyAq!V#{;tlVyM99K@k-~6=+0}32=J;jS#8JROY9x3_uOC< z6kbnzq15F5f?cpO4J-$uG;aU6*lf)#UwtmQ!794-+5M&?@-f#tTYH81ai4f^aqCa; zZWWc>2(0MKCGv@P4Nm`Zd~bQ%6hSH!KbU~7LiMkcFe{43U5?2=Jp~K~Dtov;?i`_8 zPufBn3c^8SNOX#d|&v^i(|3ulrB-Ffaag-owszs?;_P8pBo{hjzmfNJu& z93P|eaT(#1j*j9S7k8oP(+b-Lzmhv2mnhZ4H&^8N-X@86-n8HobcBO96oF;L$6Ig- zBu`5(?w3t~drj-epc7`3K^6SIyEs>Mqd+-6u8y3D7e^O_-y>b$9q&pzYG=2JY$Z63 z-hVB$Z<(dBxSf)5PYI7>K^O7yuw{^07dp41(|#AcB@2zxuMJIU7>PXB-}J+?=-P41 z646n|pFP-vBtuG76+t_<-b_#a=AxthFzMMl;qk#ms@Gw6dwEKu&`)pK+%Eg~?3Z$K z8j8P?{eP+15hOY+p{KvW?RxtAz~irZ#jifQJW5Wc_bK6poR66M=%=2O^$GlRfLDJe z?fZ=P_^;o1=iqmK8BJ!IT#o&-@7Mb`8HgI7ywE**!XHG8;-)GaRF*<8?9)7PBAD#u zT!iO>?X@Pee&Kh|0+Fy&#+x3_KkxAgovL#Ddle2F?EZE*`4>q#0Y7vD9DztMAe`gt zy3@j~+oeCB+3qG2t6^hf^U4yI3(9}gbKN}dPJP~QZp{Nu z=1oca_b?gibzIsW%=7wX=jY$1%{G!lZ42r&X^~cAS7T2d!6UY^wjS0#{?x}*I5EN^ z0z5!7Oai;aICgIx(fN5uMQ!WF@U?vt9s_|E_+V-n)xKTq`Ff>w#J_bUSUM(d8})fP zL&x`F$;o0ry^C{OTdlU^ab>)>#WLD@R$Nd}NocvK>J}_qsVIrQnDSia9+j#*Co`ix zqGz^qu{b>%f+7!5v939Fd+vpQXz5O6xf-kXBod14b>n=VHvhFR6DUKc71 z6fG-Dw=$!3WK=IIm7^qe|(Zf`eS7VK}cNU+k^yeWu}k8kQ2S=5n`y=Jn7FlsNc zXndN|?UkpSS%`aelC@jNLuQ>vQFj2XTDE`iSR<$qm>k+sJD(s_o9eNS3PnJP~fQP&a=lYA2&nDd>61i`> zO@H!u+#T;d#M70|u=FC+Z^rn&N87F9H$VE3w0GTVtb8<7WF5N|bo^LD+&IM3av$yL z7Sek7{mQ;MDv~5YA@0GuKi)APhb3NC!RwZz#=$U03>dN0B?`jSzyyVePfZOZ*Jey_0Q7g_fwUNdwTC4 zSEp0mCW@c%0}tZ4kzDL^X7xbME)R>H?vaX3-Ba*|!=0mad${EU`|CsrL8{jebOKh4 zEG*5NZU3^lwx-7c?{m2-wI>L6y)IG(Co~SL=`d@aIvmch9*;=3V;`(gD)H{SMxD?D zQCl}{yHLu9=l8a;G5q@7ec*I-4YBNIiJ)aY?4k3utWN zyeVwP8ffMF+h^bWM5wPzL8L{r08?dS7j2mb_?{$zckjGZksY|{MJJZ1QG^zLs6(r# ztE)@$D?JOl`ZjVaE<6LcH{)e)zF~g+%08Fe*GNh05?W42Nh_MtCwZpCP>>Wf&h>ki zExowD@4_t%dJzyoho={KmPYFuRL)4G3iUcScq})V4XGf`~v&e#L1!$N^~P8_&v8N2zQ&NpRa4Qg%96_+mwbO@H! zG$;yE36}8{bc`zF!RFL{LUGoxB5vnyrrFwvJ`jw)j*2mA?L#EkH9(!csX5)l4fBFG z`~v0^;PL|U%xLq@%#a{hoRJxb;1BDlG^*}hde4ALje zOW{5i0`2&ZPd2XZhJUJ$06G(9#-vXq0lnaBF~j}Wm&u_ks=+@^i@UF&tNY`O$5O9$ z=`~RS@Au}{#(kdSo7+1SqNJ6-W~Br#yPcRya9Pkx*_|w!+XJPjqE<{i3}E34R4f0a z8YrdOeLv7+LhZ7Y(A9g}s3x-&1*qcV;^bl*e#|T!7ou|2Fjh5_9(nT2vw*%iq_R?( zmYNFkOvN9<2H2~Uo~X27{;_LtDvdoTzRTdj4;6eE*ioP=T|JN=oE4h%9T@yo=-PM@ z5o#J*w3Ixbp@}Nq!yb1DfCa9v7OoCt1p?cF!{hGL;f-F}?*CLz0hH*k4;>SHFaV<} z`B|}K|s>sQ`F%y; zYkYrE`|202?D>**@akPZX4k%x(I`zOkNf$Z-@&5!X!y>G3F)`te)$Ju8w4r>ITqh> zK#x@pO-%71Ay;gBBXbdzdH`l;bIvl~Uu^^T>{LQ+N3P?B2WLO7d16!~71nZyKE3c8 zJ^uaf&IPYp*|e0Dk@$O)n?bA_Zcg=<5@&j5%B)pI4iC6y0Lc{+osF^nG?x3fJ7szy zz^7e9d?Ik-V7Dm<(X7-NUD@Wc{h*2$O=HnrIrG*leMZxb{Xvp3A}dcVI4@!MHePD{ zp*s6~^nq%-tDfz0_m+z=3Co4AUoN7xjY?2~`7QY`TC?)ZN-E4@w6p?Mn@aD{Ht#vN z-X|1Ek{YQc7&nn*j&g`Dyb_faYY-#bOR&oaaHZ%@Q+wzDlcDKjC6~8RF=?F{uK)LI z?KmGsPx1jt0uX8>59~V69mnH03mCcFq;2m0-SunuklfC5Rom>_4taK-tLL;8wW;J~ zzmw?$sE(}2=ZQHgGp>9q}_z#UegRCAh55QN|qC5ZbE()5(%;IwcNHmAi(p8d*$DnJb;2mVzrIEedWqsofHa@%6Pf7PgyiX-Sumi zUVkgGL&w$sIJa~|Sz@c%RGfULTcxCj^~}1$@1wnmBQrNF$eWPPIKP#J(P3a=D_koK zECtFA$<=69DVj2C4mk=%&nIjyKjZ1?dFJT(d@E%3z!2)dnWM4BoO?Z8&K!*~z9;1> zhOLm;^Z7p>tpMM!DOpLiwY4z*kOXcGCSmkzDGVDUU_bGD+9?XD&iwoK8E54u#}}o1 zOVzU}=-9bySC!$(o%ah26UL-w-ApcSeSPQH&C95^Q3df^91H$Aov-lo@HJXK3LWjT zcX*lCX>Z~Lh|!$OxjLebx|xO_Yvoy@fWn-R*z&~9ui_fRrqWo)RtQl|?knOc@hmDX zD^@3m*D9RO6u;V9HSn4>BtVX?gp&=0A}A>6P|!ux3OK{EZQoyb`gH5_?Cd`9l$1a) zLnqhkl(z-kt3+2aDHo8WgLGzoGa+S?>$vS!y0Xz8pOQ zPD@PCuJK6kOj~fbr19q{{g0T&s3_+U0rt|jw28?B@Wd)hlFshl5{8dR&%24 zC$5jUZ-`so&P6pBFThYwIssbJomuy}|Bt<7XSc**fOe4Ws9{6zEc#m+oFiu` zFRFNbui@rTP#J0EZs(4}p!X#J2s4Q@R#pLdPj=n#qqmj%QqaHl=Y{p0uKHG4VKl!7 z)z{5U-WVOd)w^1->X00b=yI#**R;(iY8>uAvPd+4v?31Yc*Lht0qn7y{+wsa*)i+6 zF**awHiMMX;q zqo#BT$8*(g!Z(zi8!y`_{GL2PsKpH3-dm0ec`H7{X^ANN!q?NU;j$Y9y1r{_B`3UC`MNoGC3m&%RLii_8Q>L8n(oXr1V^QKJ|m_Rs^n)T;+ zFH;?n6h6_)FPvi{gpi{$_!nuKPmN8QY8!UbVpas^}|0HvH^orSI=dU)skFyvNG;+}zSqV=3)EMs6KPaNEr=XSA#BM$#Otqx%zklrYuF@?L!vd!i7$`o#3~&|^91 zA$Q(i32rj+GwZm?RtIJ21_?3MOPV4#uNzb_Gr21L&&iI^RUV@XTrd_-!I`=#{L|A+ z?@sY`kA`kPb!BDc+c~3S8-=>iHA7T){Za7z-M!f^9FeJ`Z^T&WE;}@;3Y$XgwRIIH zBiPNb`}L-%XpAwe$8`mSi)qEuL6f3yIaU4V(I5Wl65k9d*w~0;1)wMO2KI-;@8!s3 zt>vidWtO+trqBbSHw1#u9;7sOBN=>eh3{QG;1}o?FusAYs6m*54kgPfP z#GAeL@a;Obi32))%BMu_G~J1x7YG=*Zk>#|@!NePdR(P&)I@c@L5&ROm&#n`ju=fk zH0fYV$v!+>nelXU#BUFm72+czb!CFq*^eIB9*nrFfPf?djzO{r93rKKt6`@|$%spS z!0+T1Yq%P+d@B*V^a2C44{ilGvOi3CSln;ysA216c9qTmIhx3rvl)Ow?7Xj@y#ap* z^7fQRYbwhre7lTWM0VTPX_fL=<^n(<*-ZdMR$alVZcuq<xj|hr~6herGdB&fi{EADmgU0cKi9E@h6Mo=}PqM$(3n%yo~%~)6PxmyG#+TxhNRL zf{S!%RCPN3{LMUMILaqa3f0U0vL)iF{s|XpWosLo3EzcjU!XxhOIQ2&5&xTQ+gQ}q z)qf(KAp>Sfgin$C`SJZepH%2Z%&=kb%18?#Y2UdUgV3815xpo`tB73=@qv5Eu=$H# z*ZB<#S0Gw{i6auv(D7T1usLoG0>(6bzWMBzOBKAO#xX0Gk8SvXo3CQIa3qWi{V>|< zg`11bjSsQ7B#E5#gTrpJJBa2N0#p3QpMWyQN!To+4ocN`b+I-q`~TQ`^KdNJwr%`Y zQksw^GDM~(37Ln|T|$YJk};x^IrCU66%q}`kkAU5Qpp$!2_YpiXSge5Ec4K}pVaey z&+~oXw!Q1!w%>n0o7RfB@9R3Rb2yIu*!TTF5$Yw_U!-=8b86Sxe4yE#fJC+At|wUO=39kXTfYVa)3esuWSV;+rC1pCve)%yZsJTr8`k zzut#(7!QMVfg$G~vDk+Dj7jk%v4_MhGEt652wx}bHoAz%wM}TN9+&4PA|gfDc@_Paa=F7g)M5{sSXE;oQn<7{pp%UOh-%&#$vzMb76dedKNW zzat+UrbiY1Lbbdkk14;Z$a?RlzN`2w?Nm82y+7`wxhdHQR@FEPRl^sYtw41{Y;oG{ z-8G?RTi>4*mm0|X5i889RQUDB%g!Nyon?G`Gn=;Ktwv9ryA%9rWWpT_3?QMXN~~Cr zqmm1sIRiJX*w$LBqpsbC{QL|yU{`UvL%zrPwFA|=v>LkxaVQ0;$b0bAjGsmRY<=O0 z;_9V{Qb;(`ZNIK&SneI+Jvw1I1-X(So!Fi@0Ys6rw@V1KryMv~N@=iV=Xpt#f>3#j-0(pqL_s z4ApmJgA$Q1bOjdCaXs5b_uPk7;5GaCtyuN-U9w*GDnh>A z93RP>x$H257Ia6Q&pxQ{6lJ$t&xlp}V6xJp;7H1JSfCC1;|61@R#j_wNF#6Jxx z$&x7gqhRaeZ!pemFW;ILlv%2J=uipB9_=Hp=^k!VVun|n5LtUOqpB|o>9Jrm;BwgH zYEY9gkHWj4QZ|{jh8w0)e{(iVH9Er)nmJ~sGx)5ixUh>czA<|`_tGl+KmZ}Vnr4Bw z#pdL&1J`FM(}qXdVoIcA<2jZoP|k>o-O}s|8kw%asRxn9+MS)};Zsm0)w$fFHIEHQ zIN}nic5hbWz8r}GwsFHY_4p5E&*kKXn{mF_!zQY7$Su3!i?l|N2&3ERWKhE$-OFl`-Of^&QYHIA;^=k|~50|(Y z{{5lbD6^^QP;IXt7$zn-_3i3z4>lMvz)XG^B5X`G(XSyeB6e^K|coy8@wey1q~_UjP|y z34DL9!;6*y)V=Fl8NWd;j$MFch14V0Q2iiqGzv+iL=b>OEnjy4eMnbR$js$ZNgkbW zA70CkK;H6c9M{Xh*lrLDv_WXIP>{O;saY0qK$*9LB+WU0kvcE_jrbv=?dN+h3t^@aiJ9*uq=u zeZKgMRTV-wnU3Qx>&``@azXxDnjy@hDf;c-QeJi=;hb;0{u?A$B7niX!D}TXCSIzb zDFIYO&Ve|D6c;8ZU6JBvRn(moM@@oACZMRhgiAz;Zo#D@y; z#f^By&Yt?Z+Sr;l^_NusC(Cb~K zv~Vi`86Te#WJ}|+;pu7 zf4C4=c?n&5Yp>8h_1061S34)?8>u>M}qA@E8`u%pBQ&CkPaTuh=}Y@FGb=K-k6X z2e@EoxH)rMGYkh#>gt;Gr=4I-EMG+5&Zw@M>|p=oWK1z2>s}Ft#@xz{ESg^R+1hS>vD;ss%f{T? ztz$C%bhG|Qxhpc`EZv2JAtMkMQTLKzVHnA;V7ut6l44DH zd*W$%v1*4#`C+P(rg-_(xg9E-8bwWW;TP0wQ?vqP++hXOP)K8EI5JItKciT5f6rxJ zg)!8$p)Y%?H4u{|<2s;(aNv@9a5$tUU=Oq7G$bTSKqhlKH1u)6j!{JFHE`a}s+arRkNZ1MR#M%nUNHF%1m&iK*H1slGY4F9l%7nNG}u;9iY zhT0^Kh!=!)b0Ww79I&;+-_G7tWL`q%BR;l#OmK65`Asmvc*6DgnAPR>Wmj*;(ze5E z#)m1QkYCBFB{3))p6{e$Yik|PXDuW)z_I=03>`S` z6bALMP*-H6VJ7%?swbxA$p$ep(_p?2!{G+t*ox9$DbF|J$7Kp`jo`S z;W-M_8ulfxxK|I3FL?7L{5~z^m5&hPNamkv)e-I;8+dy5Vt1C&6TEeKOujNfE7MnV zY6PPslpa(ZaiVO;(2>cbI20^Hq&Ze{$k+?@NPxaPIN^@)UM`{3l zXtKVpZvUZe4KR~`Tfgukq}J-ys;q&)4FExGTECpQrdfvM>ZK+Od@vFr02<{Y+(ae% z>$S|neI=SMG4q~^){;X5t3{s=7$12S9tMZxTQ}>`OJw+hy%Yya^}^k#3)fON(tfN} zqP7?!%-G55lVc3z455ULOceYp46pSffQARuQl%vppB~3l+NJEj%&qX#G_*%_k}2Bw zQwz7rPwo(-S2NM48V&O)&he*sRgcVA{;9XrmWq9$dK@q}H^0`)_rWMoJ86&YRh{5^ z_CK7E+mUwXVVXG>i`rC$%CbK#O(b}4#bWjWwefu02irwD3Z~{}z@JTgZg{*TFhaw) zfFv&9obXkdZ^4FB!?N`bFiUWGS!+rw-oB+Gqs4PZ4R3oU;}s=r>SqV72gN5%L&HEq*5oZSha&(fSjU$Wj0_<~ zBj5Q&m-nF8=mMX}6{O)hs`farfn=Ryjllh&%83ntaAEw}cKk!W!0V0J$z$J9(YRntUD3D!50;KS+wQA=#dJlSS?Hy0+WiGTCYu1+~F@j9ayiw*Qi@Rqs`&#$q5FODygdy(=5;}h_D&r z4Or_UQ#DV`o`T!F51z;p39$CGV6s)jg3}+NYA^!%6 zl?5G7iE%T5B6jWCbr$I>B7MId1s1aB{ov$?oZ62xpeFDc5embQ9g({f$4Ve;b7wmW z2hAgjVE%r1Ojs+}P|1^mBzh&lXv{^x4o2lco6!*5B%*KW(xo=-SJoK=86Z<|&Qbw> zK*))K8)G$N{Q@z#dr-U_%`1R&KSmW-4$U6dV}S#aA&O*hNC+%xNCm|(e zA2?~P?mrH}F^mUu5Lpv~6{OQyeb1X&H9 zJVift?ONT`eLqI3?n+b-WnJ{St>eY@{7FOJKYQ(gKe+Q1YvKr1*97#xfKwcAc-gpPalLSsStDLpBP7(91X{ z_NRD(B5##9w4d08pjuIk%jWUE*kD15I5yz3DJL5qJxUK?@J`w4A$Kb)VN9;pFS}5T z07ia|sOL#Kk!j5heK`rfxo5s914~!BaN$BNjDnF+V(pQ8fR?ChdkJtPQu0KlC!F#i ztLwJfAbia41IWO6c9W){)ty7jYhK^jl%bkYn*!8VC*RjW%6Foe)hBe^4|$% z%qtY?E`qxc&nOnrXwVorexX2hhg^c#^k4?Vqwxxf*Tl@;7AkXl zc;oy6FTlqRZCv7BU&P)gXts?k2khc=h&^g5alq>_#YpljVR&JdZx;P1lV~4T?-r|TAG-asrvWIw-r+g4 zY|Z&vc~tYwru&wSFNg^^-p?CnaNWTvI$-(v?{_Y>EAh}7iqOZYSLiN6Re9tBVnrn$ zs9J?B)0!Q-zZJtEgnj`SgtmhB@egOk@2c}B3*}7B%C1_@0N^+GXr_h`JrA|np||Qh z|6S$;3-;Ll)Ugp-k$4|OvENGfQ1G_U&&evzs}AuiTReVo zYF=m??V9?QoGM^ivL)&1&Kj0ocf3~XyKuLpe67jk?Ei-4EX4K`QI6`ru1gNVVr?mz z3$X9m87e6*o^T?fIBVvu=D5ZAHIXToyhAO;x^1Vz<6?!kS`U9t|2y}wv)aD5;XdW; z@IT^~%67@t)|oy&|z?#WCzxs@(Li26V=V43Ea zjr;Yvf0fxgQ?bxDK_8JqpRo0S`Q!XYLM{Gn(PGb)%k#fBr&}_Jdoz;?tyJ%j^<#`t zmi4|b%e?V7(a4c;NoThio0-$v@_BNbzP2@mEkhc*Pg6a9MVysnuDpUv1}{_&Y>{QCD?-ZZcmwr~f1 z?q^bEUT%AT&LSvb2P6GnXH(iQvoaq?z1`HHQ+5UINTPsm{0H;F%xu;n z{5d7&uzfKQ-U?Cs#y6L)XV!D8f1>zyD7FdYADkP6p1Z-hy@R!(ZjEGaVqv-H`E(Qw zZJ+&4uBRe?{p^av9@{vQXXr%Mf^uyxLr%h(?iZ#$p)cLrPZ8&o>uPMj0e5nF*LWuD%dKpv7|TVB?6+r~(*pEgZaliuP^9*9#sc?C%xkT-4Aj)a02?#(SB<%N zlnmK4m>4Fr-tB(<`@yKv&p#ObA=;4hMsJNoiLz5Ovkb=!zBVoF$(z>_v#_i<$gz$+ zODccg+~E6yGZU-uFm7u;1@OfqjXPbMM_QZH42TYxd@u+EEL)-XXI4??*9OQ+de_Fh zeno2>xSw1TSAL$;YH;J~j*pIWFG;1=ib)BJi~M56e@uaar@C6_%R9BHNftzMZ;rFAyYjZ~ewO$}EYs_!Ufh0bXG{3Gey!c6_h3kGlej%2*$A4BlZ&m!$hfEbmX$}up9Q*3< z`_ge8S?>x<{`~>OXn;*~(_UEW`I7w})e>j+zgT-AV0+K+$4B1b#{c;a#bTdWt83pa zRGHu9-G1&eJG_N@_2Z|wH26u(eJtEidWAvwUvmZ}n1sNgsTs~x1V~2{RP3^`3qKD3 zDSza7<%80`GXXufp8RI~`>e(@|McgXx0K91^Nfs)B8P(lcb4u_5#HBrHa#1eE`MZn zp755fTt4r8z{`7tjCSYm(RNq|vd!fBm8148Sn;P?pY37XexdbQaVSg}<~~-h0ATZb zq-#0nMmkdc%(dRI$~<#HY-g9Tt?fK@nf=GZ6|W41BbVIRf9K}#@4@tGEVAYK^IRR2 ztDB9&d~)9QjyProlo)Hq2$ZSBfBbBHLC|>kzCGDPY@Rd7=wHRX;+jkH&Rv$;{YN12 zs6C6B?$3luH6atK^T$M^zLhw;%jZ@)Tt>x1rq{`u(#{s$!rNMp&>3#5ZFj_TdMdc4 zV^mCeh^zbe{XCEfSa$2LKR2{y?uKM?I_qTq%Dbw**;nr6n~@ z!5d@8rJ~dk48k@)DKqKMAXB$={9&^>cw>qG=ZA`VQ`4Hc9c%t#=KmYpkE{57GynRZ z=50Kqe|`1mR&Re8b6n)(vs!5hawvR)QjP2%r6BstFMUREWP()yZUKloXpB${(E}_( zN8aZiup$r|-EaB;s$v6Uo0ykCX|Qqi(miuZ5aLh*4ge`Rqu(PspP})ic7@yrn;4A) z{aQ#SAc&ZgMO-ryS;9kqann%Bd|6SPofH={n^4noo#6Z_50b!~VnLEud&bi%V*Lv1 z9k?3*{c+9J{vXG25g4~DJ>dq0DS*Mb?R`(4q@mV<#LzYSi$@@64=9IPUVOh2Hzbjs z1asmK-KX7I zC*1MH&d%IQ=fo)JdcHTUzQV&A7n*+JwEu;+(!)!LN~Eq%yCLwrb=$Sf9p(AQ8^Zug z_D@OS9Wbme-#Z+0NzuMvmN(YwbC$+HXO>2FdzMCrt=<8Bj&+FtmL*>ARO-RE2C%j` zZ;&nhz((dms27Ac=+I6{M8FZ3Sa2#d4L+X-vynO3NO6FBqO7CC1~LJme9e_weIGs) z5SNgM&CfUB5)py`CHnEC&{T!l)3RvCTuv0z>u3c*gYq_lK62kb8=KD|? zL&`BZeg75)ZmI=%JCq|dyBL5)$VXA7^sb=&^`YClsQu=`bC8e#twn$ZjNlS}KI$`7 zQ^vg9Q(aB%`uM@`j&fkj&&X=eb%Ht+-ha=^YkgzG@cB{?&IS#e3pxaZHcXZxXP z*n@gTNc6Q~@_J2u{dK)tFDu}J0g~b$U|zo$*%X2lVhsh38rui(RDk9Xqiq0D781ID zjZ%$){g7KnDIT~;LUJ+{S$}y&1vDobT3TSpo@8qSkr43PptBriF=OIo%+N}+JBuC3 zK)ih$G$n<66__Rdj<(~$Q;k-cn=niuK)man@(fj7aw#NaE#MB!8vt2_C?R(a-A_-~ ze4V(e7k~v4|KL97<0hYsMBaEH;a@bzU%`X3KW%2lix(oWPesLEj5HY@q;GJ!3Oo3a ze&9+!ir6SHL#P8x2E>D5*);IyaEjO!xXN)gyp|9b#~ZO&W#(3@_@^#y%f!Ig*;JZ6 z5_4hM&vF)pjc**`UjvGhi4d*kZ@pBaME=O)PBr1!Lf5{ zYRm54y$gQEf_WVR8T*(R9^U7{3Rj*S0svZUWah33)W9!-K)>7|#8G>FtP^pA*j!YL_P|lAt1_9$w6Gu7Y#rVif;0b#yk5J+BRp$c>F#m^KEfpT|RrR2Re!LHm3|RD-pzYp)mSj(~~R;9CU<66^q|=DI$E zat7$(dfs29n7r$ck7lBQ=l-}`wyk&}*+kWA$GVfZq^EN530=MoFc>AfJ!)!d)&n6{ znoqu=hs@3B=pC|WL4m`f#9X|i*yotr91k$TvV-knp3FYqc5IPYHn^+rT9|quKw|~H zStxEQlAN6_-9471)ZO(3R2t*-H!t4~&Lneq3-zVH*WNlc_ATrfm#S`z-p7Z_>!2oT z*iSa>ndW!zC=eE)j)vML9%4l`->t>Q<{AuJf)_3ZpB5k|I1I{>%QaIaj^z&od2qB! zcn_FRSo;2|pxsVS>?t%#>n%lCL%{nqHTq@+87w{M3-g#%a$ zQ`5kV{OK;Jz2YT`;dMnnnDo~Ue7+x`n29JVOQnBzD?ej46d zEc?@-hLqd0YE~aLAHgOI+64`U>EQ8|ZDZ+;XH0)BUO?9ns5QybzOcL9;|xvJly>^& zuzmw-sk6>!Qz33by$%*!(i&&5hzs@6^Tb_Y+H0e`#NrnQJI1`63)H0@HB%oRZw$vV zF~LUWzl22j^E~YF*sl^2629H?w1*~36dSgIFv|)Kj&(h5SO>CF!a>gXyAJ-a{rAgblq;b3#dO8XDlBp!(kkiOYD+xZG-fWbilx{L09Pe?3p;xltC^E z{o1@}6bpB*sq8$|zbCHdjosy!_uUb4AxOX{!WsM*5e?uz#h_eIbCrbK{9c3geB`trLy6bqom{6X8~GhBpnn_Q#GT>{$97mT8~J{8%ak~x98|;R z2)`-{LFi6@Y4^EvC4pr;_ic`mD0M#k7(>0yvp;`|-_kQ1P zkFfF9j4z@)yy~X;gZGJ7%(mnCg2A56HB$z@PJeqa{#qXNyo6@A@25N@i%5htplF9K z1G8(tyORt@fP&gaHCc5A!@^^ zMrI-e2I}eo94E-}#za&5JM97#yLgO-dYD{x1rmsU5Mp7$*ilrvYd3qj$=&nzs6qTg zfzAW= ztip8~g>zCl09^uAK|7JXF6z=!%em!Ly9)D>IG~`^+=g@iNsGox`{0F(XaiWM6WM5W z`pP(cciO;Ao(-G{pv%MFMpPjvWDd(#_BmF(>;Pd(TI#F442>>kII7z&JIB2+9-PmE zTNZ212mjFMLHa>ApH1vwdX62Iw)@1muW!Bp>X%4zMmKHQvV~XuFQGQC^4*cEa#g49 zfB#BdUXE7SY}d;vVbGu>jxd03Wg{LXiB1k5CMg490%x3LeXP9d@SGeA&4W(N{yKjA zcm?YQ)w~=ehF68Wwk}kDX(YnqfSwpns-a#YwDjO${;6fEvZVNe&W@|stSQ?gPK&QO zd+>mM?W(0$KuR}N#P+Q-8odD`U4(+TaWZ~L6GpdU5qut-Vm-O;^2?25x$m`wrgphs zJaR6{5|Ou-mZ`4Ri4#IojUE|kG0aN$;gUgUje?rNbUc3_(o{csC3;*-{pjYIZfa$0 z-1oyY`dfseegKgh_6OSMy!UBZ`Ak!jvjVZRK;2(pX^=m@2wLv}q?`NmJBV$?|65-o zn1x)FIupsvYdol2wu))sAS$Tgx6X^K?+R5n?0+`84@oIho^U5Q{r)<#GY>1 zGNqrr;6Qfyjj4PSB_E07p+Y(xmv-+qxbg728kk^W_k-!8@@+0Q)eA*Q@Lw(JY(SLb z@klQ`Rxjh>@<^~cUt2HsW*&Z|WvXG&kg4oc0peqGwd&PuyT2pKPIu%7@Pcm(VS_5gN z6#0)2p|YK82$kpGKJd-|>rejQzBw0cP{l=dVcn#rlByjh??7LVhT<3us5>AOxC+0N znzQf>^2T2<8JO;r(Ifap2>8iDf+YDkkjT-7RpjZZfYPFM!5Y;CsAP_#X(FgiL_YyB z`;ap^gD_D6SAZ=CI(3<{$a*S1g;pyGt$tO+q)6n$Bcr-JynY(sUqIhgh@EUe6+{2x z-ri*hz-pyYn%;#XVQn#(J3VQL80E%&nKh51o(=jOJ`rgEHXzAuHnum!Iykn-%6RyY z0XrxWIPOG4xzHo{+?ra2V#7G};Ft2mG$5 z(Trr2Fe9vjhcGf7U?3mVxRIc!Ti_{gmDPULDwt&n{i>rI*w0RTD&qW}BPkIn0G>R# zdt4GB3L@sg1>+(K`vUCTK*%C;Gn3R+K1W#R-e(md*u_v%I z(HEBha5B-iPj~p+tl&QC3})dsM>hRR@rReL;U9m$$nT$}))u5k&A+|B2vtZGEDMWmjpivM>3E@A?2H|5l+dUq^J5%-zE8OPaeQ-k!EkRd>kX{TZYA`zX7N z9hPs++O>|k{y1~!YrPp3{2s*r8YeQsj`w~K%>VYPYFxm?dC?er;YrJ}yH`M<5lE1z zZ6lBJ-^LQ=Ka$UX{Kq@vOWk20;4CT#!~trxYytEkvnL~FHsGHa$sj0ub@ z=>P5J$yo5l&fe&go12>dHLCbBgoC7XEIG0_Hc+HE0{-FXuf*ubE|G{xX)+z3nf)LA zGGxh3ra++oN58D(V-*@x1Ba`BG*#zE9E>c-+~sjKVHfYd+g5mzZ*@apwuOGn3_kE(@9o^vPwUF*#d(EHa;*;QIB768hHYj|_zZj(+ zO>gjBtt_o2B4R#tL%b)Pi$6Q5e31q7d zNz0Qy-u;K`Yi@sn5CkG~sbxZwsSUfN-@5<)(;|hrTeU4y zQVVvYANfCg)N{Yw<(QkRzu(r@_J8urwNOU%iAvqrFn1dN+kUhE|9RS7%g@}3vQ8;F zk_{1~Aj zUaUp!Q9+#jke(^L^2u3tocMnLDYS7@DmoOx^-KZe`G3 zak=zxrkFRM_?-N->J>=H#3s2!-wZwFxm>zfNW&ixn)hB^{t|3Uf33svyZS|;q9j6B zqGbUNjbw&s4u~7J!y{x8LDlTU-Or}Qp}r)e1xlEq!6GKu7jZ%fMYyz{L)Q(-Ed@?x z?2Tbo82`L5Z?y3|$d#yUmcVw~7B6%P^m~0*#o8&t4F>U>H`T#w_)fGDNUfZy=rSlk z+P{+CY+z1<>U~w$`o$V3Zta>!;Lq=jt^$L|D@b(_&;(>|CHhV+SMX(41rKj*ansW? zOkA$tctAdfs@@m`_AJ~9krEesG*f2`0rg2(d?LMv5Vlha(KDo>yE6zXt&UsDTrWX6 zhgkvPQ#SgJHZl}F>Q@)|f)=7Ha=tD(tHMa}xtBR~rtmx%gk#Vt@Ye~66tLV|GqC{a zFYBkrOBG+^Ip2tkjLfP)k5&GA&K->3F*Se`m2zBI-k;UE5=}|?`$Zx5Al9;^e;m>_ zWTd3E3bc7nC~YNewGN&X;i13h+Oru69;FbfGh3)POEdaF@gIbl9xEZ%V$UQ004i%I z^3cGk5u`i@UNa7PFv|s41uH9db-5ubu}6ebt^d7SY-=WIA^7PrPpU&&_7`^KgbrD) zq~#HjL@!>+fre;Bn2)x2O2)96wXLs!izn#nuOROu-c8WH5TbY%RMv5AI}rl)z;Pd!3AF*T-Jo>lwsuI ztBA$aH5zm(w$KDGdttzx5ec^Sb$Ay$oT!2-wnRlRYIO!UR$Ei^C(R*th`uPl+dKlo zfZX(l?yu3^HUe|LF9a19eo&s*iFqDYoUtbo#+AK4AzkuCLtEwTAZ+*<&h#8d4PVfF zod)+~G#B=%b|DTjR$5wGn~*>Hg$+6OB*yz8WLjCFlWu$^4ARNB&bbp`BCb@>REwMV zg5|LwIGbUnxWQIfst77w0*A&=Nv$PAaQel_aBv#wxP##b+@MG4VQ<_ ze9}^v-UqF-{fR0#37CwUtYEl2PTS$N8I=#GR*8|nAf5ALRhNn>n(c){KnJ^GazX-g zsGgn5Te~W`9|xf{48SeryJti5Z(J)19u^_qdqY%SEW9zeWKu71*8dmmNUw0}BqGqxOtj$;s%$;;{gQEs(#I=z$@1X5#wll@e zv+vb*tU+}QV+m&k7Ho>XC$RwTj#n7Yz8X!rGgp(+ck9b|atmnS8tl|Jo#8H9JU#Ja zlS~erC;R&z9oYeL{-In>Z|6G~m0~oC86Th?G4#nK&=fj$O{3^ z6SUBhJBJlELo~%7t%${P1yV%c7A5%uDht?m8haD&eerwz)0r~U!|mv z<<;YLy4E*z0-E~P5`Upy^)Qh=#N@04%X?LPl;cTi?r|K7Pa`2|GJru__N-~Bz5v;x zNYCs2&)nd<)N3>7d;4-x``HVFO`5B2*7vRf!#-Be3qCO&Ag{08@#5!^VmcR?4COBL zhg^t7B4T^T#ThK8HSh)uLlr8d`d62r;;B6Tm_fe#=w#=MN%q>ESV-)wu+7$E5&Q&w z$+mkQqdG&-@B0xY9E+fFqgtUsfu~3DzQt%_`uE&(1D3W8#bMeHBJW&pO{H#GO2423 zs{Ww9LFfTWAeH4ooKixtQB)OuPt zbF5vxnsh2I-S+jH=XW%I+BTIAF~aTY>L}UJ89zQ&&r9BAr&q<_bA3b>LGeN+xsAZX z7|?cVYiLBJ*$`>}x2=L^=L-*#@)5y9cloB$<0I|-+&!V4=^HgEm>Jat1*z$$2{+De zl;N0%=24{@I+r{3_yP-p3~Yekp==uuM6X9HP0xxpd016?nertRR}v}te1 z1sqU(la_au2JxY7!-xK`cZkGGaPDG7U?*i@;@Z|pM4l`}5#|iVRLKLCJ6%8UTY9Jq3R0rDxX8k_o$ANOs8PO17z z+3DMY55g`&4U7V&p3!tz8YQyRuD`>-(=5AVCa zwoLsicFXlY3MYt-0&Tk2@R#3->fRAQI2daRd;P3Y&YgX&;(#W?s{rnH40!#@g~Y}D zsod(NzK9eLc;>STs{6AEqOYWOG2{0*rpIV3VcF0v*Xb{2T-S}k3!;%uW-Mu69dk>d zU*!)~#QW>Jf34NUEwuI@+lgODA*WU0Egel`{-e{K+`n$x+Q1ja4<*d(ZFnQNX6Xq> zD|&g>B6PbMrr?nS#oo*9C!Yx!yO@~7jbnS=RxyDUm@^oVm^MMS?~9wn5^-{u1k?Y- z-c%+v4)X@CoflMvRJHPogkr@sBwb>}c|2s9K_zQ?v} zkMQ!2kZ_8H@nySMJPMrzBi_$Z`l0YAd~N(7;PQ>0`F^Iu@YiH0{0aWf!t9cXqW?*@ zHLDBQZ=ZLUDqz#Go51F>Hp3s5*JpZQz}$fiKJY|qb2G=SLPvYW(8=eRgMO!quKE)D zOLOkJuRnYdM;`Oce|n}mK&KA=m_qgBh}{(M0R#YN^eWvHkQ56qY0*n?yY(U65RF~- z?!7DpKg-k7XgJ*6xFO!)2ONHUfNzIYpxuL(izgq%Jzp66Bdl^%Ucba^uD9hX!8@TV zkiPIQJgfB?%L7X{v9e)Cl_2oq4FQ{^f}V*?{C&@YIPkzCD+S&Cv$3!S9ZB~#lj4K& zFhrUj%`6D67(lR}U`l8;5f~5C;$cA2KH&|BVc*S29(;7YDJw5KO9&lcM0SP%%|4d9yqwMCn?l+I@~COoAhElGwQJmKM~ydA}f{RWquq0)02 zxS^Rgg*{~v1q`%eIJtGbv~g2GBZUL0lJNQr%TzzI${8tBEj1G&uyW@qOnJ5pe_*cQ zxGl0d?I^D*?0aCrFj!vZA8>-~iy9)H$GXpY5FdHuVK+EfM@!A^_nSNkhSjiRIu2;p z&>l4^HA)gqqv$xoy9MilI}~TS51YzzRrl>bXEctTp)rx~KADY@+l7S#q%ZuK_!_;V zLLLpcBWMT?LMp6tx*sN(qLN$@{@269(XH8%`(Hdl?YBrL1QQTekI(!s16Y2pQ;IsB z&Nd_iVTiyG6TE2g{0^04S1$>}HG8!{p}Rv>mmjH>YTL|zXOKP`!FpQdFg9i|CUAcY z_s^Qtpb%JP1aAfQT>^$~eY+-Pe!=WGt`2ss)YKouV|&Uvz%}@`g}#QyMx>`5stZ6? zxLsNW4Vw>}b@@SU?WA2jpESo8J*#;wdxhB;*@9gZN>2_mH;$wopsvz5b}Ts0<`=AK zuCrUiBcgRDN?c++eVxS0k4>!_(O+$SbjURjq`mEOb%*C$mucR_RLu~-FT5VGlkbgED~`&0xmj| z@j(23#_5sz7LfA5yx~lBf=66=Tc$3 z%ufo|3Ey)DS>P4ogXNU*(I#tD{yH9Q8sSE;vj@Nc<@4S1spw*P_GtiEe|@_1zCTUL ztcUZ^x#Keqv&9tn$g+DD6hveQ^2P?BaA$&tL%9h5Yb3<-J+7Rbk}M@Akj^80r%>KT zD{_NsqglTUm8($`JS0;^M+hdtzSU2WNZbaVgn6`BBy`uXW90cIynVanQ+412FjJu` zZVS}UFh3UI(BYej8ar|5qg;e!6m`iAT`yGy&Kt{8a4mok3U5il=*O}SG>feMoL&kA zC<<^A^QQr;ticrwu;jKx$`dAiL4e{ZD42_pAyF*lB9<0A3Wv6qnQ#_ts3MNs10tw9bJ!xw8m%9mRyUgCMkF+b_ObIBCurs*I5O6F>n+Gl^J}vod%Nf}Af=;1X z5E|?(7JhJ8ue|1=dXyimakwZ=nM8fSk^!SmUZ%g$gOcmQi+Pd|Wq0jmo>*t}h%(?> zp0UmvW}>%LHr8}(pUxgFXEc1B)#vr)2uA^L80~QH3b~Ca*kjjfOscn%wy?&t|TY7~+5hy$0rF}EO z6ylJw+1T%T0MF>8h&|K$FACbuj7U!m|JMcUSS!Ke|OJj zh@!Jvl6D4341cSC*hL(=3G7yQ>$ETMvS&Jit? z1IAfNPHqS$4BUtIA>qx+1dIb7_N}O>7PAeX-&h>QmC#XO@%p@_{!HEo6Sw;y9cHXB zaW5I~v-Y~~)atSrC4i#7SBgIdPe>8x8=NzZ77IC4P8?nJM#jADlh-ERBggY*pi=OM z>+7?Is|jKqH!=yu{>zu=4GdE61CQOVm+2{eVpvnaZ?1P1@%!-Ut{U5pu_@C94RB0! zj#SvaOH#5E(FDEC>Cq#m+CetuXCU06jhTw~WvC%zhhv0O#7Ha>odURga>K@9H^wxMo@r0yX|A z$j8IP&mB2@Ja96+>UGoPNH2_)iDB$1$dxGp&F{g2qNaXmed%OtyA!+sY`C$l!hW}p zgY$2IlLu~?GF!AhU+uNXvuw#B2{yzMp@Nn2T!C`q{R#fCMHuux+-nb1A;Q{uA(awvr6CWs?tyZjpK%RDkVSRDjc$CZ; zu&(`x-X6?K(t&%Y%W#*vWt#gJv!~fDm#se^NG3>!(B{pUlDCf!a)WZz+TlASAtID{ zZy6Z7JDvLV%71#h+C^L0a_3X6@s5mK6s+9;_}YwNu;Gx{p2 zj2!3U(6#dh4z^eAHlR3V#3w1w9CTLHX8Y=E3g;5S)CMXGg89MG&Cr@>IvW=le{O`^ zE8I*?D1`)V8p%2^@s{k+M|8wh6>&HeSh5Pj7zYX(xT6EsHk25u?S2MHQPmCw=+!M+>Dd7`9d=pmnrP^OxkYt&h+ zYQjG?G^}=3GbSigZBsQ#Ej;{FBnz&-)NExiIC^-UHZZxkTm|m zviGXh>Au1orC^1IL1$S7{WC9hlF>(~1~_?}9$GdcpU-_;RPue zXAiNMMnHjcdcc@0tYa$-H>B6~2SbNlob?&sEJrobu20iH8Z2zVze*Rb|R;}DF6X=4(d z+p~!!#K4O&E?h9q>c7Yfz^fUyani(0D$?c^BjqBrN~AS{X2m!2yv>)X?WVGDwtio!onC?$ zsB7a$+OMOb%mIT)_EZr4uw(2Vd!6LcEH2`X{GK$gKs%s8r0VuztKFVq^&#)3_)GSD z!h-cCcBD6M+!#mGBN!a7g6lH}0@D+D&gpqk&BUz@nBB7mg=yJz*{{m$t>p>JuXg{q zN-??__2(eZo8>xea@by+zZFEhLZOfoLv`U{D|TGN#moJCx7Qi&Clrd}bUV`!aH(Rw z;}dUXSq0mh=r9q!0r1_8CM2Y`1XoDXa+-u!$V~@x+h$gtiL|{c_ikl;vERv|wF=*g zoz=L&F|XIVumVLD(z&Y>#m`fr5KB#MJhw^~r!-n!N zbbzdAs!2JSvBg-%W*^2c?@;H?#Vnd_^J8ACU}cP=28+^XB=WE499grzKdESp(mB>= z%k%u!Oc60PhwKYs1o09f$1^E?;Y&TQnUO5?@!|3I`6MuHT;K1{E!sL+00I~}jDce8 z4B(Q_`Ctmi=y0jpDCgd{S z(|YXDOGNEdOYb(`_&ERa!Go0nRw;W|A0jiL_}KWs&}q7@G!e5qZNw9=ZFV%1auHBA z9qetboETQ{YojC~`~Adg^EMgQr%Fq7s%ivvH@gt78zfoqRD>Pka+05roJO+6o|R$* z=N4z+_>C6v#j0l-`O$Cdddd5T{?6s?PIh*#-(s#2H(`+WOCz5saOwL>-tTRTZx_oE zX`Nhmz2vXl{v5U08Q@IMbmF%TnTdw;GQ4WY2hBoXRkznMS2pj&HA``aU%3F&ggS9^l~vb)gtagch|VO6IlYazh+Qg+ylB1`S~`~8uhpA zg~>+u)X#5H-Q;1U{3=H}L>xybaI`BTU7Yr9lMFuyc>_geowc$T(-V(g9RkToTwTt_ z>(%!3%;DVL{?-?|vkFMnxsq5%3^5W>>erwEn81vKbEmDIDPc@MeE6{PywG_3tQ1w|7nzyP_uTP&YJNC0SrsosBb6Mrk#Dma{N{W+>6x1;izLb>qf zw8`jJ1_mDb{`@H`X|bQA6iA*fZ?_yfO)M=T!6z$9MLd0S4q7hl!$weHBtq4D^Px5C z9ipekx2soIfi*hN6xaD6>8tsA>#|d@wk72S3P*Sp4X1#Gn7}ib9A$eRI&{O`69q^X z)6LNntuNe~Ze*hIfXCG6K8p+XY?MyR`?=p@uXhb72QE7&e%EI-MIISCIOCM{*5*XX zoIyO8CZ@2>HR)&0`Hz~{3V~qu^X!WRuO8LBL#R@A_@xcnk?sT`At7O=-cA8Uh7q?b zuyqMNtr5heLXZhdpSYqvTJjTo_FvSL}PDe39Z42i(l8thy$&5rV=s`(y zYYK(rmKTEU$Ip2FyyPMU<4nX|B=Wv(|FAhkuvJ3zq;#z$vZ{uwq{%J8@M0B6M$0h~ zpmlEw&cDo>+If8ukh)%$S1$DhFn}Vr%(jI_c_U6>Bde#m|CDdK^vmN5JmK2%<;|qA zsQ}vb`?Mcb%S%8h>Ap#|z6uq?BX%{&n?Va9zR+mYNnn~IZy?#@h1)@#bC{X4g&Sc( z82Wcy+%!`=s_yy3@Yb|9Gih~>9c+3b#isFGd^-tHdPg@7jhbv=XV*XtCH)33v>Y&O zTFmUCw|Yfj#AGI#b{he*pj?D4*mIp!_b>B$f&iT=QYG@IDVLU^?n*21ky9V=dbQ(G zHr^sGfN`^!I$a@xcG?e|Ri~*st^ay4**eHoB9gT6W&ZL%sEb3 z$Kq|*hU$K87JEjVli1uZj(4WST{fd-LkaE3^yro0UX`P9mo~LJHxaME%L9X75d$@H zJ~%y9SI~!^e9x`hpSEvC5;mArg4cu{Xr`>m$?@}~cGXuK3Ev%_R!2{CC?D=xGqtex zm~Ve+9gsM_uCt#44|#e`euy3nEtpru?Nje_nW?p3qMI8sWSI`cue#*fWKju|# zt2O@pw4ZZl$_qyoWVN3*Z{NO8Wc~NY90$}D%OrX^17zqL06)MWg4VunMez^$B}hY6 z4{}xxjCEoqP$)?2gj2(Rp&7IrX4|HwCTxK56{Ux$Ls&?UB45%_pY_c);0^e&H|FNH zGoW}3*9`~Bk!_z;gBIBENlyd&u@cjDcB}nf%0RT5oMLAFk(E~_0};X#hdD}Yz?6ey z9=nfc&E+HNq_&X~^NOV8B(AkNt$m@YDwG3haD982#<;~EQZsX(4L@_wS14)gP@Tcl z7rSg?-@ku54E^>6sG*LYQlz`{$Z+>ry@aPu9+!_Yd620XDH4yS%6_k(ys`ioGM z8xPPbf5>}ND0@z;Q1={liP?1(FhulZx~^yQJ66! z)*7T`R>~k+_2-g5Nnv)()dwFXtx|^c*z4+@9u6(|endafiS)*V=VwBT1DaDxq3%XI zLmGo9>kp9XS?@N3ysXM$k z10u45k_q+AItZDTQHbSrU#b%-0u@r@pd2-AYs93m|1Y-OT&ckDRq z`4u+B&vd>bzzKDaLdE=k#lZDicsB1Ll!AVlxehS5KbjL!k7j)?>fNU~+!1D2hM?QZ zK3Eab?IzoXW>Y4F7})N43NAoF)Rg=KsaHmeCof_$e^TIiUV!vlHG&2)e~2Zv<|0x_ zzjE@3;p*9z8Imz(RA$s0Y|YepY_unW266*PLn?O!_Vtw#Rg-lUNCZe0EbCmb=g@km zaUOi1j)*fD!MU*4(-W5W>!5zn-|`hOb}((5&XWg~!^lYC%|TDWVc?%I^9hkMWUiYj zK45kEUcGu1PZ<^9(6k>IAg9FCT3Pr5vO9d|*FznU6b7gL3&4ev=tW{g%=8#3N@?}i zQLWFfltCRd)RZ9azLO_W3<@gM2HLrnD=@yYmjHxMB7raDvJQky${_4iYy$0K#}`+W zTq0e25*J*=v1g2q8%OqNRuafT zT+xqgka@29-vJV}zt0JN;0+AeIn!}UNc55iBB~Bv-jjqh(s-4WHYjNG@J!;XF{);S z)78``c*K6L+=~vV>m{usElzx`j1VMr2lzx2VND50DP2#n`zB{5slE7yMsn*4rdkHN zOyD@m67_!2G+n(JEI=PzKzScvW_A<>PGJF+z$NhX0*&OC%7E$JdG1glAa(A~{z`>s z18^LIt7YPrMQ!^ax0cUPOIQq?R@2jyLO4fUEV?yZ%hn0|pVWqJ0x*~$(-!mqj?K%{ zBkzTA)D_UBFq6fD8Za8@-tIL$ecFc*M4-cg_?_RXV8)T{I>KA%7@x#Dt{Az9S~Me+ zY9!S`%lpGVsIwgbFUJRXC-K7eX`AVHBEFx)KY(Rrp*VNrWE~1)RCDu39D^t#x=}4H zQ=pMti>x{sDjhsAluXIKq!Yi+zii%POpn`-VDWDj%q7u@eAPq}tLBcK?5kX3kP7x7Ulx|`v zrF4UWbZt^W;vG}Z`ObBn@B03N@8yTRb;(-mdFC_c9b=69zR#1Y)g=++iy}eAAwu^c zZ!C$#!;S#uQ*J|q9y?Xg38E7i3xiUu3M%`eXEMZP&yYZa_8-}e7?&h)A&}BrpSt@omwJAejnZ7Jz(VEi8;vl1@L%A zEt%$Z2|?X=BRrV}O?}Zb)elA5op>Ib9|d4e)fPaw`yEiUE`0&Y1%^8q0gwe1-f1JF zXh2Sz(U~+R_uA5%F@ygod3{p4Dy{%jRfV|8(7R#1Bl%B40lbxU1w&&eDV+- znkx@tkV`tj#~IXj<9K`Y(TSMq&8{0zM1C@Ywe$||JL6i@S?tl>yGrVJPi*hC(u$>< zI^~put9|N##^VhPXmvqKI!SbD6O2NqgX$@snz3X}>H&X?M(+9rl1`OR9Ml3L6anOF z1$T=LHUn&VM0O&9a*7anOUuhYBKJP!C)xI} zB%~9K#6Rzs0d4(g1R3f>%~(8+ALCOpK5V>D8}Okd7L-~nndR~Q4U*!+1ApC6(8J^Z zYflk&?Los*OJNT<9978c9|M3KL&PM#?+IuOG7};_6uJNL$n^*egVA8s0J!mHMroxG z;j`#+eY(JC;KVtJvAZDZ4BB&@I&rSH;+D4}{a`>WzlA^dpWCAIlQr!5S6jq)Wf~N( zajGmaPJaZoi~`wfs1vpZi1X>)XKJPgir$KNdNIJpHx&rDD49I<7p9G%@9^*0KG~Ll zjt{qJnG>{SYYj4+V#K)YOmnYxWwzR^Wo}zSCC;Qo?x!I{x`)dU#BC$bEjt zK{T|m9*WF2!>nv;dJ}QMFh?C!Co;8bS#uQ|z_s6#uQG{n0klKP!zUB6$4PFCs}FoH zdBRmXS6=EQi~Pg&$GIFQ3$F+e{X$Hk4W2n$aQTkm!=_($10M5#m?_J>CSqIKD>`J8 zAsbc!R{3cde0}>c6hg%~*v-_PrD}JFGp>TQXM{jS;cGagfY-?=zUfnul9&_y6XLP< zmbxgt7>+YV6QaL{(hUPAV80#@4GlT`N+V*%2U9ahmA+g)F5Fx&(a9 za-qOI_a%oT<8H@_m^{qG+Jt+){-8S#2oica(0TSlUPJ%_41@dei`_WHdzuI-Ynw{B zgb6#9<8my5;Eu(lA3am+TD~kP^w`eMOq6wfPo$z~!M(Ym3(nRmlLtNy%Gr3~JD!*% zW>(vBqgWY4A8CghY?fq4-37ovum5Qj_8Eg?D3_0SjFv-ATPGg3N6?HcxU7hU$yft4 z#kOuB9-_0^BGoG>{s69z_~%|v9{GfqFHcBdzZ(!wH1<*gEtmQHU+*37FNY=_m3UHkrc0B4GOAbCstC{i&7;>yV2t4;QsoDslx zhTj<3j05>NryRi#DUjIHxf~S|ThVRWB6j5qMFoW-Om5q8A%X}S!A4JPKXUqX-bbj` zFh5Ro?m`UNu$A7A-r1=wylDUQ@ORMK69FFDaj0<*WRMLpyM93* z^k!E`6tv=Zlzi4=j=c78U{hn`ff4u-;rT`k?UhzlRV~uZMDzn@rWLQjqj}T&n%Gd6n>jU6N4q*t!^{8vq#S2*&29g?4%vAbgc;l z=bhlnde%wJt6EqjS~fsj?Va^5J|$&44Bm~R;%YRJXvJ7;)_uwM&`CIq0110Lt%@FQ z?!*M&&UI4q=HT8tNnl9EP_Z|&cau?6#4<*!^va?lz6gOJi**`+A4Y=b$5Pv}(eK*2 zlq>dhtqIIeCwNk$U-c_cM?%?rVHE`DQ0t)~!fo*F=+xAnvSrE;TOIBKggt@@IDvj; zF*FJx+WZffqyXkkI=kMsP?$e~Dq-NkD5u3{qOn|sss}+lW>FyDGt8Z02ZTl1l;O8v zt&vvasnS$0?%pJ&rnU{0xS)X$TGCX#QQCkmBgh9~%ziif-7t0a+t@{P{XoRQ_Y_)( z3l}adL2)yJb`Y$~8v>j$4iXngx`hE_)#?|k?YZNu`rwG>+Jv?tG=U+kHfA=5n3{t(TT_~_c81%Z*10UAk3 zJnyc~lTvDQWk;J6)KHM#Ux>>1p2ddx)C^AjW!8bN3uq}&Je}-8(BuF53HTnAAp2jh zoyLpH|NFK7eeu6n z3DV+$v7>*QG;O~33ne#WNV%3#fIP6d{rna*cnn!Idhac#oCy`(jCfxC)#KVbq+vW1 z(9#R8EmoN?p|fDL2*?G4OeWeXFSx=_8&39BU1)&pXB#QrnfU>kG9|IDxyzno?_PVe zyPG0250X}`%RU22_9RsW3$Dn0gI4__g-YA`_0~}c+6np23kBBrUYLy29Pn^a6u`zuX?e zXu`}=-b1iL<}IiV$Vr-vbr%0d96FeVE#G6{^Nlo_PZkx=g@|k0y{Lz&1{gNx88>(g zJ~sG=LzQiZf6 z)NQsuPltW)-p8a@EcuD^0)c(D{Y}Lc0W{5+w(isRh0KYXh}_kUJAN@ z7L)K4?HYQ!zkk03g$U*5+}xaA%6iV*m_!)CMG30t(Mzg$k9dxKI;YBXsJQuGRfhIS z7%&6uf^rj)8hOJ~`hMpJ7~0!7s+IhX(c;wU(~_hcG5Kr~9Odb*PV$x$XabW06tgU>K$xwjKCDe4@2WB%=;?Vb(xGqH_)tKA|wE^WR->|Idm3e;j2r z{KjW0!~xKS_hd%IOsFI9F(@^-hf&ld)**c}?1B+soJ8{`3R+im-DOaENYg`ai}CKV z?hc}#2a>Xc0uT3->}j~qO8{(;(vsoOl}878LNRmvBhW`JV~qSFLv%q<5+xFvlVFD3 zCQ_4!vXvER9a=LgOU4P20yI)h2FD5%0dji}GK(N$w}_-#>vL zTS7q>R>J-M9smkSAkY#xuHO}O&>XMCVTeDvb`P1W!K3|dU5CrWdbBZsgaFM2N!vGQ zY{(pFQ6%bz5mN^Jf>3~9s^g;oe~RbUMT()|ixd`ELJk9=55z6Fonnj$yoUG<-L#5z z>j6AM=CJXFmCMt>z)omDi!r>4v=)WiCG&)#5hR2~r`!5_s18X`OZ*363@HQxI)Pzm zw?Q*ud;=MDN>(c@!QBKn&uf^B@{`%`@Q^@dW!Q}?p4Y_>Kl-!Ze?Xp=i|vJ(A1mNs(W!#qJmsNc48V5T z%8hMf%XCl~hYlZ<-B2W8*79RoIJ6ItL4WT6wIPdkLO0pzT_9Xb$edD;*pP^l*&C=+ z*+61i^i7fGO-IEhw$cY!zDLN?$m}uL8}t!jK4d>ot5opp8X1I)o*<7gDbrj}7~A83uCzG)3XuD!>_gJY@`H z2YBvEL8ybF;P>ki)k*k)y$GO@a8J;J9VWwAag*RLyfl?CI0l8HJd4MiT_V}ZtYSiU zAU`E-VgUT3Du5(bV^1RHqEtqSK>|oY+1sVK`qsawMlw$fJRk{AP(9IwjIi)9orU3t z1yOaqMjq?}?1PMXLSsbqNFgo6&@!?-T#^};Qo?8Usl=9f(*fV6+4d^U{JQxU_fMWQgGN_sSyPmQ4Fq5zSuJIiw5+k$eUHXlY4@Ge8GLV&CyP^Zgz3( z^CpF*%eQP=D(A;UTfh1&halHp)~(-fOPN0Yx?)f|^~HhRFZ8&o*YA~9-IVoEZg=(E zO0G4^S2t}oyJfnX<5qxJ!SOdXZnpmASE~QY*S9cgYkFcbI#GVbPVBR5WxH;*OFpVz zyxitzvt3-5ELlQboENcbzG64h+H^oC>)QR%5EKL)zVx2b1^4M-F^t2Y)x)p#_M_Pl zvLoI!=-OV-Eg&d9K&Z}J0Ydm%Zn zqY-n5TiwW3{?9DvuoCP%v$t0}+Pm$6N?#@Bo6o|_o5ezxD$y zMN*QpuJBs6-nDD5*=`IS4imNhdzeKj;#hKHqef z6ESAV6qS0^Uf2ETB>J#Br4s0~k+~bbean_Dym#+OfScY- z;0mIjHzOpO{K)Qe<$4&X?nE$nYc_8De60cj=Q)rym9H1}aW*V}#;YBJiM8e!1oL}E zP^Mv&i&^$(D)gc9nrSpn=R!k5+3}?}3z)r^Eu&DHgaJu{r}#9k1Gq*I1tV(l%F+#~ z`VC;=h1?fh+GzHQ$OGQc97n?komeHERpFOaPM*A-o0}^LbkBaM@j`oh`%%}~wNdd7jW7A5p+J&!l4MagX@1UR&IhkPVxP7^U0w)7JGcaFsL z0b${bw{G1EO!m&p&8>YUyK!oyO&k2Dnt?$i*8c_YGmZV{!UH(aIWPg%6xdmW4)xWL zDar%1Xm;@&@uwecVFJ3|oM92G9IX&cHx=6+LezN4I@!wD(@KtcP#Kyc+A)6lcVXcv zP(Ug`bnW4?%II`i+*1z!SpX>Znx>^mnBRR0w&|Tj!;2Hk=21bWn&&*V zb=R&`_)XKwP%-1nqd2+)Z8@35i{S$*lLw3F^g-a*B8y5JkNtcByfhMF1oz>?7vOVi z5jj>gY-ePQMd;NB<6&))*zeD&n1EAn4uo%@J~Gb4=`H00ix@GE@rKm++ob3So~Ajb$@MhW)TodHmv}K5&>H(w z9!=%6v^1Jc_hWdjm@i+>-@ALa4{0&Ad(a!DdBGy1(eNktVE$uq2M6};lffRT%XOM4 zXRZdp$8X>FtAlzo2VKe38uuVJy#PN@OqqKPEHYD z^w%;&9868te`l#_KisT_C#e>8)c*V6;6ZuXwQskPe&aj~_oF(zry0}cs{nLIVSCUr z+#J+W_1cowu2{8d5Q&<<|KKZfPHUXFWE*CF{Yu23wdOR#2OMT_Whh?BY)uF*E-DII z%vGoc{@>%kyOrXr;6^>QoI+WX4@ry0J2PJOl+d)R9r<>K{JOO7lRJ`HMK|zt$GR^r z$UsT>{AW}Yhat4#fsJ~3&cXD}xUPyUWT4NTz-t_OE^?I`>;U87;#lAeh!~+_>>bJ}yhCLCKXxqx5r@b(befsA9!UiXU$aY=j4>?VrWNak!!V zZ7ZGxd6O3wE9Td)a+X!^V2aDe^kLnFE~e<1ad2`*KZ4^BCPKopX4n{qSv}_aQ=+c} zA{3r-?}SZMa#S2TzBpH7+Wa=2^B4z5QTHzr5blH%mcf21%OdxK6C9W!>&4Vc*vAdGd=42pGnzV;#0f2#xTdMCAF?O%VTmEVA+Xv^P6&dhFisApTpbyH9|rQ{_gV8)B&&YU@OWE6Hl1`e}A zOgN;%Y`*Z2f59MN<|>=8zZmw3CU550w~q-^pVNvqG7FdY&&K;KQ=AFL^fg)o;@c7; zwts{{%nxg(SS`r<9fA0CKu{RibHL#n)0VLXpUbT;#p_4-`1k-NsJF8aMiMUr-@-FS zY?7a^?^iolde?hpH{@!)llBV`1taqn;u%9TMe8vtN5cvc%f_z)8w=%9tdv4UtKLk5 za~%A$N^#gE#-RlT1!XKOAayHkykPfeLRTS148;ChwUi>@vu`(r;>!#7k?wK=T~2(; ze)jeC3F-$#w^}SUhby412-ynxWPiB&A_Ny%F0An(8Ynaz5u8SP6WB;b*|n_?B5HTnFe&b9~25;NbQ7`vdiM40e8(rUt32Zrj1P6KPO*-{0r_tA5R>C3yD z5xObI`CcIlZsszuZ#O?be;10k`BMF8ufdG+)7(A;e{cM#`tsz8XA#fDlmGg7Xx%F$ zy*;fM{hmGL@mTZtrj;P1ntAzSEi6gzkB9J_=>tI-^6ktZy$c78~YF#}XR zcdD(n%aiza!uSw90^X|8ND21@1KlMkJhk5Lv*eG^vJ4f}b61C|7xCmKJS1qq5=JGx*<=2M-?9blh-ujuGP~h81l# zX=V1zLgrE2Y6(GI@CIUD^I%tayAY(%B#!bF!=9@}p8m^r4$SElQCWp!G_%>s{+%UL z{WWx#(4ZhwYF&s{>6n(kDjHB^+I-|f#9j}xU2dIjTp@^3uV=QXASTUhY!4d9@1&Lu zn<9-)&@ad@170@}qL&!oqD#OalVP_U`?A zTx6i5paACniFW@I@*r*AH>H~(YB_8>hf6lm5Q$Tb_Xoi7v{k8s6m^$kko3GOjAq?i zGnF!1D2*dzyjl8f+7N35v{w0}K zD>2;~411#KZBq7$<_lkp+N*MF4UHhD(;6qvBXokh2=4Or-E{B4gPhQbiOn3KP=Qhx%V(a^=8VL-`S~L{+6sK@FcPzjB^Bp8*$IIeH@t(kwKwvP zP-K?#bZB!a? zPu+uqA(xf>Rg<-Z6shE`Qv*h3=WR;TQubu4Gor9R^~#R>0%Nz z5}#SFHoiVV^1p1vPGyHm5HVt>3(+gvbCS$)FWp!BuHVr)v1#kpEAjKl{Tq$uBBedb z>G#%btA+!#;T56$6o2v8Y6|6w=0M3^&ygbe3kj-RE0!<+E~BtLOx)4If*uAiOshnf zW@l~TN_VQNs_Nf<0@+qH(5)tx9PLetRWWCoC&(5sFfb4Zoq$RxC&K{~X4Nnk+|~sb zX4l2yVT)cHAOsyl8U$B@NYs+`E>tm5E79D>*MqNQu+P|VkIgxs5ruJjW^-O}-5}h`Z zr{P<`GI$bkznMd=bquwQmY^E1OpP`g!1mF$7E@zX_>358>&~5K-svigHt+7(nq&2q z8-!Zp*ABPksKYvDk*1#eE2O;b_k>gg##n#aiLN5?Yjtz4U-htJ_)s6x(Ao1k{EE=9K`S=U*h&|2GCUteE{+S)jr9q)M5MyD(o zU_@4(=svCr>IK9-e4{-euB)*%-a1Vf-@AYRsMTZnb8ze}X^xLkcRC2!sc(QwuOV_= z)QuJkT4N$}pw&x6F;BZ9*vaFOssEfUJj;78{YPaL0t14#xPpg3My4KPF1aF& zPY`v{RybGwcF6Dvfzv-D8u+7_{NYUV+t z1~?mhU9HJhS=tSz7}HKV*R0=ltW`Ne)Nv#f#FHi4lIm9yX)_fTmnht^He7o`+ngiV)gf+IMB;<>|9C_m1WpO zp?nmezlv|=^^N*i{%qiOdDF#G&8LjA*P>&AI+FApR6{DX4ug0YW(KfVrc~|Lc$NLU z8n2~<%xgc^+p~^Fl@6KkOtD&xutWzkcXoExzO&Mv$NKcxaw-J;&fn3O12FSBV3leJ zl6lq(@T2dI(Sy+2rbzpZ8O~q7o^|rnDPHl#*sOXl)etGpr(k(aGl|Ig&-1vB|EpvbmeO(!}Lop1eokFI)A#KVB25 zo|bVrLWZQ*uMgQu7!<9}blkd(B1$iiUqX=ze9Em>>Jr+-U(F?}*fz}4!OrF6(SlVs zcFM!5nTSO5OiWDF0BxLv=P-vpA7c`5u5%{gGl9S6=_zI?gKInR4k-+9G8 zb!GPwg=c(W-4btzTsvKm;>9X{5y+0x{h2wWsFlODHo87+lB$?S4o@JjJl>}3eO3xt zO5o=W+9+GZasKf2+qaiUheFOW)!wUqM&o;nf(T0?DdemIIb24X@YGWPC;JGbgzG7% zeF416Qy~xWGoqC``~@s#U_5&+OOvHAxl7BtTygd<(IG>V>1Mn0An(~w(Xp#ltA8N# zHwCeTnp!{-g08c66F#l9CL2T`#dy-Pc9 zX_?f$Itx3_3`@1M5}+&pmmEs;0qu85E7$FK0lY%a9}>kBTw4MVgU62^Rbl^DG~ylc z=*}_ol$+A3Ca~bX3)dEET9@#2<|m?_*^b49-s#o=_S+*4n1ue;6TLzF&jo)E-@>U7 zhr_XE!-jf;XCAjx{0uu4c5Tvz}kp*P@%;cN&F!;FT0I#nF}_U+qIx4A5_@I#kx zYp<6AR8n2+!DxFVLxN%JRt}`$^2z}4h)I&K#xvxlhavGR9o~16ESA)Q?OZtP_U%2> zO1!{`iRq?B+sn%9dSo#Vp2R{TUwnpEn5l@C;=!i%jN(?qliAA=5a5f0&6g|aGAii4 zDqD*!nlZF=62+)lLi`xw@l2rf2D0hHyP!$wgV7BP$knD9R`|BPT(Z0O%kRiF3GWHC z2Ii$bfm43P%rh$lScbNPh$fmVbxTa-cp>R5@n*H~Mnfn2&Z0-qz9p*jmt6;rNBwNi zGyQXRb^wB^3L7>}d$@qIEJS?NieEUcHtb<%tXr+Q_Ut)v|MC^HFgy|M1>H0}A32!#2@(4CHsWv6f>f>|2mes(t~r zX<(YFns|}7Otky@>m+~|kD@i+!8;=DK3M1ksee7lwhC)#`+&Uq@$~wsc{>)!e-bw~ z;@Gq2oA*!80AlehGh5Q_C4#$j_3;2nL>OJwk6sU|NaxdY|4=le_6Vikv?S+r7>(W? z&2=h9j%8C*Mz{#-6|Oj&H+;3oC={)_(X7upaXJgNqh%W;;KC|xHZ=J?rG0@RAAJ~7 zh)n_Ai)q4*+Aemk!sC9Dq}K-6}aj+H4Ubrj`kz^HHGHO0hkGBXg2(5CbP`if(Wye> z%=@`pT{jLKT%fg#{Y&g498PX+5rginblMj<$n=bw9{_22&q|}W)$wvV8yg$bF_-Is z4J#>8fA&ov=`;t++jAXG`MKnccW~uA*0?u>asw!3KvKr*__%Etc8&O0TNVa}SIDL< zddh=oE;E1-ksCmLfnEjAR6%_XH>R=kSD$?a){+=Xoc)NN(E*?($r@=VBiv@=XIg=N z07toJsk?}-4R5xt0regY)gxcqj}crd0_gPUPYL9(%#G30pPsrnGdrub<}5+OLFHb_ zY^f;<2a|$8s2<=CCs)VlO>E%|l7x7Zeo%;k2XOd;^%Bp&Jjn!7Lo6TE!OTyb9uk$L zKjR@!Ww-trAd&(jHL*U#`W6+PE}H;Bhxp4MQZAZpG&7eu(= zF^Sa@3&Vt@D3EOySg{1WBfs?!vX^-L$?WUj~I zxpQY53ri?<2yTMF?^5%E9HJuhpIGCl3*{0?3Fbowa6y zox_=+QC135|~X$T*#hO6w?l*dhI+SgKR5XEF3jcfqc3Nz@6if9zXW- zQbS4%Y4KWHi*BH+1H^se08SjVa&mS~fJY}*jH4f5!JEuGxbS%JBas-qR(p8<>C`s6u;*t_>0RiQL zxp5B^8DEzjTKxpO4)L@FHYJH(rKE0>>TV<_p8&uCRR@IeU=oS2wnG}9T0#Iid;1O# zikMZOM!i)7SVRuaU7E*GY3>S+om?RBt7C1sPAVi8m>y}f2-XI^h6sPR{@@0<-YTdS zI5~Td0bnCrW}qoO7O7nVnA_pjY!%c(9PUMrCp}>97a^Z=i%tfR6K7TvV{FAOD5whC zqmHOc_01J|#V6G$#G+>V@ko+Gi}0^T;`7KCitmK!(xqS(r(115qw-vVR7^ERSTHT3+wU}_p?wCV*iyncW={Uts)F}6MB zRfy3hxTRxBh-VM+Ms@2`;QcnM4w`@;eIabstl{vO20k0o!&sJ2+bI7mb}zagrciF? z{Ed#`_LF@V2yv+7u?5^ViGc{~M{jD|2`B+u@CAZVGnB;ulm2*334nkgI;=tgqloid zdWWZm1gbdb@xWwFc4(suq!(J4g1J@2VX+bPL8aFK@SiV`(-0m_)XSnK#tSIim_*k9bGBZwO4y3X9%+H@!cId<4B3-(Guv*z;JFku4Gf1HZ zp&_h6?ee$DJ~VkCh?p6iEwM@IuBTm<$X-WZY2*`wHjLef8qC3c0>El#t@?*k9uKzx z++8HG1ki)P%+VAA4)}FXDK}u$^X-wKhBXvNh7iAM>N+Gdr-+sU53f?96vm9;^~#74 zM5p&Z1s+Xk;0DB-7%;~P!UGo^hZ9454!+cZo@)ZvPA-*v9i$d&C>VY0;h}=7K~xfp zthx@6z`60WO(*&hiV!Xo5F7XLM*pW7kZ=@Acu~ylJvb0Qd&I1EEm2 z@oEVc)Rd_apUkqEbewc(YUrri$e?Q8?srU@PdB z2kBuNiTo)yb?kplTM_pPf9l84hKAX*a08gY_Vjb`ZyE!?IuC_~Ij|)xDTh4y1IYHh zz*6xW@7o@Q-6S9W{Wwt>Lq+}H7uAJW$8_wA|)yE zk2b1E80>q>cY-kP!%_8Zql&mUWLX;$@d3iknV>!7*YJyl1S|)wEiEfM2p=5S0jlvV z6pod#<&>|Zw+IJFp@=1bIYUBv{dupei_2R`|0Ge^=M=G({3AX@_t6&4t<20JJu`4O zrpWtA0Cw!k3X1zKmQ_Tduv?u(GK4<^`EX_#Ss)N{9dbapC!s4C#MTl-Ejrt!{kzG( zh!H1?z|WrpCjbbEe|r0$C0ket#cw$sxv)Am{KR2q%Fkaf2|)j!p95b-H{g){=jXuJ z-yZEhFRT&$vlIUP%FU(Te{S>NFTA+<>c2npe?Q{ii}SzB`R|pWQ2y_=$XZ+_IIdqX T5E8^jeobCl>2%7;OSk_A5flPM diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/W1D3_Tutorial5_Solution_0467919d_0.png b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/W1D3_Tutorial5_Solution_0467919d_0.png new file mode 100644 index 0000000000000000000000000000000000000000..bf42ab982e105d2e01e4cff74f00348a520a938c GIT binary patch literal 72195 zcmdSBgHA|Px+x?x77OQqR_ zNVmkM<99F5`~4%{xh~If9NF<%Ydz0h&w6@8L1y<3#vPkBZQ3n+_0p|Po35HvO_`lkBDQ%FYqvT~5viR8}^o8{$6yn!s-Vd(qX$n|{4@>Di7ypWNPb zeth=<59I@wZl3pGRsQtrpO?2-{C@xKUz{v|e|DQX{mZZ0AI{ZI?J4Q6FcPksG!hA) z@zieB8IP)su$$r+7Z6{Fu;cdDJNzr&_2)nL_;?k!{olWA+QIQ^@BjVFzo)NW_`lz8 z$~m$fpYMPFblzwG?f-s%Z|DENexPlnElofDHn*yOsDQDnSki@9POowS<9flO1^o)Y zJjMbRI2RY`sKN%7dnpa<5)z$_M}OILW7t4yo6BsSh$PFt9ZwV@#ndguBgGtM z?5}Q`5oN<8{O8+Qul2%gi}tq}#to;0g;mv3v~Es+FB=#dihuE9@1w_$DXz599Xof< z+P~kl>A=;rw6ux68`Ps#;ssJUu;k?AUQD%dGhTD{IQNz*8<_%H&ah zcuo!2Tj9r|L>q0M&}gAjHIH9=edFoA;|KQbd;8(T8S?5nMXqLF-#zWfwxoRfc1J{J z*S9O%=$JpN>3YOQdoEW^7TDDUa>YM+@_Uw5mqu5CliKq`r~3ykS8)CF52JvQPq`n9 z`6)ILtB5Op%p-q4`Rn6AQ-V?$pT75CZD`Ee%0kTITjOrs29M(W ze_rG{PK4>WjvU9s8S@qC{#@*_GJfsc)eYA8jn!GjqLs;l(YCah3l}bAnl+!o^V)Ro z%O9+)Ry0k$$`?nY=(ca#k>nouu=1(iVX{zoED58{l(9FDS-(Er&mqBb=*F-`PqFrH zM#0amx?U5nmU*;t|DmX*=D*|F_2?}}w-?oF@trFdbBwmTXp zjvhVFTlnL?uvO=K>*6Kvw0ph6y}iATdtTnS5gQ+G{^Q$!alc){AriX#>Gz_!uh$*x zYSNvibn3!I6s6bZ0zfpi$tMTySB_U+xnr1(RR3ow^1-i{V4Yb-Rw8rlfO3yQg2RpZZr+Frw^C_9o`Q z)W`a#)KVgF#$&NrHOE?0TjS)z1@C=3r_Oz-F}=rQA(GW~_~@QX+YhkndaY%lQruNh zO;kR4T0Q+zU%79F+ug-w+SnMaRkt-_0{LnuAQ`rp{rib%~0tZMsT3 zCAHX$w%n0rzP!FNHOtTa&pFMsyN^!h4Gj#$P8F{x=(phpZol!d8DPKogVHY6(#RgYQWymER!!6dOTN>5AkTdN0DmmDOs5m_gL(G%=%;( zCo`Uu{wcSiNT;SWeVO<%3u<;UDF)$I`6or*{PN?k+KYTRkk?T-Rq|{H(zWT%wHP%- zS)Ov6Z#Q{A9;HxqM+mO1r`<+X}752G2?K&^H`Z5>CCfx_WXIk+!VX+fB*ew z*7W}6XiM@~`h0poR8-1o&1{pQ`baU<{jXoYl2Uv^KJ2|=O^{>B)Td9EE?>D~)(|E6 z?CDd|2UJqDL{ZnTWtq1+&KBI0!u7~&spQ<;v-p#foSawBpFf|d zmKwzd++Kj$B?%Rbn) zr$vU%!$3pIE>t{1!sXV>moLRUS94Re3*t~3NTHtS$P%D^u1p>tHkqL2C_di9)`Ay0 z$i|k+Z&=NF_U!GzQ!2M8s>#av_NKwv>Yv|Xb*8$Du)Blr-o1N(o}S}*+1ppgub(<| z=7tJIRmGQ42%T~A5$4<3xg zwisSrnnmSRb#=|ZaPi{Lr8jLXpFDbcwz`?3mSx=|#d7xEIeK25COjnNLT8&TTeh6! z=9a^c)@H~$?HpZdr{y+8h{ntgHE{9q-F)-r4UmHpzL;{JtzmGEY0ZBRw*9Q>QXU$z zva&n2Z@*dg>|k@ThZ|lgudoQ6*>q!lP0VdRY2Utm2M!;8FXk}%9zFZ=dWdt3d25O( z&VZQnkALu#Ux$Sy{`J=;hsnkMbFmw^5qK-h|-N4FYra1|2e-|$mH|CmX^8&j`CzT;QdKkY-?-t zv3rG8rJR4TRamC^hhXWMOuHck7VNX3hG;c3{a=6m^#C3P1I(($egZ1NQvV($YymL1mo#XHTBo8{ zU$l$e)zwmU0#j2{fu>^94Qmo{SQ1fAq9Xh^UB2IIMA{y^x}=!n_;C7iwA-Yp(i)zm zKz)>?x!sbcRbH7{``6g(Og1()dC&CLSC`@rORX>Jhu74H@Reqce~FP1SofSCSJz+s z(R=0GxxY;F#{B>N_gvr7P_ziX`8WGo%^WMCavuhL`{B?3aq1L02U%)+J@ocw+#|jE zAbZ`E7V_T1W(9y%mM}k_X4NI8 zR6zPVwIidfz2R*96t>EW&)=K3F#4zDfK1X@ycr+&B<^4LY{ zm-|#LQnA^6(NdX&%$h3sy=$4oIF`wGhd;l0(9AC%CYZMDh7IWZ?j3tPHAyvDBPcpU?{Imc%d8KosKL_4 z-L%_2_+y5WjrD>?zE5Rk3^*+Oq~pfmP>WjaB;TYa=(I0Nn%G)5%M#sC{KUjm$%gP` z!IV^tLDefq59{in)8=#I?Tw2wHN3t!dPBE2zWw(gxg`~?^x~9xW1M{XF@K3y6H86L zdr+zSvZ+=AQp>}~`ueK^=&}Z$fJlUNq}+IOpW0#f>He0TKHWubR9dJ?vPQG>+!q;N zR4~y_g=^QIO%DX?GNK?h%s9KQiFc(NUf2KlaQl_Jci)yy^xIqH0&MZw_Fs(wloF=q z7>7T9`joHReO5k3J;TVm>~7Uek+X{n1FD7K3B@Q2b`B2lRUNIP$B&oYPB(bUXVG>h z#-^{lz1(f>W`>cxKDwWyhq2|!T|>s%tBZa9q-@!(?sjOD7h%9U@d0T@nsL~9h(z(C zA22k=D~d1*TZByTB?4ZSOK+?Oj4d#;u~`>Rmor^Kfz7Z;b)NoSj)G@A-kB$$FR8Zf zX?t9A=z)l5v0sW?40a!b$MRg2?Mh{0FKIzaSfC)a%9S} zJ0jP)Bi5KL$~GDmr5T*MU5x!oPM%UtGLe|A72px?KEcj@82F7}%zpUJ0WMW;YWC?v z&%eBr-LKzY`LY~aSk9rn!zj|F%^U^q=lTq!4KE*KXaD33pb%Ni`l8XL84GRI=}5Iy zpOmrmum&75$&#~5nMHkJe6V#le{~>N@WjxKI5~zG>z?9rbl3m+wv$M0G?GnDxba0XV)5~Kr7L%MD3K1rDOL(X5%yD{M_m`Bz zPQ0?5KVw{%UWu$9$;v)9f+--qOe6+0Su=`V!)!(hKO=*x35Qt&44j zi-sKbeMdvQf-@7*O?RB3$ZetCRyR1ZmZr6lmf^YPTEMevG+4#{360MOt({VmW7XBs z+t7c-Zm^awOVqIXwN~@UdvNNoyNjcg)TE%LAEkTbYgXqP6%}$qRrY;HPrZ_=Q^dS3 zUCpCsS>v@n8*Rb+aA5AeR$jUiy#YCok0(xp5)EVfr4(OW=yKBavk?t{cvVhLkkxC| zroc`u z#)_`0pIs5{Gy$N#GCNqu;#WY97$|S#vz2__;>C||%(sh3aUXil2O3COo;kJs51dE| zT9cCF@o@eM3Gj^Y){E%@%`VM=6h3Z~c^nmvU4akJI*x~Y$+6arm+C{s7hBb|Uzwk{ zg63~W^9rYqJwA6+^;^JRC_+v>9ytlVfNmXH!lI(>_AksjbJH3vfdoI>zvV;o;tp;Y zHCTPlTl`_n?_KR4Pzn99mSo3@0{11`U}j}afpXtrTfD<0*xzr^y8Z#~I99l5DQKW#@`*pGGkjqcQk1i}J@z zl8p-9>y%5!%gfsrtm~P!LN@A06^=Fnq8E8iXj12Jawm#6Sv0s+QP>~q9TqNW%^S&j zomUZ6Qc_|A=o&v}cX`+G>p_AOvoUYq%1KAK&JFk4hiAXqOV6WKInUED&boQ)u6ys- zYNg*rxGXL&=S%)FQbl&+iB{=O_J!j|U%h(e!=s(wFDB)5;^^1LM$4#$+AyISOG~4n z7=NkiY?1o{E1#0&qNTcepP&A@MAQFmV{2RMP8Sn)-&k94Jf4u=G5G!acOUV|++(dP z&p^N1*K}-a4pvBw^e{Oy-#lF3kg@`LHa^N1)}`O}L61(`qONc~giqh6^zj}hKxEs* zHE0Z7y$fsLN@6rk`B&4`T&V{_*)N%yCEB!VJGImD+RQ3ioXSFFKsi{7x0kjz^yZ&b zit9J);i<|UODSNMYLy4K+PdrTi?Igo+4&H9*(YdmOv#!#^(t#19QWRD^GTjfgo44C ztdV6qYRB)>$#HajX^2h9T1$qFKnglm2@^H9aA8Z^(ZRoK_q^Ks*X)k*h`l1CUt^zp zE)CFA(rII@HD`8#@ltGY03@eB?RSM{CQwbJ1H3{4a4B7mBp4xsx1P#;laivSas`{$ z>Ko%|58;?7f9d`K4i)}vt+Nf% zsn}n#w@{p$z!V*ZKc5FNQA2O0W0OjX_F7jjaGJ{2a2lu%Tt>^JMQ^OiK-gKSnW8t}pkJKa+_lLQ z1s%L4t8cLrB>Nn|xSKMJ6`@vWfpL;UpXssUxQ5RfH}Sn_p}VomsV5nTXe87)DlP0% zCiJ$^_KbM5)|6!6Q}V7vj1zaVD$nL#6ECS!Qk0AZ(L8zN$iHjWXM?W=^F*fcf_G;F zm@A_(9_Qc~0}v*>Ifz#`1sy^o*$MI%-o^vnJhrjVE=t@<0i;<4oZ^Z)NF8~wfIRZ( z4qw_cjI#iCp}8pLxki&mjh@cFCeAvKu9Up4gU-WTAy6-IXc;IV8KwBf$jAtTArUi* zS7)w2?xPmD<*26WSgHt{d_DrmFRB%sLC1Xd&a+He5a|D%g+xPzW%xT zic!ks@UY0>luQ<{SW&j8h6|a;fxuBvb{Tp~b~6c^phrCO_07`onjde^hbY*bsw@4N znthv_uzfJ%cOZsb;1{5gGxPH;c+mF!m46U&kH&Ki*!(u{@#B~MCFer`n%*1Lg^6wq zX7pA~Sblo)M@O3eUSCJHe2b9+mDWhwV( zOC=3NSts_|L2?qXac*FtmET_44g@X_5&{(T!f~P_78;ri&V!Y&;<6Jpe`zp03nD0G zCRoSyWW!Lh>qJ(opy#S{xEBS?7ce9azfR#xcc;z`fd(}{WL*;Y>S?wZM732DdFGMOcsM^=) z=`hn@MOG{dA1I#6^JK`lCYTr0ugJn`7ROKpzayGDw1m;|_6)LPgSgf2b_{vURB>Fp ze!Ve1+S5#}AXGcg_A+?{&__;>^h_yIvn?H{mN^L^5vR2CGwY+JQ_+0PKm=8XRKAu( z>*95r@*T_zI;fBgCi8|^aZtk7TOd?-*;KHAZ6;oO&H2V|sNNKhO9NEDym)SHp@;Xg zU99MLB1;2z3FzpuU6lUw&kK5bdZdDZqs*Z!hNLLA-70o>Au1gTtD3WOE=Unnj+12- z6sm9PK@|M=q# z(Szhd`LW-dAt2C(qP@cA-+<;ff!4ew$Th`L>DH|#5J|Lv>NycKUh?LUnr;G^9E8FJ zT}!*r`6hX1qC*)sL~)PtK!z{3-XcqCLvxUGiUl%GHeH~tpS3l4sBgom}+Hl>Lj9ds3)b9 z^W;fcoVX@bpXBP)=*OARVftJ{a&mIeo$a9{Otfdn5ZMdjN0+DMR?dB$kpq>_qxPcX zrH8)hEb%I-8$hi#2Tag7Esi?$m?{r=>R@RFZkj#TmS5Am+*Lww{B#+8#>B08bey(A z&BcPL;tZ}qye$)ciRgIHRP;-|cRP0ZLPfOV*2t_Rbv8lU?Q!>qES64ZwK_trB1G-V z2lGu$P0Mp_`sBv8ZQaV~wlGM%VQIW~rs*(OsqUEZJX6x+*#3Lhfi4I)AaWWb!t5yP(xm4DHhw zJ71hS@llojGhUJb%+IcT?DOXveb|h({NB*}I+#Z-I&*Cti<}ha9{#>-mtd5~yt+OF z6G#7>MUbpx(2L@PH&%K!IqLbZNpYT?YYlit$ni=~6<_}>`Qb^;@ttp=m^im;B2 zHYMn4eO3^7YtBscK$(^ki_uaC;GQd07Td=Wah@560w$F<_Z`t}Ar zoSQfuLxRuI`=a|qtH2X|VPJ)IWIMIx7)QC34?0l@B?o!2)go$4ISgE1;IDyP$w5Km z(3mLa@7u$v4d$suK?Cb&z}XkSJ=kM0e{Hle-d}sdNXoOzS%O>+N~CP`CX-FCQCLv8)YM7_DfPbUJf*6iBVF5aQr8aG7}?2qXe|yF`0>t zZR#F<-56kzoc>m-xQ&ft7o&{NVLde#f9dsH)2^|u1#^#(eTjtiSS5VhCQ^X1j z3w`!;D4eCSj*C9+&a;cJdqWrHweIq`qC`$!K8s7$I8xk+hbmj-;qLC^$0E)Mea0Ia z-k--#ovMzVDB$02v$oL^#rG~CAVB_N@onxPA9npmI}VO7fAJjte6+Dj4~F2l%a}{l z!>iY>@q_spsab>`huR|PI`_dW1A3|BlxBKIF=X4JrbkVyFCN}5bWTg)k*R1X27Vd2 z+i*Ik^SiP{JwOoOT1YZ}+?5TGM_3GF5x&#G1a1-ifrl@@* zs`Z4~1{RyH!aXusRB$}+h56xnkH?Z}MeX_bCdYH<=WD&bne6+HiX>1bVmBz4Qx?qh z^y$-fzcmk!S2Q&pm$DFPui_fsTQV{-OhA}?S=16QulN&xNjZ++&i0z>$f6`92c6dJ zphXf-$*PD390L2G+NNu~UJ_`1?8k3kq*s4DNQgptroNFK2?icmuA3pGjyUv zW2wjZ4~^Le4;?ZZZ%ZqkbmY}7wqX381}#^V^})ts-=6S_x{b9dFUK4Y?F$#~+W@y( z1lt(yy9+=soWtanx|vWW;Z!%nsMe2%G-d6mBy4lS8>x%FyzJ@E140k{8f=W>*ZF zwJPaIgSi*0M-4i=cTTq&vKBN{R8*t{eCh1KW<5DSKKW+>4~6YMY0zEt-_bY22Xf0| zWc;&NB-_FwBxWB{=8LzvDZI5VWEx&BjVd@NX-M=M`C)aS-#(yv;q`@*4F)D*i+f^K zhKExWFP!Dy9-SDd@!h9Z+G?_BL8qX_Zcu#u=}(BSTGNuEO?@-ZJb>LApeU85 zQH;tDK`M`m{@M5W(lF3V`9w~SRUukTISh=5YlpN%pzbQI`9=7Per$1x0eiW)vI4vs z{wn5H{nO&4ddbq#QeQ5$l(TbEpuxVdT&ohJ;4Yqxs64Z)lyv{3(UUMqYph4_W!1`7 z!l5c?;Tj$?hMOYdp;%N@#1iHYUQ%WPNSrW1gn(QtRch)n*;wQUI+bMy+c@82>TfUHbWXAVx_Ps_AsCK2PY-M?AtK#=Zzkkxur6FP5Po<0$THdq!0U=| zDQXNgA_w*8DghN;mh+>{*MV0uPGtyXr*biiJIWD&p`vbL^1gWS`_7!qp&ih!zaU@m zdLrbRh?vXFJ0fk6^nqDdeinr3_h78%fVNhCf^Lv<3D+9eDFzZgrBj%jP>3!`^pGxE z8>3gZHZ_>z(W6HRaG6vM^V;cs4)a|g-^21YtA5&TIei1vsT#pX;Kq&tE~mckj132_Y8(fHm^%6HY0=ztcJc%WfGkThOeDlgJ*} zp052bW#oq5isqrLORbCxP}1*~JsUwM%vy1;{Uf`)m`%v+HFhZU8B{bk$7W0aoNiaH ziaalV%}kS*rk?45RMoAmt-bT;Yj0rf8WcjN%?zU+k!C!5@1N$g^LSIjI+Yv_zMIWR zEncn`)LBCALT`C@C=0eA9G)uW$?WS~=kqL)0_b5&kmsP6d4sj8|fqT3`_lbsVkcsuE4 zjQ7^vj0P&gUhAve!Ma7RSwz~Rq~mDGS{KoRN-c-7AU%f<^6lQavk@SJCD9lRK5i+7 z{JnTj>lXO(4HC0HrDFr-D^n%bw0x*5jC^+=meRln!u*aSjvx~>>p!qCd;6dFoPx4X z{`u#Kk~(Dm>@kDxURE*ttMH#PC`Gl>-N#3&IHEJnyK0vgrmPpX^o)V!8lFM}D^Z#d zKQ(*a8mg8;vr>+gHppiE8wU&dITsSB5HvN1V5N*dr|>&@I3kLb9(GoC*-6}HZbLkk zHbGS*Ne^UU9F07Iq)FV@$-4E(N1HPP`9nGK`N zZvku>d>Ff>4wS{BW6*+y%c32GcBG>8^P7wE+m)QeoF*eYH`bge>5w=EoWGDgm8cZ& zr_v4+PHumRXkS_dtDT99wY4=U3JuA*r+ToTreAaDk7k$Wl3zgF1M*7x*TbulegkERFRuwNs(DzaE}2! zP&EwYIMrbNKeyRPWpJFY>MTxv&y}LCv%RC-wAJIac^Y&zIB0BAlqvRmI(U@4KRGiP-$Q}kQ|NF14+;eQi9Ww?;8!FY*e ziZd5Xe-z?t*RG|v3$V{#K4{FFPTcEW2aC0<#REZ_%qbj*MUEmrMc_S zWJj^q1elgaj#VW25Mb$?h)|LbarNqlJl?a|e9h*@3tJ!t-GU?=sUaf0{rBHLqXP&5 z`}w591sg?86Yh$PRv+xXsiULg`u!7L_stH2by+%FCRet2%Hu}qyl1Y zeRG$Jka>%JqLbWUUra_tITT@%mLd`W;IKW;4{~}VKzzcy?k6DYR4YWB?Y*H$C(}=x zZ-MATX@vxrk6I<cP)$}6 zN?W+=uu(h|!c#cRr4y-Id0$7flB4qc-s=?QH;+NnA54^zVeo!~q?|cKYJ$k1LdF77 zkVFUMHQ`o(=``2K)}aj-56_^WAfx<%LTJQs2KS2y#8I*i!`T^Jj4Zkd^&=u*^bJ-K zI*o4mScKp6P~PuIO`ZDl&yT&1v0l7qdfFWI0!7n?cN92I+%hnDN7fJt0kV-0#;BV4 z)no8eL4no7%0lTm1ln9zUevCXf+kFK6|@@p$JADwfm_h!wXvp<6B%qldy&%8%V83q)3h#9 zIr8fEIlY@E#g^F$f7&g-ya~y0H;b6^{u6SQoGbWC4iKa)c-@JU(-b)dKh&1{8{k4 zDmv|?@}pjPpw$p%@*;gPzn!D*x5_Q=RwvtyqqFikBM`Y6X^b}(H|WGsB38!jKnExv zb1}7Daf;Dav<`spNpYTSyBXghzM*-XfNOp*_y4Pdcaz(Zw0h-1!TIDjq7BxR8V|m!M^bGN?sHenyUF*3e|Vx-VY|HmJCfe&-Ly^mGwHTDxB$}SY2r0PSwI_PH-p@!=l zg-|dW9Yp^`T_F0>93XX=H`V4fN5jVK_f-pg9;8NcPSs4zhqr?tFkidS-)zG-`yw&h z^<^8E$k-wF6tQ6WYuCQMlMN7rJOQzfL(L&?jQLYbnU!XYU|UU#JahK^Q)E%%9$I4U zBN^$a`}x`Sdw6&>BW@5sN)nYoURd8H?y-ia$YC_w)G}meM?yOmQ$z zJckBLPA2An1fli zzL*i~V5 zmc~gsMID00PX4Ttez3B&?2d)shlG&O@D&n$f~6iulI^CASMZlrvDKARwDKgLH|T%><^vfztW4w>zb1wTyt)=t zR3Q}T@YyJmGXD3kUm!v&28lEpu&y6RV;kJfnsR6C!e1Fh-p3OVSaZyJ$%0LZ3d$dN z`SN8^iPs8_NVE^9=clJ5XG~m99t8AJNo_oX*m88$Sm-m{da#^v8`K{^?!|Wm);i_U ztEQX{hKZBBbq{M2nry+5KNlh&7<6;QA@T@0g2%G7MQR#mKwQ-OEh9M&gHvwM`%5qK zb-%|)P#yB}n^YvG+|YJr&5B0YZ~2+NNq+G_Va*@(N(F7cU!u$)*Jz3)8Hw|M{P?j6 zGzziB_aN(pxzve1-E=nfM(y^OU|<5<^%NIYpwEPyP+LP_Bb`})9*>rf#emIY?u%B} zQgE=-?n(GNSbTfa9BX6V5BZ}WZlftR5!jezbM5^DWg&&PVy^7o%_?ylxT?7-fPZSw9GRHjmcxS)?}kXKUw^@8pA-6`Lh73PzE)$R)$C{q3m&D@nL{v#c5m~AtbY#NYv@QFupn1&^^dw*J64T#rfV!lL9&iKILIutWF_j>j zT`7((h)3q*XXnP|CMN;yE2o`u3>^%=YeIM1v~^ZM;svRSEnzF;{v=^}IDqvJsS7FnwX!(f=F z5ZCj#e>s{r#mvq-|1{gmEQ988@@h%4z?710hV;wYDan&3tD3luh(|514J23&_6fI4J3Gr@I_l zwdtDiEEhuNZ2;69M=pCfg<^1|j@=ZkpVg)R5BX>*ZE}=RQ(*hY`ncS(a%YBWz4_(P z+Wb5qyBU_)b$KM-uJC~rdU?2rbqkX7iO`lufylF5=)Pae>2eI8pXy1)^E3+e$^Al1 zDJW+Yh~1;DsnHedY%IN)-oXTo3havvmp@8dR(VZROzp{V|J+&Zs!y^EQ)Vg%Tyz{#0w7 z4-U2UbvI649^m=j8BwEyP&cAa(Ida6gX0q87N(aul>83Z->^wv@QJ7lPQ4eHl@nu1 zON$ik!gpy77*%!uBAj{vqWYS+vdTD3EbGg`7HV9#c<5+Gink@U^Kc83>TX z6o7|?K-0Tih9XHuhyDKZtWPCno5{@z>%E;se2@&K}tzk`87FT2&1Ty$f-;HJ#7R+ z5K@i9vlN%GN{hMw;DL};=h;X}H&u|&mg~Gl@1O!0x*e@Mvx^Lkm~~{DW=qP>1mK4{ z#b??mz7k_~k6@sMfLXw|+b+f}LRA3$_$`(MuF>~V6zd6K6Oa+@0_ z$k`l9s4LoFbL@3`g18WNAsw-DiTlk1`-~*Z z$vg=JmP8VT0PW{wXTOB8GGS}hO3IYUpvPV03ABo(nfVQ#By`G*ZZ#BQ8>m;Sx$KNrNDV%^YP=_a&wD za$xe#oT&Qsp;&;Zj=`=ip%sxb-8F^0KajNmElMQbrt+SZ`$>{06{_?SjbFdcf#GVL zwwvq}T|-b;)sRHjs)Ko>>5m#!*1eEjdb8e1P;|3F&8xXiy9PtiPI*;{C66CX^Qxc$ zw2?*`WKX^#Ibe^7IZ+Zw!vyuToc*&3=-3|W>Y!|}J7g&!Y;7X7XX%x01r;;}mSU9l zEF=JA_V7V^`gr8*;{Ro)5FZGHD6Bez5lSZer_n6&hU*r)173!+O=h<^IQqEDw2j_W zy5Z*q!bs3qay6OMLQ*O5!lNB@%_^_!^7jLY)QCi{n z|L=)Ckw~4t9=h<faw@{^dA|NY{JfvnqE})Oa6{64fEuvBO*Mwr_aE zQr6s*!G+MFfXY;2laBJue7?9xROm|^3Z~WpmEPc`@AU|BNFI>w-ue6QH()2p4frX9 zZE0PBwV$lBF#P2rNs{HV&Tml%^NawgFjhhCM3BN$sOv!sOt@~X!;$TTGz%JdewHty zFV~^K*qJlsyGxX+)wX5W%YNE`OD8@*gNKX@9%3*cU#+O<2Ym;3+rgWB3 z^0hj$B{4*k0^%}8jy&H@JoycXJT&z-sW`W9<8|eFj@h zSE>sYMP27iQoOnA3KgqTR|1#@7WzzaH0{y!Oum15`ZJLF~Y)#98~ZVbqyL z#AO(q-cs|Gib#eG&sygZN0lS*BWRWh5cn0O`S)1EQ8ua#d8uYyr5F16H0CR^>)2#P z(}V;KbkOBAA=!|!i6+z}sN4t#V!S1L;t#}0;KQh6R8C~K-VdO(gHM*^*eiZ63S5t_ z>}F6{+dJ7Z8?sD5$$f_2mlM5!B{ zXSdjpm0?l>t?irpa1HO^bouI|AWCrz0FuMXZwsy96&=di82rr1(M-C{*Jxu1zLLk< z!kKE|Cn+SPLfA`U3LK6Q&xV24wJKkUtckN^GKI)DbNCgu9>i9!|M&g(ceKtod?X7r z8hJz(S<(+K3)d9*6>oSTP(tE(uWI`^=`lcs?(2}dTQSW+CPF~_FC?u#D1uSQ8Djp< zvU}V2qc?BfjN>=vnf%Ye!r{4-k?s#HJ^g!|+m>k#FY5#814+mQ9aue3TRNzXF=GP+ z(_WoAe^o&N%5+9jHQD*7(x&+23etZbJ0`#XCf2_3_3sV5{K#@GNny4s#9OS|e96VF z03x28r1Bm%DT}b@pOi4uW*)}W2cuWaeeuKNKW%-ERdhg!I*CJzOqHHWHZs2P-a6#H z&<~QUK}3?#5{FJ*af2~LrchN4k+o_@wydk$(JF6rbpwp|-sQ7fo*`W`a(5MUlVmdE zM{6Wf|CyWe6ql}!NFp{igO>u44AAD)9;t91{4_=KaPo%GtZD+X1=vcF8jvql&(j#} z*;&7?`hn&VE$);;Tm&o$;hM`5UZ8gFJ&lurYC3$ox2e8x?cKB_`?vfBW>MQqdnA7R zHS+cfG;^X#imDhR3M%i!Di%sPDxohn7* zXU2f@enkJ!L^6@$(lItA;d@kkNwPRetH7~tki7}SCo|6-#YVpG7OY@@D<--(NCl~; z2E2L|G0H%LE)?==80L(5cEFp0TNtI2HzfQeAlt6=V8tqUr#U`L%8rC{RfTKU_$L%6 z(jZD9nO2jS1SqyNUmk&$Rx#H_%oF*G%LDxgx}3J%A0}RXu;&XLN^_vvlpQB!z~%%a zxz+)B$P^isIti^JVH|kZ)|qO5>3^$*sNwSwHyVT*vCe>FI8;cZ2nC9Z;UIQn`k07onn#u>c^n{-Ew_LCGgzhy3L`;64}q#i3R&4OZ4IPq`dDU%Rsx~W3qlf`3=2}^M8rlr%w}nXTNgB z+p;5*%#bEkJ5F^c6YiE=S6f0Kv6g%z?e1lzea8_m>18qjXwWCfJ~0WRMqKxYrRMiH zW8@4;xWAZfMts)gc|rXje}7Clcl;=cnnJ@Efry>n9*CLv|Kiy1$1KlJP@vmr1RAnV zO~i#;*_tI?QN)y^fc!oQ$my0OG(85h8z7~b4G0aMQtn&s&IFM=xrOY9mG?ci|jLw(@I^L-42}j9L+_)XajJ)i7(6}+AL+*{X zMQiguL+&zcvfjtnmxz4hu2T>!30)nj@IrMX2uYNglHu;|ZlS|T1UpN*-R7wf$n$8O zl`ZQNkxt!d6d4lzAP~V_QYWV%@U*5XoaPrQ>Le$P2y4MUQv`)UlZDp0u_BGIS+p4` zlp<&?aRu2u&V9QJT+2{fG{K;mijItbMn1Q(^BB5irrWC#BJ-i~PX2yclMMHdT)7W; zm#ACfsnBzjm4zi4akT+wZm&LH5jj$uoDzc_5qbiU44&Y}vq*O>!gOuCm%p(>_tmct zx9=Zky=4Xl?21sQ`~XH_-YQKyUmzv$qAQt!Mq24Jn0sWR09+lVIJqiYrd;^g7%h{T@J_LTOp>L2=ot7z;>gdnix?pdjeIH__)A)@6y1^x0Hzq zKr&+aU_75mKH}_`Nj6*Htj6S=ZnOO1k9ES;x$qDW3o;lC@D+GrJcaHjy}lD;BJ=sw7F+I#oe5oZBDHp++mi1SJS`Fys^Iuvd^I zR53d2jS;KM(9spQ(J@!1bs=XpiqQCoaU8Q?{ECvmFh843e*nx;x26cDoF*giow2Erm`#ImtInd@L=+T!KAO1h7*`x+Q5+8RII?0Sn&2Bq-lJ_0Km~ls00# zHg3tq8RSCT9{X1ntEvJSo76x6KXMNP@mL6u#`bfON*Kf66+(hnK2l5tIn~Osaf{C7FJh?E}jLw~1hixFXRVb!!wcP%`oa6)6F~cI@6AsCSrQ zBULlo;tS$=0X9()jeGY3DH6lNO3bSp5*Es6Q;N1Y4e|vJ|4R@Ua#;p!$`Pzew#y&p zklW0lWBN^fVFA*QNSo*+LqrgCqQD#8LQY7;gbuD`cnJxYO!)wIkm5m>8FI_;J;k<8 zXIQ6l`^{5zOLP#)xQY5s?rsR=)zyJ5F9yjT$XXF3pkp+HG)h>of;n|8ucEhX+cpNG zj8*7Lk_x~fC(~pML-LpxJB2tPZY{vTkE$u!rSSgeB&}2@dSxbtbYwipWiZO*>fqCw zariO`7(*g=hmh$MZyn zBsZ===*q{WMtNQnB-boUs&Kq(Z6TDJj(i6S88}i|$4NGa)rv1VFWOWSS5kECjcXmAHgjV2$3HE5`z0q zw0&Xge1k^SKT|D1t!Le{u!CHyJbE{FM1z_nPz{z6K+XKog0E|J`b4n3_?*k+^+|Mdzk z04t>ploO5Q85s(&G9_1`#braJu;dx5T_`_foLvKRE(XCVQ ziHS$_1WHFc#(vfOCPee4AyMaC|GpFBCkW4sG#MMDLJBKjZ((A?of-UxxNm>E$Ia%s z_@;MYAdr$hv6iBMYb9{;fK5INu7o1H%&|m7Ez->>2y-5!#2RimK_k!eJIO(~U*q&{ z=ChyCEgKr`!fPd5JRQ5;DXLu2jv4r;Fo*3U4+PuiFl40IHas*k}{2+b6rTAqacDUF~Aa3((WKK)3|G>rjJ z5ii<3B!Zw8W>8b8LtG<>wVLX%j`ZaD%(y<;saYDb3yLO^1c@Tj3O6Rk#wbKBN6MBi znI6~K5U5BHLinF{e&X3Vka~xNx$z;dlZ&_HMY>*(<@4KJc1KjI6|$CmfKGk~s$VKF zqp`R}&@nmS(>oV~Kthqg_6L^n?k(#znVLSN32kUT$7)-GA z(y@FESWcuIqorLZ*P5tk!_5-rm~za5eqxH8c-+z4^gg0p>$ueoe%8MHg-oEKD2q1l zqqqcD+#!>z|zmnU@NM^fN zrw`|Y++>kLcKgZn2;w%wZwe?Kx)HjxxcD9qhg3D~nqbbG7r=&bNk#zf)ETr7pPyr8 zmT>-1ygbU)_#?6&ZIFoLwEh6aFjuXiPj}OZcG83grGi@%f+{ZP^XI^Q5JG!%aH_TTq8btAX9Kn8@7PwE?L4(Xo|q|br7G!p}< z_v01}%#H#I+bn(zjX|5d%eXTaTSL@Ql=pXHltND$e##Mb15YGFs~&TG*s)(xf#=Z9 z!%px$v;*M%d1(?D-c1~v39)u$%G1cLeAs0L_QPpR2YGe0P+q>GpE!K~w4ES%Mld?G zUdsi4MP1zS8_?h$4KDw+XTcN(7V*4Mr9~^IoKU`LKyKz>2v*nEU9a7_^ADhR zGH?dL$I9ptWX_yiuVf+Tj4~qdUvR74E(}tKocKMcQzRuJ`Dhdzt^#*abwiYRa-|yL zPD6{E&tm3rW2wQ5T-t?w&+GqZW;C`e9-ld0^e1G-2@5;%uJ#&!j>JPS=W5pb@ljk_ zv$i3|TFE3hiDi?N%%tPBAnpV>)FbdlnqkF~JC|}kImM7rI`%(_>5`8POnwWCn7Q)( z9RQ3KV5Kjsqz1@ZBlc1RW#=En{3hQX`$>lk{BYe=fp(4`kpnieF}Yn#wJMOu415bJ z5|*>T;xQ05H@U9ygj~o!pds%umgSfeK#v;*$iJEZ1e2uAMDbbjk;n^`;4U5lo%TpA z{#*BQeMP+(|6L=)$QRR_^AH zMxrQ*il|5>sidOeJwL0p+V1Ch_w(NO{vF42{QPm>dvCSYcet+4aGvMqyiiCO>}s5T zcA$TA57Bxxqg>#8KxJJ=cXucjC8cwu=4Rr4P=&2@{!$Z=cPoMEb>zsA&L2yskJXg# zI!g|ncuUKwJ4+#QzX{YDg?I=V5j`K3=ZIqLIkAq28!gZ?5BY5_!?p#$RE{U%04lT} zMeStaSlFuU;0MM`^S1O6^$IYFQ75eu3bX{QeEMvsl+L0BL7yOPB`ZyUeb-P3!0BsA zb<5({i^1rUMi(ni9b5Zp`(A%|tB~s-)e(4$hWo51cXfAHVjnvId`8W^k%V|H>D$0# zIyRN91AJ16wL@La8#Dl`L&31zC8tGzy2Q~o!#?TFnh)sQHdFYb&IVGFh7)abv`+1q zxM0edg-exayuN%?1uB>q$XfE#PrC^%TxfxEzbeNohcGU1O(@!9iT(HWeJ3O_C9kk> zKmML$?gc=gsI*iQs5aTG&0e|hUHgPyFr-UFhRSRZufMVcqCa%(w=#R)q9QBbo?4^`?EEqf z!Gks-12S@9LtpmBp?ou9plx>c3aO;jG$2x+P?ORIqACsYi{2kU$Q%=yWh^~QF6b%SqW7+qJt2#3wAH+t(*n|LtU`;R|LEgWB5?TN~ zJ6TXb43OD$Bm83ar`P@B&8CD=#KtOUP`91>BO|%ze*JlxIt(orLdmH@ERk~B; zjE?s9Eoe=sOG71m@N~Wg2xpRZQ)d8j55wtk&?9rq%b@dhezY%qXMJZb87%63|Orr~Md%%X_-4_KUK=%wI4TJEGWrauQDyZL@P8Np4Or%k#hkpe4PyPp?j4{n0$ zT6;*+|R%!WMouA2;FAzGUMSvL~ zZJgs*Mlhy32?-2nLLfsCghfBsKzW3~*QKS+@IEXmDY+HYKn1GH_{Xs8OT-{~>8PX1 zCt4>l-dP&M0+q576e~t7g#}1n%$}YHByQAut=e=f(x&xjKF+U~%Jumdnv zuDT-(kJ1?V)^qN;r3Wz_mV+-h|vj5HS>wP&?Qgq+%1H@jiE` z`^%*~yZukAfER^!dk&Jh`6ARBCMFQQAUGJbaiqWtW2-He2B^dmuD8C#v zmKI$4L!kaQArd%FWXO5a&w*aKG)h&W5rry{4IlrF~g>H0(j=kdha@1eu_sBzytel9c#vDSW^=d0SO#N?tn1+Mh{+=u zr>4B9x|3Hf(6E0oZ=j3JsL-!V^>fPw7?UpxkK$ay-y5Ot(!gi2{v0(vbRl{H!(Zso z-x;gFKhOTr>7MH||K}h5)r0*b=doz@TwftP>fJaDt8@NXubnz<_MSeP$+;HByBSY7 zmhMr{R!qc4$o&!V@0hSN_vCC4&o&P}2P6aS8p+ZT@+Y^VG)6=i+5QnD(Du zV`#}mE^pyyoQq~K{`EagEM%{;sIMz#29EnW>Tj3N->r#g*MIU1B2Ry58tX0S_y6lv z@lRIZw}}g$V*9R}`k$`c00&Jq#&qU?xVyfpoBxwlVZQv|+sTM3JKwvwJVWul(&3X9 zN)t^;EVrP2rIBbr3L2z&($6A8CbADh{=5k(0O8Y^U~!4xp8W87D`M>?P*JIt_2_jS zEat2mH@smPw;eE5OoC3zL`^nyZ~?@~)Q}Pzj~0Y_FZC)ALUCWfl!HJhS=jZOW~#P= zg^ozwasKphRCq;Z!(eOoLn$+hoSCUwK8x8VFl8bjJ3%bL#S$h^_lOh>y~{smPI%)u zUm^qml)6^!-Cg)s*bg`I>UWGAl?R~O5f;2?y@2ysmC+;uk(+(}Ft>aTC(idBQZ%-W zuWf(V)kQ+RWK%j(ptn(981_`9dY6L*+WjMHaKLYF?a`vnclkB2sLb3IB<(bLoY|u2WPpnSTz8MU_)GWmNU?_TO zeY9$4h3HH1)*Rfz*ZmdMy&s{GSP?M=lWxe$#dfbzViRicF|*eY?H75aFbA`PsNk9# z7;{iyIGEnLO`g=!c0%=znB}oEXoPsQ)Fm4Ly&)vbOOrFIKsHg^!z_bIP84*Ehor7N zJo>M>;Jq&jcyqaB7=tG`y^-97O(w5S0QQwW$tsV~pB<^3-Y zA_SP+Ub!S_Gu%Bq>4)2k5@DX+`gF*1d0{eOM zelA;3n|=TqA$3!`5NZPe2?bw+l8#VNdVLFqU*Bk-tM;-|Ed>!HP+&PFIoIC?oGpIx z5f90rAemxr1vSMu zp`*X#aKaJkD+`~EtOGAk&x6uI4dSj4Ay9)Wud0ozfoT@!JLsg4 zr!JnTjk8|*1qG4U_fwgUBWOlv9TQ-ZW*kuhBL3U%e2SiOrf(qWDsHv`#Mk*OOT^`` zGic&pz?iR|Je^vbCnbZxUecoU{Ffq73=3>4SQA8(CCr_e?}uisZCT7YjWd=H@@-hqn#v=Cr$ z*+QW_i1Z{Qx|=JC6WXP(qQ9YoG2Rp(KYrYN;-hZ>{97o1ZrgKoPOb!MR`$c0d4EeDIm__*Qg+;*F z9DslbWGuia%COF$UKb|c_~(c{hBH)9g3~c5agw>qHx;|M?x?k?y{s~Z1iNOo4_yCR z*N2Xs{@BDd{u(r0pD4T%zm2hJU>JmbuMVArscaYmXagz@O^HNc$KM_Zg+wViV{XTo zW2T&1bk9xPD9Br)1o84Wd3AsPjt2HVnOGvu{P`QVpyHjGVLub2l-d{TK0>-?$Sv_K zchEe}V0^uGrrYaN6INKSQi8m?9KUk*xm%VuqE!z11@j+sx!Vt3)DqYd@u)Dn!Qm8U zZ(uS&Y^Pei^1v@^CJUa7hKT>&4l_lRhL;*vdAU6Q2rVIr|FCL|L@oI56>{q4Qj7je zp%}rn8Y0ilG<*&eW%`^saRA=rzdcilm^@)<9WlH|9u>+P1ol!6lpq>XC}luQz>V(w z$Vw2(1JFgI$4nY<#bFe|!pWS6t;Kc9=M;`hSSt^6EIQ0a=+W|taXI=+W`wr@YbI$_ zaK6bF@a3!EF_KnDr&Qt3gK-FAj0mlUMjt8Yf?+)w1#f8>&|t6}@Bv9UfcHOkdILqe z_S?5_gRh~{UcBt#=%HT++#piM9Gkn`0S%!jgp$X0#~`5pfFQ^bgBLU=hAsQpsUPMw z^nN5Ks?1qqO(!CBh}f||g3*y9c*&;+zd{r-L_!sbhWeQ0(uG-9L%r#shz`J09M5ra zD$u8cdNG_|n=ZBPkcKZ9N9p=r;QKd`uw)F>fC5QI%#%7ef5Bq*-kd*=$tuqL9W;I6 zK*`*Svlq&=liWa1SNIP6NDp8&oF}_DF6fAR@O6oSw*<4D^MiODm8Zfz(d*U2QY?gzyyoH z@gYr-rE7%%=Q6c+@iYmkVulg}Au7x=FbIy&8i~P6JJC@Q<9#Q=C)Y#!aP`g91)T2~ zt#dkJ+C!7%eQ5hLyI$&tVDy#}60p75G7faK#5IK3y-Tuiqyt3)>^-U#A$AbSU5?%T z4Ps)F>dbW)W`E@E5LVB#eWF_7d{G~;*Jqzkz!z-qZ~aktM#mLVi;g=20MoenR#?yU zxQ5eCBN)AEeK(q@A%C}9xNYzm#e|MOcDkK&k@0a~`aVcgj{mUw*?S)6qg5DuTdy5A z&HMs`J7O2GXk=#{JKcV2V6(z@2+izrmlOU-gdGG1bJdkNH+m|Pr|JE9YdZP};tLtm z^y`p@?1PvEo2+>I4R}k@BrQO`8-;V-HUSZx8!(goyWV{#g;W=jb`~|Lq)}l(fhcff z0KOfaot?Y(uz%`e0V0IjDW9UXpn70oAjePVzp?&|p8QwI{a?;o`vn#l+L52maFND4 z;x0^tzM%^Fz|`k_%=A%56~-YAj>l$6OND_~bv5U&qM{amV~%aA72+f&Mv*>#7!u!! z6joV(9+V@)kNoPR9O*L?j3FK+`%lz{jt&RTzc8W>3Jw<^jG8OYrf_;xh9sKruRem2 zK5!DegPEI!Ov=lw8ps)|4%qHFA@cOZM4PoAH~Ry<565f}79floIb{ciPb{BB{QWYcI^yT<(c1+ z9rOT&U5dR5P1vkb6iFZcdra5}e4LzyGXv*TqLYdL*QhXU8|<~<(zAV``I~fWXg?A; zY4F|ezpKm|5utU>P5CSYXb{~ZGiy}ZVxI*xVtFNw9S;4n9zLAl2cl%hkGP&4_R^{ zlOS0d#?Pv}kD3OMV*)f8SVc6Ai$!cd99``e$lpIG=RNYq2F*=|L>i(Mfh;mq2w=DY zCbOSI2~P6R@D`y(i<;rCR}Oc^8z}gg!7%&XCcoI)p}uA+=ZdHQzD{PG022fTO9_Ph z&nyGhO}T_gb}vBdaZFS!IB&=Vb)iFK9UL`p-m=`~LQFh&B7{K}+wrL34P4w=TAgN+ zSj9;}4-Ea?r#=KbMW8HTbbO#FZV3qt;iw4>$c+z^a!=)(oEC!hkTUXM8N3@hq%1iX zhx8NabLtv0R6k|`eqf^d8-_@md)n(~$@3pTt+f?vx(i23#e$L*_j^yTFM-$SYKnRl z`&7m8t3GaH$Q?%BrpgSAeKaA`WTi&hH?Ye%2W~y;Ba*gZ*;08&B`4UfgBCw2Iaqjb zF2-f&X$@r@ZDJ+|OW!k~bs@$erX5{p$r08`a`cm0U#Y;?Bqko7MZ{pC<8qbZ<9y2a z5WzRo9)haGHiIfV8xqNeri1LiO;=xS|6t_7Nc@N(Xlp!P96b~_6-J9!SDp0I}AuN9^}({#G!dxA@s%JNth9r=by~}HfN^hGQKHM z-iwhq!47!9DxNI1(KcQs+ycGKFSU>IpM~WkTx!JP@VC*eO#&3HGp>d5pph8I6-tNW~GW0yaUJGOEKr9*-YS zFBx*P_sQ$(xQNt|bq*&R(w}#0pF}fUR8MOv8)(j#3g0k)1ShC^TVXw)*7!bc=m!~$ zv}BtCGtR#-#_%)^_eTC>k8Ih((2V+b%1n`O_R5edt*p<$@mny5J20>`O|o**(C2c> z7&_WBV)DaFSQtpVl160f*PB+8nf&Em-M09x5ozNXMaKUFzCErs;G4r(@hu&f+mUAv zZn)fromttrBz5ZK#K(W8ONMc7Qfapqau7sAw#_!2Tbu9vO9ef?JLa2G_JJeQhVLBc#kPap3%<$((_^Cu@Cs!6*^CtP#y^I z7Z_f}U_BA=G(1%9s1v_^3pizhtcK1hBhZIOlyfmhl~2zKw7|JI$m?f>?sq&EYX+HA z?g$mlC(I#l1$c(F3h&W4Y|so*))}1iyW@`vp7!wtJqgBC6x9{t0@3Uli7T*KNIE98 z7?U}eZPiI^VS~2s7Bw{;tsgOc`8vUI?T|F$SQ(vQ8e<|Gz=d6tp_2fjfZ7*=9b4ul z&J|9dfuF~@O_>3WBo;4j#^XuHNVhoC-`~HC25@!PaDH~Q;pq*^V0G68Q=o);DC_PQ zh@-=+IB$y)92EUq(F9C49Js&sr8y?3s>Bn1;z3HLu=mt_FNv~~%I%Et7x0vc%`WH!3zilC zmISmA1_clB``;uzS==1Pi%vN5!hA=t6Ryv4Y7$YD7uX-~l zf<-MCnacD9*-cR~_?nN;D9EkLEtM@WX(XXSL}5V-d9%eTaBgaRIIiZ@H57}?VBk$; zZ3X^)v7}_#`$Nd(->UW}HK9K#jY$HkN<;DL_3STur_6cvi^e@iT)&Uf8JCI6!cWqK zX$AbWmhQZ}0UZ77L@%IgbSNpA>b>1B# z51N~RJBHV^1rD^uLN13~>MmwoNQ#BBpRhz!nA(_bB^NV1PJ(U0pb9Z(F()#SAIhL= z)P@U+X7*Cf&30TD48hF;L?gerLr{qlF47GaL`$}UJ zkscjEH4A15^be}$I)*7mKz6eaR*th#&!W)j6ngoYIts!M=FZ6R+dX1LRLp;;`YN+s zO3DS&Rc}Tw`rsN4{O>s(Oo2-`%LqUO_T^qcvk;JzWL|p%_0BgGtn`BfS<$2%9r^cHNq`g% zzWp+Ei*W$hEO7*9vDL>f;|1ID%1XR2kx?c^6ycmjW*A zf+27VpoBvht2LahzGcfEl)Heny}G4&S@okA46nd9pVkc=(}WUkBGK0nB*V# zabI+6LQFDe!UDY-9dYRKbz7{(Ets_;MK=v*U@gF=4mJxk@t|^S;ug$8LG%&Xz6PfQ z;&3~88~{cbj!+Zm>n&JQ2=hq>Df-iSdjA50~?h_-UVmheM;;kqSv&ea-m71ye8~C1I-Y289OR(w5iaZt_iBu(q7FVsArc zKfn&sfa#9&t!aj08Kl7lvypY{fkQvX+)_yUA&8`C;JVmkvFLE>WNab_lXiDFIDoX+ zey;fUF=OH|Jh!@XX^~{N`zxH?d0cRD`d>dEPl^ppZ#;6Lp&!1=42bMjM8FboE56%N z{eUaQ@@7Z)>-?ABgDty!y(?%H(EOeFJ(>cGfSoQKT59dLbQ!;*XHBysBuVsp=g-d` ziV;fhLnFWojz9a~p?X~Jd59Vo_E(2r!Md_OA+hX9_?l()8L0X(ZukDred8Am9~Z8r zbP?wf-|WJnxu2hPtohEkWj@uM*@220REop5Pg|H}P>%3fuE+ssT#2N7p*@pmtMTKZ zz!^Pln8+5H51anF7l$wvdbbZNSu$eGs8O%()A>(NWyW8@fhbHcU+08S-<{-_eF>C) z(=hSPxHWa&yvX)@s(w?QlJuad3wNBp5_0&LSW_#Ef@VrQOmvmlwSJhWIch8rHtqic z(tqyC5Ey>}oni|dYVaN~;(*bjk|W$a2ZoW;DDQ&~)?>`EM?~vg!NtJ!Jv|<0KC7EP zo`4<~@9O?p76k--m}h^N9uTG{xu6Q zQ%+sSUZq;;pL1-`1Gh!r&U(*-B}d7&p1d9K<#UIh7(*AbO0!k7I)PEy8#ES}(fP`4 z$0g}NUF@6`$Z45dv3$EQY7v9HG&eB*S4N8s-b>dby4m>Nt73K@o~YxCh1( z76&{Y#+rJNik4HqiD*ZM9wW3}kaN8aostj}S(GMIi%mX3bx#IHG(c!EYuU1vpRKZh zc~RC~Xb#6uK#6s(T?TSZ(&OT^gfUrlvtZ~<*7r6&T*I4i-P+94os!D|2%pUE{WynE zYmiu5p=<-2X=eW#>5tCr2AJls<0r;H@t2RC!f^Ukc*|=$&`xoc7*i<2**u?Fj&@GV zPIyW1aT=)k>QH#=L5+@k6N2#i2PV?eXo3#BAPZIW6^*fMO&+xVa^+Li;M^{1a+q7p{<;W!9Xm|dqD-0p3ru@>xN2e?oG`pzwR&bX{Ud2rsI|*KkrhP{ zNv8I1i{9@4A)@;QSZ_M^kMwq+2;Xu4hk~I0UAf&qSrVUS zk*;bK(|4pehjU$?b52E`rH`Ci691j_kqqz6R45Dv-FFMRb33+s%)p!6eovlJBSz=W zId9>A>TunvWqTF{9}^7T+m!sGSx?yJ`lSuUB@(N?#rXN`+-bBecMhvW0!*0Ctu(OTe9^t* z5zV3&?U8JPz9P?XwxV8}j-J#|R4;O4S_HRDb<#)Q5(uxrl@JhRJ$1&wL;xLti}T@} z^K|qKc_6Y6K$EInd=Fr@A5LOig_(VWrS%@ViErZ)xrc5-D-kD=O4SFfFo8m<$s9~< z00v6ep^~}>)F2u7^M+s{eLqSuFkHL3;Xc z0ok8LU8qocBesK3a z;3c+s+-iAHFNAq2V4UX@q!5P`cR%!z`u@sPOwKy7HZC!F2vscdy3X`8SVi;6o4sP% z@Y)RQpRSU1cPb9*1o-cVXJixFSPLe?~7WJpbPY1BuPhPHv;Oogot1Lm|+PKwmku5|0cg%U5Tcdr`7?qP~ zS05Dd=TozA*y3dcOo$}o?dGGD6cjXp>3+zCJB#Qh_72^I_ZwBztZ^@vF@rbC=b(_f zOa1HkZjz8^n7)|N`@<+DY-l^s0{-CXlhlGUZo*T+&KtblSo}zOgrHW##D4Mv#Vwrt&V`sr}v) z7@;`lvhEBm32({o7*o9~Z}axv_^usUPFB!5Ua66=Jn_prQksQx%yv~7Vb2`ZowNx= zQr5pd0&huhgwtr8tqr5tc;ottR z5hHG!s$o*}c%xyK=r2#>)=5xv(XD80Wo~Q=yZy#I62Zg;uuJ*R3F7JV!?8-}LN^XA zrU$E`BASYhA<)+stwuh$t73VT-54FPeY7z2T6gp%r=5JR4W~Z@!dUPVCmzMW$X@W; zWm#&HXW|fk81;Rzh=?yZIoUw@@2_5^+nNIRTre3GdwO3-alP+?KwoqSutW>Mz~>6I z2GBBM5uN~Gky2$T)pYF)K<`^m%$dOlIybFd`#Q%)an}RynTnq72_8wAs>2K4fcKUg zd~IxKJRLh~2G&jh>@)IKRT^yjatE-AKOmQt-(gW=8J+|Gfz{SE>AB{Z*(q*KTae+I zl$kda+8GBb!N}ENFf#}f#j7R5o~*8|1lo^qIUNB137DZbL_LgsgiT~6`N+I7@s|E; zss7g#gaWbqRTI@2^9x^kt(2}@$k3Xwa$oI&%7rg1#Y&_WRW8&%k*c}raZ?|VcRv_z zEd(?)6O&#i(9^1-TQdbq{KSI&&TjD!wIe)bcm))O5{qM;=;i;x*(rW21m+80WS3i< ze8MOiFECWQpW-PQN;(~p^n-t!f$$sVLu=f1neD%u)ugA5w0pR5C`8Q{V+WI8^5DTm ztcy6+J=I)UNn`tezesj=a4mJ(bZQQJk1_sRo@HpASje18ISK$*9!51x-~_Q{=g!%f zoH~I)C*jHut*1E8qr_*IGh+GOba)NoERG0Ybbkjr`{62OXg$^zR9_(w;st_NpiXWt zj~rN89AX_g^t%kp)0kmQ2P;D-6bAS{6%$YpJM-v3+8`3aEX9HLpao|hsV+L>hyBxd zAm3-jP`Zygw;f||eD+TyY|&2Z%DBI(n^W7f9w$G~JuSo(Von9qt+aY$0vG#@yqF4- z(fS@*>Lw3-*osoQ6-awLUi~e`#gi5(kPWo$$1w97-LM%c-k89%Tb9)e&L3PTJ6K(R zYCon)wdnx%Cd=cXpbCr!Izm4IR0wMny*-<9Onc7Y&m4R+oeWRuc^`yXy3KF7#n_iS zgVO2(lg?RO5TphGT(O>}r2T?hjqh~o8mp@5EA~E#IX7EOon4_wr$VXdg2z^#dkxdD z1%$eU6wU&u(0OXFa#gCEdaYN~sV{Zj4XF&f6rjR%7FE({jS z$>JV9Jw$rU6?m9rtu)dZ$9gc{_d?I^?Why<;s@Sfk%icRlYx2-XScxtZ9_EzaRAWu zFJ&$|3y1cTH(v|BiO=EL+S88?Ovbck(+$xefJs-<0Wb*;Efkn?$-srf!0@`RKWB~o zfuroDf`#C546pp2q`@Ca8-rkm*bLdrK?GvqrtNwLw(A_v?O4pt{V}7(aSGTgsD=ei zI$v{P_Sn$C>9rY;Ld*}G(Qy(+>v>{H_kt4-OV#- zC9s^IkJ+)yQqanqVBg?*)WAtw0Bx6^ zJ45A!@6SVj|Cs4x`Mme~xD4?>W`D&whxyWt7pLKTJzrm(4x(SBf1vk3OKI9DE(*^O zI4z+RaY5`JU+R!v#e3%-TGjx=^Vg2y*Iwl9>+2AHed+Gpm+y1WmEzIyfC9BF`OSAM zXNSpPkj@`uQUMe^GvO%Z=et4WctY;+1jRO>rBtL!AVR!??~t9&p;?>&X%xK2X@Kj6 zB*kL;5^3QQTqgXK*4_L1Z~_%b+re46!v~6J@O{u<-~?ECAW}1F^1$&?HD^68twa&m z^H?u?s&VgQJv+mV*Ixw=Nr&t{Ly+K)zr9h(GC~ZFJ<9GKPMJatu$w-doBKInajEw{ z)9m!MRJI*z_&3Un7~c+;Y3FlBNk^0|;lX9_!pDg|P+d9l?R9ItjfM0*+g zQ8c`2ROrs|^+k=2G;_Y*GB${_FFv0z?lYl#5ER4_v$NZ9=y9M}?A)PMH@|J+^{{*C zMK+jf)vQLN@WfD{VgoxSq&%47)sriQf$;^*;GX(ihq}l$s!>Zu-JN5Xol4ru;K074Os2Tg7WveBs z$5ede_jJeq&v^6JMZ9w1sdab}*gHVn3zm7dpV$uce2-*!+Itx+{wwg;d5%1}4;3YR zGo!YePMNbhzN#f*tU#W%#@G?wxOF@iZe`y~+{(9Pp_9b2SeiO8Ztza6P5~If*cxZO z7p0HlS;e4I1}^fqeW~#yYfIhbM~6XC)y0T zWU=G83Va*uLsA}$<~2CgGBwt4ExQt8bev(ofuixl=`W1ym-M%L;ZW*+VF!15f2t6m zCFDhw#*BwJ5R3*l7Ox*1RAgZ%GAHt9O{1OR$JOirtTlmRDkT)~_s%$hc29JvGZuz8 zCjQeKg4In>UBv$}-noR`!oF1x?n{r69q9HNTm#!#9KPIJY6^54@!^n_RSR6rK3z?J ze{7Bkyc14P{iYTY%87tG8|qFh3v6?5luZVyh$SW&wrP*#pn}2udrBt7E11D~2E&8>O*Y%olc7mnf{=|$$FbI6>>$_4Lzlm{xsc>@$P?;<(1fl6XfXFA%j?nqS zd{l2CO%(vjFTW+o3{J;cfL9UJZP7k{{OmD&<9)DR3_u{hx#`qF$Z`skJiFqga7apV z{4tSINT=!t`*Q|)p%~Nrjecw~?e9;B2b=FbTzz0jNJx3<3@>kQD}v2Ql)Klvsl*_+ zR}Gn_YnOiK;^r2<h(8XTyRX7tA4SQ?7J0P;-_uqekiz3L zh2Ucgc$SSPw`pl{q5U%c)>fx6K2@B#GB7H&8n))!;BcA*6rxwe7x!R4Cv{lQ0W(gH+M z6Drvj<)n|35>!?9NQQkv&PoY*jYsQH12-<6;`1S9?B&+59UHd%J1Fj0vnQC|iEgEC z96!K$-F(%z9k&JxNuVJ%oGge=mlqZ6>%Dy!7I8A7?5C(BNK#IlD7Y*3H)&?Ad>c9N zEm~4c0AV`iuWP3U&(okGiC#D8c|PuNJ(a#sYKs2UTUx{u)Dh(Q?P}4U)saWbKyUZ- zH7XigrRW}G-@4@=sKL~Ng-$=(QwK~WkP_`%_s*vqPr}0K?5ElsFL%)1P1_N@N;UrE zna{q!o$m@22W+Wu;ZAndzX5;Vb%^8jhbM^pVu6tyE~6hDfeDPzaq8vQPgpBYF3GMj z9Q_=eaZh`Ymw{>9*BxE!jpZBT{T z4v9cM_UIf46e*qFs1~pvU=5;ihmBp(5#mSc25hQ)Rpu2|&aj z3#tHq@>jtd**=jd)V8=tJs$ic&jvzs>a1gbokw_h#ZU{u%DE`+--@5aSko<5I>ks> zvSDIE?C>2RjusiCdiI59Ms&CNMcV@LlWC5Fkf|((BKXcpbPYZ_$b;H{IB)agukaq0{1lIF?xr|@<6PX~Q{EP(UKr1s&GLS%zOmm^tu@tg*H;>`FFsGoq$+E?mz01VJMiHK2mc(6 zgw47bCt^RjlY5s{ltwBlqKCeU6{Ryb32&XW1aI50&$0|mbs9Xz+TvY&Kz&GmUR{Q_ z^cpL85Gvqg;pxH`4=RTgh7**fpC*34mP(tO*|DNUoHX1FWk?VZf$RE%K`>VjE%^Ci znO9h9_^SOg&}39!Z;xd|6%F^xgwF8liQ@?C0_bGwynmbQ3yHrf=+mnhRIr69aUqms zPQ|@o6?1!2+V`R5KOJ!UQf8>v~JN&zir`A!oV@xK_3`2e-$%Wz=#@RulkN;lZ(4jdzT^ zTz$B*q_ID+fw7iV#|k*DAG>80eO(z-I3LEWWu>rMoJA!yf)>Gtrg-j6&XgJ9ZI<)0 z)C&%$1@i`b3MC_3QP?JzD*R-HL4>?y*gZ7twdQ)L(QFuHnj;$x zl?>Rq3#i4B`fzp|y|h<1O3|v-!;aMtL&H=umV0t@^BOvAsloQ#k>uva&elp9iLe%v z9PuEf(r6C!+Wt^oEQY!ANgUDcY2fgDT~jwzi};X==H7WO@BWY+p$hM66C5cg0bc(6 z^4-etDJy^`sSa1K_U%)9+d-S+8EUV;qh?u6b9}##LlJ|?Q!d4xKc`&5wxI)2DA2eA zN5{nKp=+0)P%Vu4rYVoV$x^I_Agk!mQ(2RRdAi65yqa?Ds`u~~l0n~Z$3)n4z>_gH z=&}ICF40S0ilBQ7up$1>_tsPs^S56VL&qP& zXLx2BZpEmgDa@~^8`&xCMR^PcR409XaaDO}(yu}}P@QlY*G%1|uP^nNavenrvc4Jd zBtr~2Q^DOu4qbVZ|1tl*vpHRNj*OWSNShC*B5jNJ@%07kph^tHw{p!OD*T`$nhgV) zlUQ~)e(u$27)hJCh?fRJW-};kCuV%Sz^{3Peb1)A3`XNdj-rk9zRY`KCiXqI5WNfKyKAN)h8|?~ zcd7)N*nWGiRtSJEb=>sSfuOXM(t+bF zJ7AWaJgoP53)a7gFr3!xFm{W90UzXJeyEjDieOO$QDs_>G~ow$@S4}m#wTIZkfnp5 z;mpU%<^B2jj^>Xy7|P~@*e8%&+B$UTAu?j0V!;>pee6YKbB*@Ukp6J?uDnIrwe1*Z zObE|GW-$9AcAJ0jT*XXoD*j(Lbym<(4Yn9PDMT*eXIB)oz?N*kHD7+~JN(y^Szj{6 zO5{49h={}(D*miYw6uIRZ{X{3+8x+@Qt!NfV*#8fE0<4}`4mM!(T^JBe13v^ai+Yq z=a2Hp0x%496b}y{KAaBZs!mw3m-!i0fwVgdn%sqZS4_MhiOahC1``-kpKs+lRYgNO zW|0L&$Xl#~ool?{a%YMsso6GR!USiTFBlap`wmF%etF}5@a{8^}cczGX_SMfT z4D#x$w~M5VSbuDewTS&kRjXw8-i{tv8JK_pq1yFRO-84aS8+**%>Ah*#eWD!$_F&u ziERCp*U8G{IWRV@*v?Pap=F4&)yQC#8L)ecEUbH;FKJy<-suNzss6YO1OQz^`*FoS z0cdLOz9DF$G~YSr=NoF%#0*qYY5EP`imYTanO`Q4H~Z_+p0d#F>7XZ!??Tp#=$f96 z@3S}D8>^AzL+YX>n6Ir-48S+4SHl$VPp8m679oUOuIqK0u8J{ej!tQnWa-NsH@t)P z$kS<_sWCAzFG~gi^yZ;M=fY9!x<6i{d7p#X>XV=%=$)AWM`(vHS0N5>sN>D)Sx52n zIm`(+W;1snK}tY;>0q{zIJRJQ@tX$+DbgX}@S#Xs;_RpnvuXHyEv71mq8p&C1IGo_ zH^*R&(>K9O)OLwM`oZwF)PdY@IMSpL2Obpj*TX1MqdLVZjK!RsGDd5>bwMY2fBp-3 zm0~x5{%{37knM2KR8@8VAZF}m+~2I141cr`SK2Wrip8^*0HF2CM|?4T_u!9q=(r%@ zmKW_5HM`eSccT4%o5wQAu%~f5OB4tD#vkXRaD~OE(68R5Um&6xEX0D{a9vk1bAU-) ztq&KRbuPUPY?@%)qpW&BV~QfmCCOTTpx+jtI@x(KB2l74und(5M3>LuWKVzgM~gA>1Ri-+OO;0 zomKnhI$iB#0eVPx`?3Ac^?r#uGTDDYpp+m!b=-3lp)=t&KwK1nP;P84&2O;EoFt4} zeW%45$BMZgOjYIP*SL73RGsu%jD!`_#SbfoRUxspdgRoYp%4^7-W9!%qkOVcy8kYjAMww=ZvG zF-gZCYGB8tgs}M%?J8gh@y~y-0_W}B6pgFFcE)K%63_xxh4n@T5f2tv%>$!HCm~-7 zbZfpDO~><={am_M*^JN*kcPp(j{_X-boA$2xIf~B;n+uC0(7YLqep>~WDx--#Lv-t zSb9gW4l6!x>(h$M8ppXvLqL&0!X(f46`1kV9WTq%Y-;IEth9WwFROT`pdj@=RH96N zOKXP_^@AVvn}r*8B+t&1%K>$^f|~tU`@{VmiU!@7bEvh{$Y%6sf}XxMbRI#DxFC+y8*g(_eiSCElu+K@o`IzkUH3oB2xD4#ZK#_SY7`q9OAI*TFbb*?gR zF~=Uhtv*J|eE&c~VSMwBY5Man7|ucwpizze5Ck)WyUP7IAJh0_Hb8*(d4uiqWIq#F zR697(UH+gRxtMt*;DT5#0O8xm6 zS3C>wy3-*qz+^8T_d+lnnc4^#6D8U!jh3Fa!GSi(ZWv9xvWV;;4TDu0tTmK9fWFj0 zIlu7;3Q)wMY-JB{4s+aEQ}i-tn*f&xiAnu2R)Fr0@u~m}Elr@~VzoX1pgHHZIv3>t z#bRcgrfln7DTM1{etYy6Xwb^`7uR+q-R#Zj2uV6Sjv=1;>55BhN@c2kE?#6E-mKvv zFRb+faD5qX8lCrGBU)rdMVTl*1@c*K_G8@vh2r!phzqAu^-w9i_pG?-sj)X@Wd9Zoyh@DoSX$pPmy*6H;C zGJHA}Z>M~GSMLEOvSX0KS{0RRe5tSCvHD5OVn-Pt7XYL6#6n=Qqp%VmC54!G6~$VH zI4j-6?7hz1+c+ipB+`BX&ObK6ZcauXC!C#xEguxBA=+?31y1lvLPe(+kX?Tu9)+cz z-|vVlU3!uvCis%#i;LTL5ZD8SNcz?hO0K4=2NbGzLl#11GvwN@c1KdXW@U~_6eZ*1 z15fX+fJ@+pYCyknd-5xCjR%8#0a50|$C?|Yy~AfTA*>DP(Teq?nTJ+d^${dxDLJ7)35(Sz`Mpb8lJn zKxFY^KUma${@M$T5&U~-!PxVQ{%mt@v7Zk;xX#V&iiKx{R=M6d)G7&#)dftQDN2qe zbN#_Tem>;Q=E7j_Nw7xIkA)-j$iqMOg-sC8VEo68j8|_~{;eXw#-Z1e*p1`MT;!rV zOTkbYxqb-Q4iys7JA{fugq+kNJuYSU0L(lKp`w~ix&iPX>k;5+INbKx+A;a4PV)C5 zH>SKW!sDd?%K{jQ+M_0~+GD`R>!<$3L(MlW#u8f5<~8WXhx0plF4*9eJil7_aWLx? z{yWiqrh(FXLu;?zMvKGEO|yVFut~TMaWViWiqHufKi-^E0>ha{5Fk)7ql=C&cTi(& znRHcBeP3$o?yB0@jHwK#HNQd%Yps^=DLT~yd;(~%*R*V;MpK+;R4u268UR4jNdg!J zNOLs{cBJ79L!zBoQmN! z3CQ8nKV3ONL`T4#5*aZZ|MPrhGOJs)OcQ$4ke~mh^>(rN3EvG~UfKA;I{FtZi+rRI zQ&ha|oJwpI7f={+uzwr-&AwxLdXZHdzF2zPe8G61SBXHXuZ_{u3}2n_#94npB@MJS zm43B71#2kYX2}`r5Zf`--+PZ06L9+2b-&~ZxS=}Vqd@?}oSqX6O08{}_hmW#Z1rFv z0&%G|1h%fXFA%YP9c*fd4B+NwwR(iHG#@&;ky}~&0U{mY%_ z400v4SFP6YN-by{73Bm95%BkX<*~^tl*d}0zhZ-T5s&tmpp{u=M0Da1 zcqHhtraX+S*TCzn8qwDe+ zwe}~p!cPEo47EesZ|z!Jl4=^;X|AlTORxr91x{e)6#@khYq6t!&v^CpE_!=p%z1>b zu+|??oNob`F)%^l9!PcX-ja(uZZZS~>#(H!kg%_dEkULXy~7ZExY2jx4txu7+3i4X z3NRB`UYfSV51Kx2gJ0p8c%P_LeKg@L;tekc@bV)^>53bWHN~{TPX7!&4%XIym%qT~ zpwtp;n9NTl)s>DP>qi%fu!36xiO7MsJ~t&|IUoWQ4!O;a9H$Nq%1mC0 zBS{{u`(1mFvl`EiqpiE_bMy7J(pW^jn<;1?TM$~RK!xnL z3=JHAp3&?+wjBb}LZZQGp%E#7ETl?fP{m7>p2#j0TLqp$+{%NuS{{4I3C|u zt6lK-D(+PMAib^!bY`c;YV#PM-y%2=h6eD{fJl}l8nkaz$U$lk0{&oB0or0mLD$7?uUZLaH{NxJOk5-l@J`2cEP zYbapSArCc0-w6t7VP4>q3#-I1``O^Dc{>wI+<1pUA!Z#~y})H2P3ojeGQZ4%>!UJ5K0X?q6)ux9)dchs5fpUv?sRT$ zL?i4FO57B9jr;0&=~&t4$jAGzov5Lwd#9?407q)|ZuFzfs8<~9RIJAo=}G#P4ELd- z+nA8%O=BszleW$T7H<|kEYeid>qj|djTcEdonAboN~poAY$rezp9Q|3s2vlS(ECwz zSoM{ahZ$P7c+~zR9LYv`)v#wk<_>@PGOylZYNN{-b+E1vA{LdWrF{42!#08QLl04VCjU~lB0AIk2PlVv~O zV&O0JU!h6R-G;A-K$_g1oY4 z4y>eLmia|)jD7ndv;HJ?$;IEN=*LYH$fNf5ZvUCczH3zl{RCwY41k`Fh#ab+t6#A` z=$1Ur3)^ep+56!+5}_0VLw%B0|NcYgY!{tj22DeYiBMS59FI?=0ucX>!Wc)DMC~jb zz}CBvip{tq!ITv;9?XC$HITm87L}M9_ARTv1yCY4X-?|rnlnPoyO?|&AEK z5vD5v@hFj?taa-9YHvQ2Jx~+$y`SRc%Zd~*c06I17N_CE1>Q}KdRkCB+O4L`cH9=S zYP2&RTGdWA^jH<~Yqscb3$e<0DYhsT=zfLPJ{+pkW|DdtP{@Pic}iBQi*bbiHQ6d&ThKaRnYh+2Du z%y&-s`l6jC%$pA@@bw2Su<2lFFwdw^g4IAA1@DE_8?po*=QG1KX8 zyxl7RA^Y|?f92ZZRFO+F4HWW`L<->FdgiKVoF2u*aip{&#Hz{b75q{q0%e+GTyT7f zn7l5=3k=ZpwV$+cUx1#Fs!hTIXRbc5-7%EqKB6?Y7f;65mtex~?C$$6aAv6|)EgzT zjLk@2Uzyuek^G6hK3gKmC)Gckuq=SkE~Mz}HvH zg^Uo{kz}eI#U={xD`3l$@gUVOR!)|<#Kf>PW_Yj3@Lcfi>7ECP7a+qt%1vYV9kj%a zS!>YyP0_j9^Tq&v3An$4md(|6K*hCLn^n&PwK3(x(1I-s}z123165wBP_gAx(6SE&*@0`8;}gzou$p$BXUa)?JdC zR(FR5kl+_t$_a##FJE6LOc~1z79#G19sq$=BZqXsqNGxhfXuq*M`g}jI<}#c_uUyG zKlCR(!2D(ts!b?X zG4R!oY#=18C76x>%@a7zHlI|gaw zVb@`=?ZH-EFtmy(4bX#MkAq3*d@|dot14A7*qyp#Qt}6{w~&yvgOZY_3=B+akekH4 zC6=`pI5M1MA{)UKOQL6xgF95n?^1yopE&4*z^w1H6cDh+B z1f1+4ynFzS2d0+1$C@jDe$o@z*g(xgS+F9S2TMP~C82DZit{AAK-Au-UbRj&Y?n## z3=Je*>6_(4%JcbcR50nU#If7Qrp%j5SR_j1`6S9B!F*Wjg;Zxx)y-2`)fMA`^#Lkg z;2@zuW!rh^4kpf1kR#*Z@~^umuyrC0e$derQYznm&&3w4)*?i~^Va2OIU2yC6=(%m z!IdibSCe2bB8Rh?uaa#1F8q(_ z>9QJ&9cFB3Yu6g$Y}0`|nUa(U4_C4lhd3xw;7WgYcmiRLMO%)VjbkBU5P7L$tUw8K z{CRlVP{^S=Kt*U^_h=!{mqF6aVx-6vHD1$9eL~D}v|T>XArt&e`zzBRYBFOJ24Xn^ z)=6|qU%gxHPo#}-m1OM6oQDE==#lD?^MmkG)~jAKJ&Tf?@H6Ulw`|`&6XnsXuDZCQ zKVPr{i`H*MesiBrG$B2`*b6#@n$i*^*C#hEI`cahGx)5VRj(%`0_|_wJl9OH*=oyC z5RJg`&EOV{@>vBbcHP#?o;Ws2pBAXu!{Y=1(2^t0lbR{ijP)#Vm{cGj?7WaQD~HOD%yM;jlOZV&_#?5@O*Ittzv3^E)t5xwOfgnu*V&WZlh%&=(RlIJZ(R45g1;9vIq}Hj1P18nUW*& z%=0djVeV0YIQKQn-QcEG=xSf|hK^H8uuSL-42^#DP;%Cjv z&;qo0*N!-RZSDHa>}e*Yb0iA@eKCgQDg?J5{Xu>pf&=%*v(g#m?l2$GLdx4Q#D-YFD(4m0z zPjG);@nHDQ@zk%i9zN{fzj6?We`pggcZkfcOk{WqH}pk3U-T{wiJ};%%*bY_nnXlL zf6zRF*%t_Xrczcr+E{a_IJ)%MtX1pSQrR6Zap7nYQ*%k1m7eYr?Lk9=cP{~!!U&ss zsY{G6(2HxWxOSZ@Rm}^Ac2uX_@Q|HosT_I#)7gEen|l{;|!erw_`NIhfd6-1!;}kKX%U9E&&yjoPL*5Pn8} zmT?^6^zC?1%>*){JV1ynMKCizJ}I7H*73vov{ zVe^4msNoFm)7Hs|%e?vYFVyQmSWy6CjD^ABw+oP9%NtPy8zAS0$V{mNvCL#urJn&q zRSYQbY*7@x!)k3_M3ouvGPqg2B8@FkxrrBt8kNeP0uTimxt^r1C;Fc!!W?gj>i6i& z>o9NReDjOpu;lnuZT}Z%qBtFn=ALT}OU)L%@>d-Z{tM5z%z!!tJrA%NI$hl{v0lE%jlt`p@*%2&nU z6X>O}u*Ud1CH`d2x?>zGZO{Xv-A?+~t){gwmyp10-O6{LDwB*$N z?M;%6xn6zklu?fU^dLZjz#v;hWE48`x3}-EU~8z0+^)OYo%W)J4S?DSush2Ke&!6y z5(UT%*6xAyT9-QPWJJwE0yKzjPBgwbk(1&cfSjs7r83zMlZf+C7@%N4z52n3JNZo; z)-?_Lf!^me?s>>thZ7VZwTJhIvm1!0W5`#FH9)aY4pUW~y;6u7tWZu53{ng#m0{FAgUqG4zd5oxvyG4p;RqL#fO(1vh<$oTY_t{F7xW@3Na2S_2^s|Y&@jejP$+~Z z{1vJI3?N|J;SuCN zoFcb2c};^EGsnV*yZj}IYw1^Cyn!24b98G=Ir&U>~NI!ip8eY3gvu>A-t(re*E zYiUpa%AG9ziG`|DIr>ssRAnffLJV2Ng1RA4IRRhOm4bp4N|4jGXldP7ET>_{#Pm9U zhq!2hgi#L}CD>~Z>#HA7$qmL@WTi!!5EXUtPgX$riqg&^v(BXWUIs@}J^O2^*T4$F z)k%k?07G6yOS2;^8vkpHWua{@{QGCE{dBCfR-(FetY4UQ6-_u z(MqaIFu!1huI25HOZ!o(Eyj=R|5Ze?!G}1 zi7!cUEFxt+{$auJHR{DYn2loMz{UnWrDe z488%W0V=*LQ0)KFb*Fn;KX~)otDVgD{QJ`}>)RZSD4^x?2I61gG&@dHQDjt*$@5Jb{0MY6!&-Ii^~L?Z*YJlB_k~ zF^+|@DrFEU8?d6p7X;b0$P^GAC{B4O+hIu;U>6fAh-pib%Md=&Hlq`&4;c-pIjEa2 zFNJv)YkUM!K0-r?V5R_W=@z*(brkn&-EC_N1KHLE-gbsKK-FxdH{lpa0n=FxDFzm) zA17L;tPRq?X-M3$U7fsOLl;M14E~*qpdA{7Bxw|IjrCN~86j=<0IH*>Z;SO*0RCV{ zf?_iW$9UtEKU&W_*(0n!K!J#e>OHnvqBWrPyr>N>jv#RwQC2-LnL!WCE>a%Gx#g|F zIM8t>;ka01FROZK9Fp`@n0BWb3_KlG+_%%sxBjFHu$wU%M5VBB=Xi8FScF3jqAwTv z91qv2$_Cqy2;@zHbeEnSr1C&=C7>CjaxSAZ5%>YxhuKgr1JZ%0LS8)U^X;9|gIG-U zP{?k`JxPQJr0MDEDYre?*=g4tj{Y&v={E#4D&yE*0v~f)60?BABDBp=+QXi*_m%`j z!crl$<@9}6)^@j)9%iaARJC#v6xIWc)Fnu;O{hb4r?|C>eN(6}8$A;46yg0)Eo zyNm~UzOuus(x;xes9xKV9 zy;LPN>hj6b4X0=ZgiIW0oQs&+USdZY5#pBF0y;W|)vhpR zX(zn(B&~8_muJCX!84fBsHfacGG}iF{vz)!{)ch^6d8?47k@tWowG{0BZUJov%q(!kVkIQc#4LnT%$Ex-HAYSJ5U% z1z!*HD*m86yN-666Cz)IIrhS8s|%^cKi_-04H+xo^GZEa$qa&20j+_RfV5d=9*|~W zcu&vRV-?A*$O7T${@50doO>3wv1T=%OVYs{RAeEt>#f2D??ieK0R_zS^T8)5{nik< z^p+hGOewVuq^I_Ro60muUET|w>mtbf>L^p z3H;c2634J>j!3GGqFVun>co^nUqSX~y*29S33Aw}1YhD+aH{|KWw zf7Ei8Z~D~uY*oWllEgAo%2-AkI80*uWrYLm5&@mfqoBAOgwYWC@KzC1BC@TWAS`dR zq6MjV+)K?ALC%K;0q%H2*Jp#8BK6p}Y#fp-+nSp*1Q0+f182?h@8E#=F-8CS%bovI z-IvE>y>4$m&C{-^GzqaOp;RJc8Z=N!h9pCUhzw;+#@Z!Kl0-6AGN(}Hp_xd^oFRK> z3L%6L-fKN*ch25t|2n_-{pa=B=X30h@AEw0?|rX(t!rJ^wRWxR)lWk<)i?n1qtcOv zA=Ih92}9k9nWpiV3JouTfMNXOEMHi47 z^B^a^JoQpu=DT6luuBUW0a8bgZo`%(@hy#IgqvdpJV(ADu!vFQ+KW&-pBjI^a7v3~ z%CAzZ2v$DxM1(?R{1`blR8oFqPx1pD;e~=*6SZ)UBk80DAW0O-{g}jw-RLb;ofxFd zkbt7crzXUr>ZTSHCBn;Nednq+v4!Kv}ggO1k|sHWl~QTy35jY!-)4MPENSUW}xCiv|A#Jy0H7 zd;Nq_QyW=Y{8`tHZ2P~nw7??y$xB`{u+Sb;r1&50EhzJ}9;38%B;R8ElZl6pXY<%# z_n-RA%3=S>bEBocKB&|0N4fRltQ%IXLCv!m;S&_j&jtAEVmch*UDDKVFkf-Rhs1;J z`_27+Z~ktV!h1-Z)4zrls_&}mA({F#sKKY2{@DAe+g zcXD{Y{LBXtyu=IWH$Z=+Li)$6?mX>MN$i&Yhuekk3Tz+3um0SAMRmt9sy6<|H+tuP zxGLnsfm+F`fSsx88aG3HF{$DqgmkAMPp1QpxtD&uaKtaB8V&8wFAGt(J-oEPbhOZD z*@x^+3jiZEXiPISA)R3g$UznVH72Dga=wIxK1>7-BnA+0Me=#D8>_B)z8i7!aIt_z z`7`SJqvNnmmRzj9Ks5w|kq^{`92NzU=Qeg86GDCk=3(=A5=5}4m z*6dDw^e{j5i!TN%YyopCN>xy5B>5{-Boj7;0Z-nDqRzjO^w%cVl(Dx+MMkah4JWTo ziyJww@B2~1^3p36n0F^29sio2ttqRis;Vm3Fhs;{YN0@H(q>CELK~RmQG@!S@gUMyb>j=+O#BMW zIJC{=3QRzgr4bJ%$0o=T+~gXF>!bW0PpgLEORtUwFT|1)R)(2V1O zAWmO;y(Y>7O3QoM zk4#rc_ZDq&0cHlTDA|LqKU@I#Hcx}{`mW96b%x9geyB3D){41AK{#VlvjA~zrl$u= zK`a{ECFgo0q(>k5Xs%oO4*&5xx?wtboWzLVC*vQ@(0)D(*m=gq|8EZL5xl2EM^|b| z_LQH>5`I3x{4bIym^O@&_M?(7NQKYw^U_cQRfB2A;waY!qF!Ax{q z9CjOSax2^^wOkDNb#4u_ISkfd`a*%>pMO)v&xw76F2lXeumHK1^%cHwbBM@oWAqEH z`^EhVowcW`6a%J192AJoG3h-bi)_)Gr|v-(;%H>@Fy}UVrnT5h=*2i+_1lT7NkuC)AcGSVf=0 zrF_}TtIh{(Rd5A}Yuu$)&NP3Y|WJC+V)K*84K4mE`wyh1c)u3p7=wNVK~dvE6d_YUU-F zyku5Za#$GwKf!D!G9uj!H#cg#w1Z76JAE-Z^pKc@EDR)um97k%NrM7IL&+b`3+iGN zQ2TyhbVQ*h^}4x34Lc#dq4_kisx@9qImMrh`#HQADCk2K+c zg17iv><6`YIahwCr>xf3`JqwJ_E@jQ}@4l-xY`@`qD2xR~&OQ?0c0du_E@=icf0 zP|sg_1;eWk5Q5*mGHg3?!1hWhH_bao&@IZ=8p+f61@YQ$ZoDrCb<9QCn8GK9nhI#e z+2_(`a1;M#1eiOVgRce48x0MGUjXR{Tv-o1iCO!&-3l<6bC2s+cG5!0P=@+%Ef%?NKbcdI>_Y3+}U@A=ZJ?(Ej`wwf%q- z*(L{)+0j!?<~>;7LSd>J#6!%`Jddj%gRtNQ*bGDxgy;&b)iThPD2cfY>!U29ML}|K z5IBtt)20XiKM!L0RMlO8z@m!GOHM9rDM+I`tu&AK^=TLXvWNR$+v*Wpx3C+`;^vl# zEOin@GX_n2JD7iG&YWqF4;m0+=VRX9M`m|l$DF>75m0h+L{cIo(6lF+SH3`Gae@sp zj4W!)k`N4$q?JiK#lVfe%-2Ae)g!81uV!e}NiaX!V!tMn$%IPcQxW&jw6m(@;1jfs z$Z@64oxCoY1`y;DM8-YDrV%J5vs-~K`^+2>=G$3sQ~ZwQ7W@Qwcpw^x%d=$?ONizS z1{$(jWQ_&g=#E4suBXo&T<*05A6hFs-f$xdpd~y!G~_wYmbk;^P6!bRQZE4a7Ix#L z(7II2e4me6?;#*#jOImN`5f)XpV8f2XKL=~(q*WKZjvbkK5_e+nXAF2N*hN3RTge; zd*Q5wj%Cs-TU^TF#QZMuOG1z6eJHaLf;3N|Z6>gPp*_r4z(@^}WZ$?5!P}@G8S1jm zD)hT&Q~iNQ0FKIevU5a7wUeU5MfSBT$W()AA_iF;@kh~NCe`u{CuZ9t_>`w~@&Vhc z@}4PnJh|8^x8MOq+%ftSEV}!zdZ=9vXi}QY2jo3N1b$rCpz4bs)EyxtgIJ*08 zy7QZt629|X%Jtfw?3Qw#?2=XzVmU+z(Q+l_2P?_DcA!VnsbOZz#3jNG@Znp3rzqcg zOmYFd8j{djVmfG`xk^0^lo=t_Twvn2zY@uR209AqHh1gnDFl*8ziTm<`G^?nO}%@# zX`5oe`rwg3Xy>FpNx{^Y8uOLtzk`YU?N@4*pj*M@#FM)URXJ8@+^`>FK8fP+B7-d+ zvtdj`qywy}dB9uUAmN}%!_^+6g zxbR?683Qo+@`{^#niESOkxROcjyN~-k@~XS zl1623My@!hU-UX^U&g}rVs*8-#`5-eO1j{|EuXFM2}@j5!)5`F0Ti1^>Zz%Dk!7sK zZD_lLL{Z;fm$tJ=+4ljcyKv`#x03F5`wQs!kplZjToK_5NSg82q!{}Zm{fMZ2)Dp( zTG?QY6UAFj4#tJN(|MQd3S2}&wS^l^0;)(Mwscs^!B*3zyUA=$ZHh}dxw|r33Z$n; zd3K3Zl9UaJH-+Ek(XEL8m3aqUP-*Edh1>Id;mmIqDLlkzA>)>5iUQX-D30_N>iBW>KVy-8-=uzR#Gv}DJH|DcA6twp%x(m${H)?&IS2K zRC^>_^2S;vE7sg-D{5!wI%Hx{Jr*_jVtB<^kV)Md(5pkYGj%gAX&zW}5V!hp@(jG| zTm+*Hub87uM?A7DLaTpN>}9vKZIjz_Fnvf-gjP&3-7%_n$jL^_hxuuXT)>V-dAc7xf6Ferfm>xST7L*7vPOxx6aY6leDIvha# zS%qdNxw(TdLlUqYhs-YHbBNDbWKa;}_2(k*utECi0wf;p;Oj9;L(1}a*tf4kKITF< zwEhbNthGbLGlvU8`)#fPemIC`KJ5)t^qWgjd{}ryz6Hl2@VBYE))WFcpF;_7*@E@D zh;g)uSNESZMZThx|1pITR1A{LgP22B{h>(mzrt{20eMo8+=!_nft!`(z*Ks=jN&F-iaAQ6 z%#6Vj+xw{sbU2?1TiXCx9)d`W>``73_-MQ!aRuu}uiMbpA0xv9j7DajXGGN_WfL)% zYLaT)FQBaTmQW`VV)6=R@fTeF%3neReba#VR93z>3R2lA!DqS)TtFOb@fEK#H>2{+ zt-Oab&?hSzLoV3fAVFaI9)Q7nfdqGC5#q+5s>t+?QnJ6cg+><;o37jpI4*55xHuuf zd7ur_8_uFuGpV7uBt!J=2j-+lmj%@7Qf`c-qEUjCvR3l&DakUNyqIo-i-RcYfq$K4 zAxcD4&>W!H6x{^44i12cl(`mm!^DwQ=ERq>JMB$8+|P!3DZZYb!L(&*OSE;O=%5Gj zg6M7@y0%i*^1b(i6$Z$WC!!%yXUrS49KfK)v5hSyD6^LA4X)jY7t7;fC3+Z1GET@s z_0G?KKUwJzGFV+GwMVzc%d1PNR4i;u7(c^zhk@Vcn>YbiAp!bQj>7uL12cNb;MOMO zsn_~U+k`dD^AUIpZHUQSc@IwysHaA9AS%hPC5ahqM1UL?4w3Pnz-4uxVqX?r$a@+E z&XPEpyr7&C)8BB*dlJ*J;L#t~_1w?leMh}G!kp}Bqj<;}BR3MhedUbpeN}nh7$G28 z?~_^Q5WJ7vNdKNrl$?3>hzuFwZ0Y z{1T}YeHElSvRgc40TdCm`TvLqB-OSL6{5LWbCiUul`P;H0at)+I>i{Lo@|(^xu2MWx%{14~ah#&>`6WhZAC_pzXz~gM)O- zGWv?|g$nW`EnvgmskqYP6t?8`_DJZ3Qu)1oph@9_a@EkDL@3Ul~!MG~6(uEP^~~sV;7{$TJz4 zPw=Pd+e5D=6T|1&&-J0$C$_cJ|Z;VFh2*RS}S!6L(` zJmb&lCptzS{sk;X>c;%~Vc5thH()zeTho4yoIt?RcKq^zZNNuk4BKG|%PdZhHhC< z`T0lb`O|k@;pxvlO8BG$FZ!Ei3xS|Q#=nF={nz_rS*NhP zn_zTH@YMKaJY_bk&U0o?O}fuxczMEvY4;WO&xzeVhi&}WDQX)O7N6GT=iWB!Jj-Ga z8)xTQpXME_ukP5mAPyOyD4Cy#2`XWF*xcipQ+Kxud%Q zi+06^;dUOa-=y&yhGA=b|NE>@!+_oH=GfQIY;K5INTy$*f3Y-bt0KJLxqz|r?>X7_I5|LV`^ugpd_HD2K7 zUd;6;??i>9KYauEcn!=JF|kY)W4m_) z3M#(2X%fvhmxLwhl>M8Z>#h3bH~qtPm^lQ81_y(qBzCXRjg<#a0T~#|uBb@Wx)|i; z6@uf?2(LJ!`MACP5pey$k@3-w-Fsfz_WmkS<}JqpIQi<;4hSi>UAc0lJ{fsw6bQQt zyLPFYm_(ye(Vy>%YVsx~?GhFD)n(`2jzg$t9nW97B${6wqpb<}R>{~n5`*8bUAc1e z;>BOmAv#x3R4gVTif*ae6_1gbDGbH|B0!srWq`ze!)a*1mI`vT|CKjxJcYT2N1QVB zU(U+&Rk0B4))pN^x$%l|8iRV*efzvAK*Dm%+qZAaQ*G~ksHqV>`uqC$L88L|m+~pI zR^tuP6KL3m!4HdZD7jdHig3t-~UM5=8^2P~l?LFkh6K%eyh3K@f5GHEzE17={V zs1v8P%yVb`zER5DE?iJ?a7fe000DZMDeM>bcOHiuWQs+df-gHUM95G!wqG0OM>o3; zUKouCE}DVIE_}P@jpn^?Rz^lf6AUPCn!@aZvr}9|q_Fa9vd|^-DszCTYETvPblG=* z(%@UVbYA&HXP)oj;r=mXqn+a7;(On}e}5)n%8VHa^1k5ms@K0rY)wR?XJ;OtdAeV6 z%=zrxDrTWbt4_QD66EK`^XAM^4e7&7yfv5E*Bt&`m3`*SnN`5MB%Qt`@ism?H+%EK z_%$ztcV;5R_rcd{5s^HnyFKUDKb|}qQ`bwdE?T4&ZR5K8f&R{jh=}@QC=C3~)IkVK zb)0i8ivF6Dle43YZiVTyP&fyH@7$F%H8JBurQzG_A1AJ{XK;v&((3gS2d_!q^e%0!~_NH_hqlH zDpg~(OR^t{%?y{gJBTaw$1w1_kdzh%iI?M6XQbYX;`!kb{L$8C&WUH=wlh)6zB{^V zT~5x|=Z$McMT?hAJ|=<-{;*b{Tuioe0O}rAu9bd5ZQcs4`xRR?7G#PK)K{IYF^Kt> z3&RL*d#5-b-}4ByaVTi5%kZ1keU$(suKD_E9L~gwSan%OMy41$abv~ryZ#L1Vi3X$ zadC0QFXG_PKFr=M3Hmo2c6a--%dS~dB%J|0Jtowb@^rD!nx!iuJjZ=>Q8FHFd1zb= z(AvP8><>3qIDlhb-%f72wO`RUiI)#k-J99i*ogRf?^Oq+l1d+?4}4$KjKx8KJpHo3 zt&5f4?p2Ct#x{Glp7&OaKQYKkY_-LKBtnAw`U0!wU!4bE!v3Q-52T&b34tk$L>gG@LuAQ z)j4_cq-RJ`k?NVVX9Yz?Mfay;&mP7Q5(7I}`c0iN!=_C5Q>R7G_fz@M<+EqcLc1~> zGv-y+npMK3>ueDY2)Y(VNh)vI;tG3>HMXr<+8GWSp0#alG*InYWBb6jZ}eYfAk|y_ z%O)H|#?=|=bn-we)C8jZzEqpLbA|Q>Lo2t<6EmEmsb9!vq0Kh^`)YO3#uWE`ar)-~ zXhq=n(7BdJdm}+J5gfw4dO!0b#NgGkg8+L%p!N9(yMJXHn*;>lF4(P<-PMG8*b(Sh zM^8^?#F2<_3c$l(M-KxT?gM+XDBf_FhLE>Iko}1h#_$&aoEMY=5w_V2XTbAw7A%N> ze@6L0WofB8rWq8W@+bzDr<Qy!~#cGm3Kb&ZV?=z5)T zb&UkWtnS3Ova-Dc=)6c;)?XBW9~b@X%o#e3`9#{MUi`kckOCD9&#TJywgz+*1NTv3-iYB*Vqx0GI>y@p^!FS_5Y=f+zbA z1bS*_>BfFU0K6k6%V6Y4;SEiFh|X;~K%1tj)I?l?+u7N<_v=^w#f#TN4PF4yITS7g zg3Fe@tZF!t(8ypc$dH%YpsF%~ktAZiXROI0vU>qzfXtB7)EOc z5(s1>ItA!|cRJeCyqK48`Swn!0R%Z*tV&Yxd>dDF&?wKX*=7d6mWwADksluEDe z;o;#h#?8frfjg-KsR_)3Q|GX-9B}gX^^E|K`w>nL1@I@?U#>Ed;T8AklZgy^>h#_f z8BZ!IG@)ox$5Shg)!T~tJ@=asdPN|MxI#%7j$MJ4khW>(Jm9Un>vvu^xG@M?QAf7xWKuh14Te<u#&QS6i#!@-BWW zwqp?r?Q8!2(V!82MCVcIz=7Zpf8k%c3ruNAxgN}3`3rKd@CPd zRrwRfk3Wot4WypRV14rZ`s+jc9q59i~HSxeKwMEP? z1?O_pyZA7yiZFoSS<*q*uZKfauDW2sf;(d7jP>okeSMLDC=HVNH@N-&`vDpFb9;v1 zRts=*-+W|K;@FS%&&wM!Uj*>vp)r|r#;)X$u#iLBW@RoORw)|J*3jmP7iZuMy|KG2=33B0B9OTvtx*9j0o?Y*I2*U4Be1OF zp@e5Zz+E^YDj?BSf<{~Q_Lk{tF?!N6>*n*7ABtFFqjR2(a4#tR)mNWy$FkBDvE?vrw-*@HR}9X&fsV6Hxz4r zJ!vqYk&Gr8Vf=e^XMcIR@*>dd1)diH)QX$f{{Co;pM5(l`VTvi!`}jgB+DjFov1y_ z^`PfEqoZ;xGfX`0;M0k>vG@(D5K7pgo>*?h37jC38g!mSv#?}4&3lPMs^n69Zz1vq zT1Wx1&N?`&&<@$NckgwW1Y9+5I|sH7PFrJ05^{2KR_CuZDN}+-OdT!pA}ote2b?f5 zAoi(r!eR9i8ijL^bOlDKgt?>ih(y#22mD{(3-GWB&#f3y-`scy<7gP#p5@ZQkOHbi zJ3$v2j{b;W9me=oIOt&3c2{FzmXI?(-OjF5<6?o%_koF*wa|SKD`mk6=HbDD54s)A z>sXEAD;^%f=mWRBPp?RxHV%n>P*4!!@?r03``&u=Yv7nG!Y-Fzt+E;+YZ)KkokupF zwRLrOz)dL?BIg@1!9`=ojVn&JG-O^h)#jmH0_Hhvn7Qt2Fv3Q~Ev*lV8p#%S(DyVn zi{vLR1yvqaywSKOq~tynrik7J2(@{_8=qU(X0uM` zkW@7?GV-*@P%_295?Id_BW}{TT)rhsjM36Si|1U3{{!Ve{`kY>$<*M=@)YYRJCoSG zQ{bft$|0;rF9gyfBvil)cPTs zVwi-w6J=ik*62g0ffFrdv71mV5SmexW@jm1fJi2)`)qPS)0js?hUvnmyLu)cdHM1s zFb%g5|DmOtNQ=dNZZll1gAnwUusZWB%#3eiz3x4C?i_tM=)&p3^l5N}pLciPgPfYT zE<}+-#9&86hyi6u2q7jDQ-Y3vm;$3MEtJPAxY%*CDWj7wKsofP202~?;*FVqUXZei zN*8EKhhKW~iA7Bf@n5T+o}SKES34u|eWe+ib^Fqe$M)7_K150*gSOQ9bLXh!zaWzc z!r>E#if-sT$hN2mi_)PBYnW__Crp~O-4m6HIBS4@!)vIV|f2!?azPBcIN!Bt%S6=HmA4LQtXT z&0A;kVgR%*aodB8qZi_5Y~OC$#GN^PdRva2EBXr|!P`@ez!EEhj`x*IAB;c3aX!^)rcY6Eg!6(@bb)%3vp8&~tYV=j4CrIzF*P*ZzL=WXnhK#iI`E;QZ7{X9*V26rkU((T= zh>;v6Gbc~J?_1+9VR>G31tUq*!XnPj0Y_}5>VCVRwRVLTsu9YYY0GG=5aKHYsA3ee zCKViqmWi%>zEK~isvta(J(>{O$~h4>jH6M==KS(5H+P?!;(m$+NR`O`<8h@c)~(Zl z)LJaJWX#N$z4piNC|q0P*k=|0DaZaR%p^k5(A(g8aGNJldp2f1MLu-}`PAZNOZOYE zXx1ugoy*Cob2#__-oB(=M~H?Kyv?*^mM_;%DUx=A?jm5HL2;-*=ZdHpK0G`2e^Hj^Dh4&LtbpfBt+_q$?{gTd{$=t}=BDLmB5RKrg9y6XEL~ zsW)ySj~27vcKpPN8yh~LE^G!AXHy|BAv|3p%v?aA**>W&r_ED$zUv?;h)qc1JiM+7 zG@;cKH{}45(`C+{S`0SNu89GMmKjEf?3awr42Mh0v-)XQqH1GKc~}%Imv~C4=MB+9 zN(S+ua0b0IiJQczJK5Fwq_ngrv@fS(5K*bO=^LouIkz9iK;4~X*NUP6m-9S{jqh^H)^F= z$z#-m4=e)XydRf@sHN9k#4501C%o{eCPVT5O+!1D0w6jL!4X)i0~B!4ax-fbiJ$@o8{y_!U4nPM21~;$pn}4 z(I_fTAoJss#~ELU@<|OmLOi@r`i6$v;}oBJ<@RY=Sv`1)&8NB}8Sds#1>J-==Ve-t zR1Kaq-`vxaXU<%!o(*7Ee!o-i9kBhBcpS%JT_aUqul#zZY_hC?cG~f`dGWnhMU32| z3vI`bAD?6)i8!9NHL6YNAkdR1j3VP($9BJb zJv}`tQ{^RklUnY2Wm|uEHsunhO!+VC@u>*}*_Y{*?#?A^7ODrx!u23Ux#t<{7)jSq zvV>zIkHI_1>mK%>@Gz-ZGlgZFN$c!bbmB$2wBJA2X*Z6;M|gh;2-CU~7egKr>gG22 zu15S8$g)HWSJl0LYF@uU?K*f*8V=4E{D+WBRE*PcYnq4%4--Lk}C5 zL>qDXyl6U=#val7E~?ViFLgjpFs&vpSM5ax`jsO_-pJA z_jlIlw2JbsXi;5G?w8a3tSTKwftdJ-jYB7 z)`D;SBVvl~Y@gS89OSF#IWz4=^jgbepVrlkS+Bv0k^zUkZylKS*9E7YCxV!m(Q32} zG!G2m2aPZ=^5$LOdT=6Vf_b%B=#GxAu3}i`O#752dHhPn{tXt@uO~2S4bu&OJ2)TN zMuJA3(q5J)y0P1GrfkHaz$JO~cbBRR%WDtqlYq!!!usL(d2d0*4};n<$qLTSIqZO8 zc{aIv{|JUHpXkGlK%IPZW56pk?%Wa`1%HJKX`K|SW&Ki& z@RdSyFYxg2ys|*A16jtNkadVB)H{f;#3!!83!O=b_wjrwDXM~vYO4I@-PL}Z7!kA~ zzGMy!!a}|}lYMSzVsm~0In99^JW2r- zVgnKi zSHpgwoYhh^@ZfwHzK1;kV@r7f7g9w6man39K+zLnba zA21X%vmM6iv0ccxO(x!+w%Y02@#PwEayX69lN!a)@f5eBq16ZN*aMiPQC_3`I`T4- zX*MHwq?_VX=ZnuieN;8;y%US)7YLP_F!D(mT^gi4#Q?gfzyPEFDr!9tyzUUX=x3gB zMYXTzbEQuTAtR`A1*N1k@lldcdVw>3U)c}by+iI}7&g31mMjF1brmmLWB${pPhnpW zatz_l!EvYW>gUmGjf?ot>S^}aVhQu)*`p?R#R4zvymay6tI{lRwes@qzkJ;N`*GaG z7uf6?wa#JK{c{^r7{Sq48moZ8?+>v*fL(qGInTT!X?ULO$OT`QHXxAuqd74syvQj93^g;G2=W;!D$3cu8ldNxqnt{@@-r(bXRi>xfEXo#<8J=Kk!b7JPq z>C;7V#QFD`?JlTR0iXRKd}ykpn;RRAFt?yG!g(fxVYWF$Z|nheKwMoQry1km(Ub!({7;h7<(k=U_gagsv@kU3IGNM?Bi~jibA#z?Fwt8 zp4)TeHN_pi9<@9J43q8B^Oeyh&&Iio0|ZJtgw_t=B<*C2&7#Id1OOA*jNp@Gf4(;) zG*m!9K!J*M5c6uAo1+jlA^=(%m&OEaciNi*h>H$$hzgo9=dh!@8+Z31U~&;uxI1w@ z{`OU4f`#OF?kqtqZjJ6P)%;FQPE&vRMHQ}ZMH!Cvw!89Qpj2h!;JAyPv;YuqRD%RH z5cp~4GM&NihYk495rnvk#9c_v_8{t5qpFMqLebIQeyryUgWx4{{Y7k!M6Ya literal 0 HcmV?d00001 diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial4.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial4.ipynb index 43a8c3119..5f8510a44 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial4.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial4.ipynb @@ -17,7 +17,7 @@ "execution": {} }, "source": [ - "# Tutorial 4: Representational geometry & noise\n", + "# (Bonus) Tutorial 4: Representational geometry & noise\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", @@ -25,9 +25,9 @@ "\n", "__Content creators:__ Wenxuan Guo, Heiko Schütt\n", "\n", - "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk\n", + "__Content reviewers:__ Alish Dipani, Samuele Bolotta, Yizhou Chen, RyeongKyung Yoon, Ruiyi Zhang, Lily Chamakura, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n", + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n", "\n", "Acknowledgments: the tutorial outline was written by Heiko Schütt. The content was greatly improved by discussions with Heiko, Hlib, and Alish, and the insightful illustrations presented in the paper by Walther et al. (2016)\n" ] @@ -61,7 +61,7 @@ "\n", "5. Using random projections to estimate distances. This section introduces the Johnson–Lindenstrauss Lemma, which states that random projections maintain the integrity of distance estimates in a lower-dimensional space. This concept is crucial for reducing dimensionality while preserving the relational structure of the data.\n", "\n", - "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures. " + "We will adhere to the notational conventions established by [Walther et al. (2016)](https://pubmed.ncbi.nlm.nih.gov/26707889/) for all discussed distance measures." ] }, { @@ -644,6 +644,72 @@ "display(tabs)" ] }, + { + "cell_type": "markdown", + "id": "b64eaea5", + "metadata": { + "execution": {} + }, + "source": [ + "The video below is additional information in more detail which was previously part of the introductory video for this course day. It provides some useful further information on the technical details mentioned during these tutorials. Please feel free to check it out and use it as a resource if you want to learn more or if you want to get a deeper understanding on some of the important details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "200235dc", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Video 2 (BONUS): Extended Intro Video\n", + "\n", + "from ipywidgets import widgets\n", + "from IPython.display import YouTubeVideo\n", + "from IPython.display import IFrame\n", + "from IPython.display import display\n", + "\n", + "class PlayVideo(IFrame):\n", + " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", + " self.id = id\n", + " if source == 'Bilibili':\n", + " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", + " elif source == 'Osf':\n", + " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", + " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "\n", + "def display_videos(video_ids, W=400, H=300, fs=1):\n", + " tab_contents = []\n", + " for i, video_id in enumerate(video_ids):\n", + " out = widgets.Output()\n", + " with out:\n", + " if video_ids[i][0] == 'Youtube':\n", + " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", + " height=H, fs=fs, rel=0)\n", + " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", + " else:\n", + " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", + " height=H, fs=fs, autoplay=False)\n", + " if video_ids[i][0] == 'Bilibili':\n", + " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", + " elif video_ids[i][0] == 'Osf':\n", + " print(f'Video available at https://osf.io/{video.id}')\n", + " display(video)\n", + " tab_contents.append(out)\n", + " return tab_contents\n", + "\n", + "#assert 1 == 0, \"Upload this video\"\n", + "video_ids = [('Youtube', 'm9srqTx5ci0'), ('Bilibili', 'BV1meVjz3Eeh')]\n", + "tab_contents = display_videos(video_ids, W=854, H=480)\n", + "tabs = widgets.Tab()\n", + "tabs.children = tab_contents\n", + "for i in range(len(tab_contents)):\n", + " tabs.set_title(i, video_ids[i][0])\n", + "display(tabs)" + ] + }, { "cell_type": "code", "execution_count": null, @@ -1461,6 +1527,20 @@ "3. Cross-validated distance estimators (cross-validated Euclidean or Mahalanobis distance) can remove the positive bias introduced by noise.\n", "4. The Johnson–Lindenstrauss Lemma shows that random projections preserve the Euclidean distance with some distortions. Crucially, the distortion does not depend on the dimensionality of the original space." ] + }, + { + "cell_type": "markdown", + "id": "40936ec4", + "metadata": { + "execution": {} + }, + "source": [ + "# The Big Picture\n", + "\n", + "The goal of this tutorial is to provide you with some mathematical tools for your NeuroAI researcher toolkit. What happens when you pull out the Euclidean metric from your toolkit and, while this has worked well in the past, suddenly in different scenarios it doesn't seem to perform so well. Aha, you spot the potential for correlated noise and you reach deeper into your toolkit and pull out the Mahalanobis metric, which implicitly undoes the correlated noise in the model. Perhaps you can't even tell if there is any correlated noise in your data and you try with both metrics, and Mahalanobis works well but Euclidean does not, that can be a hunch that leads you to confirm the presence of correlated noise. \n", + "\n", + "Sometimes you might be faced with dimensionalities that are just too high to practically deal with in your use case. Then, why not recall what you learned about how random projections can reduce the dimensionality of a feature space and be largely resistant to corrupting the applicability of distance metrics. These metrics also might work better in this lower dimensional space. If you apply this idea and need to justify it, just reach into your NeuroAI toolkit and pull out the Johnson-Lindenstrauss Lemma as your justification." + ] } ], "metadata": { @@ -1491,7 +1571,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4, diff --git a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial5.ipynb b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial5.ipynb index 77480d80e..191d2f283 100644 --- a/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial5.ipynb +++ b/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/student/W1D3_Tutorial5.ipynb @@ -17,17 +17,17 @@ "execution": {} }, "source": [ - "# Bonus Material: Dynamical similarity analysis (DSA)\n", + "# Tutorial 5: Dynamical Similarity Analysis (DSA)\n", "\n", "**Week 1, Day 3: Comparing Artificial And Biological Networks**\n", "\n", "**By Neuromatch Academy**\n", "\n", - "__Content creators:__ Mitchell Ostrow\n", + "__Content creators:__ Mitchell Ostrow, Alex Murphy\n", "\n", - "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk\n", + "__Content reviewers:__ Xaq Pitkow, Hlib Solodzhuk, Alex Murphy\n", "\n", - "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault\n" + "__Production editors:__ Konstantine Tsafatinos, Ella Batty, Spiros Chavlis, Samuele Bolotta, Hlib Solodzhuk, Patrick Mineault, Alex Murphy\n" ] }, { @@ -52,7 +52,7 @@ "source": [ "# @title Install and import feedback gadget\n", "\n", - "!pip install vibecheck --quiet\n", + "!pip install vibecheck rsatoolbox --quiet\n", "\n", "from vibecheck import DatatopsContentReviewContainer\n", "def content_review(notebook_section: str):\n", @@ -67,7 +67,2087 @@ " ).render()\n", "\n", "\n", - "feedback_prefix = \"W1D3_Bonus\"" + "feedback_prefix = \"W1D5_DSA\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ef9abaa3", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Helper functions\n", + "\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "\n", + "def generate_2d_random_process(A, B, T=1000):\n", + " \"\"\"\n", + " Generates a 2D random process with the equation x(t+1) = A.x(t) + B.noise.\n", + "\n", + " Args:\n", + " A: 2x2 transition matrix.\n", + " B: 2x2 noise scaling matrix.\n", + " T: Number of time steps.\n", + "\n", + " Returns:\n", + " A NumPy array of shape (T+1, 2) representing the trajectory.\n", + " \"\"\"\n", + " # Assuming equilibrium distribution is zero mean and identity covariance for simplicity.\n", + " # You may adjust this according to your actual equilibrium distribution\n", + " x = np.zeros(2)\n", + "\n", + " trajectory = [x.copy()] # Initialize with x(0)\n", + " for t in range(T):\n", + " noise = np.random.normal(size=2) # Standard normal noise\n", + " x = np.dot(A, x) + np.dot(B, noise)\n", + " trajectory.append(x.copy())\n", + " return np.array(trajectory)\n", + "\n", + "\"\"\"This module computes the Havok DMD model for a given dataset.\"\"\"\n", + "import torch\n", + "\n", + "def embed_signal_torch(data, n_delays, delay_interval=1):\n", + " \"\"\"\n", + " Create a delay embedding from the provided tensor data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : torch.tensor\n", + " The data from which to create the delay embedding. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + " \"\"\"\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " device = data.device\n", + "\n", + " if data.shape[int(data.ndim==3)] - (n_delays - 1)*delay_interval < 1:\n", + " raise ValueError(\"The number of delays is too large for the number of time points in the data!\")\n", + "\n", + " # initialize the embedding\n", + " if data.ndim == 3:\n", + " embedding = torch.zeros((data.shape[0], data.shape[1] - (n_delays - 1)*delay_interval, data.shape[2]*n_delays)).to(device)\n", + " else:\n", + " embedding = torch.zeros((data.shape[0] - (n_delays - 1)*delay_interval, data.shape[1]*n_delays)).to(device)\n", + "\n", + " for d in range(n_delays):\n", + " index = (n_delays - 1 - d)*delay_interval\n", + " ddelay = d*delay_interval\n", + "\n", + " if data.ndim == 3:\n", + " ddata = d*data.shape[2]\n", + " embedding[:,:, ddata: ddata + data.shape[2]] = data[:,index:data.shape[1] - ddelay]\n", + " else:\n", + " ddata = d*data.shape[1]\n", + " embedding[:, ddata:ddata + data.shape[1]] = data[index:data.shape[0] - ddelay]\n", + "\n", + " return embedding\n", + "\n", + "class DMD:\n", + " \"\"\"DMD class for computing and predicting with DMD models.\n", + " \"\"\"\n", + " def __init__(\n", + " self,\n", + " data,\n", + " n_delays,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=False,\n", + " lamb=0,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=False,\n", + " steps_ahead=1\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " reduced_rank_reg : bool\n", + " Determines whether to use reduced rank regression (True) or principal component regression (False)\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0.\n", + "\n", + " device: string, int, or torch.device\n", + " A string, int or torch.device object to indicate the device to torch.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + " \"\"\"\n", + "\n", + " self.device = device\n", + " self._init_data(data)\n", + "\n", + " self.n_delays = n_delays\n", + " self.delay_interval = delay_interval\n", + " self.rank = rank\n", + " self.rank_thresh = rank_thresh\n", + " self.rank_explained_variance = rank_explained_variance\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.lamb = lamb\n", + " self.verbose = verbose\n", + " self.send_to_cpu = send_to_cpu\n", + " self.steps_ahead = steps_ahead\n", + "\n", + " # Hankel matrix\n", + " self.H = None\n", + "\n", + " # SVD attributes\n", + " self.U = None\n", + " self.S = None\n", + " self.V = None\n", + " self.S_mat = None\n", + " self.S_mat_inv = None\n", + "\n", + " # DMD attributes\n", + " self.A_v = None\n", + " self.A_havok_dmd = None\n", + "\n", + " def _init_data(self, data):\n", + " # check if the data is an np.ndarry - if so, convert it to Torch\n", + " if isinstance(data, np.ndarray):\n", + " data = torch.from_numpy(data)\n", + " self.data = data\n", + " # create attributes for the data dimensions\n", + " if self.data.ndim == 3:\n", + " self.ntrials = self.data.shape[0]\n", + " self.window = self.data.shape[1]\n", + " self.n = self.data.shape[2]\n", + " else:\n", + " self.window = self.data.shape[0]\n", + " self.n = self.data.shape[1]\n", + " self.ntrials = 1\n", + "\n", + " def compute_hankel(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " ):\n", + " \"\"\"\n", + " Computes the Hankel matrix from the provided data.\n", + "\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults\n", + " to 1 time step. Defaults to None - provide only if you want\n", + " to override the value of n_delays from the init.\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing Hankel matrix ...\")\n", + "\n", + " # if parameters are provided, overwrite them from the init\n", + " self.data = self.data if data is None else self._init_data(data)\n", + " self.n_delays = self.n_delays if n_delays is None else n_delays\n", + " self.delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " self.data = self.data.to(self.device)\n", + "\n", + " self.H = embed_signal_torch(self.data, self.n_delays, self.delay_interval)\n", + "\n", + " if self.verbose:\n", + " print(\"Hankel matrix computed!\")\n", + "\n", + " def compute_svd(self):\n", + " \"\"\"\n", + " Computes the SVD of the Hankel matrix.\n", + " \"\"\"\n", + "\n", + " if self.verbose:\n", + " print(\"Computing SVD on Hankel matrix ...\")\n", + " if self.H.ndim == 3: #flatten across trials for 3d\n", + " H = self.H.reshape(self.H.shape[0] * self.H.shape[1], self.H.shape[2])\n", + " else:\n", + " H = self.H\n", + " # compute the SVD\n", + " U, S, Vh = torch.linalg.svd(H.T, full_matrices=False)\n", + "\n", + " # update attributes\n", + " V = Vh.T\n", + " self.U = U\n", + " self.S = S\n", + " self.V = V\n", + "\n", + " # construct the singuar value matrix and its inverse\n", + " # dim = self.n_delays * self.n\n", + " # s = len(S)\n", + " # self.S_mat = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " # self.S_mat_inv = torch.zeros(dim, dim,dtype=torch.float32).to(self.device)\n", + " self.S_mat = torch.diag(S).to(self.device)\n", + " self.S_mat_inv= torch.diag(1 / S).to(self.device)\n", + "\n", + " # compute explained variance\n", + " exp_variance_inds = self.S**2 / ((self.S**2).sum())\n", + " cumulative_explained = torch.cumsum(exp_variance_inds, 0)\n", + " self.cumulative_explained_variance = cumulative_explained\n", + "\n", + " #make the X and Y components of the regression by staggering the hankel eigen-time delay coordinates by time\n", + " if self.reduced_rank_reg:\n", + " V = self.V\n", + " else:\n", + " V = self.V\n", + "\n", + " if self.ntrials > 1:\n", + " if V.numel() < self.H.numel():\n", + " raise ValueError(\"The dimension of the SVD of the Hankel matrix is smaller than the dimension of the Hankel matrix itself. \\n \\\n", + " This is likely due to the number of time points being smaller than the number of dimensions. \\n \\\n", + " Please reduce the number of delays.\")\n", + "\n", + " V = V.reshape(self.H.shape)\n", + "\n", + " #first reshape back into Hankel shape, separated by trials\n", + " newshape = (self.H.shape[0]*(self.H.shape[1]-self.steps_ahead),self.H.shape[2])\n", + " self.Vt_minus = V[:,:-self.steps_ahead].reshape(newshape)\n", + " self.Vt_plus = V[:,self.steps_ahead:].reshape(newshape)\n", + " else:\n", + " self.Vt_minus = V[:-self.steps_ahead]\n", + " self.Vt_plus = V[self.steps_ahead:]\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"SVD complete!\")\n", + "\n", + " def recalc_rank(self,rank,rank_thresh,rank_explained_variance):\n", + " '''\n", + " Parameters\n", + " ----------\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used. Provide only if you want to override the value from the init.\n", + "\n", + " rank_thresh : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want\n", + " to override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + " '''\n", + " # if an argument was provided, overwrite the stored rank information\n", + " none_vars = (rank is None) + (rank_thresh is None) + (rank_explained_variance is None)\n", + " if none_vars != 3:\n", + " self.rank = None\n", + " self.rank_thresh = None\n", + " self.rank_explained_variance = None\n", + "\n", + " self.rank = self.rank if rank is None else rank\n", + " self.rank_thresh = self.rank_thresh if rank_thresh is None else rank_thresh\n", + " self.rank_explained_variance = self.rank_explained_variance if rank_explained_variance is None else rank_explained_variance\n", + "\n", + " none_vars = (self.rank is None) + (self.rank_thresh is None) + (self.rank_explained_variance is None)\n", + " if none_vars < 2:\n", + " raise ValueError(\"More than one value was provided between rank, rank_thresh, and rank_explained_variance. Please provide only one of these, and ensure the others are None!\")\n", + " elif none_vars == 3:\n", + " self.rank = len(self.S)\n", + "\n", + " if self.reduced_rank_reg:\n", + " S = self.proj_mat_S\n", + " else:\n", + " S = self.S\n", + "\n", + " if rank_thresh is not None:\n", + " if S[-1] > rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < rank_thresh))\n", + "\n", + " if rank_explained_variance is not None:\n", + " self.rank = int(torch.argmax((self.cumulative_explained_variance > rank_explained_variance).type(torch.int)).cpu().numpy())\n", + "\n", + " if self.rank > self.H.shape[-1]:\n", + " self.rank = self.H.shape[-1]\n", + "\n", + " if self.rank is None:\n", + " if S[-1] > self.rank_thresh:\n", + " self.rank = len(S)\n", + " else:\n", + " self.rank = torch.argmax(torch.arange(len(S), 0, -1).to(self.device)*(S < self.rank_thresh))\n", + "\n", + " def compute_havok_dmd(self,lamb=None):\n", + " \"\"\"\n", + " Computes the Havok DMD matrix (Principal Component Regression)\n", + "\n", + " Parameters\n", + " ----------\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to 0 - provide only if you want\n", + " to override the value of n_delays from the init.\n", + "\n", + " \"\"\"\n", + " if self.verbose:\n", + " print(\"Computing least squares fits to HAVOK DMD ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " A_v = (torch.linalg.inv(self.Vt_minus[:, :self.rank].T @ self.Vt_minus[:, :self.rank] + self.lamb*torch.eye(self.rank).to(self.device)) \\\n", + " @ self.Vt_minus[:, :self.rank].T @ self.Vt_plus[:, :self.rank]).T\n", + " self.A_v = A_v\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1], :self.rank] @ self.A_v @ self.S_mat_inv[:self.rank, :self.U.shape[1]] @ self.U.T\n", + "\n", + " if self.verbose:\n", + " print(\"Least squares complete! \\n\")\n", + "\n", + " def compute_proj_mat(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Projector Matrix for Reduced Rank Regression\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + "\n", + " self.proj_mat = self.Vt_plus.T @ self.Vt_minus @ torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus +\n", + " self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ \\\n", + " self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.proj_mat_S, self.proj_mat_V = torch.linalg.eigh(self.proj_mat)\n", + " #todo: more efficient to flip ranks (negative index) in compute_reduced_rank_regression but also less interpretable\n", + " self.proj_mat_S = torch.flip(self.proj_mat_S, dims=(0,))\n", + " self.proj_mat_V = torch.flip(self.proj_mat_V, dims=(1,))\n", + "\n", + " if self.verbose:\n", + " print(\"Projector Matrix computed! \\n\")\n", + "\n", + " def compute_reduced_rank_regression(self,lamb=None):\n", + " if self.verbose:\n", + " print(\"Computing Reduced Rank Regression ...\")\n", + "\n", + " self.lamb = self.lamb if lamb is None else lamb\n", + " proj_mat = self.proj_mat_V[:,:self.rank] @ self.proj_mat_V[:,:self.rank].T\n", + " B_ols = torch.linalg.inv(self.Vt_minus.T @ self.Vt_minus + self.lamb*torch.eye(self.Vt_minus.shape[1]).to(self.device)) @ self.Vt_minus.T @ self.Vt_plus\n", + "\n", + " self.A_v = B_ols @ proj_mat\n", + " self.A_havok_dmd = self.U @ self.S_mat[:self.U.shape[1],:self.A_v.shape[1]] @ self.A_v.T @ self.S_mat_inv[:self.A_v.shape[0], :self.U.shape[1]] @ self.U.T\n", + "\n", + "\n", + " if self.verbose:\n", + " print(\"Reduced Rank Regression complete! \\n\")\n", + "\n", + " def fit(\n", + " self,\n", + " data=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance=None,\n", + " lamb=None,\n", + " device=None,\n", + " verbose=None,\n", + " steps_ahead=None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " ----------\n", + " data : np.ndarray or torch.tensor\n", + " The data to fit the DMD model to. Must be either: (1) a\n", + " 2-dimensional array/tensor of shape T x N where T is the number\n", + " of time points and N is the number of observed dimensions\n", + " at each time point, or (2) a 3-dimensional array/tensor of shape\n", + " K x T x N where K is the number of \"trials\" and T and N are\n", + " as defined above. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " n_delays : int\n", + " Parameter that controls the size of the delay embedding. Explicitly,\n", + " the number of delays to include. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " delay_interval : int\n", + " The number of time steps between each delay in the delay embedding. Defaults to None -\n", + " provide only if you want to override the value from the init.\n", + "\n", + " rank : int\n", + " The rank of V in fitting HAVOK DMD - i.e., the number of columns of V to\n", + " use to fit the DMD model. Defaults to None, in which case all columns of V\n", + " will be used - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_thresh : int\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " rank_explained_variance : float\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None -\n", + " provide only if you want to overried the value from the init.\n", + "\n", + " lamb : float\n", + " Regularization parameter for ridge regression. Defaults to None - provide only if you want to\n", + " override the value from the init.\n", + "\n", + " device: string or int\n", + " A string or int to indicate the device to torch. For example, can be 'cpu' or 'cuda',\n", + " or alternatively 0 if the intenion is to use GPU device 0. Defaults to None - provide only\n", + " if you want to override the value from the init.\n", + "\n", + " verbose: bool\n", + " If True, print statements will be provided about the progress of the fitting procedure.\n", + " Defaults to None - provide only if you want to override the value from the init.\n", + "\n", + " steps_ahead: int\n", + " The number of time steps ahead to predict. Defaults to 1.\n", + "\n", + " \"\"\"\n", + " # if parameters are provided, overwrite them from the init\n", + " self.steps_ahead = self.steps_ahead if steps_ahead is None else steps_ahead\n", + " self.device = self.device if device is None else device\n", + " self.verbose = self.verbose if verbose is None else verbose\n", + "\n", + " self.compute_hankel(data, n_delays, delay_interval)\n", + " self.compute_svd()\n", + "\n", + " if self.reduced_rank_reg:\n", + " self.compute_proj_mat(lamb)\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_reduced_rank_regression(lamb)\n", + " else:\n", + " self.recalc_rank(rank,rank_thresh,rank_explained_variance)\n", + " self.compute_havok_dmd(lamb)\n", + "\n", + " if self.send_to_cpu:\n", + " self.all_to_device('cpu') #send back to the cpu to save memory\n", + "\n", + " def predict(\n", + " self,\n", + " test_data=None,\n", + " reseed=None,\n", + " full_return=False\n", + " ):\n", + " \"\"\"\n", + " Returns\n", + " -------\n", + " pred_data : torch.tensor\n", + " The predictions generated by the HAVOK model. Of the same shape as test_data. Note that the first\n", + " (self.n_delays - 1)*self.delay_interval + 1 time steps of the generated predictions are by construction\n", + " identical to the test_data.\n", + "\n", + " H_test_havok_dmd : torch.tensor (Optional)\n", + " Returned if full_return=True. The predicted Hankel matrix generated by the HAVOK model.\n", + " H_test : torch.tensor (Optional)\n", + " Returned if full_return=True. The true Hankel matrix\n", + " \"\"\"\n", + " # initialize test_data\n", + " if test_data is None:\n", + " test_data = self.data\n", + " if isinstance(test_data, np.ndarray):\n", + " test_data = torch.from_numpy(test_data).to(self.device)\n", + " ndim = test_data.ndim\n", + " if ndim == 2:\n", + " test_data = test_data.unsqueeze(0)\n", + " H_test = embed_signal_torch(test_data, self.n_delays, self.delay_interval)\n", + " steps_ahead = self.steps_ahead if self.steps_ahead is not None else 1\n", + "\n", + " if reseed is None:\n", + " reseed = 1\n", + "\n", + " H_test_havok_dmd = torch.zeros(H_test.shape).to(self.device)\n", + " H_test_havok_dmd[:, :steps_ahead] = H_test[:, :steps_ahead]\n", + "\n", + " A = self.A_havok_dmd.unsqueeze(0)\n", + " for t in range(steps_ahead, H_test.shape[1]):\n", + " if t % reseed == 0:\n", + " H_test_havok_dmd[:, t] = (A @ H_test[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " else:\n", + " H_test_havok_dmd[:, t] = (A @ H_test_havok_dmd[:, t - steps_ahead].transpose(-2, -1)).transpose(-2, -1)\n", + " pred_data = torch.hstack([test_data[:, :(self.n_delays - 1)*self.delay_interval + steps_ahead], H_test_havok_dmd[:, steps_ahead:, :self.n]])\n", + "\n", + " if ndim == 2:\n", + " pred_data = pred_data[0]\n", + "\n", + " if full_return:\n", + " return pred_data, H_test_havok_dmd, H_test\n", + " else:\n", + " return pred_data\n", + "\n", + " def all_to_device(self,device='cpu'):\n", + " for k,v in self.__dict__.items():\n", + " if isinstance(v, torch.Tensor):\n", + " self.__dict__[k] = v.to(device)\n", + "\n", + "from typing import Literal\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "from typing import Literal\n", + "import torch.nn.utils.parametrize as parametrize\n", + "from scipy.stats import wasserstein_distance\n", + "\n", + "def pad_zeros(A,B,device):\n", + "\n", + " with torch.no_grad():\n", + " dim = max(A.shape[0],B.shape[0])\n", + " A1 = torch.zeros((dim,dim)).float()\n", + " A1[:A.shape[0],:A.shape[1]] += A\n", + " A = A1.float().to(device)\n", + "\n", + " B1 = torch.zeros((dim,dim)).float()\n", + " B1[:B.shape[0],:B.shape[1]] += B\n", + " B = B1.float().to(device)\n", + "\n", + " return A,B\n", + "\n", + "class LearnableSimilarityTransform(nn.Module):\n", + " \"\"\"\n", + " Computes the similarity transform for a learnable orthonormal matrix C\n", + " \"\"\"\n", + " def __init__(self, n,orthog=True):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + " n : int\n", + " dimension of the C matrix\n", + " \"\"\"\n", + " super(LearnableSimilarityTransform, self).__init__()\n", + " #initialize orthogonal matrix as identity\n", + " self.C = nn.Parameter(torch.eye(n).float())\n", + " self.orthog = orthog\n", + "\n", + " def forward(self, B):\n", + " if self.orthog:\n", + " return self.C @ B @ self.C.transpose(-1, -2)\n", + " else:\n", + " return self.C @ B @ torch.linalg.inv(self.C)\n", + "\n", + "class Skew(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a skew-symmetric matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X - X.transpose(-1, -2)\n", + "\n", + "class Matrix(nn.Module):\n", + " def __init__(self,n,device):\n", + " \"\"\"\n", + " Computes a matrix X from some parameters (also called X)\n", + "\n", + " \"\"\"\n", + " super().__init__()\n", + "\n", + " self.L1 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L2 = nn.Linear(n,n,bias = False, device = device)\n", + " self.L3 = nn.Linear(n,n,bias = False, device = device)\n", + "\n", + " def forward(self, X):\n", + " X = torch.tanh(self.L1(X))\n", + " X = torch.tanh(self.L2(X))\n", + " X = self.L3(X)\n", + " return X\n", + "\n", + "class CayleyMap(nn.Module):\n", + " \"\"\"\n", + " Maps a skew-symmetric matrix to an orthogonal matrix in O(n)\n", + " \"\"\"\n", + " def __init__(self, n, device):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " n : int\n", + " dimension of the matrix we want to map\n", + "\n", + " device : {'cpu','cuda'} or int\n", + " hardware device on which to send the matrix\n", + " \"\"\"\n", + " super().__init__()\n", + " self.register_buffer(\"Id\", torch.eye(n,device = device))\n", + "\n", + " def forward(self, X):\n", + " # (I + X)(I - X)^{-1}\n", + " return torch.linalg.solve(self.Id + X, self.Id - X)\n", + "\n", + "class SimilarityTransformDist:\n", + " \"\"\"\n", + " Computes the Procrustes Analysis over Vector Fields\n", + " \"\"\"\n", + " def __init__(self,\n", + " iters = 200,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 0.01,\n", + " device: Literal[\"cpu\",\"cuda\"] = 'cpu',\n", + " verbose = False,\n", + " group: Literal[\"O(n)\",\"SO(n)\",\"GL(n)\"] = \"O(n)\",\n", + " wasserstein_compare = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " _________\n", + " iters : int\n", + " number of iterations to perform gradient descent\n", + "\n", + " score_method : {\"angular\",\"euclidean\",\"wasserstein\"}\n", + " specifies the type of metric to use\n", + " \"wasserstein\" will compare the singular values or eigenvalues\n", + " of the two matrices as in Redman et al., (2023)\n", + "\n", + " lr : float\n", + " learning rate\n", + "\n", + " device : {'cpu','cuda'} or int\n", + "\n", + " verbose : bool\n", + " prints when finished optimizing\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + "\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.verbose = verbose\n", + " self.device = device\n", + " self.C_star = None\n", + " self.A = None\n", + " self.B = None\n", + " self.group = group\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " def fit(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " group = None,\n", + " ):\n", + " \"\"\"\n", + " Computes the optimal matrix C over specified group\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " Returns\n", + " _______\n", + " None\n", + " \"\"\"\n", + " assert A.shape[0] == A.shape[1]\n", + " assert B.shape[0] == B.shape[1]\n", + "\n", + " A = A.to(self.device)\n", + " B = B.to(self.device)\n", + " self.A,self.B = A,B\n", + " lr = self.lr if lr is None else lr\n", + " iters = self.iters if iters is None else iters\n", + " group = self.group if group is None else group\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if group == \"O(n)\":\n", + " #permute the first row and column of B then rerun the optimization\n", + " P = torch.eye(B.shape[0],device=self.device)\n", + " if P.shape[0] > 1:\n", + " P[[0, 1], :] = P[[1, 0], :]\n", + " losses, C_star, sim_net = self.optimize_C(A,\n", + " P @ B @ P.T,\n", + " lr,iters,\n", + " orthog=True,\n", + " verbose=self.verbose)\n", + " if losses[-1] < self.losses[-1]:\n", + " self.losses = losses\n", + " self.C_star = C_star @ P\n", + " self.sim_net = sim_net\n", + " if group == \"GL(n)\":\n", + " self.losses, self.C_star, self.sim_net = self.optimize_C(A,\n", + " B,\n", + " lr,iters,\n", + " orthog=False,\n", + " verbose=self.verbose)\n", + "\n", + " def optimize_C(self,A,B,lr,iters,orthog,verbose):\n", + " #parameterize mapping to be orthogonal\n", + " n = A.shape[0]\n", + " sim_net = LearnableSimilarityTransform(n,orthog=orthog).to(self.device)\n", + " if orthog:\n", + " parametrize.register_parametrization(sim_net, \"C\", Skew(n,self.device))\n", + " parametrize.register_parametrization(sim_net, \"C\", CayleyMap(n,self.device))\n", + " else:\n", + " parametrize.register_parametrization(sim_net, \"C\", Matrix(n,self.device))\n", + "\n", + " simdist_loss = nn.MSELoss(reduction = 'sum')\n", + "\n", + " optimizer = optim.Adam(sim_net.parameters(), lr=lr)\n", + " # scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.999)\n", + "\n", + " losses = []\n", + " A /= torch.linalg.norm(A)\n", + " B /= torch.linalg.norm(B)\n", + " for _ in range(iters):\n", + " # Zero the gradients of the optimizer.\n", + " optimizer.zero_grad()\n", + " # Compute the Frobenius norm between A and the product.\n", + " loss = simdist_loss(A, sim_net(B))\n", + "\n", + " loss.backward()\n", + "\n", + " optimizer.step()\n", + " # if _ % 99:\n", + " # scheduler.step()\n", + " losses.append(loss.item())\n", + "\n", + " if verbose:\n", + " print(\"Finished optimizing C\")\n", + "\n", + " C_star = sim_net.C.detach()\n", + " return losses, C_star,sim_net\n", + "\n", + " def score(self,A=None,B=None,score_method=None,group=None):\n", + " \"\"\"\n", + " Given an optimal C already computed, calculate the metric\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor or None\n", + " first data matrix, if None defaults to the saved matrix in fit\n", + " B : np.array or torch.tensor or None\n", + " second data matrix if None, defaults to the savec matrix in fit\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + " \"\"\"\n", + " assert self.C_star is not None\n", + " A = self.A if A is None else A\n", + " B = self.B if B is None else B\n", + " assert A is not None\n", + " assert B is not None\n", + " assert A.shape == self.C_star.shape\n", + " assert B.shape == self.C_star.shape\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + " with torch.no_grad():\n", + " if not isinstance(A,torch.Tensor):\n", + " A = torch.from_numpy(A).float().to(self.device)\n", + " if not isinstance(B,torch.Tensor):\n", + " B = torch.from_numpy(B).float().to(self.device)\n", + " C = self.C_star.to(self.device)\n", + "\n", + " if group in {\"SO(n)\", \"O(n)\"}:\n", + " Cinv = C.T\n", + " elif group in {\"GL(n)\"}:\n", + " Cinv = torch.linalg.inv(C)\n", + " else:\n", + " raise AssertionError(\"Need proper group name\")\n", + " if score_method == 'angular':\n", + " num = torch.trace(A.T @ C @ B @ Cinv)\n", + " den = torch.norm(A,p = 'fro')*torch.norm(B,p = 'fro')\n", + " score = torch.arccos(num/den).cpu().numpy()\n", + " if np.isnan(score): #around -1 and 1, we sometimes get NaNs due to arccos\n", + " if num/den < 0:\n", + " score = np.pi\n", + " else:\n", + " score = 0\n", + " else:\n", + " score = torch.norm(A - C @ B @ Cinv,p='fro').cpu().numpy().item() #/ A.numpy().size\n", + "\n", + " return score\n", + "\n", + " def fit_score(self,\n", + " A,\n", + " B,\n", + " iters = None,\n", + " lr = None,\n", + " score_method = None,\n", + " zero_pad = True,\n", + " group = None):\n", + " \"\"\"\n", + " for efficiency, computes the optimal matrix and returns the score\n", + "\n", + " Parameters\n", + " __________\n", + " A : np.array or torch.tensor\n", + " first data matrix\n", + " B : np.array or torch.tensor\n", + " second data matrix\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : {'angular','euclidean'} or None\n", + " overwrites parameter in the class\n", + " zero_pad : bool\n", + " if True, then the smaller matrix will be zero padded so its the same size\n", + " Returns\n", + " _______\n", + "\n", + " score : float\n", + " similarity of the data under the similarity transform w.r.t C\n", + "\n", + " \"\"\"\n", + " score_method = self.score_method if score_method is None else score_method\n", + " group = self.group if group is None else group\n", + "\n", + " if isinstance(A,np.ndarray):\n", + " A = torch.from_numpy(A).float()\n", + " if isinstance(B,np.ndarray):\n", + " B = torch.from_numpy(B).float()\n", + "\n", + " assert A.shape[0] == B.shape[1] or self.wasserstein_compare is not None\n", + " if A.shape[0] != B.shape[0]:\n", + " if self.wasserstein_compare is None:\n", + " raise AssertionError(\"Matrices must be the same size unless using wasserstein distance\")\n", + " else: #otherwise resort to L2 Wasserstein over singular or eigenvalues\n", + " print(f\"resorting to wasserstein distance over {self.wasserstein_compare}\")\n", + "\n", + " if self.score_method == \"wasserstein\":\n", + " assert self.wasserstein_compare in {\"sv\",\"eig\"}\n", + " if self.wasserstein_compare == \"sv\":\n", + " a = torch.svd(A).S.view(-1,1)\n", + " b = torch.svd(B).S.view(-1,1)\n", + " elif self.wasserstein_compare == \"eig\":\n", + " a = torch.linalg.eig(A).eigenvalues\n", + " a = torch.vstack([a.real,a.imag]).T\n", + "\n", + " b = torch.linalg.eig(B).eigenvalues\n", + " b = torch.vstack([b.real,b.imag]).T\n", + " else:\n", + " raise AssertionError(\"wasserstein_compare must be 'sv' or 'eig'\")\n", + " device = a.device\n", + " a = a#.cpu()\n", + " b = b#.cpu()\n", + " M = ot.dist(a,b)#.numpy()\n", + " a,b = torch.ones(a.shape[0])/a.shape[0],torch.ones(b.shape[0])/b.shape[0]\n", + " a,b = a.to(device),b.to(device)\n", + "\n", + " score_star = ot.emd2(a,b,M)\n", + " #wasserstein_distance(A.cpu().numpy(),B.cpu().numpy())\n", + "\n", + " else:\n", + "\n", + " self.fit(A, B,iters,lr,group)\n", + " score_star = self.score(self.A,self.B,score_method=score_method,group=group)\n", + "\n", + " return score_star\n", + "\n", + "class DSA:\n", + " \"\"\"\n", + " Computes the Dynamical Similarity Analysis (DSA) for two data matrices\n", + " \"\"\"\n", + " def __init__(self,\n", + " X,\n", + " Y=None,\n", + " n_delays=1,\n", + " delay_interval=1,\n", + " rank=None,\n", + " rank_thresh=None,\n", + " rank_explained_variance = None,\n", + " lamb = 0.0,\n", + " send_to_cpu = True,\n", + " iters = 1500,\n", + " score_method: Literal[\"angular\", \"euclidean\",\"wasserstein\"] = \"angular\",\n", + " lr = 5e-3,\n", + " group: Literal[\"GL(n)\", \"O(n)\", \"SO(n)\"] = \"O(n)\",\n", + " zero_pad = False,\n", + " device = 'cpu',\n", + " verbose = False,\n", + " reduced_rank_reg = False,\n", + " kernel=None,\n", + " num_centers=0.1,\n", + " svd_solver='arnoldi',\n", + " wasserstein_compare: Literal['sv','eig',None] = None\n", + " ):\n", + " \"\"\"\n", + " Parameters\n", + " __________\n", + "\n", + " X : np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " first data matrix/matrices\n", + "\n", + " Y : None or np.array or torch.tensor or list of np.arrays or torch.tensors\n", + " second data matrix/matrices.\n", + " * If Y is None, X is compared to itself pairwise\n", + " (must be a list)\n", + " * If Y is a single matrix, all matrices in X are compared to Y\n", + " * If Y is a list, all matrices in X are compared to all matrices in Y\n", + "\n", + " DMD parameters:\n", + "\n", + " n_delays : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " number of delays to use in constructing the Hankel matrix\n", + "\n", + " delay_interval : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " interval between samples taken in constructing Hankel matrix\n", + "\n", + " rank : int or list or tuple/list: (int,int), (list,list),(list,int),(int,list)\n", + " rank of DMD matrix fit in reduced-rank regression\n", + "\n", + " rank_thresh : float or list or tuple/list: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by dictating a threshold\n", + " of singular values to use. Explicitly, the rank of V will be the number of singular\n", + " values greater than rank_thresh. Defaults to None.\n", + "\n", + " rank_explained_variance : float or list or tuple: (float,float), (list,list),(list,float),(float,list)\n", + " Parameter that controls the rank of V in fitting HAVOK DMD by indicating the percentage of\n", + " cumulative explained variance that should be explained by the columns of V. Defaults to None.\n", + "\n", + " lamb : float\n", + " L-1 regularization parameter in DMD fit\n", + "\n", + " send_to_cpu: bool\n", + " If True, will send all tensors in the object back to the cpu after everything is computed.\n", + " This is implemented to prevent gpu memory overload when computing multiple DMDs.\n", + "\n", + " NOTE: for all of these above, they can be single values or lists or tuples,\n", + " depending on the corresponding dimensions of the data\n", + " If at least one of X and Y are lists, then if they are a single value\n", + " it will default to the rank of all DMD matrices.\n", + " If they are (int,int), then they will correspond to an individual dmd matrix\n", + " OR to X and Y respectively across all matrices\n", + " If it is (list,list), then each element will correspond to an individual\n", + " dmd matrix indexed at the same position\n", + "\n", + " SimDist parameters:\n", + "\n", + " iters : int\n", + " number of optimization iterations in Procrustes over vector fields\n", + "\n", + " score_method : {'angular','euclidean'}\n", + " type of metric to compute, angular vs euclidean distance\n", + "\n", + " lr : float\n", + " learning rate of the Procrustes over vector fields optimization\n", + "\n", + " group : {'SO(n)','O(n)', 'GL(n)'}\n", + " specifies the group of matrices to optimize over\n", + "\n", + " zero_pad : bool\n", + " whether or not to zero-pad if the dimensions are different\n", + "\n", + " device : 'cpu' or 'cuda' or int\n", + " hardware to use in both DMD and PoVF\n", + "\n", + " verbose : bool\n", + " whether or not print when sections of the analysis is completed\n", + "\n", + " wasserstein_compare : {'sv','eig',None}\n", + " specifies whether to compare the singular values or eigenvalues\n", + " if score_method is \"wasserstein\", or the shapes are different\n", + " \"\"\"\n", + " self.X = X\n", + " self.Y = Y\n", + " if self.X is None and isinstance(self.Y,list):\n", + " self.X, self.Y = self.Y, self.X #swap so code is easy\n", + "\n", + " self.check_method()\n", + " if self.method == 'self-pairwise':\n", + " self.data = [self.X]\n", + " else:\n", + " self.data = [self.X, self.Y]\n", + "\n", + " self.n_delays = self.broadcast_params(n_delays,cast=int)\n", + " self.delay_interval = self.broadcast_params(delay_interval,cast=int)\n", + " self.rank = self.broadcast_params(rank,cast=int)\n", + " self.rank_thresh = self.broadcast_params(rank_thresh)\n", + " self.rank_explained_variance = self.broadcast_params(rank_explained_variance)\n", + " self.lamb = self.broadcast_params(lamb)\n", + " self.send_to_cpu = send_to_cpu\n", + " self.iters = iters\n", + " self.score_method = score_method\n", + " self.lr = lr\n", + " self.device = device\n", + " self.verbose = verbose\n", + " self.zero_pad = zero_pad\n", + " self.group = group\n", + " self.reduced_rank_reg = reduced_rank_reg\n", + " self.kernel = kernel\n", + " self.wasserstein_compare = wasserstein_compare\n", + "\n", + " if kernel is None:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[DMD(Xi,\n", + " self.n_delays[i][j],\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " rank_thresh=self.rank_thresh[i][j],\n", + " rank_explained_variance=self.rank_explained_variance[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " device=self.device,\n", + " verbose=self.verbose,\n", + " send_to_cpu=self.send_to_cpu) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + " else:\n", + " #get a list of all DMDs here\n", + " self.dmds = [[KernelDMD(Xi,\n", + " self.n_delays[i][j],\n", + " kernel=self.kernel,\n", + " num_centers=num_centers,\n", + " delay_interval=self.delay_interval[i][j],\n", + " rank=self.rank[i][j],\n", + " reduced_rank_reg=self.reduced_rank_reg,\n", + " lamb=self.lamb[i][j],\n", + " verbose=self.verbose,\n", + " svd_solver=svd_solver,\n", + " ) for j,Xi in enumerate(dat)] for i,dat in enumerate(self.data)]\n", + "\n", + " self.simdist = SimilarityTransformDist(iters,score_method,lr,device,verbose,group,wasserstein_compare)\n", + "\n", + " def check_method(self):\n", + " '''\n", + " helper function to identify what type of dsa we're running\n", + " '''\n", + " tensor_or_np = lambda x: isinstance(x,(np.ndarray,torch.Tensor))\n", + "\n", + " if isinstance(self.X,list):\n", + " if self.Y is None:\n", + " self.method = 'self-pairwise'\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'bipartite-pairwise'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'list-to-one'\n", + " self.Y = [self.Y] #wrap in a list for iteration\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " elif tensor_or_np(self.X):\n", + " self.X = [self.X]\n", + " if self.Y is None:\n", + " raise ValueError('only one element provided')\n", + " elif isinstance(self.Y,list):\n", + " self.method = 'one-to-list'\n", + " elif tensor_or_np(self.Y):\n", + " self.method = 'default'\n", + " self.Y = [self.Y]\n", + " else:\n", + " raise ValueError('unknown type of Y')\n", + " else:\n", + " raise ValueError('unknown type of X')\n", + "\n", + " def broadcast_params(self,param,cast=None):\n", + " '''\n", + " aligns the dimensionality of the parameters with the data so it's one-to-one\n", + " '''\n", + " out = []\n", + " if isinstance(param,(int,float,np.integer)) or param is None: #self.X has already been mapped to [self.X]\n", + " out.append([param] * len(self.X))\n", + " if self.Y is not None:\n", + " out.append([param] * len(self.Y))\n", + " elif isinstance(param,(tuple,list,np.ndarray)):\n", + " if self.method == 'self-pairwise' and len(param) >= len(self.X):\n", + " out = [param]\n", + " else:\n", + " assert len(param) <= 2 #only 2 elements max\n", + "\n", + " #if the inner terms are singly valued, we broadcast, otherwise needs to be the same dimensions\n", + " for i,data in enumerate([self.X,self.Y]):\n", + " if data is None:\n", + " continue\n", + " if isinstance(param[i],(int,float)):\n", + " out.append([param[i]] * len(data))\n", + " elif isinstance(param[i],(list,np.ndarray,tuple)):\n", + " assert len(param[i]) >= len(data)\n", + " out.append(param[i][:len(data)])\n", + " else:\n", + " raise ValueError(\"unknown type entered for parameter\")\n", + "\n", + " if cast is not None and param is not None:\n", + " out = [[cast(x) for x in dat] for dat in out]\n", + "\n", + " return out\n", + "\n", + " def fit_dmds(self,\n", + " X=None,\n", + " Y=None,\n", + " n_delays=None,\n", + " delay_interval=None,\n", + " rank=None,\n", + " rank_thresh = None,\n", + " rank_explained_variance=None,\n", + " reduced_rank_reg=None,\n", + " lamb = None,\n", + " device='cpu',\n", + " verbose=False,\n", + " send_to_cpu=True\n", + " ):\n", + " \"\"\"\n", + " Recomputes only the DMDs with a single set of hyperparameters. This will not compare, that will need to be done with the full procedure\n", + " \"\"\"\n", + " X = self.X if X is None else X\n", + " Y = self.Y if Y is None else Y\n", + " n_delays = self.n_delays if n_delays is None else n_delays\n", + " delay_interval = self.delay_interval if delay_interval is None else delay_interval\n", + " rank = self.rank if rank is None else rank\n", + " lamb = self.lamb if lamb is None else lamb\n", + " data = []\n", + " if isinstance(X,list):\n", + " data.append(X)\n", + " else:\n", + " data.append([X])\n", + " if Y is not None:\n", + " if isinstance(Y,list):\n", + " data.append(Y)\n", + " else:\n", + " data.append([Y])\n", + "\n", + " dmds = [[DMD(Xi,n_delays,delay_interval,\n", + " rank,rank_thresh,rank_explained_variance,reduced_rank_reg,\n", + " lamb,device,verbose,send_to_cpu) for Xi in dat] for dat in data]\n", + "\n", + " for dmd_sets in dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return dmds\n", + "\n", + " def fit_score(self):\n", + " \"\"\"\n", + " Standard fitting function for both DMDs and PoVF\n", + "\n", + " Parameters\n", + " __________\n", + "\n", + " Returns\n", + " _______\n", + "\n", + " sims : np.array\n", + " data matrix of the similarity scores between the specific sets of data\n", + " \"\"\"\n", + " for dmd_sets in self.dmds:\n", + " for dmd in dmd_sets:\n", + " dmd.fit()\n", + "\n", + " return self.score()\n", + "\n", + " def score(self,iters=None,lr=None,score_method=None):\n", + " \"\"\"\n", + " Rescore DSA with precomputed dmds if you want to try again\n", + "\n", + " Parameters\n", + " __________\n", + " iters : int or None\n", + " number of optimization steps, if None then resorts to saved self.iters\n", + " lr : float or None\n", + " learning rate, if None then resorts to saved self.lr\n", + " score_method : None or {'angular','euclidean'}\n", + " overwrites the score method in the object for this application\n", + "\n", + " Returns\n", + " ________\n", + " score : float\n", + " similarity score of the two precomputed DMDs\n", + " \"\"\"\n", + "\n", + " iters = self.iters if iters is None else iters\n", + " lr = self.lr if lr is None else lr\n", + " score_method = self.score_method if score_method is None else score_method\n", + "\n", + " ind2 = 1 - int(self.method == 'self-pairwise')\n", + " # 0 if self.pairwise (want to compare the set to itself)\n", + "\n", + " self.sims = np.zeros((len(self.dmds[0]),len(self.dmds[ind2])))\n", + " for i,dmd1 in enumerate(self.dmds[0]):\n", + " for j,dmd2 in enumerate(self.dmds[ind2]):\n", + " if self.method == 'self-pairwise':\n", + " if j >= i:\n", + " continue\n", + " if self.verbose:\n", + " print(f'computing similarity between DMDs {i} and {j}')\n", + "\n", + " self.sims[i,j] = self.simdist.fit_score(dmd1.A_v,dmd2.A_v,iters,lr,score_method,zero_pad=self.zero_pad)\n", + "\n", + " if self.method == 'self-pairwise':\n", + " self.sims[j,i] = self.sims[i,j]\n", + "\n", + "\n", + " if self.method == 'default':\n", + " return self.sims[0,0]\n", + "\n", + " return self.sims" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eced3162", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Helper functions (Bonus Section)\n", + "\n", + "import contextlib\n", + "import io\n", + "import argparse\n", + "# Standard library imports\n", + "from collections import OrderedDict\n", + "import logging\n", + "\n", + "# External libraries: General utilities\n", + "import argparse\n", + "import numpy as np\n", + "\n", + "# PyTorch related imports\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import torch.optim as optim\n", + "from torch.optim.lr_scheduler import StepLR\n", + "from torchvision import datasets, transforms\n", + "from torchvision.models.feature_extraction import create_feature_extractor, get_graph_node_names\n", + "from torchvision.utils import make_grid\n", + "\n", + "# Matplotlib for plotting\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "# SciPy for statistical functions\n", + "from scipy import stats\n", + "\n", + "# Scikit-Learn for machine learning utilities\n", + "from sklearn.decomposition import PCA\n", + "from sklearn import manifold\n", + "\n", + "# RSA toolbox specific imports\n", + "import rsatoolbox\n", + "from rsatoolbox.data import Dataset\n", + "from rsatoolbox.rdm.calc import calc_rdm\n", + "\n", + "class Net(nn.Module):\n", + " \"\"\"\n", + " A neural network model for image classification, consisting of two convolutional layers,\n", + " followed by two fully connected layers with dropout regularization.\n", + "\n", + " Methods:\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self):\n", + " \"\"\"\n", + " Initializes the network layers.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 32 output channels, and a 3x3 kernel.\n", + " - conv2: Second convolutional layer with 32 input channels, 64 output channels, and a 3x3 kernel.\n", + " - dropout1: Dropout layer with a dropout probability of 0.25.\n", + " - dropout2: Dropout layer with a dropout probability of 0.5.\n", + " - fc1: First fully connected layer with 9216 input features and 128 output features.\n", + " - fc2: Second fully connected layer with 128 input features and 10 output features.\n", + " \"\"\"\n", + " super(Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 32, 3, 1)\n", + " self.conv2 = nn.Conv2d(32, 64, 3, 1)\n", + " self.dropout1 = nn.Dropout(0.25)\n", + " self.dropout2 = nn.Dropout(0.5)\n", + " self.fc1 = nn.Linear(9216, 128)\n", + " self.fc2 = nn.Linear(128, 10)\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " x = self.conv1(input)\n", + " x = F.relu(x)\n", + " x = self.conv2(x)\n", + " x = F.relu(x)\n", + " x = F.max_pool2d(x, 2)\n", + " x = self.dropout1(x)\n", + " x = torch.flatten(x, 1)\n", + " x = self.fc1(x)\n", + " x = F.relu(x)\n", + " x = self.dropout2(x)\n", + " x = self.fc2(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "class recurrent_Net(nn.Module):\n", + " \"\"\"\n", + " A recurrent neural network model for image classification, consisting of two convolutional layers\n", + " with recurrent connections and a readout layer.\n", + "\n", + " Methods:\n", + " - __init__(time_steps=5): Initializes the network layers and sets the number of time steps for recurrence.\n", + " - forward(input): Defines the forward pass of the network.\n", + " \"\"\"\n", + "\n", + " def __init__(self, time_steps=5):\n", + " \"\"\"\n", + " Initializes the network layers and sets the number of time steps for recurrence.\n", + "\n", + " Layers:\n", + " - conv1: First convolutional layer with 1 input channel, 16 output channels, and a 3x3 kernel with a stride of 3.\n", + " - conv2: Second convolutional layer with 16 input channels, 16 output channels, and a 3x3 kernel with padding of 1.\n", + " - readout: A sequential layer containing:\n", + " - dropout: Dropout layer with a dropout probability of 0.25.\n", + " - avgpool: Adaptive average pooling layer to reduce spatial dimensions to 1x1.\n", + " - flatten: Flatten layer to convert the 2D pooled output to 1D.\n", + " - linear: Fully connected layer with 16 input features and 10 output features.\n", + " - time_steps (int): Number of time steps for the recurrent connection.\n", + " \"\"\"\n", + " super(recurrent_Net, self).__init__()\n", + " self.conv1 = nn.Conv2d(1, 16, 3, 3)\n", + " self.conv2 = nn.Conv2d(16, 16, 3, 1, padding=1)\n", + " self.readout = nn.Sequential(OrderedDict([\n", + " ('dropout', nn.Dropout(0.25)),\n", + " ('avgpool', nn.AdaptiveAvgPool2d(1)),\n", + " ('flatten', nn.Flatten()),\n", + " ('linear', nn.Linear(16, 10))\n", + " ]))\n", + " self.time_steps = time_steps\n", + "\n", + " def forward(self, input):\n", + " \"\"\"\n", + " Defines the forward pass of the network.\n", + "\n", + " Inputs:\n", + " - input (torch.Tensor): Input tensor of shape (batch_size, 1, height, width).\n", + "\n", + " Outputs:\n", + " - output (torch.Tensor): Output tensor of shape (batch_size, 10) representing the class probabilities for each input sample.\n", + " \"\"\"\n", + " input = self.conv1(input)\n", + " x = input\n", + " for t in range(0, self.time_steps):\n", + " x = input + self.conv2(x)\n", + " x = F.relu(x)\n", + "\n", + " x = self.readout(x)\n", + " output = F.softmax(x, dim=1)\n", + " return output\n", + "\n", + "\n", + "def train_one_epoch(args, model, device, train_loader, optimizer, epoch):\n", + " \"\"\"\n", + " Trains the model for one epoch.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Arguments for training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - device (torch.device): The device to use for training (CPU/GPU).\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + " - epoch (int): The current epoch number.\n", + " \"\"\"\n", + " model.train()\n", + " for batch_idx, (data, target) in enumerate(train_loader):\n", + " data, target = data.to(device), target.to(device)\n", + " optimizer.zero_grad()\n", + " output = model(data)\n", + " output = torch.log(output) # to make it a log_softmax\n", + " loss = F.nll_loss(output, target)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if batch_idx % args.log_interval == 0:\n", + " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", + " epoch, batch_idx * len(data), len(train_loader.dataset),\n", + " 100. * batch_idx / len(train_loader), loss.item()))\n", + " if args.dry_run:\n", + " break\n", + "\n", + "def test(model, device, test_loader, return_features=False):\n", + " \"\"\"\n", + " Evaluates the model on the test dataset.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be evaluated.\n", + " - device (torch.device): The device to use for evaluation (CPU/GPU).\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " - return_features (bool): If True, returns the features from the model. Default is False.\n", + " \"\"\"\n", + " model.eval()\n", + " test_loss = 0\n", + " correct = 0\n", + " with torch.no_grad():\n", + " for data, target in test_loader:\n", + " data, target = data.to(device), target.to(device)\n", + " output = model(data)\n", + " output = torch.log(output)\n", + " test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(target.view_as(pred)).sum().item()\n", + "\n", + " test_loss /= len(test_loader.dataset)\n", + "\n", + " print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n", + " test_loss, correct, len(test_loader.dataset),\n", + " 100. * correct / len(test_loader.dataset)))\n", + "\n", + "def build_args():\n", + " \"\"\"\n", + " Builds and parses command-line arguments for training.\n", + " \"\"\"\n", + " parser = argparse.ArgumentParser(description='PyTorch MNIST Example')\n", + " parser.add_argument('--batch-size', type=int, default=64, metavar='N',\n", + " help='input batch size for training (default: 64)')\n", + " parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',\n", + " help='input batch size for testing (default: 1000)')\n", + " parser.add_argument('--epochs', type=int, default=2, metavar='N',\n", + " help='number of epochs to train (default: 14)')\n", + " parser.add_argument('--lr', type=float, default=1.0, metavar='LR',\n", + " help='learning rate (default: 1.0)')\n", + " parser.add_argument('--gamma', type=float, default=0.7, metavar='M',\n", + " help='Learning rate step gamma (default: 0.7)')\n", + " parser.add_argument('--no-cuda', action='store_true', default=False,\n", + " help='disables CUDA training')\n", + " parser.add_argument('--no-mps', action='store_true', default=False,\n", + " help='disables macOS GPU training')\n", + " parser.add_argument('--dry-run', action='store_true', default=False,\n", + " help='quickly check a single pass')\n", + " parser.add_argument('--seed', type=int, default=1, metavar='S',\n", + " help='random seed (default: 1)')\n", + " parser.add_argument('--log-interval', type=int, default=50, metavar='N',\n", + " help='how many batches to wait before logging training status')\n", + " parser.add_argument('--save-model', action='store_true', default=False,\n", + " help='For Saving the current Model')\n", + " args = parser.parse_args('')\n", + "\n", + " use_cuda = torch.cuda.is_available() #not args.no_cuda and\n", + "\n", + " if use_cuda:\n", + " device = torch.device(\"cuda\")\n", + " else:\n", + " device = torch.device(\"cpu\")\n", + "\n", + " args.use_cuda = use_cuda\n", + " args.device = device\n", + " return args\n", + "\n", + "def fetch_dataloaders(args):\n", + " \"\"\"\n", + " Fetches the data loaders for training and testing datasets.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + "\n", + " Outputs:\n", + " - train_loader (torch.utils.data.DataLoader): DataLoader for the training data.\n", + " - test_loader (torch.utils.data.DataLoader): DataLoader for the test data.\n", + " \"\"\"\n", + " train_kwargs = {'batch_size': args.batch_size}\n", + " test_kwargs = {'batch_size': args.test_batch_size}\n", + " if args.use_cuda:\n", + " cuda_kwargs = {'num_workers': 1,\n", + " 'pin_memory': True,\n", + " 'shuffle': True}\n", + " train_kwargs.update(cuda_kwargs)\n", + " test_kwargs.update(cuda_kwargs)\n", + "\n", + " transform=transforms.Compose([\n", + " transforms.ToTensor(),\n", + " transforms.Normalize((0.1307,), (0.3081,))\n", + " ])\n", + " with contextlib.redirect_stdout(io.StringIO()): #to suppress output\n", + " dataset1 = datasets.MNIST('../data', train=True, download=True,\n", + " transform=transform)\n", + " dataset2 = datasets.MNIST('../data', train=False,\n", + " transform=transform)\n", + " train_loader = torch.utils.data.DataLoader(dataset1, **train_kwargs)\n", + " test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)\n", + " return train_loader, test_loader\n", + "\n", + "def train_model(args, model, optimizer):\n", + " \"\"\"\n", + " Trains the model using the specified arguments and optimizer.\n", + "\n", + " Inputs:\n", + " - args (Namespace): Parsed arguments with training configuration.\n", + " - model (torch.nn.Module): The model to be trained.\n", + " - optimizer (torch.optim.Optimizer): Optimizer for updating the model parameters.\n", + "\n", + " Outputs:\n", + " - None: The function trains the model and optionally saves it.\n", + " \"\"\"\n", + " scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)\n", + " for epoch in range(1, args.epochs + 1):\n", + " train_one_epoch(args, model, args.device, train_loader, optimizer, epoch)\n", + " test(model, args.device, test_loader)\n", + " scheduler.step()\n", + "\n", + " if args.save_model:\n", + " torch.save(model.state_dict(), \"mnist_cnn.pt\")\n", + "\n", + "\n", + "def calc_rdms(model_features, method='correlation'):\n", + " \"\"\"\n", + " Calculates representational dissimilarity matrices (RDMs) for model features.\n", + "\n", + " Inputs:\n", + " - model_features (dict): A dictionary where keys are layer names and values are features of the layers.\n", + " - method (str): The method to calculate RDMs, e.g., 'correlation'. Default is 'correlation'.\n", + "\n", + " Outputs:\n", + " - rdms (pyrsa.rdm.RDMs): RDMs object containing dissimilarity matrices.\n", + " - rdms_dict (dict): A dictionary with layer names as keys and their corresponding RDMs as values.\n", + " \"\"\"\n", + " ds_list = []\n", + " for l in range(len(model_features)):\n", + " layer = list(model_features.keys())[l]\n", + " feats = model_features[layer]\n", + "\n", + " if type(feats) is list:\n", + " feats = feats[-1]\n", + "\n", + " if args.use_cuda:\n", + " feats = feats.cpu()\n", + "\n", + " if len(feats.shape) > 2:\n", + " feats = feats.flatten(1)\n", + "\n", + " feats = feats.detach().numpy()\n", + " ds = Dataset(feats, descriptors=dict(layer=layer))\n", + " ds_list.append(ds)\n", + "\n", + " rdms = calc_rdm(ds_list, method=method)\n", + " rdms_dict = {list(model_features.keys())[i]: rdms.get_matrices()[i] for i in range(len(model_features))}\n", + "\n", + " return rdms, rdms_dict\n", + "\n", + "def fgsm_attack(image, epsilon, data_grad):\n", + " \"\"\"\n", + " Performs FGSM attack on an image.\n", + "\n", + " Inputs:\n", + " - image (torch.Tensor): Original image.\n", + " - epsilon (float): Perturbation magnitude.\n", + " - data_grad (torch.Tensor): Gradient of the data.\n", + "\n", + " Outputs:\n", + " - perturbed_image (torch.Tensor): Perturbed image after FGSM attack.\n", + " \"\"\"\n", + " sign_data_grad = data_grad.sign()\n", + " perturbed_image = image + epsilon * sign_data_grad\n", + " perturbed_image = torch.clamp(perturbed_image, 0, 1)\n", + " return perturbed_image\n", + "\n", + "def denorm(batch, mean=[0.1307], std=[0.3081]):\n", + " \"\"\"\n", + " Converts a batch of normalized tensors to their original scale.\n", + "\n", + " Inputs:\n", + " - batch (torch.Tensor): Batch of normalized tensors.\n", + " - mean (torch.Tensor or list): Mean used for normalization.\n", + " - std (torch.Tensor or list): Standard deviation used for normalization.\n", + "\n", + " Outputs:\n", + " - torch.Tensor: Batch of tensors without normalization applied to them.\n", + " \"\"\"\n", + " if isinstance(mean, list):\n", + " mean = torch.tensor(mean).to(batch.device)\n", + " if isinstance(std, list):\n", + " std = torch.tensor(std).to(batch.device)\n", + "\n", + " return batch * std.view(1, -1, 1, 1) + mean.view(1, -1, 1, 1)\n", + "\n", + "def generate_adversarial(model, imgs, targets, epsilon):\n", + " \"\"\"\n", + " Generates adversarial examples using FGSM attack.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to attack.\n", + " - imgs (torch.Tensor): Batch of images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " - epsilon (float): Perturbation magnitude.\n", + "\n", + " Outputs:\n", + " - adv_imgs (torch.Tensor): Batch of adversarial images.\n", + " \"\"\"\n", + " adv_imgs = []\n", + "\n", + " for img, target in zip(imgs, targets):\n", + " img = img.unsqueeze(0)\n", + " target = target.unsqueeze(0)\n", + " img.requires_grad = True\n", + "\n", + " output = model(img)\n", + " output = torch.log(output)\n", + " loss = F.nll_loss(output, target)\n", + "\n", + " model.zero_grad()\n", + " loss.backward()\n", + "\n", + " data_grad = img.grad.data\n", + " data_denorm = denorm(img)\n", + " perturbed_data = fgsm_attack(data_denorm, epsilon, data_grad)\n", + " perturbed_data_normalized = transforms.Normalize((0.1307,), (0.3081,))(perturbed_data)\n", + "\n", + " adv_imgs.append(perturbed_data_normalized.detach())\n", + "\n", + " return torch.cat(adv_imgs)\n", + "\n", + "def test_adversarial(model, imgs, targets):\n", + " \"\"\"\n", + " Tests the model on adversarial examples and prints the accuracy.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model to be tested.\n", + " - imgs (torch.Tensor): Batch of adversarial images.\n", + " - targets (torch.Tensor): Batch of target labels.\n", + " \"\"\"\n", + " correct = 0\n", + " output = model(imgs)\n", + " output = torch.log(output)\n", + " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n", + " correct += pred.eq(targets.view_as(pred)).sum().item()\n", + "\n", + " final_acc = correct / float(len(imgs))\n", + " print(f\"adversarial test accuracy = {correct} / {len(imgs)} = {final_acc}\")\n", + "\n", + "def extract_features(model, imgs, return_layers, plot='none'):\n", + " \"\"\"\n", + " Extracts features from specified layers of the model.\n", + "\n", + " Inputs:\n", + " - model (torch.nn.Module): The model from which to extract features.\n", + " - imgs (torch.Tensor): Batch of input images.\n", + " - return_layers (list): List of layer names from which to extract features.\n", + " - plot (str): Option to plot the features. Default is 'none'.\n", + "\n", + " Outputs:\n", + " - model_features (dict): A dictionary with layer names as keys and extracted features as values.\n", + " \"\"\"\n", + " if return_layers == 'all':\n", + " return_layers, _ = get_graph_node_names(model)\n", + " elif return_layers == 'layers':\n", + " layers, _ = get_graph_node_names(model)\n", + " return_layers = [l for l in layers if 'input' in l or 'conv' in l or 'fc' in l]\n", + "\n", + " feature_extractor = create_feature_extractor(model, return_nodes=return_layers)\n", + " model_features = feature_extractor(imgs)\n", + "\n", + " return model_features" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be4a4946", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Plotting functions (Bonus)\n", + "\n", + "def sample_images(data_loader, n=5, plot=False):\n", + " \"\"\"\n", + " Samples a specified number of images from a data loader.\n", + "\n", + " Inputs:\n", + " - data_loader (torch.utils.data.DataLoader): Data loader containing images and labels.\n", + " - n (int): Number of images to sample per class.\n", + " - plot (bool): Whether to plot the sampled images using matplotlib.\n", + "\n", + " Outputs:\n", + " - imgs (torch.Tensor): Sampled images.\n", + " - labels (torch.Tensor): Corresponding labels for the sampled images.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " imgs, targets = next(iter(data_loader))\n", + "\n", + " imgs_o = []\n", + " labels = []\n", + " for value in range(10):\n", + " cat_imgs = imgs[np.where(targets == value)][0:n]\n", + " imgs_o.append(cat_imgs)\n", + " labels.append([value]*len(cat_imgs))\n", + "\n", + " imgs = torch.cat(imgs_o, dim=0)\n", + " labels = torch.tensor(labels).flatten()\n", + "\n", + " if plot:\n", + " plt.imshow(torch.moveaxis(make_grid(imgs, nrow=5, padding=0, normalize=False, pad_value=0), 0,-1))\n", + " plt.axis('off')\n", + "\n", + " return imgs, labels\n", + "\n", + "\n", + "def plot_rdms(model_rdms):\n", + " \"\"\"\n", + " Plots the Representational Dissimilarity Matrices (RDMs) for each layer of a model.\n", + "\n", + " Inputs:\n", + " - model_rdms (dict): A dictionary where keys are layer names and values are the corresponding RDMs.\n", + " \"\"\"\n", + "\n", + " with plt.xkcd():\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, len(model_rdms))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " for l in range(len(model_rdms)):\n", + "\n", + " layer = list(model_rdms.keys())[l]\n", + " rdm = np.squeeze(model_rdms[layer])\n", + "\n", + " if len(rdm.shape) < 2:\n", + " rdm = rdm.reshape( (int(np.sqrt(rdm.shape[0])), int(np.sqrt(rdm.shape[0]))) )\n", + "\n", + " rdm = rdm / np.max(rdm)\n", + "\n", + " ax = plt.subplot(gs[0,l])\n", + " ax_ = ax.imshow(rdm, cmap='magma_r')\n", + " ax.set_title(f'{layer}')\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " cbar_ax.text(-2.3, 0.05, 'Normalized euclidean distance', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax)\n", + "\n", + " plt.show()\n", + "\n", + "def rep_path(model_features, model_colors, labels=None, rdm_calc_method='euclidean', rdm_comp_method='cosine'):\n", + " \"\"\"\n", + " Represents paths of model features in a reduced-dimensional space.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each model.\n", + " - model_colors (dict): Dictionary mapping model names to colors for visualization.\n", + " - labels (array-like, optional): Array of labels corresponding to the model features.\n", + " - rdm_calc_method (str, optional): Method for calculating RDMS ('euclidean' or 'correlation').\n", + " - rdm_comp_method (str, optional): Method for comparing RDMS ('cosine' or 'corr').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + " path_len = []\n", + " path_colors = []\n", + " rdms_list = []\n", + " ax_ticks = []\n", + " tick_colors = []\n", + " model_names = list(model_features.keys())\n", + " for m in range(len(model_names)):\n", + " model_name = model_names[m]\n", + " features = model_features[model_name]\n", + " path_colors.append(model_colors[model_name])\n", + " path_len.append(len(features))\n", + " ax_ticks.append(list(features.keys()))\n", + " tick_colors.append([model_colors[model_name]]*len(features))\n", + " rdms, _ = calc_rdms(features, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + "\n", + " path_len = np.insert(np.cumsum(path_len),0,0)\n", + "\n", + " if labels is not None:\n", + " rdms, _ = calc_rdms({'labels' : F.one_hot(labels).float().to(device)}, method=rdm_calc_method)\n", + " rdms_list.append(rdms)\n", + " ax_ticks.append(['labels'])\n", + " tick_colors.append(['m'])\n", + " idx_labels = -1\n", + "\n", + " rdms = rsatoolbox.rdm.concat(rdms_list)\n", + "\n", + " #Flatten the list\n", + " ax_ticks = [l for model_layers in ax_ticks for l in model_layers]\n", + " tick_colors = [l for model_layers in tick_colors for l in model_layers]\n", + " tick_colors = ['k' if tick == 'input' else color for tick, color in zip(ax_ticks, tick_colors)]\n", + "\n", + " rdms_comp = rsatoolbox.rdm.compare(rdms, rdms, method=rdm_comp_method)\n", + " if rdm_comp_method == 'cosine':\n", + " rdms_comp = np.arccos(rdms_comp)\n", + " rdms_comp = np.nan_to_num(rdms_comp, nan=0.0)\n", + "\n", + " # Symmetrize\n", + " rdms_comp = (rdms_comp + rdms_comp.T) / 2.0\n", + "\n", + " # reduce dim to 2\n", + " transformer = manifold.MDS(n_components = 2, max_iter=1000, n_init=10, normalized_stress='auto', dissimilarity=\"precomputed\")\n", + " dims= transformer.fit_transform(rdms_comp)\n", + "\n", + " # remove duplicates of the input layer from multiple models\n", + " remove_duplicates = np.where(np.array(ax_ticks) == 'input')[0][1:]\n", + " for index in remove_duplicates:\n", + " del ax_ticks[index]\n", + " del tick_colors[index]\n", + " rdms_comp = np.delete(np.delete(rdms_comp, index, axis=0), index, axis=1)\n", + "\n", + " fig = plt.figure(figsize=(8, 4))\n", + " gs = fig.add_gridspec(1, 2)\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " ax = plt.subplot(gs[0,0])\n", + " ax_ = ax.imshow(rdms_comp, cmap='viridis_r')\n", + " fig.subplots_adjust(left=0.2)\n", + " cbar_ax = fig.add_axes([-0.01, 0.2, 0.01, 0.5])\n", + " #cbar_ax.text(-7, 0.05, 'dissimilarity between rdms', size=10, rotation=90)\n", + " fig.colorbar(ax_, cax=cbar_ax,location='left')\n", + " ax.set_title('Dissimilarity between layer rdms', fontdict = {'fontsize': 14})\n", + " ax.set_xticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7, rotation=83)\n", + " ax.set_yticks(np.arange(len(ax_ticks)), labels=ax_ticks, fontsize=7)\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.xaxis.get_ticklabels())]\n", + " [t.set_color(i) for (i,t) in zip(tick_colors, ax.yaxis.get_ticklabels())]\n", + "\n", + " ax = plt.subplot(gs[0,1])\n", + " amin, amax = dims.min(), dims.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + "\n", + " for i in range(len(rdms_list)-1):\n", + "\n", + " path_indices = np.arange(path_len[i], path_len[i+1])\n", + " ax.plot(dims[path_indices, 0], dims[path_indices, 1], color=path_colors[i], marker='.')\n", + " ax.set_title('Representational geometry path', fontdict = {'fontsize': 14})\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.set_xlabel(f\"dim 1\")\n", + " ax.set_ylabel(f\"dim 2\")\n", + "\n", + " # if idx_input is not None:\n", + " idx_input = 0\n", + " ax.plot(dims[idx_input, 0], dims[idx_input, 1], color='k', marker='s')\n", + "\n", + " if labels is not None:\n", + " ax.plot(dims[idx_labels, 0], dims[idx_labels, 1], color='m', marker='*')\n", + "\n", + " ax.legend(model_names, fontsize=8)\n", + " fig.tight_layout()\n", + "\n", + "def plot_dim_reduction(model_features, labels, transformer_funcs):\n", + " \"\"\"\n", + " Plots the dimensionality reduction results for model features using various transformers.\n", + "\n", + " Inputs:\n", + " - model_features (dict): Dictionary containing model features for each layer.\n", + " - labels (array-like): Array of labels corresponding to the model features.\n", + " - transformer_funcs (list): List of dimensionality reduction techniques to apply ('PCA', 'MDS', 't-SNE').\n", + " \"\"\"\n", + " with plt.xkcd():\n", + "\n", + " transformers = []\n", + " for t in transformer_funcs:\n", + " if t == 'PCA': transformers.append(PCA(n_components=2))\n", + " if t == 'MDS': transformers.append(manifold.MDS(n_components = 2, normalized_stress='auto'))\n", + " if t == 't-SNE': transformers.append(manifold.TSNE(n_components = 2, perplexity=40, verbose=0))\n", + "\n", + " fig = plt.figure(figsize=(8, 2.5*len(transformers)))\n", + " # and we add one plot per reference point\n", + " gs = fig.add_gridspec(len(transformers), len(model_features))\n", + " fig.subplots_adjust(wspace=0.2, hspace=0.2)\n", + "\n", + " return_layers = list(model_features.keys())\n", + "\n", + " for f in range(len(transformer_funcs)):\n", + "\n", + " for l in range(len(return_layers)):\n", + " layer = return_layers[l]\n", + " feats = model_features[layer].detach().cpu().flatten(1)\n", + " feats_transformed= transformers[f].fit_transform(feats)\n", + "\n", + " amin, amax = feats_transformed.min(), feats_transformed.max()\n", + " amin, amax = (amin + amax) / 2 - (amax - amin) * 5/8, (amin + amax) / 2 + (amax - amin) * 5/8\n", + " ax = plt.subplot(gs[f,l])\n", + " ax.set_xlim([amin, amax])\n", + " ax.set_ylim([amin, amax])\n", + " ax.axis(\"off\")\n", + " #ax.set_title(f'{layer}')\n", + " if f == 0: ax.text(0.5, 1.12, f'{layer}', size=16, ha=\"center\", transform=ax.transAxes)\n", + " if l == 0: ax.text(-0.3, 0.5, transformer_funcs[f], size=16, ha=\"center\", transform=ax.transAxes)\n", + " # Create a discrete color map based on unique labels\n", + " num_colors = len(np.unique(labels))\n", + " cmap = plt.get_cmap('viridis_r', num_colors) # 10 discrete colors\n", + " norm = mpl.colors.BoundaryNorm(np.arange(-0.5,num_colors), cmap.N)\n", + " ax_ = ax.scatter(feats_transformed[:, 0], feats_transformed[:, 1], c=labels, cmap=cmap, norm=norm)\n", + "\n", + " fig.subplots_adjust(right=0.9)\n", + " cbar_ax = fig.add_axes([1.01, 0.18, 0.01, 0.53])\n", + " fig.colorbar(ax_, cax=cbar_ax, ticks=np.linspace(0,9,10))\n", + " plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21f68945", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Data retrieval\n", + "\n", + "import os\n", + "import requests\n", + "import hashlib\n", + "\n", + "# Variables for file and download URL\n", + "fnames = [\"standard_model.pth\", \"adversarial_model.pth\", \"recurrent_model.pth\"] # The names of the files to be downloaded\n", + "urls = [\"https://osf.io/s5rt6/download\", \"https://osf.io/qv5eb/download\", \"https://osf.io/6hnwk/download\"] # URLs from where the files will be downloaded\n", + "expected_md5s = [\"2e63c2cd77bc9f1fa67673d956ec910d\", \"25fb34497377921b54368317f68a7aa7\", \"ee5cea3baa264cb78300102fa6ed66e8\"] # MD5 hashes for verifying files integrity\n", + "\n", + "for fname, url, expected_md5 in zip(fnames, urls, expected_md5s):\n", + " if not os.path.isfile(fname):\n", + " try:\n", + " # Attempt to download the file\n", + " r = requests.get(url) # Make a GET request to the specified URL\n", + " except requests.ConnectionError:\n", + " # Handle connection errors during the download\n", + " print(\"!!! Failed to download data !!!\")\n", + " else:\n", + " # No connection errors, proceed to check the response\n", + " if r.status_code != requests.codes.ok:\n", + " # Check if the HTTP response status code indicates a successful download\n", + " print(\"!!! Failed to download data !!!\")\n", + " elif hashlib.md5(r.content).hexdigest() != expected_md5:\n", + " # Verify the integrity of the downloaded file using MD5 checksum\n", + " print(\"!!! Data download appears corrupted !!!\")\n", + " else:\n", + " # If download is successful and data is not corrupted, save the file\n", + " with open(fname, \"wb\") as fid:\n", + " fid.write(r.content) # Write the downloaded content to a file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "93aeca0a", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Figure settings\n", + "\n", + "logging.getLogger('matplotlib.font_manager').disabled = True\n", + "\n", + "%matplotlib inline\n", + "%config InlineBackend.figure_format = 'retina' # perfrom high definition rendering for images and plots\n", + "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/main/nma.mplstyle\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dd8052d5", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Set device (GPU or CPU)\n", + "\n", + "# inform the user if the notebook uses GPU or CPU.\n", + "\n", + "def set_device():\n", + " \"\"\"\n", + " Determines and sets the computational device for PyTorch operations based on the availability of a CUDA-capable GPU.\n", + "\n", + " Outputs:\n", + " - device (str): The device that PyTorch will use for computations ('cuda' or 'cpu'). This string can be directly used\n", + " in PyTorch operations to specify the device.\n", + " \"\"\"\n", + "\n", + " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + " if device != \"cuda\":\n", + " print(\"GPU is not enabled in this notebook. \\n\"\n", + " \"If you want to enable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `GPU` from the dropdown menu\")\n", + " else:\n", + " print(\"GPU is enabled in this notebook. \\n\"\n", + " \"If you want to disable it, in the menu under `Runtime` -> \\n\"\n", + " \"`Hardware accelerator.` and select `None` from the dropdown menu\")\n", + "\n", + " return device\n", + "\n", + "device = set_device()" ] }, { @@ -75,84 +2155,532 @@ "execution_count": null, "id": "c28a92e7-e76c-48de-b574-15a1272717cf", "metadata": { - "cellView": "form", + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Load Slides\n", + "\n", + "from IPython.display import IFrame\n", + "from ipywidgets import widgets\n", + "out = widgets.Output()\n", + "\n", + "link_id = \"8fx23\"\n", + "\n", + "with out:\n", + " print(f\"If you want to download the slides: https://osf.io/download/{link_id}/\")\n", + " display(IFrame(src=f\"https://mfr.ca-1.osf.io/render?url=https://osf.io/{link_id}/?direct%26mode=render%26action=download%26mode=render\", width=730, height=410))\n", + "display(out)" + ] + }, + { + "cell_type": "markdown", + "id": "407ace26", + "metadata": { + "execution": {} + }, + "source": [ + "---\n", + "\n", + "# Intro\n", + "\n", + "Welcome to Tutorial 5 of Day 3 (W1D3) of the NeuroAI course. In this tutorial we are going to look at an exciting method that measures similarity from a slightly different perspective, a temporal one. The prior methods we have looked at were centeed around geometry and spatial representations, where we looked at metrics such as the Euclidean and Mahalanobis distance metrics. However, one thing we often want to study in neuroscience and in AI separately - is the temporal domain. Even more so in our own field of NeuroAI, we often deal with time series of neuronal / biological recordings. One thing you should already have a broad level of awareness of is that end structures can end up looking the same even though the paths taken to arrive at those end structures were very different.\n", + "\n", + "In NeuroAI, we're often confronted with systems that seem to have some sort of overlap and we want to study whether this implies there is a shared computation pairs up with the shared task (we looked at this in detail yesterday in our *Comparing Tasks* day). Today, we will begin by watching a short intro video by Mitchell Ostrow, who will describe his method to compare representations over temporal sequences (the method is called Dynamic Similarity Analysis). Then we are going to introduce three simple dynamical systems and we will explore them from the perspective of Dynamic Similarity Analysis and also describe the conceptual relationship to Representational Similarity Analysis. You will have a short coding exercise on the topic of temporal similarity analysis on three different types of trajectories. \n", + "\n", + "At the end of the tutorial, we will finally look at a further aspect of temporal sequences using RNNs. This is an adaptation of the ideas introduced in Tutorial 2 but now based around recurrent representations from RNNs. We hope you enjoy this tutorial today and that it gets you thinking not just what similarity values mean, but which ones are appropriate (here, from a spatial or temporal perspective). We aim to continually expand the tools necessary in the NeuroAI researcher's toolkit. Complementary tools, when applicable, can often tell a far richer story than just using a single method." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b5d6178f-ddf5-41ae-b676-15e452dc8b78", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Video 1: Dynamical Similarity Analysis\n", + "\n", + "from ipywidgets import widgets\n", + "from IPython.display import YouTubeVideo\n", + "from IPython.display import IFrame\n", + "from IPython.display import display\n", + "\n", + "class PlayVideo(IFrame):\n", + " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", + " self.id = id\n", + " if source == 'Bilibili':\n", + " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", + " elif source == 'Osf':\n", + " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", + " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "\n", + "def display_videos(video_ids, W=400, H=300, fs=1):\n", + " tab_contents = []\n", + " for i, video_id in enumerate(video_ids):\n", + " out = widgets.Output()\n", + " with out:\n", + " if video_ids[i][0] == 'Youtube':\n", + " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", + " height=H, fs=fs, rel=0)\n", + " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", + " else:\n", + " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", + " height=H, fs=fs, autoplay=False)\n", + " if video_ids[i][0] == 'Bilibili':\n", + " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", + " elif video_ids[i][0] == 'Osf':\n", + " print(f'Video available at https://osf.io/{video.id}')\n", + " display(video)\n", + " tab_contents.append(out)\n", + " return tab_contents\n", + "\n", + "video_ids = [('Youtube', 'FHikIsQFQvM'), ('Bilibili', 'BV1qm421g7hV')]\n", + "tab_contents = display_videos(video_ids, W=854, H=480)\n", + "tabs = widgets.Tab()\n", + "tabs.children = tab_contents\n", + "for i in range(len(tab_contents)):\n", + " tabs.set_title(i, video_ids[i][0])\n", + "display(tabs)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d2ce83bc-7e86-44d3-a40a-4ad46fd5a6df", + "metadata": { + "cellView": "form", + "execution": {} + }, + "outputs": [], + "source": [ + "# @title Submit your feedback\n", + "content_review(f\"{feedback_prefix}_DSA_video\")" + ] + }, + { + "cell_type": "markdown", + "id": "937041e9", + "metadata": { + "execution": {} + }, + "source": [ + "## Section 1: Visualization of Three Temporal Sequences\n", + "\n", + "We are going to be working with the analysis of three temporal sequences today:\n", + "\n", + "* The circular time series (`Circle`)\n", + "* The oval time series (`Oval`)\n", + "* The random walk (`R-Walk`)\n", + "\n", + "The random walk is going to be broadly *oval shaped*. Now, what do you think, from a geometric perspective, might result from a spatial analysis of these three different *representations*? You will probably assume because the random walk has an oval shape and there is also an oval time series (that's not a random walk) that these would result in a higher spatial similarity. You'd be right to assume this. However, what we're going to do with the `Circle` and `Oval` time series is to include an oscillator at a specific frequency, shared amongst these two time series. In effect, this means that although when plotted in totality the shapes are different, during the dynamic (temporal) evolution of these time series, a very similar shared pattern is emerging. We want methods that are sensitive to these changes to give higher scores for time series sharing similar temporal patterns (e.g. both containing oscillating patterns at similar frequences) rather than just a random walk that resembles (geometrically) one of the other shapes (`R-Walk`). Before we continue, we'll just define this random walk in a little more detail. A random walk at a specific location / timepoint takes a random step of fixed length in a specific direction, but this can be broadly controlled to resemble geometric shapes. We've taken a random walk and then reframed it to be similar in shape to `Oval`. \n", + "\n", + "Let's now visualize these three temporal sequences, to make the previous paragraph a little clearer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b57dfe1a", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "# Circle\n", + "r = .1; # rotation\n", + "A = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_circle = trajectory\n", + "\n", + "# Oval\n", + "r = .1; # rotation\n", + "s = 4; # scaling\n", + "S = np.array([[1, 0], [0, s]])\n", + "Si = np.array([[1, 0], [0, 1/s]])\n", + "V = np.array([[1, 1], [-1, 1]])/np.sqrt(2)\n", + "Vi = np.array([[1, -1], [1, 1]])/np.sqrt(2)\n", + "R = np.array([[np.cos(r), np.sin(r)], [-np.sin(r), np.cos(r)]])\n", + "A = np.linalg.multi_dot([V,Si,R,S,Vi])\n", + "B = np.array([[1, 0], [0, 1]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_oval = trajectory\n", + "\n", + "# R-Walk (random walk)\n", + "r = .1; # rotation\n", + "A = np.array([[.9, 0], [0, .9]])\n", + "c = -.95; # correlation coefficient\n", + "B = np.array([[1, c], [0, np.sqrt(1-c*c)]])\n", + "\n", + "trajectory = generate_2d_random_process(A, B)\n", + "trajectory_walk = trajectory" + ] + }, + { + "cell_type": "markdown", + "id": "113a0dee", + "metadata": { + "execution": {} + }, + "source": [ + "Can you see how the spatial / geometric similarity of `R-Walk` and `Oval` are more similar, but the oscillations during the temporal sequence are shared between `Circle` and `Oval`? Let's run Dynamic Similarity Analysis on these temporal sequences and see what scores are returned.\n", + "\n", + "We calcularted `trajectory_oval` and `trajectory_circle` above, so let's plug these into the `DSA` function imported earlier (in the helper function cell) and see what the similarity score is." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3e36d59", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "# Define the DSA computation class\n", + "dsa = DSA(X=trajectory_oval, Y=trajectory_circle, n_delays=1)\n", + "\n", + "# Call the fit method and save the result\n", + "similarities_oval_circle = dsa.fit_score()\n", + "\n", + "print(f\"DSA similarity between Oval and Circle: {similarities_oval_circle:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "9f1fb622", + "metadata": { + "execution": {} + }, + "source": [ + "## Multi-way Comparison\n", + "\n", + "We're now going to run DSA on our three trajectories and fit the model, returning the scores which we can investigate by plotting a confusion matrix with a heatmap to show the DSA scores." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6ee9e8e8", + "metadata": { "execution": {} }, "outputs": [], "source": [ - "# @title Bonus material slides\n", + "n_delays = 1\n", + "delay_interval = 1\n", "\n", - "from IPython.display import IFrame\n", - "from ipywidgets import widgets\n", - "out = widgets.Output()\n", + "models = [trajectory_circle, trajectory_oval, trajectory_walk]\n", + "dsa = DSA(models, n_delays=n_delays, delay_interval=delay_interval)\n", + "similarities = dsa.fit_score()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18318ddb", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "labels = ['Circle', 'Oval', 'Walk']\n", + "data = np.random.rand(len(labels), len(labels))\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "ffd49b4b", + "metadata": { + "execution": {} + }, + "source": [ + "This heatmap across the three model comparisons shows that the DSA scores between (`Walk` and `Circle`) and (`Walk` and `Oval`) to be (relatively) high, while the comparison between (`Circle` and `Oval`) is very low. Please note that this confusion matrix is symmetrical, meaning that the analysis between `trajectory_A` and `trajectory_B` returns the same dynamic similarity score as `trajectory_B` and `trajectory_A`. This is a common feature we have also seen in comparison metrics in standard RSA. One thing to note in the calculation of DSA is that comparisons among identical trajectories is `0`. This is unlike in RSA where we expect the correlation among the same stimuli to be `1.0`. This is why we see black squares along the diagonal.\n", "\n", - "link_id = \"8fx23\"\n", + "Let's put our thinking caps on for a moment: This isn't really the result we would have expected, right? What do you think might be going on here? Have a look back at the *hyperparameters* and try to make an educated guess!" + ] + }, + { + "cell_type": "markdown", + "id": "d0ff5faa", + "metadata": { + "execution": {} + }, + "source": [ + "## DSA Hyperparameters (`n_delays` and `delay_interval`)\n", "\n", - "with out:\n", - " print(f\"If you want to download the slides: https://osf.io/download/{link_id}/\")\n", - " display(IFrame(src=f\"https://mfr.ca-1.osf.io/render?url=https://osf.io/{link_id}/?direct%26mode=render%26action=download%26mode=render\", width=730, height=410))\n", - "display(out)" + "We'll now give you a hint as to why the setting of these hyperparameters is important when considering dynamic similarity analysis. The oscillators we have placed in the trajectories of `Circle` and `Oval` are not immediately apparent if you study only the previous time step for each element. It's only when considering the recurring pattern across a few different temporal delays and at what delay interval you want those to be, that we would expect to be able to detect recurring oscillations that provide us with the information we need to conclude that `Oval` and `Circle` are actually *dynamically* similar.\n", + "\n", + "You should change the values below to be more sensible hyperparameter settings and re-run the model and plot the new confusion matrix. Try using `n_delays` equal to `20` and `delay_interval` equal to `10`. Don't forget to define `models` (see above example if you get stuck)." ] }, { "cell_type": "code", "execution_count": null, - "id": "b5d6178f-ddf5-41ae-b676-15e452dc8b78", + "id": "9d8d4c03", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "#################################################\n", + "## TODO for students: fill in the missing parts ##\n", + "raise NotImplementedError(\"Student exercise\")\n", + "#################################################\n", + "\n", + "n_delays = ...\n", + "delay_interval = ...\n", + "\n", + "models = ...\n", + "dsa = DSA(...)\n", + "similarities = ...\n", + "\n", + "labels = ['Circle', 'Oval', 'Walk']\n", + "ax = sns.heatmap(similarities, xticklabels=labels, yticklabels=labels)\n", + "cbar = ax.collections[0].colorbar\n", + "cbar.ax.set_ylabel('DSA Score');\n", + "plt.title(\"Dynamic Similarity Analysis Score among Trajectories\");" + ] + }, + { + "cell_type": "markdown", + "id": "a6377c65", + "metadata": { + "colab_type": "text", + "execution": {} + }, + "source": [ + "[*Click for solution*](https://github.com/neuromatch/NeuroAI_Course/tree/main/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/solutions/W1D3_Tutorial5_Solution_0467919d.py)\n", + "\n", + "*Example output:*\n", + "\n", + "Solution hint\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "04b0e32f", + "metadata": { + "execution": {} + }, + "source": [ + "What do you see now? We now see a much more sensible result. The DSA scores have now correctly identified that `Oval` and `Circle` are very dynamically similar! They have the highest color score according to the colorbar on the side. As is always good practice in science, let's have a look inside the `similarities` variable to look at the exact values and confirm what we see in the figure above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "55fa4065", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "similarities" + ] + }, + { + "cell_type": "markdown", + "id": "59cb799f", + "metadata": { + "execution": {} + }, + "source": [ + "## Comparison with RSA\n", + "\n", + "At the start of this exercise, we saw three different trajectories and pointed out that the random walk and oval shapes were most similar from a geometric perspective, both ellipse-like but not similar in their dynamic similarity. To better show the difference between DSA and RSA, we encourage you to run another comparison where we consider each time step to be a pair in the X,Y space and we will look at the the similarity between of `Oval` with both `Circle` and `Walk`. If our understanding is correct, then RSA should indicate a higher geometric similarity between (`Oval` and `Walk`) than with (`Oval` and `Circle`)." + ] + }, + { + "cell_type": "markdown", + "id": "87cf4e6e", + "metadata": { + "execution": {} + }, + "source": [ + "---\n", + "# (Bonus) Representational Geometry of Recurrent Models\n", + "\n", + "Transformations of representations can occur across space and time, e.g., layers of a neural network and steps of recurrent computation. We've looked at the temporal dimension today and earlier today in the other tutorials we looked mainly at spatial representations.\n", + "\n", + "Just as the layers in a feedforward DNN can change the representational geometry to perform a task, steps in a recurrent network can reuse the same layer to reach the same computational depth.\n", + "\n", + "In this section, we look at a very simple recurrent network with only 2650 trainable parameters." + ] + }, + { + "cell_type": "markdown", + "id": "3d613edd", + "metadata": { + "execution": {} + }, + "source": [ + "Here is a diagram of this network:\n", + "\n", + "![Recurrent convolutional neural network](https://github.com/neuromatch/NeuroAI_Course/blob/main/tutorials/W1D3_ComparingArtificialAndBiologicalNetworks/static/rcnn_tutorial.png?raw=true)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2f0443d3", "metadata": { "cellView": "form", "execution": {} }, "outputs": [], "source": [ - "# @title Video 1: Dynamical Similarity Analysis\n", + "# @title Grab a recurrent model\n", "\n", - "from ipywidgets import widgets\n", - "from IPython.display import YouTubeVideo\n", - "from IPython.display import IFrame\n", - "from IPython.display import display\n", + "args = build_args()\n", + "train_loader, test_loader = fetch_dataloaders(args)\n", + "path = \"recurrent_model.pth\"\n", + "model_recurrent = torch.load(path, map_location=args.device, weights_only=False)" + ] + }, + { + "cell_type": "markdown", + "id": "d463c3a9", + "metadata": { + "execution": {} + }, + "source": [ + "
We can first look at the computational steps in this network. As we see below, the `conv2` operation is repeated for 5 times." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6bfabacd", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "train_nodes, _ = get_graph_node_names(model_recurrent)\n", + "print('The computational steps in the network are: \\n', train_nodes)" + ] + }, + { + "cell_type": "markdown", + "id": "1d410c3a", + "metadata": { + "execution": {} + }, + "source": [ + "Plotting the RDMs after each application of the `conv2` operation shows the same progressive emergence of the blockwise structure around the diagonal, mediating the correct classification in this task." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "30249608", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=20)\n", + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", "\n", - "class PlayVideo(IFrame):\n", - " def __init__(self, id, source, page=1, width=400, height=300, **kwargs):\n", - " self.id = id\n", - " if source == 'Bilibili':\n", - " src = f'https://player.bilibili.com/player.html?bvid={id}&page={page}'\n", - " elif source == 'Osf':\n", - " src = f'https://mfr.ca-1.osf.io/render?url=https://osf.io/download/{id}/?direct%26mode=render'\n", - " super(PlayVideo, self).__init__(src, width, height, **kwargs)\n", + "rdms, rdms_dict = calc_rdms(model_features)\n", + "plot_rdms(rdms_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "248329c3", + "metadata": { + "execution": {} + }, + "source": [ + "We can also look at how the different dimensionality reduction techniques capture the dynamics of changing geometry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b0e2cdf", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "return_layers = ['conv2', 'conv2_1', 'conv2_2', 'conv2_3', 'conv2_4']\n", "\n", - "def display_videos(video_ids, W=400, H=300, fs=1):\n", - " tab_contents = []\n", - " for i, video_id in enumerate(video_ids):\n", - " out = widgets.Output()\n", - " with out:\n", - " if video_ids[i][0] == 'Youtube':\n", - " video = YouTubeVideo(id=video_ids[i][1], width=W,\n", - " height=H, fs=fs, rel=0)\n", - " print(f'Video available at https://youtube.com/watch?v={video.id}')\n", - " else:\n", - " video = PlayVideo(id=video_ids[i][1], source=video_ids[i][0], width=W,\n", - " height=H, fs=fs, autoplay=False)\n", - " if video_ids[i][0] == 'Bilibili':\n", - " print(f'Video available at https://www.bilibili.com/video/{video.id}')\n", - " elif video_ids[i][0] == 'Osf':\n", - " print(f'Video available at https://osf.io/{video.id}')\n", - " display(video)\n", - " tab_contents.append(out)\n", - " return tab_contents\n", + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features = extract_features(model_recurrent, imgs.to(device), return_layers)\n", "\n", - "video_ids = [('Youtube', 'FHikIsQFQvM'), ('Bilibili', 'BV1qm421g7hV')]\n", - "tab_contents = display_videos(video_ids, W=854, H=480)\n", - "tabs = widgets.Tab()\n", - "tabs.children = tab_contents\n", - "for i in range(len(tab_contents)):\n", - " tabs.set_title(i, video_ids[i][0])\n", - "display(tabs)" + "plot_dim_reduction(model_features, labels, transformer_funcs =['PCA', 'MDS', 't-SNE'])" + ] + }, + { + "cell_type": "markdown", + "id": "1aaf5f4a", + "metadata": { + "execution": {} + }, + "source": [ + "## Representational geometry paths for recurrent models\n", + "\n", + "We can look at the model's recurrent computational steps as a path in the representational geometry space." ] }, { "cell_type": "code", "execution_count": null, - "id": "d2ce83bc-7e86-44d3-a40a-4ad46fd5a6df", + "id": "7f88274a", + "metadata": { + "execution": {} + }, + "outputs": [], + "source": [ + "imgs, labels = sample_images(test_loader, n=50) #grab 500 samples from the test set\n", + "model_features_recurrent = extract_features(model_recurrent, imgs.to(device), return_layers='all')\n", + "\n", + "#rdms, rdms_dict = calc_rdms(model_features)\n", + "features = {'recurrent model': model_features_recurrent}\n", + "model_colors = {'recurrent model': 'y'}\n", + "\n", + "rep_path(features, model_colors, labels)" + ] + }, + { + "cell_type": "markdown", + "id": "5c3fbd44", + "metadata": { + "execution": {} + }, + "source": [ + "We can also look at the paths taken by the feedforward and the recurrent models and compare them." + ] + }, + { + "cell_type": "markdown", + "id": "b25a8cc6", + "metadata": { + "execution": {} + }, + "source": [ + "If you recall back to Tutorial 2, we compared a standard feedward model's representations. We can extend our analysis of the analysis of the recurrent model's representations by making a side-by-side comparison. We can also look at the paths taken by the feedforward and the recurrent models and compare them. What we see above in the case of the recurrent model is the fast-shifting path through the geometric space from inputs to labels. This illustration serves to show that models take many different paths and can have very diverse underlying mechanisms but still arrive at a superficially similar output at the end of training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c904e840", "metadata": { "cellView": "form", "execution": {} @@ -160,7 +2688,19 @@ "outputs": [], "source": [ "# @title Submit your feedback\n", - "content_review(f\"{feedback_prefix}_DSA_video\")" + "content_review(f\"{feedback_prefix}_recurrent_models\")" + ] + }, + { + "cell_type": "markdown", + "id": "3ed56061", + "metadata": { + "execution": {} + }, + "source": [ + "# The Big Picture\n", + "\n", + "Today, you've looked at what it means to measure representations from different systems. These systems can be of the same type (multiple brain systems, multiple artificial models) as well as with representations between these systems. In NeuroAI, we're especially interested in such comparisons, comparing representational systems in deep learning networks, for instance, to brain recordings recorded while those biological systems experienced / perceived the same set of stimuli. Comparisons can be geometric / spatial or they can be temporal. Today, we looked at Dynamic Similarity Analysis, a method used to be able to capture the dependencies among trajectories, not just capturing the similarity of the full temporal sequence upon completion of the temporal sequence. It's often important to take into account multiple dimensions of representational similarity. A combination of tools is definitely required in the NeuroAI researcher's toolkit. We hope you have many chances to use these tools in your future work as NeuroAI researchers." ] } ], @@ -191,7 +2731,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.19" + "version": "3.9.22" } }, "nbformat": 4,