diff --git a/f22/andy_lec_notes/lec35_Dec02_Pandas3/lec_35_pandas3_data_transformation_template.ipynb b/f22/andy_lec_notes/lec35_Dec02_Pandas3/lec_35_pandas3_data_transformation_template.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..5dca29134b6cd63dcad697303c9044ff6d79bc2b
--- /dev/null
+++ b/f22/andy_lec_notes/lec35_Dec02_Pandas3/lec_35_pandas3_data_transformation_template.ipynb
@@ -0,0 +1,1210 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "CeWtFirwteFY"
+ },
+ "outputs": [],
+ "source": [
+ "# known import statements\n",
+ "import pandas as pd\n",
+ "import sqlite3 as sql # note that we are renaming to sql\n",
+ "import os\n",
+ "\n",
+ "# new import statement\n",
+ "import numpy as np"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "RHvDCo4fhXBx"
+ },
+ "source": [
+ "# Lecture 35 Pandas 3: Data Transformation\n",
+ "* Data transformation is the process of changing the format, structure, or values of data. \n",
+ "* Often needed during data cleaning and sometimes during data analysis\n",
+ "\n",
+ "Possible data transformation: \n",
+ "* Parsing/Extraction\n",
+ " * Parse CSV to Pandas DataFrame\n",
+ "* Missing Value Manipulations\n",
+ " * Dropping\n",
+ " * Imputation: replace missing value with substitute values\n",
+ "* Typecasting, Formating, Renaming\n",
+ " * Typecasting: covert one column from int to float \n",
+ " * Formating: format the time column to datatime object \n",
+ " * Renaming: rename column and index names \n",
+ "* Applying/Mapping \n",
+ "* Filtering, Aggregation, Grouping, and Summarization\n",
+ " * Covered in Pandas 1 & 2 lectures"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yoLGptrqhbBo"
+ },
+ "source": [
+ "# Today's Learning Objectives: \n",
+ "\n",
+ "* Identify, drop, or fill missing values with Pandas .isna, .dropna, and .fillna\n",
+ "* Apply a function to Pandas Series and DataFrame rows/columns \n",
+ "* Replace all target values to Pandas Series and DataFrame rows/columns\n",
+ "* Filter, Aggregate, Group, and Summarize information in a DataFrame with .groupby\n",
+ "* Convert .groupby examples to SQL "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FgnTeNRIswsm"
+ },
+ "source": [
+ "# The dataset: Spotify songs\n",
+ "Adapted from https://www.kaggle.com/datasets/mrmorj/dataset-of-songs-in-spotify.\n",
+ "\n",
+ "If you are interested in digging deeper in this dataset, here's a [blog post](https://medium.com/@boplantinga/what-do-spotifys-audio-features-tell-us-about-this-year-s-eurovision-song-contest-66ad188e112a) that explain each column in details. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### WARMUP 1: Establish a connection to the spotify.db database"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 232
+ },
+ "id": "8y9scvgCnTHl",
+ "outputId": "c72388f8-576c-4cf2-ef51-352cd11b6c92"
+ },
+ "outputs": [],
+ "source": [
+ "# open up the spotify database\n",
+ "db_pathname = \"spotify.db\"\n",
+ "assert os.path.exists(db_pathname)\n",
+ "conn = sql.connect(db_pathname)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def qry(sql):\n",
+ " return pd.read_sql(sql, conn)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### WARMUP 2: Identify the table name(s) inside the database"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 112
+ },
+ "id": "ybTqbDSOnR2f",
+ "outputId": "8dcc943b-9382-4abb-ef78-6c6d56ad89eb"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "<div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>type</th>\n",
+ " <th>name</th>\n",
+ " <th>tbl_name</th>\n",
+ " <th>rootpage</th>\n",
+ " <th>sql</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>table</td>\n",
+ " <td>spotify</td>\n",
+ " <td>spotify</td>\n",
+ " <td>1527</td>\n",
+ " <td>CREATE TABLE spotify(\\nid TEXT PRIMARY KEY,\\nt...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>index</td>\n",
+ " <td>sqlite_autoindex_spotify_1</td>\n",
+ " <td>spotify</td>\n",
+ " <td>1528</td>\n",
+ " <td>None</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "</div>"
+ ],
+ "text/plain": [
+ " type name tbl_name rootpage \\\n",
+ "0 table spotify spotify 1527 \n",
+ "1 index sqlite_autoindex_spotify_1 spotify 1528 \n",
+ "\n",
+ " sql \n",
+ "0 CREATE TABLE spotify(\\nid TEXT PRIMARY KEY,\\nt... \n",
+ "1 None "
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "qry(\"select * from sqlite_master\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### WARMUP 3: Use pandas lookup expression to identify the column names and the types: use .iloc"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CREATE TABLE spotify(\n",
+ "id TEXT PRIMARY KEY,\n",
+ "title BLOB,\n",
+ "song_name BLOB, \n",
+ "genre TEXT,\n",
+ "duration_ms INTEGER, \n",
+ "key INTEGER, \n",
+ "mode INTEGER, \n",
+ "time_signature INTEGER, \n",
+ "tempo REAL,\n",
+ "acousticness REAL, \n",
+ "danceability REAL, \n",
+ "energy REAL, \n",
+ "instrumentalness REAL, \n",
+ "liveness REAL, \n",
+ "loudness REAL, \n",
+ "speechiness REAL, \n",
+ "valence REAL)\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(qry(\"select * from sqlite_master\")[\"sql\"].iloc[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### WARMUP 4: Store the data inside `spotify` table inside a variable called `df`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 632
+ },
+ "id": "txAH9OIjnoQv",
+ "outputId": "ac9152ba-32df-4fb2-d4e0-a97f50fe58fb"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "<div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>id</th>\n",
+ " <th>title</th>\n",
+ " <th>song_name</th>\n",
+ " <th>genre</th>\n",
+ " <th>duration_ms</th>\n",
+ " <th>key</th>\n",
+ " <th>mode</th>\n",
+ " <th>time_signature</th>\n",
+ " <th>tempo</th>\n",
+ " <th>acousticness</th>\n",
+ " <th>danceability</th>\n",
+ " <th>energy</th>\n",
+ " <th>instrumentalness</th>\n",
+ " <th>liveness</th>\n",
+ " <th>loudness</th>\n",
+ " <th>speechiness</th>\n",
+ " <th>valence</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>7pgJBLVz5VmnL7uGHmRj6p</td>\n",
+ " <td></td>\n",
+ " <td>Pathology</td>\n",
+ " <td>Dark Trap</td>\n",
+ " <td>224427</td>\n",
+ " <td>8</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>115.080</td>\n",
+ " <td>0.401000</td>\n",
+ " <td>0.719</td>\n",
+ " <td>0.493</td>\n",
+ " <td>0.000000</td>\n",
+ " <td>0.1180</td>\n",
+ " <td>-7.230</td>\n",
+ " <td>0.0794</td>\n",
+ " <td>0.1240</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>0vSWgAlfpye0WCGeNmuNhy</td>\n",
+ " <td></td>\n",
+ " <td>Symbiote</td>\n",
+ " <td>Dark Trap</td>\n",
+ " <td>98821</td>\n",
+ " <td>5</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>218.050</td>\n",
+ " <td>0.013800</td>\n",
+ " <td>0.850</td>\n",
+ " <td>0.893</td>\n",
+ " <td>0.000004</td>\n",
+ " <td>0.3720</td>\n",
+ " <td>-4.783</td>\n",
+ " <td>0.0623</td>\n",
+ " <td>0.0391</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>7EL7ifncK2PWFYThJjzR25</td>\n",
+ " <td></td>\n",
+ " <td>BRAINFOOD</td>\n",
+ " <td>Dark Trap</td>\n",
+ " <td>101172</td>\n",
+ " <td>8</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>189.938</td>\n",
+ " <td>0.187000</td>\n",
+ " <td>0.864</td>\n",
+ " <td>0.365</td>\n",
+ " <td>0.000000</td>\n",
+ " <td>0.1160</td>\n",
+ " <td>-10.219</td>\n",
+ " <td>0.0655</td>\n",
+ " <td>0.0478</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>1umsRbM7L4ju7rn9aU8Ju6</td>\n",
+ " <td></td>\n",
+ " <td>Sacrifice</td>\n",
+ " <td>Dark Trap</td>\n",
+ " <td>96062</td>\n",
+ " <td>10</td>\n",
+ " <td>0</td>\n",
+ " <td>4</td>\n",
+ " <td>139.990</td>\n",
+ " <td>0.145000</td>\n",
+ " <td>0.767</td>\n",
+ " <td>0.576</td>\n",
+ " <td>0.000003</td>\n",
+ " <td>0.0968</td>\n",
+ " <td>-9.683</td>\n",
+ " <td>0.2560</td>\n",
+ " <td>0.1870</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>4SKqOHKYU5pgHr5UiVKiQN</td>\n",
+ " <td></td>\n",
+ " <td>Backpack</td>\n",
+ " <td>Dark Trap</td>\n",
+ " <td>135079</td>\n",
+ " <td>5</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>128.014</td>\n",
+ " <td>0.007700</td>\n",
+ " <td>0.765</td>\n",
+ " <td>0.726</td>\n",
+ " <td>0.000000</td>\n",
+ " <td>0.6190</td>\n",
+ " <td>-5.580</td>\n",
+ " <td>0.1910</td>\n",
+ " <td>0.2700</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>35872</th>\n",
+ " <td>46bXU7Sgj7104ZoXxzz9tM</td>\n",
+ " <td>Euphoric Hardstyle</td>\n",
+ " <td></td>\n",
+ " <td>hardstyle</td>\n",
+ " <td>269208</td>\n",
+ " <td>4</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>150.013</td>\n",
+ " <td>0.031500</td>\n",
+ " <td>0.528</td>\n",
+ " <td>0.693</td>\n",
+ " <td>0.000345</td>\n",
+ " <td>0.1210</td>\n",
+ " <td>-5.148</td>\n",
+ " <td>0.0304</td>\n",
+ " <td>0.3940</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>35873</th>\n",
+ " <td>0he2ViGMUO3ajKTxLOfWVT</td>\n",
+ " <td>Greatest Hardstyle Playlist</td>\n",
+ " <td></td>\n",
+ " <td>hardstyle</td>\n",
+ " <td>210112</td>\n",
+ " <td>0</td>\n",
+ " <td>0</td>\n",
+ " <td>4</td>\n",
+ " <td>149.928</td>\n",
+ " <td>0.022500</td>\n",
+ " <td>0.517</td>\n",
+ " <td>0.768</td>\n",
+ " <td>0.000018</td>\n",
+ " <td>0.2050</td>\n",
+ " <td>-7.922</td>\n",
+ " <td>0.0479</td>\n",
+ " <td>0.3830</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>35874</th>\n",
+ " <td>72DAt9Lbpy9EUS29OzQLob</td>\n",
+ " <td>Best of Hardstyle 2020</td>\n",
+ " <td></td>\n",
+ " <td>hardstyle</td>\n",
+ " <td>234823</td>\n",
+ " <td>8</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>154.935</td>\n",
+ " <td>0.026000</td>\n",
+ " <td>0.361</td>\n",
+ " <td>0.821</td>\n",
+ " <td>0.000242</td>\n",
+ " <td>0.3850</td>\n",
+ " <td>-3.102</td>\n",
+ " <td>0.0505</td>\n",
+ " <td>0.1240</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>35875</th>\n",
+ " <td>6HXgExFVuE1c3cq9QjFCcU</td>\n",
+ " <td>Euphoric Hardstyle</td>\n",
+ " <td></td>\n",
+ " <td>hardstyle</td>\n",
+ " <td>323200</td>\n",
+ " <td>6</td>\n",
+ " <td>0</td>\n",
+ " <td>4</td>\n",
+ " <td>150.042</td>\n",
+ " <td>0.000551</td>\n",
+ " <td>0.477</td>\n",
+ " <td>0.921</td>\n",
+ " <td>0.029600</td>\n",
+ " <td>0.0575</td>\n",
+ " <td>-4.777</td>\n",
+ " <td>0.0392</td>\n",
+ " <td>0.4880</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>35876</th>\n",
+ " <td>6MAAMZImxcvYhRnxDLTufD</td>\n",
+ " <td>Best of Hardstyle 2020</td>\n",
+ " <td></td>\n",
+ " <td>hardstyle</td>\n",
+ " <td>162161</td>\n",
+ " <td>9</td>\n",
+ " <td>1</td>\n",
+ " <td>4</td>\n",
+ " <td>155.047</td>\n",
+ " <td>0.001890</td>\n",
+ " <td>0.529</td>\n",
+ " <td>0.945</td>\n",
+ " <td>0.000055</td>\n",
+ " <td>0.4140</td>\n",
+ " <td>-5.862</td>\n",
+ " <td>0.0615</td>\n",
+ " <td>0.1340</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>35877 rows × 17 columns</p>\n",
+ "</div>"
+ ],
+ "text/plain": [
+ " id title song_name \\\n",
+ "0 7pgJBLVz5VmnL7uGHmRj6p Pathology \n",
+ "1 0vSWgAlfpye0WCGeNmuNhy Symbiote \n",
+ "2 7EL7ifncK2PWFYThJjzR25 BRAINFOOD \n",
+ "3 1umsRbM7L4ju7rn9aU8Ju6 Sacrifice \n",
+ "4 4SKqOHKYU5pgHr5UiVKiQN Backpack \n",
+ "... ... ... ... \n",
+ "35872 46bXU7Sgj7104ZoXxzz9tM Euphoric Hardstyle \n",
+ "35873 0he2ViGMUO3ajKTxLOfWVT Greatest Hardstyle Playlist \n",
+ "35874 72DAt9Lbpy9EUS29OzQLob Best of Hardstyle 2020 \n",
+ "35875 6HXgExFVuE1c3cq9QjFCcU Euphoric Hardstyle \n",
+ "35876 6MAAMZImxcvYhRnxDLTufD Best of Hardstyle 2020 \n",
+ "\n",
+ " genre duration_ms key mode time_signature tempo \\\n",
+ "0 Dark Trap 224427 8 1 4 115.080 \n",
+ "1 Dark Trap 98821 5 1 4 218.050 \n",
+ "2 Dark Trap 101172 8 1 4 189.938 \n",
+ "3 Dark Trap 96062 10 0 4 139.990 \n",
+ "4 Dark Trap 135079 5 1 4 128.014 \n",
+ "... ... ... ... ... ... ... \n",
+ "35872 hardstyle 269208 4 1 4 150.013 \n",
+ "35873 hardstyle 210112 0 0 4 149.928 \n",
+ "35874 hardstyle 234823 8 1 4 154.935 \n",
+ "35875 hardstyle 323200 6 0 4 150.042 \n",
+ "35876 hardstyle 162161 9 1 4 155.047 \n",
+ "\n",
+ " acousticness danceability energy instrumentalness liveness \\\n",
+ "0 0.401000 0.719 0.493 0.000000 0.1180 \n",
+ "1 0.013800 0.850 0.893 0.000004 0.3720 \n",
+ "2 0.187000 0.864 0.365 0.000000 0.1160 \n",
+ "3 0.145000 0.767 0.576 0.000003 0.0968 \n",
+ "4 0.007700 0.765 0.726 0.000000 0.6190 \n",
+ "... ... ... ... ... ... \n",
+ "35872 0.031500 0.528 0.693 0.000345 0.1210 \n",
+ "35873 0.022500 0.517 0.768 0.000018 0.2050 \n",
+ "35874 0.026000 0.361 0.821 0.000242 0.3850 \n",
+ "35875 0.000551 0.477 0.921 0.029600 0.0575 \n",
+ "35876 0.001890 0.529 0.945 0.000055 0.4140 \n",
+ "\n",
+ " loudness speechiness valence \n",
+ "0 -7.230 0.0794 0.1240 \n",
+ "1 -4.783 0.0623 0.0391 \n",
+ "2 -10.219 0.0655 0.0478 \n",
+ "3 -9.683 0.2560 0.1870 \n",
+ "4 -5.580 0.1910 0.2700 \n",
+ "... ... ... ... \n",
+ "35872 -5.148 0.0304 0.3940 \n",
+ "35873 -7.922 0.0479 0.3830 \n",
+ "35874 -3.102 0.0505 0.1240 \n",
+ "35875 -4.777 0.0392 0.4880 \n",
+ "35876 -5.862 0.0615 0.1340 \n",
+ "\n",
+ "[35877 rows x 17 columns]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "df = qry(\"select * from spotify\")\n",
+ "df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Setting a column as row indices for the `DataFrame`\n",
+ "\n",
+ "- Syntax: `df.set_index(\"<COLUMN>\")`\n",
+ "- Returns a new DataFrame object instance reference.\n",
+ "- WARNING: executing this twice will result in `KeyError` being thrown. Once you set a column as row index, it will no longer be a column within the `DataFrame`. If you tried this, go back and execute the above cell and update `df` once more and then execute the below cell exactly once."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set the id column as row indices\n",
+ "df = \n",
+ "df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Not a Number\n",
+ "\n",
+ "- `np.NaN` is the floating point representation of Not a Number\n",
+ "- You do not need to know / learn the details about the `numpy` package \n",
+ "\n",
+ "### Replacing / modifying values within the `DataFrame`\n",
+ "\n",
+ "Syntax: `df.replace(<TARGET>, <REPLACE>)`\n",
+ "- Your target can be `str`, `int`, `float`, `None` (there are other possiblities, but those are too advanced for this course)\n",
+ "- Returns a new DataFrame object instance reference.\n",
+ "\n",
+ "Let's now replace the missing values (empty strings) with `np.NAN`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df = \n",
+ "df.head(10) # title is the album name"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Checking for missing values\n",
+ "\n",
+ "Syntax: `Series.isna()`\n",
+ "- Returns a boolean Series\n",
+ "\n",
+ "Let's check if any of the \"song_name\"(s) are missing"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "JqzSwG5PEZRq",
+ "outputId": "05529a3d-4a5c-4654-fe05-d04b2c10ae6c"
+ },
+ "outputs": [],
+ "source": [
+ "df[\"song_name\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Review: `Pandas.Series.value_counts()`\n",
+ "- Returns a new `Series` with unique values from the original `Series` as keys and the count of those unique values as values. \n",
+ "- Return value `Series` is ordered using descending order of counts"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "uCLDr8EIGMeJ",
+ "outputId": "241d6181-d525-4019-a8f2-689939b2ab33"
+ },
+ "outputs": [],
+ "source": [
+ "# count the number of missing values for song name\n",
+ "df[\"song_name\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Missing value manipulation\n",
+ "Syntax: `df.fillna(<REPLACE>)`\n",
+ "- Returns a new DataFrame object instance reference."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "pJ2CIqq9HWvN",
+ "outputId": "2895e862-18e5-4742-9750-31b130aae668"
+ },
+ "outputs": [],
+ "source": [
+ "# use .fillna to replace missing values\n",
+ "df[\"song_name\"]\n",
+ "\n",
+ "# to replace the original DataFrame's column, you need to explicitly update that object instance\n",
+ "# TODO: uncomment the below lines and update the code\n",
+ "#df[\"song_name\"] = ???\n",
+ "#df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Dropping missing values\n",
+ "Syntax: `df.dropna()`\n",
+ "- Returns a new DataFrame object instance reference."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 145
+ },
+ "id": "O_1ZeHG8N-rB",
+ "outputId": "3b112da2-2b3c-4fb8-c7ae-dc2f2127856d"
+ },
+ "outputs": [],
+ "source": [
+ "# .dropna will drop all rows that contain NaN in them\n",
+ "df.dropna()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ggttXEqUbI_E"
+ },
+ "source": [
+ "### Review: `Pandas.Series.apply(...)`\n",
+ "Syntax: `Series.apply(<FUNCTION OBJECT REFERENCE>)`\n",
+ "- applies input function to every element of the Series.\n",
+ "- Returns a new `Series` object instance reference.\n",
+ "\n",
+ "Let's apply transformation function to `mode` column `Series`:\n",
+ "- mode = 1 means major modality (sounds happy)\n",
+ "- mode = 0 means minor modality (sounds sad)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def replace_mode(m): \n",
+ " if m == 1: \n",
+ " return \"major\"\n",
+ " else: \n",
+ " return \"minor\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[\"mode\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `lambda` recap\n",
+ "\n",
+ "Let's write a `lambda` function instead of the `replace_mode` function"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "9AJ3p-_TarnN",
+ "outputId": "a087df5d-2002-417c-e99c-5e6fc8ea9809"
+ },
+ "outputs": [],
+ "source": [
+ "df[\"mode\"].apply(???)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Typically transformed columns are added as new columns within the DataFrame.\n",
+ "Let's add a new `modified_mode` column."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[\"modified_mode\"] = df[\"mode\"].apply(lambda m: \"major\" if m == 1 else \"minor\")\n",
+ "df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Let's go back to the original table from the SQL database"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZoiyUleiyhMg"
+ },
+ "outputs": [],
+ "source": [
+ "df = qry(\"SELECT * FROM spotify\")\n",
+ "df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Extract just the \"genre\" and \"duration_ms\" columns from `df`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[???]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Pandas.DataFrame.groupby(...)`\n",
+ "\n",
+ "Syntax: `DataFrame.groupby(<COLUMN>)`\n",
+ "- Returns a `groupby` object instance reference\n",
+ "- Need to apply aggregation methods to use the return value of `groupby`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 551
+ },
+ "id": "trRMgGMysdkb",
+ "outputId": "d02098c3-7722-4505-c599-5897bb8ace19"
+ },
+ "outputs": [],
+ "source": [
+ "df[[\"genre\", \"duration_ms\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### What is the average duration for each genre ordered based on decreasing order of averages?\n",
+ "#### v1: using `df` (`pandas`) to answer the question"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[[\"genre\", \"duration_ms\"]]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[[\"genre\", \"duration_ms\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "One way to check whether `groupby` works would be to use `value_counts` on the same column `Series`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "df[\"genre\"].value_counts()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### What is the average duration for each genre ordered based on decreasing order of averages?\n",
+ "#### v2: using SQL query to answer the question"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 551
+ },
+ "id": "89hMTXCKxWG8",
+ "outputId": "5737da11-aa8a-4ed0-9b05-cd379b28904b"
+ },
+ "outputs": [],
+ "source": [
+ "# SQL equivalent query of the above Pandas query\n",
+ "avg_duration_per_genre = qry(\"\"\"\n",
+ "\n",
+ "\"\"\")\n",
+ "\n",
+ "# How can we get make the SQL query output to be exactly same as df.groupby?\n",
+ "avg_duration_per_genre = avg_duration_per_genre.set_index(\"genre\")\n",
+ "avg_duration_per_genre"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "12ZdqYoIy_8U"
+ },
+ "source": [
+ "### What is the average speechiness for each mode, time signature pair?\n",
+ "#### v1: pandas"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 332
+ },
+ "id": "fVejD2KPyveX",
+ "outputId": "fe5c8fda-29a2-4f1a-8ff4-de9ad2a3cde0"
+ },
+ "outputs": [],
+ "source": [
+ "# use a list to indicate all the columns you want to groupby \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 300
+ },
+ "id": "ImYEuOMox-ps",
+ "outputId": "2674dabd-3ff7-4099-fdc3-54e5ba0e2628"
+ },
+ "outputs": [],
+ "source": [
+ "# SQL equivalent query of the above Pandas query\n",
+ "qry(\"\"\"\n",
+ "\n",
+ "\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sEDc5zGu0bc9"
+ },
+ "source": [
+ "### Self-practice"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Which songs have a tempo greater than 150 and what are their genre?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v1: pandas\n",
+ "fast_songs = "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v2: SQL\n",
+ "\n",
+ "qry(\"\"\"\n",
+ "\n",
+ "\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### What is the sum of danceability and liveness for \"Hiphop\" genre songs?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v1: pandas\n",
+ "hiphop_songs = "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v2: SQL\n",
+ "hiphop_songs = qry(\"\"\"\n",
+ "\n",
+ "\"\"\")\n",
+ "hiphop_songs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Find all song_name ordered by ascending order of duration_ms. Eliminate songs which don't have a song_name"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v1: pandas\n",
+ "songs_by_duration = "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v2\n",
+ "songs_by_duration = qry(\"\"\"\n",
+ "\n",
+ "\"\"\")\n",
+ "songs_by_duration"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### How many distinct \"genre\"s are there in the dataset?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v1: pandas\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v2: SQL\n",
+ "genres = qry(\"\"\"\n",
+ "\n",
+ "\"\"\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Considering only songs with energy greater than 0.5, what is the maximum energy for each \"genre\" with song count greater than 2000?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "genre_groups = "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v1: pandas\n",
+ "high_energy_songs = ???\n",
+ "genre_groups = ???\n",
+ "max_energy = ???\n",
+ "max_energy[\"energy\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "genre_counts = ???\n",
+ "genre_counts[\"energy_max\"] = max_energy[\"energy\"]\n",
+ "filtered_genre_counts = ???\n",
+ "filtered_genre_counts"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# v2: SQL\n",
+ "qry(\"\"\"\n",
+ "\n",
+ "\"\"\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}