"import sqlite3 as sql # note that we are renaming to sql\n",
"import os\n",
"\n",
"# new import statement\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RHvDCo4fhXBx"
},
"source": [
"# Lecture 35 Pandas 3: Data Transformation\n",
"* Data transformation is the process of changing the format, structure, or values of data. \n",
"* Often needed during data cleaning and sometimes during data analysis\n",
"\n",
"Possible data transformation: \n",
"* Parsing/Extraction\n",
" * Parse CSV to Pandas DataFrame\n",
"* Missing Value Manipulations\n",
" * Dropping\n",
" * Imputation: replace missing value with substitute values\n",
"* Typecasting, Formating, Renaming\n",
" * Typecasting: covert one column from int to float \n",
" * Formating: format the time column to datatime object \n",
" * Renaming: rename column and index names \n",
"* Applying/Mapping \n",
"* Filtering, Aggregation, Grouping, and Summarization\n",
" * Covered in Pandas 1 & 2 lectures"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yoLGptrqhbBo"
},
"source": [
"# Today's Learning Objectives: \n",
"\n",
"* Identify, drop, or fill missing values with Pandas .isna, .dropna, and .fillna\n",
"* Apply a function to Pandas Series and DataFrame rows/columns \n",
"* Replace all target values to Pandas Series and DataFrame rows/columns\n",
"* Filter, Aggregate, Group, and Summarize information in a DataFrame with .groupby\n",
"* Convert .groupby examples to SQL "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FgnTeNRIswsm"
},
"source": [
"# The dataset: Spotify songs\n",
"Adapted from https://www.kaggle.com/datasets/mrmorj/dataset-of-songs-in-spotify.\n",
"\n",
"If you are interested in digging deeper in this dataset, here's a [blog post](https://medium.com/@boplantinga/what-do-spotifys-audio-features-tell-us-about-this-year-s-eurovision-song-contest-66ad188e112a) that explain each column in details. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### WARMUP 1: Establish a connection to the spotify.db database"
"35876 6MAAMZImxcvYhRnxDLTufD Best of Hardstyle 2020 \n",
"\n",
" genre duration_ms key mode time_signature tempo \\\n",
"0 Dark Trap 224427 8 1 4 115.080 \n",
"1 Dark Trap 98821 5 1 4 218.050 \n",
"2 Dark Trap 101172 8 1 4 189.938 \n",
"3 Dark Trap 96062 10 0 4 139.990 \n",
"4 Dark Trap 135079 5 1 4 128.014 \n",
"... ... ... ... ... ... ... \n",
"35872 hardstyle 269208 4 1 4 150.013 \n",
"35873 hardstyle 210112 0 0 4 149.928 \n",
"35874 hardstyle 234823 8 1 4 154.935 \n",
"35875 hardstyle 323200 6 0 4 150.042 \n",
"35876 hardstyle 162161 9 1 4 155.047 \n",
"\n",
" acousticness danceability energy instrumentalness liveness \\\n",
"0 0.401000 0.719 0.493 0.000000 0.1180 \n",
"1 0.013800 0.850 0.893 0.000004 0.3720 \n",
"2 0.187000 0.864 0.365 0.000000 0.1160 \n",
"3 0.145000 0.767 0.576 0.000003 0.0968 \n",
"4 0.007700 0.765 0.726 0.000000 0.6190 \n",
"... ... ... ... ... ... \n",
"35872 0.031500 0.528 0.693 0.000345 0.1210 \n",
"35873 0.022500 0.517 0.768 0.000018 0.2050 \n",
"35874 0.026000 0.361 0.821 0.000242 0.3850 \n",
"35875 0.000551 0.477 0.921 0.029600 0.0575 \n",
"35876 0.001890 0.529 0.945 0.000055 0.4140 \n",
"\n",
" loudness speechiness valence \n",
"0 -7.230 0.0794 0.1240 \n",
"1 -4.783 0.0623 0.0391 \n",
"2 -10.219 0.0655 0.0478 \n",
"3 -9.683 0.2560 0.1870 \n",
"4 -5.580 0.1910 0.2700 \n",
"... ... ... ... \n",
"35872 -5.148 0.0304 0.3940 \n",
"35873 -7.922 0.0479 0.3830 \n",
"35874 -3.102 0.0505 0.1240 \n",
"35875 -4.777 0.0392 0.4880 \n",
"35876 -5.862 0.0615 0.1340 \n",
"\n",
"[35877 rows x 17 columns]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = qry(\"select * from spotify\")\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Setting a column as row indices for the `DataFrame`\n",
"\n",
"- Syntax: `df.set_index(\"<COLUMN>\")`\n",
"- Returns a new DataFrame object instance reference.\n",
"- WARNING: executing this twice will result in `KeyError` being thrown. Once you set a column as row index, it will no longer be a column within the `DataFrame`. If you tried this, go back and execute the above cell and update `df` once more and then execute the below cell exactly once."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set the id column as row indices\n",
"df = \n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Not a Number\n",
"\n",
"- `np.NaN` is the floating point representation of Not a Number\n",
"- You do not need to know / learn the details about the `numpy` package \n",
"\n",
"### Replacing / modifying values within the `DataFrame`\n",
"\n",
"Syntax: `df.replace(<TARGET>, <REPLACE>)`\n",
"- Your target can be `str`, `int`, `float`, `None` (there are other possiblities, but those are too advanced for this course)\n",
"- Returns a new DataFrame object instance reference.\n",
"\n",
"Let's now replace the missing values (empty strings) with `np.NAN`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = \n",
"df.head(10) # title is the album name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Checking for missing values\n",
"\n",
"Syntax: `Series.isna()`\n",
"- Returns a boolean Series\n",
"\n",
"Let's check if any of the \"song_name\"(s) are missing"
importsqlite3assql# note that we are renaming to sql
importos
# new import statement
importnumpyasnp
```
%% Cell type:markdown id: tags:
# Lecture 35 Pandas 3: Data Transformation
* Data transformation is the process of changing the format, structure, or values of data.
* Often needed during data cleaning and sometimes during data analysis
Possible data transformation:
* Parsing/Extraction
* Parse CSV to Pandas DataFrame
* Missing Value Manipulations
* Dropping
* Imputation: replace missing value with substitute values
* Typecasting, Formating, Renaming
* Typecasting: covert one column from int to float
* Formating: format the time column to datatime object
* Renaming: rename column and index names
* Applying/Mapping
* Filtering, Aggregation, Grouping, and Summarization
* Covered in Pandas 1 & 2 lectures
%% Cell type:markdown id: tags:
# Today's Learning Objectives:
* Identify, drop, or fill missing values with Pandas .isna, .dropna, and .fillna
* Apply a function to Pandas Series and DataFrame rows/columns
* Replace all target values to Pandas Series and DataFrame rows/columns
* Filter, Aggregate, Group, and Summarize information in a DataFrame with .groupby
* Convert .groupby examples to SQL
%% Cell type:markdown id: tags:
# The dataset: Spotify songs
Adapted from https://www.kaggle.com/datasets/mrmorj/dataset-of-songs-in-spotify.
If you are interested in digging deeper in this dataset, here's a [blog post](https://medium.com/@boplantinga/what-do-spotifys-audio-features-tell-us-about-this-year-s-eurovision-song-contest-66ad188e112a) that explain each column in details.
%% Cell type:markdown id: tags:
### WARMUP 1: Establish a connection to the spotify.db database
%% Cell type:code id: tags:
``` python
# open up the spotify database
db_pathname="spotify.db"
assertos.path.exists(db_pathname)
conn=sql.connect(db_pathname)
```
%% Cell type:code id: tags:
``` python
defqry(sql):
returnpd.read_sql(sql,conn)
```
%% Cell type:markdown id: tags:
### WARMUP 2: Identify the table name(s) inside the database
%% Cell type:code id: tags:
``` python
qry("select * from sqlite_master")
```
%% Output
type name tbl_name rootpage \
0 table spotify spotify 1527
1 index sqlite_autoindex_spotify_1 spotify 1528
sql
0 CREATE TABLE spotify(\nid TEXT PRIMARY KEY,\nt...
1 None
%% Cell type:markdown id: tags:
### WARMUP 3: Use pandas lookup expression to identify the column names and the types: use .iloc
%% Cell type:code id: tags:
``` python
print(qry("select * from sqlite_master")["sql"].iloc[0])
```
%% Output
CREATE TABLE spotify(
id TEXT PRIMARY KEY,
title BLOB,
song_name BLOB,
genre TEXT,
duration_ms INTEGER,
key INTEGER,
mode INTEGER,
time_signature INTEGER,
tempo REAL,
acousticness REAL,
danceability REAL,
energy REAL,
instrumentalness REAL,
liveness REAL,
loudness REAL,
speechiness REAL,
valence REAL)
%% Cell type:markdown id: tags:
### WARMUP 4: Store the data inside `spotify` table inside a variable called `df`
35874 72DAt9Lbpy9EUS29OzQLob Best of Hardstyle 2020
35875 6HXgExFVuE1c3cq9QjFCcU Euphoric Hardstyle
35876 6MAAMZImxcvYhRnxDLTufD Best of Hardstyle 2020
genre duration_ms key mode time_signature tempo \
0 Dark Trap 224427 8 1 4 115.080
1 Dark Trap 98821 5 1 4 218.050
2 Dark Trap 101172 8 1 4 189.938
3 Dark Trap 96062 10 0 4 139.990
4 Dark Trap 135079 5 1 4 128.014
... ... ... ... ... ... ...
35872 hardstyle 269208 4 1 4 150.013
35873 hardstyle 210112 0 0 4 149.928
35874 hardstyle 234823 8 1 4 154.935
35875 hardstyle 323200 6 0 4 150.042
35876 hardstyle 162161 9 1 4 155.047
acousticness danceability energy instrumentalness liveness \
0 0.401000 0.719 0.493 0.000000 0.1180
1 0.013800 0.850 0.893 0.000004 0.3720
2 0.187000 0.864 0.365 0.000000 0.1160
3 0.145000 0.767 0.576 0.000003 0.0968
4 0.007700 0.765 0.726 0.000000 0.6190
... ... ... ... ... ...
35872 0.031500 0.528 0.693 0.000345 0.1210
35873 0.022500 0.517 0.768 0.000018 0.2050
35874 0.026000 0.361 0.821 0.000242 0.3850
35875 0.000551 0.477 0.921 0.029600 0.0575
35876 0.001890 0.529 0.945 0.000055 0.4140
loudness speechiness valence
0 -7.230 0.0794 0.1240
1 -4.783 0.0623 0.0391
2 -10.219 0.0655 0.0478
3 -9.683 0.2560 0.1870
4 -5.580 0.1910 0.2700
... ... ... ...
35872 -5.148 0.0304 0.3940
35873 -7.922 0.0479 0.3830
35874 -3.102 0.0505 0.1240
35875 -4.777 0.0392 0.4880
35876 -5.862 0.0615 0.1340
[35877 rows x 17 columns]
%% Cell type:markdown id: tags:
### Setting a column as row indices for the `DataFrame`
- Syntax: `df.set_index("<COLUMN>")`
- Returns a new DataFrame object instance reference.
- WARNING: executing this twice will result in `KeyError` being thrown. Once you set a column as row index, it will no longer be a column within the `DataFrame`. If you tried this, go back and execute the above cell and update `df` once more and then execute the below cell exactly once.
%% Cell type:code id: tags:
``` python
# Set the id column as row indices
df=
df
```
%% Cell type:markdown id: tags:
### Not a Number
-`np.NaN` is the floating point representation of Not a Number
- You do not need to know / learn the details about the `numpy` package
### Replacing / modifying values within the `DataFrame`
Syntax: `df.replace(<TARGET>, <REPLACE>)`
- Your target can be `str`, `int`, `float`, `None` (there are other possiblities, but those are too advanced for this course)
- Returns a new DataFrame object instance reference.
Let's now replace the missing values (empty strings) with `np.NAN`
%% Cell type:code id: tags:
``` python
df=
df.head(10)# title is the album name
```
%% Cell type:markdown id: tags:
### Checking for missing values
Syntax: `Series.isna()`
- Returns a boolean Series
Let's check if any of the "song_name"(s) are missing
%% Cell type:code id: tags:
``` python
df["song_name"]
```
%% Cell type:markdown id: tags:
### Review: `Pandas.Series.value_counts()`
- Returns a new `Series` with unique values from the original `Series` as keys and the count of those unique values as values.
- Return value `Series` is ordered using descending order of counts
%% Cell type:code id: tags:
``` python
# count the number of missing values for song name
df["song_name"]
```
%% Cell type:markdown id: tags:
### Missing value manipulation
Syntax: `df.fillna(<REPLACE>)`
- Returns a new DataFrame object instance reference.
%% Cell type:code id: tags:
``` python
# use .fillna to replace missing values
df["song_name"]
# to replace the original DataFrame's column, you need to explicitly update that object instance
# TODO: uncomment the below lines and update the code
#df["song_name"] = ???
#df
```
%% Cell type:markdown id: tags:
### Dropping missing values
Syntax: `df.dropna()`
- Returns a new DataFrame object instance reference.
%% Cell type:code id: tags:
``` python
# .dropna will drop all rows that contain NaN in them
