17-visualization_001.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "471a762b",
   "metadata": {},
   "source": [
    "# Visualization 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2478daaa-cb6c-4d73-92af-01ae91e773fe",
   "metadata": {},
   "source": [
    "### Geographic Data / Maps\n",
    "\n",
    "#### Installation\n",
    "```python\n",
    "pip3 install --upgrade pip\n",
    "pip3 install geopandas shapely descartes geopy netaddr\n",
    "sudo apt install -y python3-rtree\n",
    "```\n",
    "\n",
    "- `import geopandas as gpd`\n",
    "- `.shp` => Shapefile\n",
    "- `gpd.datasets.get_path(<shp file path>)`:\n",
    "    - example: `gpd.datasets.get_path(\"naturalearth_lowres\")`\n",
    "- `gpd.read_file(<path>)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6f50cc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import math\n",
    "import requests\n",
    "import re\n",
    "import os\n",
    "\n",
    "# new import statements\n",
    "import geopandas as gpd\n",
    "from shapely.geometry import Point, Polygon, box"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "257671bc-1e79-47c2-aa7a-1725562d23ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /home/gurmail.singh/.local/lib/python3.8/site-packages/geopandas/datasets/naturalearth_lowres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7269706-188a-48e3-882a-ac068170b1af",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /home/gurmail.singh/.local/lib/python3.8/site-packages/geopandas/datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "273ed288",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the path for \"naturalearth_lowres\"\n",
    "path = \n",
    "# Read the shapefile for \"naturalearth_lowres\" and\n",
    "# set index using \"name\" column\n",
    "gdf = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf4d871c-d6d6-4d99-be96-75faf804d4a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c983208-7851-4d25-92e7-3f110039411a",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(gdf).__mro__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "653edec9-8b82-4684-ae82-a9fa399459ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# All shapefiles have a column called \"geometry\"\n",
    "gdf[\"geometry\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2eb071a-b29a-4edd-8747-50b02fd43108",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(gdf[\"geometry\"]).__mro__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da5b5783-d78e-460f-bd22-b41e7f24f871",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First country's name and geometry\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f4da997-567d-47f6-8594-f0a37b92b9e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Second country's name geometry\n",
    "print(gdf.index[1])\n",
    "gdf[\"geometry\"].iat[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d380b32f-c401-4601-b6d3-02ac344b3f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Geometry for \"United States of America\"\n",
    "# gdf.at[<row_index>, <column_name>]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3528f252-36b7-4ca5-9108-0951367837cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type of Tanzania's geometry\n",
    "print(gdf.index[1], type(gdf[\"geometry\"].iat[1]))\n",
    "\n",
    "# Type of United States of America's geometry\n",
    "print(\"United States of America\", type(gdf.at[\"United States of America\", \"geometry\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47711b72-a24c-4da8-a804-3faf88a5fe8b",
   "metadata": {},
   "source": [
    "- `gdf.plot(figsize=(<width>, <height>), column=<column name>)`\n",
    "- `ax.set_axis_off()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9874c7de-226e-4296-af60-45e091836a19",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = gdf.plot(figsize=(8,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5547cc5-3404-421e-aeed-4b4cf8ee8382",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set facecolor=\"0.7\", edgecolor=\"black\"\n",
    "ax = gdf.plot(figsize=(8,4))\n",
    "# Turn off the axes\n",
    "# ax.set_axis_off()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58da716e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Color the map based on population column, column=\"pop_est\" and set cmap=\"cool\" and legend=True\n",
    "ax = gdf.plot(figsize=(8,4))\n",
    "# Turn off the axes\n",
    "ax.set_axis_off()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a85431-bdab-46ab-8e29-5f91d8a2e0bc",
   "metadata": {},
   "source": [
    "#### Create a map where countries with >100M people are red, others are gray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "962a552b-94a6-4690-8018-f82173dd6096",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a map where countries with >100M people are red, others are gray\n",
    "\n",
    "# Add a new column called color to gdf and set default value to \"lightgray\"\n",
    "\n",
    "# Boolean indexing to set color to red for countries with \"pop_est\" > 1e8\n",
    "\n",
    "# Create the plot\n",
    "# ax = gdf.plot(figsize=(8,4), color=gdf[\"color\"])\n",
    "# ax.set_axis_off()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a32c52c-509f-4f7f-8dd6-bc7fe53c64fd",
   "metadata": {},
   "source": [
    "### All shapefile geometries are shapely shapes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e92def-a7db-4d90-adc8-a3d01d472ac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(gdf[\"geometry\"].iat[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39c95c23",
   "metadata": {},
   "source": [
    "### Shapely shapes\n",
    "\n",
    "- `from shapely.geometry import Point, Polygon, box`\n",
    "- `Polygon([(<x1>, <y1>), (<x2>, <y2>), (<x3>, <y3>), ...])`\n",
    "- `box(minx, miny, maxx, maxy)`\n",
    "- `Point(<x>, <y>)`\n",
    "- `<shapely object>.buffer(<size>)`\n",
    "    - example: `Point(5, 5).buffer(3)` creates a circle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61716db9",
   "metadata": {},
   "outputs": [],
   "source": [
    "triangle =    # triangle\n",
    "triangle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a36021a-8653-4698-a0ba-818c14091d79",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(triangle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bddd958d-6fde-42df-8ae3-459e25fa3f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "box1 =     # not a type; just a function that creates box\n",
    "box1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7ade8d-96d2-4770-bce9-b3e35996b0b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(box1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5308c61-b1fa-433b-bb05-485ca7bd23da",
   "metadata": {},
   "outputs": [],
   "source": [
    "point = \n",
    "point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c669619-af78-477d-b807-3e6d99278f2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f015d27e-8fd8-446f-992f-4f7a88cc582d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use buffer to create a circle from a point\n",
    "circle = \n",
    "circle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39fe5752-8038-46cc-b4a6-320e6a78bbd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(circle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "800cc48d-c241-4439-9161-ff309e50373f",
   "metadata": {},
   "outputs": [],
   "source": [
    "triangle_buffer = triangle.buffer(3)\n",
    "triangle_buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b9478d3-7cc9-4e80-ae84-e56d7bfd0b71",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(triangle_buffer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a340443-d750-43d5-99cd-28e7034ce898",
   "metadata": {},
   "source": [
    "#### Shapely methods\n",
    "\n",
    "- Shapely methods:\n",
    "    - `union`:  any point that is in either shape (OR)\n",
    "    - `intersection`: any point that is in both shapes (AND)\n",
    "    - `difference`: subtraction\n",
    "    - `intersects`: do they overlap? returns True / False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1c5f9f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# union triangle and box1\n",
    "# it will give any point that is in either shape (OR)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2d3357",
   "metadata": {},
   "outputs": [],
   "source": [
    "# intersection triangle and box1\n",
    "# any point that is in both shapes (AND)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f327bb99-be34-4a00-a216-a6f40df48268",
   "metadata": {},
   "outputs": [],
   "source": [
    "# difference of triangle and box1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9082b54a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# difference of box1 and triangle\n",
    "box1.difference(triangle)   # subtraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0def4359-78f6-4f14-80ae-05c25ff64bc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check whether triangle intersects box1\n",
    "# the is, check do they overlap?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a82cf3-fe28-46d2-a019-d87f6a0f41b6",
   "metadata": {},
   "source": [
    "Is the point \"near\" (<6 units) the triangle?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a87b70f",
   "metadata": {},
   "outputs": [],
   "source": [
    "triangle.union(point.buffer(6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e04daa60",
   "metadata": {},
   "outputs": [],
   "source": [
    "triangle.intersects(point.buffer(6))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf500303",
   "metadata": {},
   "source": [
    "#### Extracting \"Europe\" data from \"naturalearth_lowres\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "410b08cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Europe bounding box\n",
    "eur_window = box(-10.67, 34.5, 31.55, 71.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45fdfcb-b244-4eff-a9d6-5cc4fefdfbd8",
   "metadata": {},
   "source": [
    "Can we use `intersects` method?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24f4a32b-9ed4-468b-a194-ebe0395fcdb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdf.intersects(eur_window)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff455178-84da-45bf-b5b9-a71a7f9160f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Incorrect v1\n",
    "gdf[gdf.intersects(eur_window)].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03aa8da8-72f4-45fd-8931-e410d1204226",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Incorrect v2\n",
    "gdf[~gdf.intersects(eur_window)].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4173f3c5-017f-44ff-b713-158219fcf3e5",
   "metadata": {},
   "source": [
    "Can we use `intersection` method?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7226fd9-3bb2-48c3-a5e9-5ce1ca0dd88f",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdf.intersection(eur_window)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "108b8b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "gdf.intersection(eur_window).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f13e4e16-286a-4681-8bc5-9bd3fcf599a7",
   "metadata": {},
   "source": [
    "How can we get rid of empty polygons (and remove the warning)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c0dd051-a808-4fd4-b5ff-fee6ed580753",
   "metadata": {},
   "outputs": [],
   "source": [
    "eur = gdf.intersection(eur_window)\n",
    "eur.is_empty"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acb1b886-f7c8-41d4-979a-2309c2b973bd",
   "metadata": {},
   "source": [
    "Remove all the empty polygons using `is_empty`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55a76a00",
   "metadata": {},
   "outputs": [],
   "source": [
    "eur = eur[~eur.is_empty]\n",
    "eur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08c59df7",
   "metadata": {},
   "outputs": [],
   "source": [
    "eur.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a12d23",
   "metadata": {},
   "source": [
    "#### Centroids of European countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb5c83b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the centroids\n",
    "ax = eur.plot(facecolor=\"lightgray\", edgecolor=\"k\")\n",
    "eur.centroid.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbfab5e7-5941-438a-baf2-e05369106004",
   "metadata": {},
   "source": [
    "### Lat / Lon CRS\n",
    "\n",
    "- Lon is x-coord\n",
    "- Lat is y-coord\n",
    "    - tells you where the point on Earth is\n",
    "- **IMPORTANT**: degrees are not a unit of distance. 1 degree of longitute near the equator is a lot farther than moving 1 degree of longitute near the north pole\n",
    "\n",
    "Using `.crs` to access CRS of a gdf.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89251896",
   "metadata": {},
   "outputs": [],
   "source": [
    "eur.crs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a6ba447-6f1a-4d31-8c33-ec1eeb9e0deb",
   "metadata": {},
   "source": [
    "#### Single CRS doesn't work for the whole earth\n",
    "\n",
    "- Setting a different CRS for Europe that is based on meters.\n",
    "- https://spatialreference.org/ref/?search=europe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "586038b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting CRS to \"EPSG:3035\"\n",
    "eur2 = eur.to_crs(\"EPSG:3035\")\n",
    "eur2.crs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d46c124c-9aff-4c2f-81fe-08885b606800",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = eur2.plot(facecolor=\"lightgray\", edgecolor=\"k\")\n",
    "eur2.centroid.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "045b9c33",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = eur2.plot(facecolor=\"lightgray\", edgecolor=\"k\")\n",
    "eur2.centroid.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0634941f",
   "metadata": {},
   "source": [
    "#### How much error does lat/long computation introduce?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0b72aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = eur2.plot(facecolor=\"lightgray\", edgecolor=\"k\")\n",
    "eur2.centroid.plot(ax=ax, color=\"k\") # black => correct\n",
    "eur.centroid.to_crs(\"EPSG:3035\").plot(ax=ax, color=\"r\")  # red => miscalculated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca9e306e",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(eur2.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f489c88d-5964-4358-b8c4-fc80d28b5491",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(eur2).__mro__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85880c73",
   "metadata": {},
   "source": [
    "#### Area of European countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e4874d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "eur2.area # area in sq meters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f55824",
   "metadata": {},
   "source": [
    "What is the area in **sq miles**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85ee20c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Conversion: / 1000 / 1000 / 2.59\n",
    "(eur2.area / 1000 / 1000 / 2.59).sort_values(ascending=False)\n",
    "# careful!  some countries (e.g., Russia) were cropped when we did intersection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd600837",
   "metadata": {},
   "outputs": [],
   "source": [
    "# area on screen, not real area\n",
    "eur.area"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf1245f-9939-468d-9a5f-34225c2dbc51",
   "metadata": {},
   "source": [
    "### CRS\n",
    "\n",
    "- `<GeoDataFrame object>.crs`: gives you information about current CRS.\n",
    "- `<GeoDataFrame object>.to_crs(<TARGET CRS>)`: changes CRS to `<TARGET CRS>`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da3ee4c",
   "metadata": {},
   "source": [
    "### Madison area emergency services\n",
    "\n",
    "- Data source: https://data-cityofmadison.opendata.arcgis.com/\n",
    "    - Search for:\n",
    "        - \"City limit\"\n",
    "        - \"Lakes and rivers\"\n",
    "        - \"Fire stations\"\n",
    "        - \"Police stations\"\n",
    "\n",
    "- CRS for Madison area: https://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system#/media/File:Universal_Transverse_Mercator_zones.svg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6f80847",
   "metadata": {},
   "outputs": [],
   "source": [
    "city = gpd.read_file(\"City_Limit.zip\").to_crs(\"epsg:32616\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f8595be",
   "metadata": {},
   "outputs": [],
   "source": [
    "city.crs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ebd361f",
   "metadata": {},
   "outputs": [],
   "source": [
    "water = gpd.read_file(\"Lakes_and_Rivers.zip\").to_crs(city.crs)\n",
    "fire = gpd.read_file(\"Fire_Stations.zip\").to_crs(city.crs)\n",
    "police = gpd.read_file(\"Police_Stations.zip\").to_crs(city.crs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "723e2bff-545f-4a8c-9f72-8a9906a99b1b",
   "metadata": {},
   "source": [
    "#### Run this on your virtual machine\n",
    "\n",
    "`sudo sh -c \"echo 'Options = UnsafeLegacyRenegotiation' >> /etc/ssl/openssl.cnf\"`\n",
    "\n",
    "then restart notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6069860-7dd9-43f2-970e-fd15980135ef",
   "metadata": {},
   "source": [
    "#### GeoJSON\n",
    "\n",
    "How to find the below URL?\n",
    "\n",
    "- Go to info page of a dataset, for example: https://data-cityofmadison.opendata.arcgis.com/datasets/police-stations/explore?location=43.081769%2C-89.391550%2C12.81\n",
    "- Then click on \"I want to use this\" > \"View API Resources\" > \"GeoJSON\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a095d5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://maps.cityofmadison.com/arcgis/rest/services/Public/OPEN_DATA/MapServer/2/query?outFields=*&where=1%3D1&f=geojson\"\n",
    "police2 = gpd.read_file(url).to_crs(city.crs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "248be81e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = city.plot(color=\"lightgray\")\n",
    "water.plot(color=\"lightblue\", ax=ax)\n",
    "fire.plot(color=\"red\", ax=ax, marker=\"+\", label=\"Fire\")\n",
    "police2.plot(color=\"blue\", ax=ax, label=\"Police\")\n",
    "ax.legend(loc=\"upper left\", frameon=False)\n",
    "ax.set_axis_off()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a609d81",
   "metadata": {},
   "outputs": [],
   "source": [
    "fire.to_file(\"fire.geojson\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77600884",
   "metadata": {},
   "source": [
    "### Geocoding: street address => lat / lon\n",
    "\n",
    "\n",
    "- `gpd.tools.geocode(<street address>, provider=<geocoding service name>, user_agent=<user agent name>)`: converts street address into lat/long\n",
    "\n",
    "\n",
    "#### Daily incident reports: https://www.cityofmadison.com/fire/daily-reports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b0b2aa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://www.cityofmadison.com/fire/daily-reports\"\n",
    "r = requests.get(url)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bee28b41",
   "metadata": {},
   "outputs": [],
   "source": [
    "r.raise_for_status() # give me an exception if not 200 (e.g., 404)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bae00e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# doesn't work\n",
    "# pd.read_html(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47173ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f166c5",
   "metadata": {},
   "source": [
    "Find all **span** tags with **streetAddress** using regex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac7b9482-5512-49e4-b0b8-7f9ed34b5844",
   "metadata": {},
   "outputs": [],
   "source": [
    "# <p>1700 block Thierer Road<br>\n",
    "# addrs = re.findall(r'<p>1700 block Thierer Road<br>', r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e9b49d2-3e0d-4a39-9dcf-13b288a165e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "addrs = re.findall(r'', r.text)\n",