Skip to content
Snippets Groups Projects
Commit 549d5b40 authored by gsingh58's avatar gsingh58
Browse files

Lec29 updated

parent 83446f20
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
f23/Gurmail_Lecture_Notes/29_Web-1/campus_weather.csv 0 → 100644
+ 15
− 0
View file @ 549d5b40
number,name,startTime,endTime,isDaytime,temperature,temperatureUnit,temperatureTrend,probabilityOfPrecipitation,dewpoint,relativeHumidity,windSpeed,windDirection,icon,shortForecast,detailedForecast
1,Tonight,2023-11-14T22:00:00-06:00,2023-11-15T06:00:00-06:00,False,43,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -1.1111111111111112}","{'unitCode': 'wmoUnit:percent', 'value': 57}",10 to 15 mph,SW,https://api.weather.gov/icons/land/night/sct?size=medium,Partly Cloudy,"Partly cloudy, with a low around 43. Southwest wind 10 to 15 mph."
2,Wednesday,2023-11-15T06:00:00-06:00,2023-11-15T18:00:00-06:00,True,64,F,falling,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': 6.666666666666667}","{'unitCode': 'wmoUnit:percent', 'value': 65}",5 to 10 mph,SW,https://api.weather.gov/icons/land/day/skc?size=medium,Sunny,"Sunny. High near 64, with temperatures falling to around 57 in the afternoon. Southwest wind 5 to 10 mph."
3,Wednesday Night,2023-11-15T18:00:00-06:00,2023-11-16T06:00:00-06:00,False,43,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': 6.666666666666667}","{'unitCode': 'wmoUnit:percent', 'value': 89}",5 to 10 mph,S,https://api.weather.gov/icons/land/night/sct?size=medium,Partly Cloudy,"Partly cloudy, with a low around 43. South wind 5 to 10 mph."
4,Thursday,2023-11-16T06:00:00-06:00,2023-11-16T18:00:00-06:00,True,63,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': 7.777777777777778}","{'unitCode': 'wmoUnit:percent', 'value': 86}",10 to 25 mph,S,https://api.weather.gov/icons/land/day/wind_bkn?size=medium,Partly Sunny,"Partly sunny, with a high near 63. South wind 10 to 25 mph, with gusts as high as 40 mph."
5,Thursday Night,2023-11-16T18:00:00-06:00,2023-11-17T06:00:00-06:00,False,34,F,,"{'unitCode': 'wmoUnit:percent', 'value': 30}","{'unitCode': 'wmoUnit:degC', 'value': 7.777777777777778}","{'unitCode': 'wmoUnit:percent', 'value': 69}",15 to 25 mph,W,"https://api.weather.gov/icons/land/night/rain_showers,30?size=medium",Chance Rain Showers,"A chance of rain showers. Mostly cloudy, with a low around 34. West wind 15 to 25 mph, with gusts as high as 40 mph. Chance of precipitation is 30%."
6,Friday,2023-11-17T06:00:00-06:00,2023-11-17T18:00:00-06:00,True,44,F,falling,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -2.7777777777777777}","{'unitCode': 'wmoUnit:percent', 'value': 67}",10 to 15 mph,NW,https://api.weather.gov/icons/land/day/few?size=medium,Sunny,"Sunny. High near 44, with temperatures falling to around 38 in the afternoon. Northwest wind 10 to 15 mph."
7,Friday Night,2023-11-17T18:00:00-06:00,2023-11-18T06:00:00-06:00,False,30,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -5.555555555555555}","{'unitCode': 'wmoUnit:percent', 'value': 63}",5 to 10 mph,W,https://api.weather.gov/icons/land/night/few?size=medium,Mostly Clear,"Mostly clear, with a low around 30. West wind 5 to 10 mph."
8,Saturday,2023-11-18T06:00:00-06:00,2023-11-18T18:00:00-06:00,True,49,F,falling,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -0.5555555555555556}","{'unitCode': 'wmoUnit:percent', 'value': 63}",10 to 15 mph,W,https://api.weather.gov/icons/land/day/few?size=medium,Sunny,"Sunny. High near 49, with temperatures falling to around 43 in the afternoon. West wind 10 to 15 mph."
9,Saturday Night,2023-11-18T18:00:00-06:00,2023-11-19T06:00:00-06:00,False,28,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -1.1111111111111112}","{'unitCode': 'wmoUnit:percent', 'value': 92}",10 to 15 mph,NW,https://api.weather.gov/icons/land/night/few?size=medium,Mostly Clear,"Mostly clear, with a low around 28."
10,Sunday,2023-11-19T06:00:00-06:00,2023-11-19T18:00:00-06:00,True,45,F,falling,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -1.1111111111111112}","{'unitCode': 'wmoUnit:percent', 'value': 92}",10 mph,N,https://api.weather.gov/icons/land/day/few?size=medium,Sunny,"Sunny. High near 45, with temperatures falling to around 40 in the afternoon."
11,Sunday Night,2023-11-19T18:00:00-06:00,2023-11-20T06:00:00-06:00,False,31,F,,"{'unitCode': 'wmoUnit:percent', 'value': None}","{'unitCode': 'wmoUnit:degC', 'value': -1.1111111111111112}","{'unitCode': 'wmoUnit:percent', 'value': 85}",5 to 10 mph,SE,https://api.weather.gov/icons/land/night/sct?size=medium,Partly Cloudy,"Partly cloudy, with a low around 31."
12,Monday,2023-11-20T06:00:00-06:00,2023-11-20T18:00:00-06:00,True,44,F,,"{'unitCode': 'wmoUnit:percent', 'value': 30}","{'unitCode': 'wmoUnit:degC', 'value': 1.1111111111111112}","{'unitCode': 'wmoUnit:percent', 'value': 82}",5 to 10 mph,SE,"https://api.weather.gov/icons/land/day/bkn/rain_showers,30?size=medium",Mostly Cloudy then Chance Rain Showers,"A chance of rain showers after noon. Mostly cloudy, with a high near 44. Chance of precipitation is 30%."
13,Monday Night,2023-11-20T18:00:00-06:00,2023-11-21T06:00:00-06:00,False,34,F,,"{'unitCode': 'wmoUnit:percent', 'value': 50}","{'unitCode': 'wmoUnit:degC', 'value': 2.2222222222222223}","{'unitCode': 'wmoUnit:percent', 'value': 92}",10 mph,SW,"https://api.weather.gov/icons/land/night/rain_showers,40/rain_showers,50?size=medium",Chance Rain Showers,"A chance of rain showers. Mostly cloudy, with a low around 34. Chance of precipitation is 50%."
14,Tuesday,2023-11-21T06:00:00-06:00,2023-11-21T18:00:00-06:00,True,44,F,,"{'unitCode': 'wmoUnit:percent', 'value': 40}","{'unitCode': 'wmoUnit:degC', 'value': 2.2222222222222223}","{'unitCode': 'wmoUnit:percent', 'value': 92}",10 to 15 mph,NW,"https://api.weather.gov/icons/land/day/rain_showers,40/rain_showers,30?size=medium",Chance Rain Showers,"A chance of rain showers. Partly sunny, with a high near 44. Chance of precipitation is 40%."
f23/Gurmail_Lecture_Notes/29_Web-1/lec_29_web1.ipynb
+ 448
− 395
View file @ 549d5b40
This diff is collapsed.
f23/Gurmail_Lecture_Notes/29_Web-1/lec_29_web1_template_Gurmail_lec1.ipynb 0 → 100644
+ 604
− 0
View file @ 549d5b40
This diff is collapsed.
f23/Gurmail_Lecture_Notes/29_Web-1/lec_29_web1_template.ipynb → f23/Gurmail_Lecture_Notes/29_Web-1/lec_29_web1_template_Gurmail_lec2.ipynb
+ 44
− 29
View file @ 549d5b40
%% Cell type:markdown id: tags:
# Announcements - Wednesday, November 15
* Download ALL files for today's lecture
* Q8 released tonight at 5:00 pm - due on Friday at 11:59 pm
* P10 due tonight at 11:59 pm
* <b>If you have any problem with P6 grades, please send me (Gurmail.Singh@wisc.edu) an email by November 15.</b>
* If you have questions, it is almost always faster to
* Post on Piazza
* Go to [office hours](https://sites.google.com/wisc.edu/cs220-oh-f23/home?pli=1)
* Exam2 results - you may get your grades by the evening
### Conflict Form
* [Final - December 19, 7:45 am](https://cs220.cs.wisc.edu/f23/surveys.html)
%% Cell type:markdown id: tags:
# Web 1 - How to get data from the Internet
%% Cell type:code id: tags:
``` python
# New module
import requests
# Known modules
import json
import pandas as pd
from pandas import Series, DataFrame
```
%% Cell type:markdown id: tags:
### P10 check-in
%% Cell type:code id: tags:
``` python
# It is very important to check auto-grader test results on p10 in a timely manner.
# Take a few minutes to verify if you hardcoded the slashes in P10 rather than using os.path.join?
# Your code won't clear auto-grader if you hardcode either "/" or "\"
# for *ANY* relative path in the entire project
# Check your code and check the autograder as soon as possible.
```
%% Cell type:markdown id: tags:
### Warmup 1: Read the data from "IMDB-Movie-Data.csv" into a pandas DataFrame called "movies"
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 2: fixing duplicate index columns
Notice that there are two index columns
- That happened because when you write a csv from pandas to a file, it writes a new index column
- So if the DataFrame already contains an index, you are going to get two index columns
- Let's fix that problem
%% Cell type:code id: tags:
``` python
#use slicing to retain all the rows and columns excepting for column with integer position 0
movies = movies.iloc[:, 1:]
movies
```
%% Cell type:code id: tags:
``` python
movies.to_csv("better_movies.csv", index = False)
```
%% Cell type:markdown id: tags:
### Warmup 3: Which movie has highest rating?
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 4: Which movies were released in 2020?
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 5a: What does this function do?
%% Cell type:code id: tags:
``` python
def format_revenue(revenue):
if type(revenue) == float: # need this in here if we run code multiple times
return revenue
elif revenue[-1] == 'M': # some have an "M" at the end
return float(revenue[:-1]) * 1e6
else: # otherwise, assume millions.
return float(revenue) * 1e6
```
%% Cell type:markdown id: tags:
### Warmup 5b: Using the above function, create a new column called "Revenue in dollars" by applying appropriate conversion to Revenue column.
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 6: What are the top 10 highest-revenue movies?
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 7: Which shortest movies (below average runtime) have highest rating?
### Warmup 7: What is total revenue from all movies?
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Warmup 8: Which shortest movies (below average runtime) have highest rating?
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
### Learning Objectives
- Make a request for data using requests.get(URL)
- Check the status of a request/response
- Extract the text of a response
- Create a json file from a response
- State and practice good etiquette when getting data
%% Cell type:markdown id: tags:
### Core Ideas:
- Network structure
- Client / server
- Request / response
![Client_server.png](attachment:Client_server.png)
- HTTP protocol
- URL
- Headers
- Status Codes
- The requests module
%% Cell type:markdown id: tags:
## HTTP Status Codes you need to know
- 200: success
- 404: not found
Here is a list of all status codes, you do NOT need to memorize it: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
%% Cell type:markdown id: tags:
## requests.get : Simple string example
- URL: https://cs220.cs.wisc.edu/hello.txt
%% Cell type:code id: tags:
``` python
url = "https://cs220.cs.wisc.edu/hello.txt"
r = requests.get(url) # r is the response
print(r.status_code)
print(r.text)
```
%% Cell type:code id: tags:
``` python
# Q: What if the web site does not exist?
typo_url = "https://cs220.cs.wisc.edu/hello.txttttt"
r = requests.get(typo_url)
print(r.status_code)
print(r.text)
# A:
```
%% Cell type:code id: tags:
``` python
# We can check for a status_code error by using an assert
typo_url = "https://cs220.cs.wisc.edu/hello.txttttt"
r = requests.get(typo_url)
assert r.status_code == 200
print(r.status_code)
print(r.text)
```
%% Cell type:code id: tags:
``` python
# Instead of using an assert, we often use raise_for_status()
r = requests.get(typo_url)
r.raise_for_status() #similar to asserting r.status_code == 200
r.text
# Note the error you get.... We will use this in the next cell
```
%% Cell type:code id: tags:
``` python
# Let's try to catch that error
try:
except:
print("oops!!", e)
```
%% Cell type:code id: tags:
``` python
# we often need to prepend the names of exceptions with the name of the module
# fix the error from above
try:
except:
print("oops!!", e)
```
%% Cell type:markdown id: tags:
## requests.get : JSON file example
- URL: https://cs220.cs.wisc.edu/scores.json
- `json.load` (FILE_OBJECT)
- `json.loads` (STRING)
%% Cell type:code id: tags:
``` python
# GETting a JSON file, the long way
url = "https://cs220.cs.wisc.edu/scores.json"
r = requests.get(url)
r.raise_for_status()
urltext = r.text
print(urltext)
d = json.loads(urltext)
print(type(d), d)
```
%% Cell type:code id: tags:
``` python
# GETting a JSON file, the shortcut way
url = "https://cs220.cs.wisc.edu/scores.json"
#Shortcut to bypass using json.loads()
r = requests.get(url)
r.raise_for_status()
d2 = r.json()
print(type(d2), d2)
```
%% Cell type:markdown id: tags:
## Good GET Etiquette
Don't make a lot of requests to the same server all at once.
- Requests use up the server's time
- Major websites will often ban users who make too many requests
- You can break a server....similar to DDoS attacks (DON'T DO THIS)
In CS220 we will usually give you a link to a copied file to avoid overloading the site.
%% Cell type:markdown id: tags:
### Explore real-world JSON
How to explore an unknown JSON?
- If you run into a `dict`, try `.keys()` method to look at the keys of the dictionary, then use lookup process to explore further
- If you run into a `list`, iterate over the list and print each item
### Weather for UW-Madison campus
- URL: https://api.weather.gov/gridpoints/MKX/37,63/forecast
%% Cell type:code id: tags:
``` python
# TODO: GET the forecast
# TODO: explore the type of the data structure
# display the data
```
%% Cell type:code id: tags:
``` python
# TODO: display the keys of the weather_data dict
# TODO: lookup the value corresponding to the 'properties'
# TODO: you know what to do next ... explore type again
```
%% Cell type:code id: tags:
``` python
# TODO: display the keys of the properties dict
# TODO: lookup the value corresponding to the 'periods'
# TODO: you know what to do next ... explore type again
```
%% Cell type:code id: tags:
``` python
# TODO: extract periods list into a variable
# TODO: create a DataFrame using periods_list
# TODO: What does each inner data structure represent in your DataFrame?
# Keep in mind that outer data structure is a list.
```
%% Cell type:markdown id: tags:
#### What is the maximum and minimum observed temperatures? Include the temperatureUnit in your display
%% Cell type:code id: tags:
``` python
min_temp =
idx_min =
min_unit =
max_temp =
idx_max =
max_unit =
print("Minimum observed temperature is: {} degree {}".format(min_temp, min_unit))
print("Maximum observed temperature is: {} degree {}".format(max_temp, max_unit))
```
%% Cell type:markdown id: tags:
#### Which days `detailedForecast` contains `snow`?
#### Which days `detailedForecast` contains `sunny`?
%% Cell type:code id: tags:
``` python
snow_days_df =
snow_days_df
sunny_days_df =
sunny_days_df
```
%% Cell type:code id: tags:
``` python
# Extract only the name column information for the subset DataFrame
```
%% Cell type:markdown id: tags:
#### Which day's `detailedForecast` has the most lengthy description?
%% Cell type:code id: tags:
``` python
idx_max_desc =
periods_df.iloc[idx_max_desc]['name']
```
%% Cell type:code id: tags:
``` python
# What was that forecast?
```
%% Cell type:markdown id: tags:
### Write it out to a CSV file on your drive
You now have your own copy!
%% Cell type:code id: tags:
``` python
# Write it all out to a single CSV file
periods_df.to_csv("campus_weather.csv", index=False)
```
%% Cell type:markdown id: tags:
### Other Cool APIs
%% Cell type:markdown id: tags:
- City of Madison Transit: http://transitdata.cityofmadison.com/
- Reddit: https://reddit.com/r/UWMadison.json
- Lord of the Rings: https://the-one-api.dev/
- Pokemon: https://pokeapi.co/
Remember: Be judicious when making requests; don't overwhelm the server! :)
%% Cell type:markdown id: tags:
## Next Time
What other documents can we get via the Web? HTML is very popular! We'll explore this.
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment