Skip to content
Snippets Groups Projects
Commit 403ffc14 authored by Anna Meyer's avatar Anna Meyer
Browse files

lec 13 notes

parent 024050ed
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
sum23/lecture_materials/11_JSON/lec_11_json_notes.ipynb 0 → 100644
+ 645
− 0
View file @ 403ffc14
%% Cell type:markdown id:1d888215 tags:
# JSON
%% Cell type:markdown id:7ccf358c tags:
### Learning Objectives:
- Interpret JSON formatted data and recognize differences between JSON and Python
- Deserialize data from JSON for use in Python programs (read)
- Serialize data into JSON for long term storage (write)
%% Cell type:code id:5990f3d6 tags:
``` python
import csv
# TODO: import json module
import json
```
%% Cell type:code id:5662ace1 tags:
``` python
# Deserialize
def read_json(path):
with open(path, encoding = "utf-8") as f: # f is a varaible
return json.load(f) # f represents data in the JSON file (dict, list, etc)
# Serialize
def write_json(path, data):
with open(path, 'w', encoding = "utf-8") as f:
json.dump(data, f, indent = 2)
```
%% Cell type:markdown id:a35312c9 tags:
### Example 1: Sum of numbers (simple JSON)
%% Cell type:code id:d241dc5c tags:
``` python
# TODO 1: Create a new "numsA.json".
# Add the list [1, 2, 3, 4] to "numsA.json" file.
# Use jupyter notebook to create and edit the new file
# TODO 2: Use input built-in function go get JSON file name from user
# Read the JSON file using read_json; capture return value into a variable
my_nums = read_json('nums.json')
# TODO 3: Print type of data returned by function that reads JSON file.
print(type(my_nums))
# TODO 4: Using Python built-in function sum(...), calculate total of numbers in numsA.json, print the total.
sum(my_nums['numsA'])
# TODO 5: Create a new JSON file "numsB.json" and try out the following data:
# [-1, 10, 4,]
# Does that work?
# Change the data to [-1, 10, 4] and try to run the program by providing input as numsB.json
# TODO 6: Create a new JSON file "simple.json" and try out the following data.
# What kind of error do you get with this?
# Fix the error by commenting the line of code that causes the error!
# 3.14
# True
# true
# 'hello'
# "hello"
```
%% Output
<class 'dict'>
73
%% Cell type:markdown id:e830f23f tags:
### Example 2: Score Tracker
%% Cell type:code id:d7655636 tags:
``` python
my_str = "the quick brown fox"
my_str.split(" ")
```
%% Output
['the', 'quick', 'brown', 'fox']
%% Cell type:code id:10e00558 tags:
``` python
player_details = input("Enter player name and score: ")
# TODO 1: extract player name and score into variables
player_name = player_details.split(" ")[0]
player_score = int(player_details.split(" ")[1])
# TODO 2: Define an empty "scores" dictionary to keep track of players'
# scores.
# KEY: player name VALUE: player scores list
scores = dict()
input_file = "score_history.json"
# TODO 3: Check if player name is a key in the scores dictionary.
# If not, create a new key for player name and value as empty list
# to keep track of that player's scores.
if not player_name in scores:
scores[player_name] = []
# TODO 4: Add player's score to the player's list in scores dictionary
scores[player_name].append(player_score)
# TODO 5: Create a "score_history.json" file and popluate that file with
# empty dictionary {}
write_json("score_history.json", {})
# TODO 6: Read "score_history.json" to populate initial "scores" dict,
# instead of the empty dict created in TODO 2.
scores = read_json("score_history.json")
print(scores)
# copying this here
if not player_name in scores:
scores[player_name] = []
# TODO 4: Add player's score to the player's list in scores dictionary
scores[player_name].append(player_score)
# TODO 7: Calculate average score for that player
# print("Average score for {} is {}."???)
total = 0
count = 0
for score in scores[player_name]:
total += score
count += 1
avg_score = total/count
print("Average score for {} is {}".format(player_name,str(avg_score)))
# TODO 8: At the end of the program, write the updated scores from dict
# into the "score_history.json" file
write_json("score_history.json", scores)
# That's it, now you have a program that helps you keep track
# of player scores permanently.
```
%% Output
Enter player name and score: anna 10
{}
Average score for anna is 10.0
%% Cell type:markdown id:b920d6e0 tags:
### Example 3: Kiva.com Micro-lending site
Many Web Sites have APIs that allow you to get their data.
%% Cell type:code id:dc891f5a tags:
``` python
# TODO: read "kiva.json"
kiva_dict = read_json("kiva.json")
# TODO: explore the type of the data structure returned by read_json
type(kiva_dict)
```
%% Output
dict
%% Cell type:markdown id:e29310b6 tags:
How to explore an unknown JSON?
- If you run into a `dict`, try `.keys()` method to look at the keys of the dictionary, then use lookup process to explore further
- If you run into a `list`, iterate over the list and print each item
%% Cell type:code id:31fe630f tags:
``` python
kiva_dict.keys()
# TODO: lookup the value corresponding to the key data
type(kiva_dict['data'])
kiva_dict['data'].keys()
# TODO: you know what to do next ... explore type again
type(kiva_dict['data']['lend'])
kiva_dict['data']['lend'].keys()
type(kiva_dict['data']['lend']['loans'])
kiva_dict['data']['lend']['loans'].keys()
type(kiva_dict['data']['lend']['loans']['values'])
```
%% Output
list
%% Cell type:code id:e09b064f tags:
``` python
kiva_dict['data']['lend']['loans']['values']
```
%% Output
[{'name': 'Polikseni',
'description': "Polikseni is 70 years old and married. She and her husband are both retired and their main income is a retirement pension of $106 a month for Polikseni and disability income for her husband of $289 a month. <br /><br />Polikseni's husband, even though disabled, works in a very small shop as a watchmaker on short hours, just to provide additional income for his family and to feel useful. Polikseni's husband needs constant medical treatment due to his health problems. She requested another loan, which she will use to continue paying for the therapy her husband needs. With a part of the loan, she is going to pay the remainder of the previous loan.",
'loanAmount': '1325.00',
'geocode': {'city': 'Korce',
'country': {'name': 'Albania',
'region': 'Eastern Europe',
'fundsLentInCountry': 9051250}}},
{'name': 'Safarmo',
'description': "Safarmo is 47 years old. She lives with her husband and her children in Khuroson district. <br /><br />Safarmo is a seamstress. She has been engaged in sewing for 10 years. She learned this activity with help of her mother and elder sister. <br /><br />Safarmo's sewing machine is old and she cannot work well. Her difficulty is lack of money. That’s why she applied for a loan to buy a new modern sewing machine. <br /><br />Safarmo needs your support.",
'loanAmount': '1075.00',
'geocode': {'city': 'Khuroson',
'country': {'name': 'Tajikistan',
'region': 'Asia',
'fundsLentInCountry': 64243075}}},
{'name': 'Elizabeth',
'description': 'Elizabeth is a mom blessed with five lovely children, who are her greatest motivation in life. She lives in the Natuu area of Kenya. Elizabeth is one of the most hardworking women in sub-Saharan Africa. Being a mother and living in a poor country has never been an excuse for Elizabeth, who has practiced mixed farming for the past few years.<br /><br />The cultural expectations in her area contribute to the notion that men should support their families. However, Elizabeth works independently for the success of her children. She perseveres because she wants to provide a better future for them.<br /><br />Elizabeth has always loved farming. She is a very proud farmer and enjoys milking her dairy cows. Elizabeth keeps poultry and grows crops, but she has not been making a good profit because of poor farming inputs. <br /><br />Elizabeth will use this loan to buy farm inputs and purchase high-quality seeds and good fertilizer to improve her crop production. Modern farming requires the use of modern techniques, and, therefore, using high-quality seeds will assure her of a bumper harvest and increased profit levels.<br /><br />Elizabeth is very visionary. Her goal for the season is to boost her crop production over the previous year.',
'loanAmount': '800.00',
'geocode': {'city': 'Matuu',
'country': {'name': 'Kenya',
'region': 'Africa',
'fundsLentInCountry': 120841775}}},
{'name': 'Ester',
'description': 'Ester believes that this year is her year of prosperity. Ester is a hardworking, progressive and honest farmer from a very remote village in the Kitale area of Kenya. This area is very fertile, with favorable weather patterns that support farming activities. Ester is happily married and the proud mother of lovely children. Together, they live on a small piece of land that she really treasures. Her primary sources of income are eggs and milk.<br /><br />Although this humble and industrious mother makes a profit, she faces the challenge of not being able to produce enough to meet the readily available market. Therefore, she is seeking funds from Kiva lenders to buy farm inputs such as good fertilizer and good-quality seeds. Through this loan, Ester should double her production, and this will translate into increased income. She then intends to save more money in the future so that she can develop her farming.<br /><br />One objective that Juhudi Kilimo aims at fulfilling is increasing the ease of accessing farm inputs and income-generating assets for farmers. Through the intervention of Juhudi Kilimo and Kiva, inputs such as fertilizers and pesticides have become more accessible to its members than buying a bottle of water. Ester is very optimistic and believes this loan will change her life completely.',
'loanAmount': '275.00',
'geocode': {'city': 'Kitale',
'country': {'name': 'Kenya',
'region': 'Africa',
'fundsLentInCountry': 120841775}}},
{'name': 'Cherifa',
'description': 'Cherifa is married, 57 years old with two children. She caters and also sells the local drink. She asks for credit to buy the necessities, in particular bags of anchovies, bags of maize and bundles of firewood. She wants to have enough income to run the house well.',
'loanAmount': '875.00',
'geocode': {'city': 'Agoe',
'country': {'name': 'Togo',
'region': 'Africa',
'fundsLentInCountry': 13719125}}}]
%% Cell type:code id:2a0b65d5 tags:
``` python
kiva_dict['data']['lend']['loans']['values'][0].keys()
```
%% Output
dict_keys(['name', 'description', 'loanAmount', 'geocode'])
%% Cell type:code id:b6cb9c28 tags:
``` python
# TODO: iterate over loans_list and print every borrower's name, loan amount and country details
for person_dict in kiva_dict['data']['lend']['loans']['values']:
borrower_name = person_dict['name']
loan_amount = person_dict['loanAmount']
country_details = person_dict['geocode']
print("Borrower name:", borrower_name)
print("Loan amount: $", loan_amount, sep = "")
print("Country details:", country_details)
print("------------------------------------------------------------------------------------------------")
```
%% Output
Borrower name: Polikseni
Loan amount: $1325.00
Country details: {'city': 'Korce', 'country': {'name': 'Albania', 'region': 'Eastern Europe', 'fundsLentInCountry': 9051250}}
------------------------------------------------------------------------------------------------
Borrower name: Safarmo
Loan amount: $1075.00
Country details: {'city': 'Khuroson', 'country': {'name': 'Tajikistan', 'region': 'Asia', 'fundsLentInCountry': 64243075}}
------------------------------------------------------------------------------------------------
Borrower name: Elizabeth
Loan amount: $800.00
Country details: {'city': 'Matuu', 'country': {'name': 'Kenya', 'region': 'Africa', 'fundsLentInCountry': 120841775}}
------------------------------------------------------------------------------------------------
Borrower name: Ester
Loan amount: $275.00
Country details: {'city': 'Kitale', 'country': {'name': 'Kenya', 'region': 'Africa', 'fundsLentInCountry': 120841775}}
------------------------------------------------------------------------------------------------
Borrower name: Cherifa
Loan amount: $875.00
Country details: {'city': 'Agoe', 'country': {'name': 'Togo', 'region': 'Africa', 'fundsLentInCountry': 13719125}}
------------------------------------------------------------------------------------------------
%% Cell type:markdown id:26b4c70c tags:
### Let's write student information dataset into various JSON files
%% Cell type:code id:c120d0eb tags:
``` python
# inspired by https://automatetheboringstuff.com/2e/chapter16/
def process_csv(filename):
exampleFile = open(filename, encoding="utf-8")
exampleReader = csv.reader(exampleFile)
exampleData = list(exampleReader)
exampleFile.close()
return exampleData
survey_data = process_csv('cs220_survey_data.csv')
cs220_header = survey_data[0]
cs220_data = survey_data[1:]
```
%% Cell type:code id:47cb92e9 tags:
``` python
def cell(row_idx, col_name):
"""
Returns the data value (cell) corresponding to the row index and
the column name of a CSV file.
"""
col_idx = cs220_header.index(col_name)
val = cs220_data[row_idx][col_idx]
# handle missing values, by returning None
if val == '':
return None
# handle type conversions
if col_name in ["Age",]:
return int(val)
return val
```
%% Cell type:code id:9a84cfb8 tags:
``` python
def bucketize(bucket_column):
"""
generates and returns bucketized data based on bucket_column
"""
# Key: unique bucketize column value; Value: list of lists (rows having that unique column value)
buckets = dict()
for row_idx in range(len(cs220_data)):
row = cs220_data[row_idx]
col_value = cell(row_idx, bucket_column)
if col_value not in buckets:
# create a new bucket when there is no existing bucket
buckets[col_value] = []
buckets[col_value].append(row)
return buckets
# TODO: create lecture based buckets and store result into lecture_buckets
lec_buckets = bucketize('Lecture')
# TODO: What is the type of lec_buckets? A dictionary of lists
type(lec_buckets)
type(lec_buckets['LEC001'])
# TODO: write lec_buckets into a JSON file called "lecture_cs220_data.json"
write_json("lecture_cs220_data.json", lec_buckets)
# TODO: create major based buckets and store result into major_buckets
# TODO: write major_buckets into a JSON file called "major_cs220_data.json"
```
%% Cell type:code id:cf1a8133 tags:
``` python
```
%% Cell type:code id:f36a584a tags:
``` python
def transform(header, data):
"""
Transform data into a list of dictionaries
"""
transformed_data = [] #should be defined outside the for loop, because it stores the entire data
for row in data:
#should be defined inside the for loop, because it represents one row as a dictionary
row_dict = {} # Key: header; Value: data
for idx in range(len(row)):
row_dict[header[idx]] = row[idx]
transformed_data.append(row_dict)
return transformed_data
transformed_data = transform(cs220_header, cs220_data)
# TODO: What is the type of transformed_data? A ______ of _________
# TODO: write transformed_data into a JSON file called "cs220_survey_data.json"
```
%% Cell type:markdown id:5279cdab tags:
### Self-practice: explore real-world JSON
### Weather for UW-Madison campus
%% Cell type:code id:7a4efc8c tags:
``` python
# TODO: read "weather.json"
# TODO: explore the type of the data structure returned by read_json
# display the data from "weather.json"
```
%% Cell type:code id:c1b98641 tags:
``` python
# TODO: display the keys of the weather.json dict
# TODO: lookup the value corresponding to the 'properties'
# TODO: you know what to do next ... explore type again
```
%% Cell type:code id:ca457052 tags:
``` python
# TODO: display the keys of the properties dict
# TODO: lookup the value corresponding to the 'periods'
# TODO: you know what to do next ... explore type again
```
%% Cell type:code id:34f6dfa4 tags:
``` python
# TODO: extract periods list into a variable
# TODO: iterate over loans_list and print every periods's startTime, endTime, temperature, and temperatureUnit
for period_dict in periods_list:
print("Start time:", start_time)
print("End time:", end_time)
print("Temperature: {} degree {}".format(temperature, temperature_unit))
print("------------------------------------------------------------------------------------------------")
```
sum23/lecture_materials/12_Objects/lec_12_objects_references_notes.ipynb 0 → 100644
+ 925
− 0
View file @ 403ffc14
This diff is collapsed.
sum23/lecture_materials/13_Copying/lec_13_Copying_notes.ipynb 0 → 100644
+ 592
− 0
View file @ 403ffc14
Source diff could not be displayed: it is too large. Options to address this: view the blob.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment