lec12 and lec13 updated

ef73606a · gsingh58 · b4c88179 · ef73606a · ef73606a · ef73606a
Commit ef73606a authored 8 months ago by gsingh58
--- a/lecture_material/12-Web_2/12-web.ipynb
+++ b/lecture_material/12-Web_2/12-web.ipynb
@@ -97,10 +97,24 @@
    "friendly_get(base_url + \"slow\").text"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "id": "c65a4d53-fbe5-48da-827b-6a917e4ac356",
+   "metadata": {},
+   "source": [
+    "### `urllib.robotparser`\n",
+    "\n",
+    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html\n",
+    "- A few websites with robots.txt\n",
+    "    - https://en.wikipedia.org/robots.txt\n",
+    "    - https://www.reddit.com/robots.txt\n",
+    "    - https://cs320.cs.wisc.edu/su24/robots.txt"
+   ]
+  },
  {
   "cell_type": "code",
   "execution_count": null,
-   "id": "0d951114-6eee-4c67-b781-547b387a8b10",
+   "id": "0bdb9ac8-a390-40e2-b675-a23cc08b26a5",
   "metadata": {},
   "outputs": [],
   "source": []

 %% Cell type:markdown id:cf313adf tags:

 # Web 2: Flask

 %% Cell type:code id:d55e4bb4-9f29-4f4f-bba6-05054718259b tags:

 ``` python
 import requests
 import time
 import urllib.robotparser
 ```

 %% Cell type:markdown id:527600aa tags:

 ### Rate-limited webpage parsing


 - `requests` module:
    - `resp = requests.get(<URL>)` method: enables us to send HTTP GET request
    - `resp.status_code`: status code of the response
    - `resp.text`: `str` text content of the response
    - `resp.headers`: `dict` content of response headers

 - `@` operator is called a "decorator"
 - `flask.Response`: enables us to create a response object instance
    - Arguments: `str` representing reponse, `headers` dict representing metadata, `status` representing status code.
    - ex:
    ```python
    flask.Response("<b>go away</b>",
                              status=429,
                              headers={"Retry-After": "3"})
    ```

    ```python
    flask.Response("""User-Agent: *
    Disallow: /never
    """, headers={"Content-Type": "text/plain"})
    ```

 - `flask.request.remote_addr`: enables us to take action based on the IP address from which we receive the request

 - 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

 %% Cell type:code id:8241e51c tags:

 ``` python
 base_url = "http://35.226.223.87:5000/"
 ```

 %% Cell type:code id:6cc81b85 tags:

 ``` python
 def friendly_get(url):
    while True:
        resp = requests.get(url)
        if resp.status_code == 429:
            seconds = int(resp.headers.get("Retry-After", 1))
            print(f"sleep {seconds}")
            time.sleep(seconds)
            continue
        resp.raise_for_status() # raise exception if not 200
        return resp

 friendly_get(base_url + "slow").text
 ```

 %% Output

    'welcome!'

-%% Cell type:code id:0d951114-6eee-4c67-b781-547b387a8b10 tags:
+%% Cell type:markdown id:c65a4d53-fbe5-48da-827b-6a917e4ac356 tags:
+
+### `urllib.robotparser`
+
+- Documentation: https://docs.python.org/3/library/urllib.robotparser.html
+- A few websites with robots.txt
+    - https://en.wikipedia.org/robots.txt
+    - https://www.reddit.com/robots.txt
+    - https://cs320.cs.wisc.edu/su24/robots.txt
+
+%% Cell type:code id:0bdb9ac8-a390-40e2-b675-a23cc08b26a5 tags:

 ``` python
 ```

--- a/lecture_material/12-Web_2/12-web_001.ipynb
+++ b/lecture_material/12-Web_2/12-web_001.ipynb
@@ -79,6 +79,20 @@
    "    \n",
    "friendly_get(base_url + \"/slow\").text"
   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "583b80eb-56ce-4c50-9354-bab71472b443",
+   "metadata": {},
+   "source": [
+    "### `urllib.robotparser`\n",
+    "\n",
+    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html\n",
+    "- A few websites with robots.txt\n",
+    "    - https://en.wikipedia.org/robots.txt\n",
+    "    - https://www.reddit.com/robots.txt\n",
+    "    - https://cs320.cs.wisc.edu/su24/robots.txt"
+   ]
  }
 ],
 "metadata": {

 %% Cell type:markdown id:cf313adf tags:

 # Web 2: Flask

 %% Cell type:code id:d55e4bb4-9f29-4f4f-bba6-05054718259b tags:

 ``` python
 import requests
 import time
 ```

 %% Cell type:markdown id:527600aa tags:

 ### Rate-limited webpage parsing


 - `requests` module:
    - `resp = requests.get(<URL>)` method: enables us to send HTTP GET request
    - `resp.status_code`: status code of the response
    - `resp.text`: `str` text content of the response
    - `resp.headers`: `dict` content of response headers

 - `@` operator is called a "decorator"
 - `flask.Response`: enables us to create a response object instance
    - Arguments: `str` representing reponse, `headers` dict representing metadata, `status` representing status code.
    - ex:
    ```python
    flask.Response("<b>go away</b>",
                              status=429,
                              headers={"Retry-After": "3"})
    ```

    ```python
    flask.Response("""User-Agent: *
    Disallow: /never
    """, headers={"Content-Type": "text/plain"})
    ```

 - `flask.request.remote_addr`: enables us to take action based on the IP address from which we receive the request

 - 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

 %% Cell type:code id:8241e51c tags:

 ``` python
 base_url = "http://35.226.223.87:5000/"
 ```

 %% Cell type:code id:6cc81b85 tags:

 ``` python
 def friendly_get(url):
    while True:
        resp = requests.get(url)
        resp.raise_for_status() # raise exception if not 200
        return resp

 friendly_get(base_url + "/slow").text
 ```
+
+%% Cell type:markdown id:583b80eb-56ce-4c50-9354-bab71472b443 tags:
+
+### `urllib.robotparser`
+
+- Documentation: https://docs.python.org/3/library/urllib.robotparser.html
+- A few websites with robots.txt
+    - https://en.wikipedia.org/robots.txt
+    - https://www.reddit.com/robots.txt
+    - https://cs320.cs.wisc.edu/su24/robots.txt

--- a/lecture_material/13-Web_3/13-web.ipynb
+++ b/lecture_material/13-Web_3/13-web.ipynb
@@ -57,7 +57,11 @@
   "source": [
    "### `urllib.robotparser`\n",
    "\n",
-    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html"
+    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html\n",
+    "- A few websites with robots.txt\n",
+    "    - https://en.wikipedia.org/robots.txt\n",
+    "    - https://www.reddit.com/robots.txt\n",
+    "    - https://cs320.cs.wisc.edu/su24/robots.txt"
   ]
  },
  {

 %% Cell type:markdown id:cf313adf tags:

 # Web 3: More Flask and A/B testing

 %% Cell type:code id:d55e4bb4-9f29-4f4f-bba6-05054718259b tags:

 ``` python
 import requests
 import time
 import urllib.robotparser

 import pandas as pd
 # new import statement: requires pip3 install scipy
 from scipy import stats
 ```

 %% Cell type:markdown id:725234ef-a02b-47da-bfec-aaa16e852a58 tags:

 - `flask.request.args`: enables us to get the arguments passed as part of the URL
    - How do we pass arguments?
        - at the end of the URL, add a "?"
        - then separate argument-value pair by "="
        - use "&" as delimiter between two argument-value pairs
    - examples:
        - http://35.226.223.87:5000/add?x=10&y=20
        - http://35.226.223.87:5000/survey?major=CS
        - http://35.226.223.87:5000/survey?major=Mechanical_Engineering

 %% Cell type:code id:8241e51c tags:

 ``` python
 base_url = "http://35.226.223.87:5000/"
 ```

 %% Cell type:markdown id:23ba100b tags:

 ### `urllib.robotparser`

 - Documentation: https://docs.python.org/3/library/urllib.robotparser.html
+- A few websites with robots.txt
+    - https://en.wikipedia.org/robots.txt
+    - https://www.reddit.com/robots.txt
+    - https://cs320.cs.wisc.edu/su24/robots.txt

 %% Cell type:code id:379c3ae5-7344-45b1-88c3-b35f0bd8eb5b tags:

 ``` python
 rp = urllib.robotparser.RobotFileParser()
 rp.set_url(base_url + "/robots.txt")
 rp.read()
 rp.can_fetch("cs320bot", base_url + "/slow")
 ```

 %% Output

    True

 %% Cell type:code id:2e3fb01c-4281-4cbf-8828-98e04d27d09a tags:

 ``` python
 rp.can_fetch("cs320bot", base_url + "never")
 ```

 %% Output

    False

 %% Cell type:code id:6cc81b85 tags:

 ``` python
 def friendly_get(url):
    if not rp.can_fetch("cs320bot", url):
        raise Exception("you're not supposed to visit that page")
    while True:
        resp = requests.get(url)
        if resp.status_code == 429:
            seconds = int(resp.headers.get("Retry-After", 1))
            print(f"sleep {seconds}")
            time.sleep(seconds)
            continue
        resp.raise_for_status() # raise exception if not 200
        return resp

 friendly_get(base_url + "slow").text
 ```

 %% Output

    'welcome!'

 %% Cell type:markdown id:1dbd2fad-1bf5-437f-9ebb-6694ba860e27 tags:

 # A/B testing

 %% Cell type:code id:6922661b-af1b-41d5-bcb0-f2759e890f3e tags:

 ``` python
 df = pd.DataFrame({
    "click":    {"A": 50, "B": 55},
    "no-click": {"A": 50, "B": 45}
 })
 df
 # Which has the higher CTR A or B?
 ```

 %% Output

       click  no-click
    A     50        50
    B     55        45

 %% Cell type:code id:31b0de1b-f7fb-42ee-a087-3d884cc31590 tags:

 ``` python
 _, pvalue = stats.fisher_exact(df)
 pvalue
 # no evidence that A and B are difference because pvalue is not less than 5%
 ```

 %% Output

    np.float64(0.5712421394829712)

 %% Cell type:markdown id:5dde0ee6-b03e-4bbf-b896-1794a2bff610 tags:

 ### Two situations when pvalue will be lower than significance threshold

 1. Sample size is the same, but skew is very heavy --- unlikely to have that by chance
 2. Sample size is large, but skew is small

 %% Cell type:code id:b8719b9e-f366-47a0-9d96-50b4fdd8fb27 tags:

 ``` python
 # Scenario 1:
 # Sample size is the same, but skew is very heavy ---
 # unlikely to have that by chance

 df = pd.DataFrame({
    "click":    {"A": 50, "B": 75},
    "no-click": {"A": 50, "B": 25}
 })
 _, pvalue = stats.fisher_exact(df)
 pvalue
 ```

 %% Output

    np.float64(0.00042033045869994034)

 %% Cell type:code id:72a18c43-0f07-4e7f-bb88-89a43563f295 tags:

 ``` python
 # Scenario 2:
 # Sample size is large, but skew is small

 df = pd.DataFrame({
    "click":    {"A": 500, "B": 550},
    "no-click": {"A": 500, "B": 450}
 })
 _, pvalue = stats.fisher_exact(df)
 pvalue
 ```

 %% Output

    np.float64(0.02820356890423392)

 %% Cell type:code id:d63d0930-3560-45bf-9c2c-3a9f14175040 tags:

 ``` python
 ```

--- a/lecture_material/13-Web_3/13-web_001.ipynb
+++ b/lecture_material/13-Web_3/13-web_001.ipynb
@@ -56,7 +56,11 @@
   "source": [
    "### `urllib.robotparser`\n",
    "\n",
-    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html"
+    "- Documentation: https://docs.python.org/3/library/urllib.robotparser.html\n",
+    "- A few websites with robots.txt\n",
+    "    - https://en.wikipedia.org/robots.txt\n",
+    "    - https://www.reddit.com/robots.txt\n",
+    "    - https://cs320.cs.wisc.edu/su24/robots.txt"
   ]
  },
  {

 %% Cell type:markdown id:cf313adf tags:

 # Web 3: More Flask and A/B testing

 %% Cell type:code id:d55e4bb4-9f29-4f4f-bba6-05054718259b tags:

 ``` python
 import requests
 import time

 import pandas as pd
 # new import statement: requires pip3 install scipy
 from scipy import stats
 ```

 %% Cell type:markdown id:a5f26f67-000a-4d68-8033-3e46023b8196 tags:

 - `flask.request.args`: enables us to get the arguments passed as part of the URL
    - How do we pass arguments?
        - at the end of the URL, add a "?"
        - then separate argument-value pair by "="
        - use "&" as delimiter between two argument-value pairs
    - examples:
        - http://35.226.223.87:5000/add?x=10&y=20
        - http://35.226.223.87:5000/survey?major=CS
        - http://35.226.223.87:5000/survey?major=Mechanical_Engineering

 %% Cell type:code id:8241e51c tags:

 ``` python
 base_url = "http://35.226.223.87:5000/"
 ```

 %% Cell type:markdown id:23ba100b tags:

 ### `urllib.robotparser`

 - Documentation: https://docs.python.org/3/library/urllib.robotparser.html
+- A few websites with robots.txt
+    - https://en.wikipedia.org/robots.txt
+    - https://www.reddit.com/robots.txt
+    - https://cs320.cs.wisc.edu/su24/robots.txt

 %% Cell type:code id:379c3ae5-7344-45b1-88c3-b35f0bd8eb5b tags:

 ``` python
 ```

 %% Cell type:code id:2e3fb01c-4281-4cbf-8828-98e04d27d09a tags:

 ``` python
 ```

 %% Cell type:code id:0eb41e31-490e-419b-af15-db66e0dd6fd1 tags:

 ``` python
 def friendly_get(url):
    while True:
        resp = requests.get(url)
        if resp.status_code == 429:
            seconds = int(resp.headers.get("Retry-After", 1))
            print(f"sleep {seconds}")
            time.sleep(seconds)
            continue
        resp.raise_for_status() # raise exception if not 200
        return resp

 friendly_get(base_url + "slow").text
 ```

 %% Cell type:markdown id:fcfa857a-10e7-4e74-adcc-9c4d384e9986 tags:

 #  A/B testing

 %% Cell type:code id:d4839880-5642-49ce-b1cf-938ddf409229 tags:

 ``` python
 df = pd.DataFrame({
    "click":    {"A": 50, "B": 55},
    "no-click": {"A": 50, "B": 45}
 })
 df
 # Which has the higher CTR A or B?
 ```

 %% Cell type:code id:341e9430-d7c2-4cba-a2f1-b5fb95a1efa6 tags:

 ``` python
 _, pvalue = stats.fisher_exact(df)
 pvalue
 # no evidence that A and B are difference because pvalue is not less than 5%
 ```

 %% Cell type:markdown id:3fa3b996-ef70-4492-8221-dbe597af8541 tags:

 ### Two situations when pvalue will be lower than significance threshold

 1. Sample size is the same, but skew is very heavy --- unlikely to have that by chance
 2. Sample size is large, but skew is small

 %% Cell type:code id:245ac3cf-a7b4-4bd4-81d8-caf7fc4897fe tags:

 ``` python
 # Scenario 1:
 # Sample size is the same, but skew is very heavy ---
 # unlikely to have that by chance

 df = pd.DataFrame({
    "click":    {"A": 50, "B": 75},
    "no-click": {"A": 50, "B": 25}
 })
 _, pvalue = stats.fisher_exact(df)
 pvalue
 ```

 %% Cell type:code id:e3a1b056-3315-494d-9667-3bfa5ebfa863 tags:

 ``` python
 # Scenario 2:
 # Sample size is large, but skew is small

 df = pd.DataFrame({
    "click":    {"A": 500, "B": 550},
    "no-click": {"A": 500, "B": 450}
 })
 _, pvalue = stats.fisher_exact(df)
 pvalue
 ```