Python APIs, web scraping and Flask

Request and response: the language of the web

Every time a browser or an app talks to a server it sends a request and gets a response.

An API (Application Programming Interface) is a URL that returns data, usually JSON, for programs rather than people.

Calling a JSON API with requests

pip install requests

import requests

r = requests.get("https://api.github.com/users/python", timeout=10)
print(r.status_code)        # 200
data = r.json()             # the JSON body as a Python dict
print(data["name"], data["public_repos"])

Send query parameters with params= instead of building the URL by hand:

r = requests.get("https://api.example.com/rates", params={"base": "INR"}, timeout=10)

Handling status codes, timeouts and bad data

Networks fail, servers go down and APIs change. Good code expects that:

try:
    r = requests.get(url, timeout=10)
    r.raise_for_status()                 # raises an error for 4xx and 5xx codes
    data = r.json()
    temp = data.get("main", {}).get("temp")
except requests.exceptions.Timeout:
    print("The server took too long")
except requests.exceptions.RequestException as e:
    print("Request failed:", e)

Always set a timeout. Without one, a program can wait forever for a server that never answers. Keep API keys out of your code: read them from an environment variable with os.environ.

Reading web pages with BeautifulSoup

When a site has no API, you can read its HTML (web scraping). Check the site's terms first and do not overload it with requests.

import requests
from bs4 import BeautifulSoup          # pip install beautifulsoup4

html = requests.get("https://example.com", timeout=10).text
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)
for link in soup.find_all("a"):
    print(link.get("href"))

Your first website with Flask

Flask is a small framework for building websites and APIs in Python.

from flask import Flask               # pip install flask

app = Flask(__name__)

@app.route("/")
def home():
    return "<h1>Welcome to our coaching centre</h1>"

@app.route("/course/<name>")
def course(name):
    return f"Details of the {name} course"

if __name__ == "__main__":
    app.run(debug=True)

Run the file and open http://127.0.0.1:5000/ in a browser. @app.route connects a URL to a function; <name> captures part of the URL as a parameter. Use debug=True only on your own computer.

Templates: fill HTML with Python data (Jinja2)

Put HTML files in a folder called templates:

<!-- templates/students.html -->
<h1>{{ title }}</h1>
<ul>
{% for s in students %}
  <li>{{ s.name }}: {{ s.marks }}</li>
{% endfor %}
</ul>

from flask import render_template

@app.route("/students")
def students():
    rows = [{"name": "Asha", "marks": 91}, {"name": "Ravi", "marks": 78}]
    return render_template("students.html", title="Toppers", students=rows)

{{ ... }} prints a value (escaped, so it is safe); {% ... %} runs loops and ifs.

Handling forms: GET shows the form, POST receives the data

from flask import request

@app.route("/enquiry", methods=["GET", "POST"])
def enquiry():
    if request.method == "POST":
        name = request.form["name"]
        return f"Thank you, {name}. We will call you."
    return render_template("enquiry.html")

Query strings such as /search?q=asha are read with request.args.get("q").

Build a JSON API with Flask

from flask import jsonify

todos = [{"id": 1, "task": "Revise loops", "done": False}]

@app.get("/api/todos")
def list_todos():
    return jsonify(todos)

@app.post("/api/todos")
def add_todo():
    item = request.get_json()
    todo = {"id": len(todos) + 1, "task": item["task"], "done": False}
    todos.append(todo)
    return jsonify(todo), 201

The same Flask app can serve web pages for people and JSON for mobile apps.

Common mistakes

• No timeout on requests.get.
• Calling .json() on an error page without checking the status code first.
• Forgetting methods=["GET", "POST"], so the form gets 405 Method Not Allowed.
• Putting templates somewhere other than a folder named templates.

Key points

• A request has a URL, a method and maybe a body; a response has a status code and a body.
• requests.get(url, timeout=10).json() calls a JSON API.
• @app.route maps a URL to a Python function in Flask.
• render_template fills HTML with data; jsonify returns JSON.

Practice questions

@app.route("/square/<int:n>")
def square(n):
    return str(n * n)