Data analysis with NumPy, Pandas and Matplotlib

The three libraries

By convention they are imported as np, pd and plt.

NumPy arrays: maths on whole lists at once

import numpy as np

marks = np.array([78, 91, 65, 88])
print(marks + 5)              # [83 96 70 93] -- adds 5 to every item
print(marks * 1.1)            # scale every item
print(marks.mean(), marks.max(), marks.std())
print(marks[marks >= 80])     # [91 88] -- filter with a condition

With a Python list, [78, 91] + 5 is an error and * 2 repeats the list. NumPy does the maths item by item, and much faster for large data.

Pandas DataFrame: a table in Python

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Ravi", "Meena", "Kiran"],
    "class": [10, 10, 9, 9],
    "marks": [91, 78, 85, 62],
})
df = pd.read_csv("marks.csv")          # or load a CSV (read_excel for .xlsx)

print(df.head())          # first 5 rows
print(df.shape)           # (rows, columns)
df.info()                 # column types and missing values
print(df.describe())      # count, mean, min, max of number columns

A DataFrame is the table; each column on its own is a Series.

Selecting, filtering and sorting rows

df["marks"]                         # one column (a Series)
df[["name", "marks"]]               # several columns
df[df["marks"] >= 80]               # rows where the condition is true
df[(df["class"] == 10) & (df["marks"] > 80)]    # use & and | with brackets, not and/or
df.sort_values("marks", ascending=False)
df.loc[0, "name"]                   # row label 0, column "name"

New columns, groupby and summaries

df["with_bonus"] = df["marks"] + 5               # maths on a whole column
df["result"] = df["marks"].apply(lambda m: "Pass" if m >= 33 else "Fail")

print(df.groupby("class")["marks"].mean())          # average per class
print(df.groupby("class")["marks"].agg(["count", "mean", "max"]))
print(df["result"].value_counts())                  # how many Pass and Fail

groupby is the Pandas version of SQL's GROUP BY and of a pivot table in Excel.

Cleaning messy data

Real data has extra spaces, mixed capitals, blanks, duplicates and numbers stored as text.

df["name"] = df["name"].str.strip().str.title()     # spaces and capitals
df = df.drop_duplicates()                           # repeated rows
print(df.isna().sum())                              # blanks in each column
df["marks"] = pd.to_numeric(df["marks"], errors="coerce")   # bad text becomes NaN
df["marks"] = df["marks"].fillna(0)                 # or df.dropna() to remove those rows

NaN ("not a number") is how Pandas marks a missing value.

Matplotlib: line and bar charts

import matplotlib.pyplot as plt

months = ["Jun", "Jul", "Aug", "Sep"]
rain = [210, 340, 300, 180]
plt.plot(months, rain, marker="o")
plt.title("Rainfall in Ranchi (mm)")
plt.xlabel("Month")
plt.ylabel("mm")
plt.savefig("rain.png")          # save to a file
plt.show()                       # open a window

plt.bar(["Asha", "Ravi", "Meena"], [91, 78, 85])
plt.show()

Plot from a DataFrame and export to Excel

avg = df.groupby("class")["marks"].mean()
avg.plot(kind="bar", title="Average marks by class")
plt.tight_layout()
plt.savefig("average.png")

df.to_excel("report.xlsx", index=False)    # needs: pip install openpyxl
df.to_csv("clean.csv", index=False)

Example: class result analysis

import pandas as pd
df = pd.read_csv("marks.csv")
df["total"] = df[["maths", "science", "english"]].sum(axis=1)
df["rank"] = df["total"].rank(ascending=False).astype(int)
print(df.sort_values("rank")[["name", "total", "rank"]].head(3))

Common mistakes

• Using and / or in Pandas filters: use & / | with brackets around each condition.
• Forgetting that most Pandas methods return a new DataFrame: write df = df.drop_duplicates().
• Doing maths on a column that was read as text: convert it with pd.to_numeric.
• Calling plt.show() before plt.savefig(): the saved file comes out blank.

Key points

• NumPy does fast maths on whole arrays at once.
• A Pandas DataFrame is a table: filter with conditions, sort, add columns and groupby.
• Clean data with str.strip(), drop_duplicates(), fillna() and to_numeric().
• Matplotlib draws charts with plot, bar and pie; save with savefig.

Practice questions

df.groupby("category")["amount"].sum().plot(kind="bar")
plt.show()