{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "EOr-DwYgRsUa" }, "source": [ "# Initiation to data collection with web scraping and APIs\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "All required packages are available.\n" ] } ], "source": [ "# Run this cell only if some packages are missing.\n", "import importlib.util\n", "import subprocess\n", "import sys\n", "\n", "packages = {\n", " \"bs4\": \"beautifulsoup4\",\n", " \"pandas\": \"pandas\",\n", " \"lxml\": \"lxml\",\n", " \"folium\": \"folium\",\n", "}\n", "\n", "for import_name, package_name in packages.items():\n", " if importlib.util.find_spec(import_name) is None:\n", " print(f\"Installing {package_name}...\")\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", package_name])\n", "\n", "print(\"All required packages are available.\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "executionInfo": { "elapsed": 10389, "status": "ok", "timestamp": 1737535342998, "user": { "displayName": "Lau re", "userId": "03302099944040145915" }, "user_tz": -60 }, "id": "E4zarSeAQdHm" }, "outputs": [], "source": [ "import bs4\n", "import pandas as pd\n", "\n", "from io import StringIO\n", "from urllib.parse import quote, urljoin\n", "from urllib.request import Request, urlopen\n" ] }, { "cell_type": "markdown", "metadata": { "id": "t4wzROqzT36G" }, "source": [ "## Scrape a table on a Wikipedia page\n", "\n", "We would like to display on a map the location of the Summer Olympic Games since 1896. We will use a [Wikipedia page](https://fr.wikipedia.org/wiki/Jeux_olympiques) to scrape the associated table.\n", "\n", "Below is the code to download the page with `urllib.request` and display its title with `BeautifulSoup`.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1274, "status": "ok", "timestamp": 1737535344270, "user": { "displayName": "Lau re", "userId": "03302099944040145915" }, "user_tz": -60 }, "id": "Nl9QICzwTGG7", "outputId": "b0c7f6a4-c1f3-4f6a-cc9f-da86a47bbcfe" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Jeux olympiques — Wikipédia\n" ] } ], "source": [ "url = \"https://fr.wikipedia.org/wiki/Jeux_olympiques\"\n", "\n", "req = Request(url, headers={\"User-Agent\": \"Mozilla/5.0\"})\n", "html = urlopen(req).read()\n", "\n", "page = bs4.BeautifulSoup(html, \"lxml\")\n", "\n", "print(page.find(\"title\").get_text(strip=True))" ] }, { "cell_type": "markdown", "metadata": { "id": "U_18h3E9UiYR" }, "source": [ "Our objective here is to extract the information from the first table, \"Jeux olympiques d'été\", and build a dataframe.\n", "\n", "To proceed, we will follow two steps:\n", "* Find the table in the HTML page with Beautiful Soup.\n", "* Convert the HTML table into a pandas dataframe.\n", "\n", "Check the quick start of Beautiful Soup to understand what is going on: https://beautiful-soup-4.readthedocs.io/en/latest/\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 4, "status": "ok", "timestamp": 1737535344270, "user": { "displayName": "Lau re", "userId": "03302099944040145915" }, "user_tz": -60 }, "id": "M5oVOOZuKxD_", "outputId": "963cee44-646a-46fe-e688-76db7b0c2934" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AnnéeOlympiade (Nº de l’olympiade[NB 1])Ville hôtePaysContinent
01896IAthènes (1)Grèce (1)Europe (1)
11900IIParis (1)France (1)Europe (2)
21904IIISaint-Louis (1)États-Unis (1)Amérique du Nord (1)
31906• (Intercalaires[NB 2])Athènes (2)Grèce (2)Europe
41908IVLondres (1)Royaume-Uni (1)Europe (3)
\n", "
" ], "text/plain": [ " Année Olympiade (Nº de l’olympiade[NB 1]) Ville hôte \\\n", "0 1896 I Athènes (1) \n", "1 1900 II Paris (1) \n", "2 1904 III Saint-Louis (1) \n", "3 1906 • (Intercalaires[NB 2]) Athènes (2) \n", "4 1908 IV Londres (1) \n", "\n", " Pays Continent \n", "0 Grèce (1) Europe (1) \n", "1 France (1) Europe (2) \n", "2 États-Unis (1) Amérique du Nord (1) \n", "3 Grèce (2) Europe \n", "4 Royaume-Uni (1) Europe (3) " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "table = page.find(\"table\", class_=\"wikitable center\")\n", "\n", "# pandas can read HTML tables directly.\n", "olympics = pd.read_html(StringIO(str(table)))[0]\n", "\n", "olympics.head()\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Qw6i-yLueZvG" }, "source": [ "## Exercise 2: Locate places on a map\n", "\n", "The objective here is to identify organizer cities on a map.\n", "\n", "Here are the steps:\n", "* For each city in the table, collect the URL of its corresponding page.\n", "* Find the coordinates of the city in the HTML page of the city\n", "* Store cities and coordinates in a dataframe.\n", "\n", "Tips:\n", "First, check manually how the page of a city is organized. With your favorite browser, check the source of the page and locate the code of the infobox on the topr right of the page. In that infobox, check where are the coordinates of the city and how this is coded in HTML. You will find that the coordinates are written in the HTML source at this location.\n", "\n", "Read the following code and understand it line by line. Add comments to explain each of the parts.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 35916, "status": "ok", "timestamp": 1737535380184, "user": { "displayName": "Lau re", "userId": "03302099944040145915" }, "user_tz": -60 }, "id": "7xaYx79JMF_N", "outputId": "61045df9-7802-4fdb-c596-5767531be764" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameurllatitudelongitude
0Athèneshttps://fr.wikipedia.org/wiki/Athènes37.96666723.716667
1Parishttps://fr.wikipedia.org/wiki/Paris48.8565782.351828
2Saint-Louishttps://fr.wikipedia.org/wiki/Saint-Louis_(Mis...38.648056-90.212220
3Athèneshttps://fr.wikipedia.org/wiki/Athènes37.96666723.716667
4Londreshttps://fr.wikipedia.org/wiki/Londres51.507222-0.127500
\n", "
" ], "text/plain": [ " name url latitude \\\n", "0 Athènes https://fr.wikipedia.org/wiki/Athènes 37.966667 \n", "1 Paris https://fr.wikipedia.org/wiki/Paris 48.856578 \n", "2 Saint-Louis https://fr.wikipedia.org/wiki/Saint-Louis_(Mis... 38.648056 \n", "3 Athènes https://fr.wikipedia.org/wiki/Athènes 37.966667 \n", "4 Londres https://fr.wikipedia.org/wiki/Londres 51.507222 \n", "\n", " longitude \n", "0 23.716667 \n", "1 2.351828 \n", "2 -90.212220 \n", "3 23.716667 \n", "4 -0.127500 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base_url = \"https://fr.wikipedia.org\"\n", "rows = table.find_all(\"tr\")\n", "\n", "cities = []\n", "coordinates_cache = {}\n", "\n", "for row in rows[1:]:\n", " cells = row.find_all(\"td\")\n", " if len(cells) < 3:\n", " continue\n", "\n", " # BeautifulSoup can directly look for links with an href attribute.\n", " city_link = cells[2].find(\"a\", href=True)\n", " if city_link is None:\n", " continue\n", "\n", " name = city_link.get_text(strip=True)\n", " city_url = urljoin(base_url, city_link[\"href\"])\n", "\n", " if city_url not in coordinates_cache:\n", " safe_city_url = quote(city_url, safe=\"/:?=&%\")\n", " req = Request(safe_city_url, headers={\"User-Agent\": \"Mozilla/5.0\"})\n", " city_html = urlopen(req).read()\n", " city_page = bs4.BeautifulSoup(city_html, \"lxml\")\n", "\n", " map_link = city_page.select_one(\n", " \"table[class*=infobox] a.mw-kartographer-maplink[data-lat][data-lon]\"\n", " )\n", "\n", " if map_link is None:\n", " map_link = city_page.select_one(\"a.mw-kartographer-maplink[data-lat][data-lon]\")\n", "\n", " coordinates_cache[city_url] = (\n", " map_link.get(\"data-lat\") if map_link else None,\n", " map_link.get(\"data-lon\") if map_link else None,\n", " )\n", "\n", " latitude, longitude = coordinates_cache[city_url]\n", " cities.append({\n", " \"name\": name,\n", " \"url\": city_url,\n", " \"latitude\": latitude,\n", " \"longitude\": longitude,\n", " })\n", "\n", "data = pd.DataFrame(cities).dropna()\n", "data[\"latitude\"] = pd.to_numeric(data[\"latitude\"])\n", "data[\"longitude\"] = pd.to_numeric(data[\"longitude\"])\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "j3_3JmXggIEk" }, "source": [ "Then we can plot the map. The map is also saved as an HTML file, so it can be opened in a browser if it does not display correctly inside the notebook.\n" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 711 }, "executionInfo": { "elapsed": 957, "status": "ok", "timestamp": 1737535381121, "user": { "displayName": "Lau re", "userId": "03302099944040145915" }, "user_tz": -60 }, "id": "93_lzZI3Wkpt", "outputId": "45d5a209-06c2-49b3-996c-5295bc1d60e2" }, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from pathlib import Path\n", "\n", "import folium\n", "from IPython.display import display\n", "\n", "center = data[[\"latitude\", \"longitude\"]].mean().tolist()\n", "\n", "m = folium.Map(location=center, zoom_start=2, tiles=\"OpenStreetMap\")\n", "\n", "for _, city in data.iterrows():\n", " folium.Marker(\n", " [city[\"latitude\"], city[\"longitude\"]],\n", " popup=city[\"name\"],\n", " ).add_to(m)\n", "\n", "sw = data[[\"latitude\", \"longitude\"]].min().tolist()\n", "ne = data[[\"latitude\", \"longitude\"]].max().tolist()\n", "m.fit_bounds([sw, ne])\n", "\n", "Path(\"leaflet\").mkdir(exist_ok=True)\n", "m.save(\"leaflet/olympic_cities_map.html\")\n", "\n", "display(m)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Questions\n", "* Do the same for the Winter Olympic Games.\n", "* Plot Summer and Winter Olympic cities on the same map, using a different color for each.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 3: Use a REST API with OpenFoodFacts\n", "\n", "In the previous exercises, we scraped HTML pages. Here we use a REST API: the server returns structured JSON directly.\n", "\n", "We will use the [OpenFoodFacts API](https://openfoodfacts.github.io/documentation/). It is open and does not require an account for read requests. A good habit is to send a clear `User-Agent` header, so the service can identify educational requests if needed.\n", "\n", "First check quickly the search documentation: https://openfoodfacts.github.io/documentation/docs/Product-Opener/v2/search/get-search/\n", "Use this documentation to understand the code below.\n", "\n", "Our mini mission: become snack detectives and find which chocolate products in the database look the sweetest.\n", "\n", "Add comments to check that you understand everything that is going on.\n" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data loaded from OpenFoodFacts.\n", "Number of products: 20\n" ] } ], "source": [ "import json\n", "from urllib.error import HTTPError, URLError\n", "from urllib.parse import urlencode\n", "from urllib.request import Request, urlopen\n", "\n", "api_url = \"https://world.openfoodfacts.org/api/v2/search\"\n", "\n", "params = urlencode({\n", " \"categories_tags_en\": \"chocolates\",\n", " \"page_size\": 20,\n", " \"fields\": \"code,product_name,brands,nutriscore_grade,nova_group,nutriments\",\n", "})\n", "\n", "request = Request(\n", " f\"{api_url}?{params}\",\n", " headers={\"User-Agent\": \"DALAS-course/1.0 (student exercise)\"},\n", ")\n", "\n", "try:\n", " with urlopen(request) as response:\n", " results = json.load(response)\n", " print(\"Data loaded from OpenFoodFacts.\")\n", "except (HTTPError, URLError) as error:\n", " print(\"OpenFoodFacts is temporarily unavailable. Please run this cell again later.\")\n", " print(error)\n", " results = {\"products\": []}\n", "\n", "print(\"Number of products:\", len(results[\"products\"]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The response is a Python dictionary. The list of products is stored in `results[\"products\"]`.\n", "\n", "Each product is also a dictionary, and nutrition values are nested inside the `nutriments` dictionary.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "products = []\n", "\n", "for product in results[\"products\"]:\n", " nutriments = product.get(\"nutriments\", {})\n", " code = product.get(\"code\")\n", "\n", " products.append({\n", " \"name\": product.get(\"product_name\"),\n", " \"brand\": product.get(\"brands\"),\n", " \"nutriscore\": product.get(\"nutriscore_grade\"),\n", " \"nova_group\": product.get(\"nova_group\"),\n", " \"sugars_100g\": nutriments.get(\"sugars_100g\"),\n", " \"fat_100g\": nutriments.get(\"fat_100g\"),\n", " \"energy_kcal_100g\": nutriments.get(\"energy-kcal_100g\"),\n", " \"url\": f\"https://world.openfoodfacts.org/product/{code}\" if code else None,\n", " })\n", "\n", "columns = [\"name\", \"brand\", \"nutriscore\", \"nova_group\", \"sugars_100g\", \"fat_100g\", \"energy_kcal_100g\", \"url\"]\n", "food = pd.DataFrame(products, columns=columns)\n", "food.head()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can ask a small data question: among the returned products, which ones have the most sugar per 100g?\n" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namebrandsugars_100gfat_100gnutriscoreurl
19Noir 70% Intensecote d or56.046.5ehttps://world.openfoodfacts.org/product/762240...
11Chocolat au laitMilka55.031.0ehttps://world.openfoodfacts.org/product/304514...
16Excellence Noir à la Pointe de Fleur de SelLindt EXCELLENCE47.032.0ehttps://world.openfoodfacts.org/product/304692...
7Nestlé dessert NoirNestlé, NESTLE DESSERT, Tablettes46.434.0ehttps://world.openfoodfacts.org/product/761303...
15chocolat noir noisettes entierescote d\\'or40.039.0ehttps://world.openfoodfacts.org/product/762221...
9Excellence 70% Cocoa Intense DarkLindt30.041.0ehttps://world.openfoodfacts.org/product/304692...
1Noir IntenseLindt30.041.0ehttps://world.openfoodfacts.org/product/304692...
13Buo Cioccolato Fondente 70%Lidl Organic28.041.6ehttps://world.openfoodfacts.org/product/40896243
10Cioccolato fondenteJd Gross, chocolat noir27.240.0ehttps://world.openfoodfacts.org/product/20095291
6Extra Dark 74% CocoaFin Carré26.042.0ehttps://world.openfoodfacts.org/product/20022464
\n", "
" ], "text/plain": [ " name \\\n", "19 Noir 70% Intense \n", "11 Chocolat au lait \n", "16 Excellence Noir à la Pointe de Fleur de Sel \n", "7 Nestlé dessert Noir \n", "15 chocolat noir noisettes entieres \n", "9 Excellence 70% Cocoa Intense Dark \n", "1 Noir Intense \n", "13 Buo Cioccolato Fondente 70% \n", "10 Cioccolato fondente \n", "6 Extra Dark 74% Cocoa \n", "\n", " brand sugars_100g fat_100g nutriscore \\\n", "19 cote d or 56.0 46.5 e \n", "11 Milka 55.0 31.0 e \n", "16 Lindt EXCELLENCE 47.0 32.0 e \n", "7 Nestlé, NESTLE DESSERT, Tablettes 46.4 34.0 e \n", "15 cote d\\'or 40.0 39.0 e \n", "9 Lindt 30.0 41.0 e \n", "1 Lindt 30.0 41.0 e \n", "13 Lidl Organic 28.0 41.6 e \n", "10 Jd Gross, chocolat noir 27.2 40.0 e \n", "6 Fin Carré 26.0 42.0 e \n", "\n", " url \n", "19 https://world.openfoodfacts.org/product/762240... \n", "11 https://world.openfoodfacts.org/product/304514... \n", "16 https://world.openfoodfacts.org/product/304692... \n", "7 https://world.openfoodfacts.org/product/761303... \n", "15 https://world.openfoodfacts.org/product/762221... \n", "9 https://world.openfoodfacts.org/product/304692... \n", "1 https://world.openfoodfacts.org/product/304692... \n", "13 https://world.openfoodfacts.org/product/40896243 \n", "10 https://world.openfoodfacts.org/product/20095291 \n", "6 https://world.openfoodfacts.org/product/20022464 " ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sweetest = (\n", " food.dropna(subset=[\"sugars_100g\"])\n", " .sort_values(\"sugars_100g\", ascending=False)\n", " .head(10)\n", ")\n", "\n", "sweetest[[\"name\", \"brand\", \"sugars_100g\", \"fat_100g\", \"nutriscore\", \"url\"]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Questions\n", "\n", "* Change `categories_tags_en` to another category: `breakfast-cereals`, `sodas`, `biscuits`, `ice-creams`, or `plant-based-milks`.\n", "* Change `page_size` to get more products.\n", "* Sort by another column, for example `fat_100g` or `energy_kcal_100g`.\n", "* Add one extra field from the API response to the dataframe.\n", "* Try to find the product that looks like the best snack: not too much sugar, not too much fat, and a decent Nutri-Score.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Going further" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* To get the full power of an API, you often have to read the documentation. For OpenFoodFacts: https://openfoodfacts.github.io/openfoodfacts-server/api/\n", "* Try to use another API. Here is a list of public APIs on many topics: https://github.com/public-apis/public-apis\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "colab": { "authorship_tag": "ABX9TyNG2Abj51Xn36xr00MfsR38", "provenance": [] }, "kernelspec": { "display_name": "dalas (3.13.1)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.1" } }, "nbformat": 4, "nbformat_minor": 0 }