## Accessing Databases via Web APIs --- ## What is an API? * API stands for **Application Programming Interface** * a set of rules and procedures that facilitate interactions between computers and their applications Note: We can consider an API a set of rules and procedures that allow computers to interact with applications ---  Note: One way we can think of an API is like a restaurant. The customer makes a request to the waiter, which delivers the order to the kitchen. The kitchen returns the order via the waiter back to the customer. The customer is the application, the waiter is the API, and the kitchen is the database. --- ## Web APIs * allows users to query a remote database over the internet * take on a variety of formats * majority adhere to a particular style known as **Representational State Transfer** or **REST** * "RESTful" APIs are convenient because we can use them to query databases using URLs Note: A common type of API is a Web API, which allows users to query (or browse) databases over the internet. The majority of Web APIs are in a form known as "Representational State Transfer", or REST - often called RESTful APIs. The main idea behind a RESTful system is that the best way to share large amounts of data between multiple parties is to make that data on-demand by sharing *references* to that data rather than a complete copy of the data itself. We can think of RESTful systems when watching movies at home. An example of a NON-RESTful system is a traditional home movie collection. Results in a lot of waste since more copies exist than are in use at any given moment. Streaming video is the RESTful counterpart to the traditional collection - instead of a complete copy stored in the home, a film is referred to by its title only and content is streamed on-demand. In this case, we're getting a representation of that data (or movie) rather than the actual physical copy of the movie. --- ## RESTful Web APIs are All Around You... Consider a simple Google search. Go ahead and search something. Ever wonder what all that extra stuff in the address bar was all about? Note: When you make a search online, ever wonder what all the characters in the address bar mean? --- ## RESTful Web APIs are All Around You... It looks like Google makes its query by taking the search terms, separating each of them with a "`+`", and appending them to the link: `https://www.google.com/#q=` So that we have `https://www.google.com/#q=search1+search2` So, we can change our Google search by adding some terms to the URL. Note: One very common example of a RESTful Web API is a Google search. For example, a Google search might look like 'google.com/#q=' followed by some search terms added to the end . --- ## Some Basic Terminology * Uniform Resource Location (URL) * Hypertext Transfer Protocol (HTTP) Note: Let's go over some common terms/jargon we'll encounter. 'URL' stands for Uniform Resource Location and is a string of characters. HTTP (which you have seen at beginning of URLs) is used to send and receive webpages and files on the internet. When HTTP interprets URLS, it points to a data source, such as files written in HTML (such as a web page). --- ## HTTP Methods / Verbs - GET Requests * *GET*: requests a representation of a data resource corresponding to a particular URL. The process of executing the GET method is often referred to as a "GET request" and is the main method used for querying RESTful databases. Note: Here are some additional terms we'll see. HTTP has methods, or various actions we can take. GET is a method that requests a representation of a data resource corresponding to a URL. A "GET request" is the main method to query RESTful databases. Other common methods include: HEAD, POST, PUT, DELETE --- ## How Do GET Requests Work? * Surfing the Web = Making a bunch of GET Requests * For instance, I open my web browser and type in http://www.wikipedia.org. Once I hit return, I'd see a webpage. * Several different processes occured, however, between me hitting "return" and the page finally being rendered. Note: One example of GET requests is surfing the web. When you're browsing the internet, you're making a bunch of GET requests. However, between hitting 'return' and page loading, several different processes occurred. Now I'll turn it over to Tim. --- ### RESTful Database Querying: Request & Response * To construct a GET request, we supply a URL in a format such that the result can be interpreted by the server. * The URL contains **search terms and/or filtering parameters**. * The server responds to our request in a format we can parse to extract data we want to keep. * Response formats include `.csv`, `.xml`, and `.json` files. Note: The server responds to our request in a format we can parse to extract data we want to keep --- ## The Question ### How Much More or Less Popular is Duke Ellington? Ellington was at peak popularity from roughly 1930 to 1945. **YOUR CHALLENGE:** Characterize the popularity of Duke Ellington over the past 5-10 years. Specifically, is he "trending"? Note: Tim Here's the question we'll be exploring today: How much/less popular is Duke Ellington according to the NYT? How will we find out? --- ### STEP 1: Finding Data Resources * Popularity = How frequently or widely something is referenced over time * Popularity = Frequency in Newspapers * Popularity = Frequency in New York Times [NYT Article API](http://developer.nytimes.com/) Note: Step 1 - let's find some data sources. Popularity can be considered 'how frequently/widely something is referenced over time'. One way to measure popularity is its frequency in newspapers, like the NYT. A way for us to find this information is using the NYT Article API. --- ### STEP 2: Getting API Access * For most APIs, a key or other user credentials are required * Most APIs are set up for developers, so you’ll likely be asked to register an "application" * rate limits = total number of calls / time * NYT API limits: 4,000 requests per day and 10 requests per minute. You should **sleep 6 seconds** between calls to avoid hitting the per minute rate limit. [NYT Article API Keys](http://developer.nytimes.com/apps/mykeys) --- ### STEP 3: Constructing API GET Request Most GET request URLs for API querying have three or four components: 1. **Base URL**: a link that will be at the beginning of all calls 2. **Authenication Key/Token** 3. **Response Format**: indicating how the response should be formatted 4. **Search Parameters**: a character string telling the server what to extract from the database Note: Now that we have API access, let's construct API GET requests. 1. **Base URL**: a link that will be at the beginning of all calls 2. **Authenication Key/Token**: a user-specific character string 3. **Response Format**: a character string indicating how the response should be formatted; usually one of .csv, .json, or .xml 4. **Search Parameters**: a character string telling the server what to extract from the database; basically a series of filters used to point to specific parts of a database We'll see specific examples of this later today. --- ### STEP 4: Constructing API GET Request * APIs have common architectures, but each API has its own unique quirks. * Carefully reviewing the API documentation is critical!! * Fortunately, the NYT Article API is [very well documented!](https://developer.nytimes.com/docs/articlesearch-product/1/routes/articlesearch.json/get) * NYT Article API has an interactive form to test out the API Note: Last but not least: * APIs are built similarly, but each API has its own quirks. * We highly reading the documentation for the API you're using. * Why not run the API in the browser test function? We often want to run large amounts of queries. We also cannot analyze the date in the browser, but we can in Python in our notebook. --- ## Try it Out https://developer.nytimes.com/apis ---