Tim Berners Lee, the leader of a group at CERN that developed http, looking fondly upon his creation at work.

What is http?

Hypertext Transfer Protocol(http) is a communications protocol used to facilitate the retrieval of data (a resource in http speak) from a server by a client over a network. A client sends a request for data over the network to a specified web address (a uniform resource locator or url). The server, listening for requests over the network, receives the request, processes it…maybe it retrieves an html file or grabs some records from a database, then it generates a response, and sends that response back to the client over the network.

It is important to note that http does not facilitate bidirectional communication. A client always initiates communication with the server. Servers do not send requests to clients and clients don’t listen for requests and generate responses.

What is a client?

A client is a software application that makes http requests and receives http responses. The first http client was a crude web browser that was released in the early 1990s with the birth of the web. Today web browsers such as Chrome and Firefox are the most common http client you interact with. However, clients can also be programs running on computers in a data center like the web crawlers google uses to map the web for search.

Screenshot of first web browser developed Tim Berners-Lee at CERN (from CERN website).

What is a server?

A server is a software application that listens for http requests and generates http responses. This software can be run on high performance computers in a data center owned by a cloud provider like AWS, like those serving you Medium’s content. You can also run an http server on your laptop using Python3’s http.server or a mini computer like raspberry pi. Apache and Nginx are two of the most popular server software packages today.

What kinds of data do clients request from servers?

Pretty much anything you can imagine…audio, video, image, executable code, text data, etc. Since a client is making a request for data having only the url, it does not know what form the data will take. However, the response returned by the server contains metadata with this information—the ‘content-type’. This metadata is returned as a key/value pair with content-type = a media type or MIME type. Media types are an international standard used to communicate the type of data returned by a server. Here are some examples:

'Content-Type': 'text/html'  
'Content-Type': 'image/jpeg'  
'Content-Type': 'video/mp4'

Why is the data I’m requesting using http called a resource?

The term ‘resource’ or ‘web resource’ is intentionally abstract so that it can represent all the flavors of data you can retrieve over the web using http. A resource is basically anything on the web that is identifiable or has a web address. A resource could be a document, javascript file, or some service that provides stock prices.

The term resource also contains a notion of dynamism. A resource is something that can be modified over time while keeping its same address. (The opposite would be a version control system where each time a document changes, it gets a new address to preserve history). Resources are not static, they represent the state of the object you are requesting at that point in time.

What does the code for an http request look like?

I’m typing hooli.com into my Chrome address bar, then opening devtools (command+option+i on mac) and navigating to the networking tab. I’ll click on the first name, hooli.com, that appears in the left hand ‘Name’ panel and inspect the request headers in the right panel, clicking view source to see the raw code.

Here is what that looks like.

Here is the raw code I see:

GET / HTTP/1.1  
Host: hooli.com   
Connection: keep-alive   
Pragma: no-cache   
Cache-Control: no-cache   
Upgrade-Insecure-Requests: 1   
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_15\_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36   
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,\*/\*;q=0.8,application/signed-exchange;v=b3;q=0.9   
Accept-Encoding: gzip, deflate   
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7   
Cookie: countryCode=US; \_ga=GA1.2.1557595774.1590521338; \_gcl\_au=1.1.392376046.1590521338; s\_sess=%20s\_cc%3Dtrue%3B%20s\_ria%3Dflash%2520not%2520detected%257C%3B; \_gid=GA1.2.1229338683.1590632937; OptanonConsent=isIABGlobal=false&datestamp=Wed+May+27+2020+22%3A28%3A57+GMT-0400+(Eastern+Daylight+Time)&version=5.9.0&landingPath=http%3A%2F%2Fhooli.com%2F&groups=pfv%3A1%2Cadv%3A1%2CBG150%3A1%2Cpf%3A1%2Cad%3A1%2Creq%3A1&hosts=&consentId=bd161554-d1a8-4843-8273-36f2fc6756b4&interactionCount=0

What is all of this?

An http request can be separated into three blocks. The first block is a single line containing the request method and its parameters for the resource path and version of http. The second block contains the http headers of the request. Headers are key value pairs passed along with the request or response and contain metadata that are used to facilitate the communication exchange. The third block is called the body and contains the actual data retrieved (or passed to the server) in the communication exchange. Since we aren’t passing any data to the server in our request, there is nothing in the body.

Let’s go through block by block, line by line and look at this request.

GET / HTTP/1.1

This single line is the first block. GET is the http request method I’m using when making the request, ‘/’ tells you I’m asking for the home page of hooli.com/ and ‘HTTP/1.1’ indicates we are using http version 1.1 when making the request.

But what exactly is an http request method?

An http request method defines the way the client wishes to interact with the resource it is requesting from the server. Below are some of the most common methods.

• GET tells the server you want data for the specified resource and that is all. This is the method you use most frequently when clicking on links and surfing the web.
• POST tells the server you wish to submit data to the specified resource and will result in a server side change in state. POST is used to post a photo or make a reservation for a hotel room on a booking site.
• PUT tells the server you wish to replace the current state or representation of the resource with data you pass in the request. This is the method used when you are updating something like your user profile in a web app.
• DELETE tells the server you wish to delete the resource you are requesting. An example would be deleting a photo you posted previously.

Let’s continue with our line by line analysis with the headers in the second block of the request…

Host: hooli.com

The host header is required when using http 1.1, and provides the host name and port number of the server we are trying to reach. Since the port number is absent, it is assumed to be 80, the default port for http.

Connection: keep-alive

The connection header has two possible values: keep-alive and close. These values determine whether the TCP connection between client and server will be persistent, so that multiple messages can be sent over the connection, or single-use and closed after the server responses to the clients request.

Pragma: no-cache  
Cache-Control: no-cache

These two headers are doing similar things, so let’s look at them together. In any http communication over a network, there can be many things between a client and a server that pass on or interact with the request / response message— these things are give the general term proxies. One type of proxy is a cache. A cache, sitting in between you and the server holding the resource you want, can store the resource you are requesting and serve it to you instead of the host server.

Both of these headers, pragma and cache-control, are used to tell caches that interact with the request they can server this request with the appropriate cached resource, but only after the request has been validated by the origin server (the server that actually hosts the resource). This means the request will be sent to the host server each time, but the data/body of the response will be fill by the cache. There are two headers present because pragma allows backwards compatibility with caches using http 1.0. Cache-control has a number of other possible values that control cache behavior in different ways.

Upgrade-Insecure-Requests: 1

This header has to do with the insecure nature of http. Http communication is completely unencrypted, which means anyone can intercept and read requests and responses. To prevent this, modern websites have upgraded to https and web browsers may block or warn you before allowing you to make requests with http. However, the fact that a single web page may hold content or links stored on different hosts presents a problem for websites looking to upgrade to https. It is not always easy to upgrade every link on a page that uses http to https because there may be thousands across a large site. So, what this headers does is states the client’s preference to use https and its ability to support the protocol. If there is a secure version of the resource you are requesting (i.e. even though the link hasn’t been updated the site hosting the resource has upgraded to use https), your request will be made with https. If there is no secure version, I’m not exactly sure what happens. Perhaps the response will be block or a warning will be displayed (this post is helpful it getting a sense of this works.)

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_15\_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36

This header identifies your user agent to the server fulfilling your request and any proxies. A user agent string contains things that will be familiar like the type of computer you are using, your operating system and browser. There are other entities named in the string that are beyond the scope of this post and are there for historical and compatibility reasons. The user agent is helpful because it can help the server determine which version of the requested resource is best suited to the hardware and software you are running—this is called ‘content negotiation’. In addition, user agents can be used to identify and block unwanted traffic, such as bot traffic (you may not want your website to be crawled and indexed by google).

Accept: text/html,  
        application/xhtml+xml,  
        application/xml;q=0.9,  
        image/webp,  
        image/apng,  
        \*/\*;q=0.8,  
        application/signed-exchange;v=b3;q=0.9   
Accept-Encoding: gzip, deflate   
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7

These related headers all deal with what kinds of data the client is willing to accept from the server and are useful in content negotiation. The accept header lets the server know what kind of content-types (using standardized media types…some having additional parameters) it will take and the server uses this information to send data in one of these accepted formats. The browser populates this header with different values depending on the type of request being made. The accept-encoding header is similar except that it deals with compression algorithms and what the client will accept. The accept-language is self-explanatory. I’ve told Chrome not to translate Mandarin for me in the past and I also have a Mandarin input method on my mac. This is likely the reason Chrome makes it clear I’m willing to accept English or Mandarin data.

Cookie:   
countryCode=US;   
\_ga=GA1.2.1557595774.1590521338;  
\_gcl\_au=1.1.392376046.1590521338; s\_sess=%20s\_cc%3Dtrue%3B%20s\_ria%3Dflash%2520not%2520detected%257C%3B;   
\_gid=GA1.2.1229338683.1590632937;   
OptanonConsent=isIABGlobal=false&datestamp=Wed+May+27+2020+22%3A28%3A57+GMT-0400+(Eastern+Daylight+Time)&version=5.9.0&landingPath=http%3A%2F%2Fhooli.com%2F&groups=pfv%3A1%2Cadv%3A1%2CBG150%3A1%2Cpf%3A1%2Cad%3A1%2Creq%3A1&hosts=&consentId=bd161554-d1a8-4843-8273-36f2fc6756b4&interactionCount=0

The last header in the second block of my http request is the cookie header, which provides a list of cookies to the server. In this request there are 6 cookies. Cookies are set by a server using the set-cookie header and then sent back to the server in subsequent requests to help with session management (allow you to remain logged into your account on a website or keep items in a shopping cart), personalization, and tracking. In this case, the countyCode cookie is tells the site when I’m located. The _ga, _gcl_au, _gid cookies are all used for tracking or advertising by Google . OptanonConsent is a cookie that records what cookies the site uses and whether a user has consented to their use.

What does the code for an http response look like?

HTTP/1.1 200 OK   
Content-Type: text/html   
Transfer-Encoding: chunked   
Connection: keep-alive   
Last-Modified: Thu, 19 Dec 2019 21:30:35 GMT   
x-amz-meta-s3cmd-attrs: md5:9eec834ce56eae7000d8258fe189b910  
Server: AmazonS3   
CloudFront-Viewer-Country: US   
Content-Encoding: gzip   
Date: Wed, 27 May 2020 11:59:22 GMT   
Vary: Accept-Encoding   
X-Cache: Hit from cloudfront   
Via: 1.1 3b1807627d3f1dc0cdeb157fc313627b.cloudfront.net (CloudFront)   
X-Amz-Cf-Pop: EWR52-C1   
X-Amz-Cf-Id: KFv4quFUhn0CT63hncQHUhwtmMFkMnTZZ6aA4tTFkhZ2SMlM7jDq6g==   
Age: 52626

The structure of a response is almost identical to that of a request. The first block contains the status line, which provides the protocol version and a status code and status text explaining the code for informational purposes. In this case the request was successful, so a 200 status code was returned. The second block contains response headers. The third block is the response body. The third block is not included above because it is a 375 line html file.

Let’s analyze each new header…

Content-Type: text/html

As mentioned above, the content-type header tells use the media type of the data in the response body.

Transfer-Encoding: chunked

This header states what encoding was used to transfer the body to the client. It is set by each node in the connection between client and server, so the value can change as the message makes its way to the client. In this case, the body of the response was large so it was sent in chunks.

Last-Modified: Thu, 19 Dec 2019 21:30:35 GMT

The date the resource was last modified.

x-amz-meta-s3cmd-attrs: md5:9eec834ce56eae7000d8258fe189b910

This is an AWS S3 specific header. What it does it out of the scope of this post, but it tells us the html file is stored in s3.

Server: AmazonS3

Our suspicions are confirmed. This header provides information about the software used on the server hosting out requested resource.

CloudFront-Viewer-Country: US

CloudFront is a content distribution service provided by AWS. This seems to provide the server with some information about where the request is made from. This information may be used to modify the response. And it tells us the site we are interacting with uses caching to improve performance.

Date: Wed, 27 May 2020 11:59:22 GMT

Date the http response message was created.

Vary: Accept-Encoding

This header is meant for the caches sitting between the client and server. It tells a cache when it can serve cached responses for matching requests and when they must forward on requests to the origin server. The value informs the cache which request header it should use to determine if an appropriate cache response is available. With this header the server is telling the cache it can serve cached responses if the accept-encoding header in the request matches the content-encoding of the cached response.

X-Cache: Hit from cloudfront   
Via: 1.1 3b1807627d3f1dc0cdeb157fc313627b.cloudfront.net (CloudFront)   
X-Amz-Cf-Pop: EWR52-C1   
X-Amz-Cf-Id: KFv4quFUhn0CT63hncQHUhwtmMFkMnTZZ6aA4tTFkhZ2SMlM7jDq6g==  
Age: 52626

All of these headers have to do with proxies / caching, so let’s look at them together. When a header is prefixed with an ‘X’ that means it is a non-standard proprietary header. X-Cache indicates whether the request was served from a cache or not (hit for yes and miss for no) and is followed by the name of the caching service. In this case, our response was served from a CloudFront cache.

Via is used for tracking message forwarding and it takes value takes the form of protocol version used by the sender of the request, host, and alias here. So the protocol is http 1.1 and the host is some CloudFront server. So it seems even though our response was served from the cache, our request was still forwarded to the origin server by this CloudFront server. That makes sense since the Cache-Control header in our request had a value of no-cache meaning the CloudFront proxy still needed to forward our request to the origin server for validation.

X-Amz-Cf-Pop is another proprietary header that tells us from which point-of-presence (pop) or physical server location a response is sent from. Pretty cool. EWR52-C1 is the code used for Newark Liberty International Airport in New Jersey, not far from my apartment in Brooklyn.

X-Amz-Cf-Id is a header CloudFront adds to the request before forwarding it to the origin server. The value uniquely identifies the request.

Age is a standard response header that tells the client how long in seconds the resource has been stored in the cache. In this case, the html file for the hooli homepage I requested has been in the CloudFront cache for around 14.5 hours.

What does it mean that http is an application protocol?

The layers of software that make the internet hum are organized into 4 different levels in the Internet Protocol Suite (aka TCP/IP model). The top level of the suite is the application layer. The transport and internet layers follow, with the link layer at the bottom. Communications protocols like http belong to the application layer because they facilitate process to process or application to application communication. The software in lower layers of the suite deals with facilitating communication with entities that are at a lower level of abstraction than applications.

The layers of software in the Internet Protocol Suite that make up the internet.

Where did http come from?

Http was developed at CERN by Tim Berners-Lee and his colleagues and announced to the world along side html and httpd (the first server implementation) in August, 1991.

Why is it called Hypertext Transfer Protocol?

Hypertext refers to documents that contain links (hyperlinks)to other documents. The prefix ‘hyper’ is used in words like hypertext, hyperlink and hypermedia to convey extension and generality in the mathematical sense of the word, according to wikipedia. Hypertext documents can be almost infinitely extended as you follow links into other realms of knowledge on the web. Hypertext Transfer Protocol is what allows one to link or transfer from one hypertext document to the next.

The idea for hypertext was not introduced in the 1990s, but rather was dreamed up by Vannevar Bush when he wrote about a fictional machine that would use associative indexing to retrieve documents in 1945.

What does it mean that http is a stateless protocol?

Http is stateless because each request/response communication is self-enclosed and shares no link to other request/response communications on the same connection between the same client and server. Because servers don’t need to track multiple messages in a conversation with specific clients, they are able to more easily scale and handle many requests. Instead, cookies can be passed back and forth between client and server to allow for stateful sessions.

Corrections: As always, if you find any inaccuracies in my post, please comment below with corrections.