# Introduction Introduction **What is a Cloud Native Network Function (CNF)?** In order to talk about CNFs, we need to define [cloud native](/cloud-native-principles) \[1]. Cloud native systems are, among other things, a set of loosely coupled services. These services, also known as microservices, are deployed onto immutable infrastructure while being managed by an orchestrator. This paper includes four links to other papers that detail the definitions of cloud native, microservices, immutable infrastructure, and CNFs from an OSI layer perspective. **How are** [**cloud native**](/cloud-native-principles) **systems loosely coupled?** Cloud native systems have a clear separation between their processes \[2]. They utilize the Unix philosophy of doing one thing and doing it well. These microservices usually use technologies such as containers and aim for one process per container \[3]. As such, cloud native applications should have all of their dependencies packaged in the container during the build phase and leveraged during deployment \[4]. **How is** [**immutable infrastructure**](/cloud-native-immutable-infrastructure-principles) **provisioned?** [Immutable infrastructure](/cloud-native-immutable-infrastructure-principles) (the orchestrator and all of the software and hardware that it depends on) is provisioned using **baked** and **versioned** **templates** \[5] (e.g. server images produced from packer \[10]) or a combination of templates and **bootstrapping** \[6] (some repeatable and versioned process that is applied to the template e.g. Kubeadm). The underlying infrastructure is not changed after it is made ready for use. New changes to the infrastructure are rolled out as new instances of infrastructure. **How are cloud native systems deployed?** Cloud native applications are deployed onto immutable infrastructure (generic host servers that support orchestration \[7]). Cloud native applications have not changed after deployment. New features for an application are rolled out as new artifacts and configuration (e.g., containers) **How are cloud native systems configured?** Cloud native systems are configured declaratively \[8]. This means that the system configuration declares "what" a loosely coupled system should look like, not "how" it should be created, updated, or deleted. The tooling determines the "how" of the application (e.g., the orchestrator, operators, and CRDs). **So what is a CNF, actually?** A CNF is network functionality delivered in software via cloud native development and delivery practices. This functionality lives within the layers of the [OSI Model](/cloud-native-declarative-osi-principles) \[9], which is used to define a network's stack. The lower layers (layers 1 and in some cases, layer 2) are provisioned for the higher layers (2-7) to provide transport. These higher layers in this instance act as applications that act upon a network payload (frames, packets datagrams etc). A physical layer 1 networking device should be "flashed" with a complete replacement of its artifacts for updates. The configuration for physical layer 1 is done with an atomic application of a versioned configuration file, which replaces the configuration on the device at once. Virtual Layer 1 (and some layer 2) is managed via templated images and bootstrapping. In contrast, layers 2 through 7 are managed by higher-level orchestration or an established control plane (an orchestrator pushing configuration versus a network protocol modifying a route table). **Why is this relevant to Service Providers?** Service providers currently find themselves at a unique transition point within the industry. Their push towards normalization within the world of NFV has finally begun to bear fruit, yet cloud native software approaches are already being pushed by a plethora of vendors, each with their unique approach. Providers now find themselves in a situation where they must find ways to achieve a return on their investment into NFV while also managing the industry's paradigm shift with regards to software development. **LICENSE** \ This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). You are free to: * Share — copy and redistribute the material in any medium or format * Adapt — remix, transform, and build upon the material for any purpose, even commercially. Under the following terms: * Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. * No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications Bill Mulligan Loodse Daniel Bernier Bell Canada ## Endnotes > 1. "CNCF Cloud Native Definition v1.0", TOC: 2018-06-11, , **Cloud** **native** technologies empower organizations to build and run scalable applications in modern, **dynamic** environments such as public, private, and hybrid clouds. **Containers, service meshes**, **microservices**, **immutable infrastructure,** and **declarative APIs** exemplify this approach. These techniques enable loosely coupled systems that are **resilient**, **manageable**, and **observable**. Combined with robust **automation**, they allow engineers to make high-impact changes frequently and predictably with minimal toil. > 2. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 10–11 > > “**Codebase** \[...] Each deployable app is **tracked** as one codebase tracked in **revision** control. It may have many deployed instances across multiple environments. > > **Dependencies** \[...] An app explicitly declares and **isolates dependencie**s via appropriate tooling (e.g., Maven, Bundler, NPM) rather than depending on implicitly realized dependencies in its deployment environment. > > **Config** \[...] Configuration, or **anything** that is likely to **differ** between deployment **environments** (e.g., development, staging, production) is **injected** via operating system-level **environment** **variables**. > > **Backing services** \[...] Backing services, such as **databases** or message brokers, are treated as **attached resources** and consumed **identically** across all environments. > > **Build, release, run** \[...] The **stages** of building a **deployable** app artifact, **combining** that **artifact** with **configuration**, and **starting** one or more **processes** from that artifact/configuration combination, are strictly **separated**. > > **Processes** \[...] The app executes as one or more **stateless** **processes** (e.g., master/workers) that **share** **nothing**. Any necessary state is externalized to **backing** **services** (cache, object store, etc.). > > **Port binding** \[...] The app is self-contained and **exports** any/all **services** via **port binding** (including HTTP). > > **Concurrency** \[...] Concurrency is usually accomplished by **scaling out app processes horizontally** (though processes may also multiplex work via internally managed threads if desired). > > **Disposability** \[...] Robustness is maximized via **processes** that **start up** quickly and **shut down gracefully**. These aspects allow for **rapid elastic scaling**, deployment of changes, and **recovery** from crashes. > > **Dev/prod parity** \[...] Continuous delivery and deployment are enabled by **keeping** **development**, **staging**, and **production** environments as **similar** as possible. > > **Logs** Rather than managing logfiles, **treat logs as event streams**, allowing the execution environment to **collect**, **aggregate**, **index**, and **analyze** the **events** via **centralized** services. > > **Admin processes** \[...] Administrative or **management tasks**, such as database migrations, are executed as **one-off processes** in environments identical to the app’s long-running processes.” > 3. The best way to think of a **container** is as a **method** to **package** a **service**, application, or job. It's an RPM on steroids, taking the application and adding in its dependencies, as well as providing a standard way for its **host** system to **manage** its **runtime** environment . Rather than a single container running multiple processes, aim for **multiple** **containers**, each running **one** **process**. These processes then become **independent**, **loosely** **coupled** entities. This makes containers a nice match for microservice application architectures. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1708-1711). O'Reilly Media. Kindle Edition. > 4. The benefits of **decoupling** **runtime** **requirements** from the **host** **system** are particularly powerful for infrastructure management. It creates a clean **separation** of concerns between **infrastructure** and **applications**. The host system **only** needs to have the **container** **runtime** **software** installed, and then it can run nearly any container image. Applications, services, and jobs are packaged into containers along with all of their dependencies \[...]. These dependencies can include operating system packages, language runtimes, libraries, and system files. **Different** **containers** may have different, even **conflicting** **dependencies**, but still run on the **same** **host** without issues. **Changes** to the **dependencies** can be made **without** any **changes** to the **host** system. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1652-1658). O'Reilly Media. Kindle Edition. > 5. The **immutable server pattern** mentioned in "Server Change Management Models" **doesn't make configuration updates to existing servers**. Instead, changes are made by **building a new server** with the new configuration. With **immutable servers**, **configuration** is **usually** **baked** into the **server template**. When the configuration is updated, a new template is **packaged**. **New instances** of **existing servers** are built from the **new template** and used to **replace** the **older servers**. This approach **treats** **server templates** like **software artifacts**. Each build is versioned and tested before being deployed for production use. This creates a high level of confidence in the consistency of the server configuration between testing and production. **Advocates** of **immutable server**s view making a **change** to the **configuration** of a **production** **server** as **bad** practice, no better than modifying the source code of software directly on a production server. Immutable servers can also **simplify configuration** management, by **reducing** the area of the server that **needs** to be managed by **definition files**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2239-2247). O'Reilly Media. Kindle Edition. > 6. **Bootstrap Configuration** with **Immutable Servers**: The **purest use** of **immutable servers** is to **bake** everything onto the server **template** and **change nothing**, even when creating server instances from the template. **But some teams have found that for certain types of changes, the turnaround time needed to build a new template is too slow.** **An emerging practice is to put almost everything into the server template, but add one or two elements when bootstrapping a new server.** This might be a **configuration setting that is only known when the server is created, or it might be a frequently changing element such as an application build for testing**. A small development team using continuous integration (CI) or continuous delivery (CD) is likely to deploy dozens of builds of their application a day, so **building a new server template for every build may be unacceptably slow**. Having a **standard server template image** that can **pull** **in** and **start** a **specified** **application** **build** when it is started is particularly useful for **microservices**. This still follows the **immutable** **server** **pattern**, in that **any** **change** to the server’s **configuration** (such as a new version of the microservice) is carried out by **building a new server instance**. It shortens the turnaround time for changes to a microservice, because it **doesn't** **require** building a **new** **server** **template**. However, this practice arguably **weakens** the **testing** **benefits** from the **pure immutable model**. Ideally, a given **combination** of **server** **template** **version** and **microservice** **version** will have been **tested** through each stage of a change management **pipeline**. But there is some **risk** that the process of installing a microservice, or making other changes, when creating a server will behave slightly differently when done for different servers. This could cause unexpected behavior. So **this practice trades some of the consistency benefits of baking everything into a template and using it unchanged in every instance in order to speed up turnaround times for changes made in this way. In many cases, such as those involving frequent changes, this trade-off works quite well.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3099-3116). O'Reilly Media. Kindle Edition. > 7. **Containerized services** works by packaging applications and services in **lightweight containers** (as popularized by Docker). This **reduces coupling** between **server configuration** and the things that **run on** the **servers**. **So host servers tend to be very simple, with a lower rate of change.** One of the other change management **models** still needs to be **applied** to these **hosts**, but their implementation becomes much simpler and easier to maintain. **Most effort and attention goes into packaging, testing, distributing, and orchestrating the services and applications**, but this follows something similar to the immutable infrastructure model, which again is simpler than managing the configuration of full-blown virtual machines and servers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1617-1621). O'Reilly Media. Kindle Edition. > 8. **Declarative** **configuration** is **different** from **imperative** **configuration** , where you simply take a series of actions (e.g., apt-get install foo ) to modify the world. Years of production experience have taught us that maintaining a written **record** of the system's **desired** **state** leads to a more **manageable**, **reliable** system. Declarative configuration enables numerous **advantages**, including **code** **review** for configurations as well as **documenting** the **current** **state** of the world for distributed teams. Additionally, it is the **basis** for all of the **self-healing** behaviors in Kubernetes that keep applications running **without user action.**" Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 892-896). Kindle Edition. > 9. \[...] **OSI reference model** with up to seven layers, where each layer provides a different level of abstraction and performs a set of well-defined functions. These seven layers are as follows. **1. Physical layer:** These protocols employ methods for bit transmission over physical media and include such typical functions as signal processing, timing, and encoding. **2. Data Link Control (DLC) layer:** Its protocols establish point-to-point communication over a physical or logical link, performing such functions as organization of bits in data units (frames) organization, error detection, and flow control. **3. Network layer:** These protocols deliver data units over a network composed of the links established through the DLC protocols of layer 2. Part of these protocols is identification of the route the data units will follow to reach their target. **4. Transport layer:** Transport protocols establish end-to-end communication between end systems over the network defined by a layer 3 protocol. Often, transport layer protocols provide reliability, which refers to complete and correct data transfer between end systems. Reliability can be achieved through mechanisms for end-to-end error detection, retransmissions, and flow control. **5. Session layer:** This layer enables and manages sessions for complete data exchange between end nodes. Sessions may consist of multiple transport layer connections. **6. Presentation layer:** This layer is responsible for the presentation of exchanged data in formats that can be consumed by the application layer. **7. Application layer**: The application layer includes protocols that implement or facilitate end-to-end distributed applications over the network. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (pp. 13-14). Elsevier Science. Kindle Edition. > 10. , “Packer is an open source tool for creating **identical machine images** for **multiple** **platforms** from a single source **configuration**. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, **Packer** is able to use tools like Chef or Puppet to **install** **software** onto the **image.** A ***machine image*** is a single static unit that contains a **pre-configured operating system** and **installed software** which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include [AMIs](https://en.wikipedia.org/wiki/Amazon_Machine_Image) for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.." # Cloud Native Networking Preamble Introduction **What is a Cloud Native Network Function (CNF)?** In order to talk about CNFs, we need to define [cloud native](/cloud-native-principles) \[1]. Cloud native systems are, among other things, a set of loosely coupled services. These services, also known as microservices, are deployed onto immutable infrastructure while being managed by an orchestrator. This paper includes four links to other papers that detail the definitions of cloud native, microservices, immutable infrastructure, and CNFs from an OSI layer perspective. **How are** [**cloud native**](/cloud-native-principles) **systems loosely coupled?** Cloud native systems have a clear separation between their processes \[2]. They utilize the Unix philosophy of doing one thing and doing it well. These microservices usually use technologies such as containers and aim for one process per container \[3]. As such, cloud native applications should have all of their dependencies packaged in the container during the build phase and leveraged during deployment \[4]. **How is** [**immutable infrastructure**](/cloud-native-immutable-infrastructure-principles) **provisioned?** [Immutable infrastructure](/cloud-native-immutable-infrastructure-principles) (the orchestrator and all of the software and hardware that it depends on) is provisioned using **baked** and **versioned** **templates** \[5] (e.g. server images produced from packer \[10]) or a combination of templates and **bootstrapping** \[6] (some repeatable and versioned process that is applied to the template e.g. Kubeadm). The underlying infrastructure is not changed after it is made ready for use. New changes to the infrastructure are rolled out as new instances of infrastructure. **How are cloud native systems deployed?** Cloud native applications are deployed onto immutable infrastructure (generic host servers that support orchestration \[7]). Cloud native applications have not changed after deployment. New features for an application are rolled out as new artifacts and configuration (e.g., containers) **How are cloud native systems configured?** Cloud native systems are configured declaratively \[8]. This means that the system configuration declares "what" a loosely coupled system should look like, not "how" it should be created, updated, or deleted. The tooling determines the "how" of the application (e.g., the orchestrator, operators, and CRDs). **So what is a CNF, actually?** A CNF is network functionality delivered in software via cloud native development and delivery practices. This functionality lives within the layers of the [OSI Model](/cloud-native-declarative-osi-principles) \[9], which is used to define a network's stack. The lower layers (layers 1 and in some cases, layer 2) are provisioned for the higher layers (2-7) to provide transport. These higher layers in this instance act as applications that act upon a network payload (frames, packets datagrams etc). A physical layer 1 networking device should be "flashed" with a complete replacement of its artifacts for updates. The configuration for physical layer 1 is done with an atomic application of a versioned configuration file, which replaces the configuration on the device at once. Virtual Layer 1 (and some layer 2) is managed via templated images and bootstrapping. In contrast, layers 2 through 7 are managed by higher-level orchestration or an established control plane (an orchestrator pushing configuration versus a network protocol modifying a route table). **Why is this relevant to Service Providers?** Service providers currently find themselves at a unique transition point within the industry. Their push towards normalization within the world of NFV has finally begun to bear fruit, yet cloud native software approaches are already being pushed by a plethora of vendors, each with their unique approach. Providers now find themselves in a situation where they must find ways to achieve a return on their investment into NFV while also managing the industry's paradigm shift with regards to software development. **LICENSE** This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). You are free to: * Share — copy and redistribute the material in any medium or format * Adapt — remix, transform, and build upon the material for any purpose, even commercially. Under the following terms: * Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. * No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications Bill Mulligan Loodse Daniel Bernier Bell Canada ## Endnotes > 1. "CNCF Cloud Native Definition v1.0", TOC: 2018-06-11, , **Cloud** **native** technologies empower organizations to build and run scalable applications in modern, **dynamic** environments such as public, private, and hybrid clouds. **Containers, service meshes**, **microservices**, **immutable infrastructure,** and **declarative APIs** exemplify this approach. These techniques enable loosely coupled systems that are **resilient**, **manageable**, and **observable**. Combined with robust **automation**, they allow engineers to make high-impact changes frequently and predictably with minimal toil. > 2. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 10–11 > > “**Codebase** \[...] Each deployable app is **tracked** as one codebase tracked in **revision** control. It may have many deployed instances across multiple environments. > > **Dependencies** \[...] An app explicitly declares and **isolates dependencie**s via appropriate tooling (e.g., Maven, Bundler, NPM) rather than depending on implicitly realized dependencies in its deployment environment. > > **Config** \[...] Configuration, or **anything** that is likely to **differ** between deployment **environments** (e.g., development, staging, production) is **injected** via operating system-level **environment** **variables**. > > **Backing services** \[...] Backing services, such as **databases** or message brokers, are treated as **attached resources** and consumed **identically** across all environments. > > **Build, release, run** \[...] The **stages** of building a **deployable** app artifact, **combining** that **artifact** with **configuration**, and **starting** one or more **processes** from that artifact/configuration combination, are strictly **separated**. > > **Processes** \[...] The app executes as one or more **stateless** **processes** (e.g., master/workers) that **share** **nothing**. Any necessary state is externalized to **backing** **services** (cache, object store, etc.). > > **Port binding** \[...] The app is self-contained and **exports** any/all **services** via **port binding** (including HTTP). > > **Concurrency** \[...] Concurrency is usually accomplished by **scaling out app processes horizontally** (though processes may also multiplex work via internally managed threads if desired). > > **Disposability** \[...] Robustness is maximized via **processes** that **start up** quickly and **shut down gracefully**. These aspects allow for **rapid elastic scaling**, deployment of changes, and **recovery** from crashes. > > **Dev/prod parity** \[...] Continuous delivery and deployment are enabled by **keeping** **development**, **staging**, and **production** environments as **similar** as possible. > > **Logs** Rather than managing logfiles, **treat logs as event streams**, allowing the execution environment to **collect**, **aggregate**, **index**, and **analyze** the **events** via **centralized** services. > > **Admin processes** \[...] Administrative or **management tasks**, such as database migrations, are executed as **one-off processes** in environments identical to the app’s long-running processes.” > 3. The best way to think of a **container** is as a **method** to **package** a **service**, application, or job. It's an RPM on steroids, taking the application and adding in its dependencies, as well as providing a standard way for its **host** system to **manage** its **runtime** environment . Rather than a single container running multiple processes, aim for **multiple** **containers**, each running **one** **process**. These processes then become **independent**, **loosely** **coupled** entities. This makes containers a nice match for microservice application architectures. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1708-1711). O'Reilly Media. Kindle Edition. > 4. The benefits of **decoupling** **runtime** **requirements** from the **host** **system** are particularly powerful for infrastructure management. It creates a clean **separation** of concerns between **infrastructure** and **applications**. The host system **only** needs to have the **container** **runtime** **software** installed, and then it can run nearly any container image. Applications, services, and jobs are packaged into containers along with all of their dependencies \[...]. These dependencies can include operating system packages, language runtimes, libraries, and system files. **Different** **containers** may have different, even **conflicting** **dependencies**, but still run on the **same** **host** without issues. **Changes** to the **dependencies** can be made **without** any **changes** to the **host** system. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1652-1658). O'Reilly Media. Kindle Edition. > 5. The **immutable server pattern** mentioned in "Server Change Management Models" **doesn't make configuration updates to existing servers**. Instead, changes are made by **building a new server** with the new configuration. With **immutable servers**, **configuration** is **usually** **baked** into the **server template**. When the configuration is updated, a new template is **packaged**. **New instances** of **existing servers** are built from the **new template** and used to **replace** the **older servers**. This approach **treats** **server templates** like **software artifacts**. Each build is versioned and tested before being deployed for production use. This creates a high level of confidence in the consistency of the server configuration between testing and production. **Advocates** of **immutable server**s view making a **change** to the **configuration** of a **production** **server** as **bad** practice, no better than modifying the source code of software directly on a production server. Immutable servers can also **simplify configuration** management, by **reducing** the area of the server that **needs** to be managed by **definition files**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2239-2247). O'Reilly Media. Kindle Edition. > 6. **Bootstrap Configuration** with **Immutable Servers**: The **purest use** of **immutable servers** is to **bake** everything onto the server **template** and **change nothing**, even when creating server instances from the template. **But some teams have found that for certain types of changes, the turnaround time needed to build a new template is too slow.** **An emerging practice is to put almost everything into the server template, but add one or two elements when bootstrapping a new server.** This might be a **configuration setting that is only known when the server is created, or it might be a frequently changing element such as an application build for testing**. A small development team using continuous integration (CI) or continuous delivery (CD) is likely to deploy dozens of builds of their application a day, so **building a new server template for every build may be unacceptably slow**. Having a **standard server template image** that can **pull** **in** and **start** a **specified** **application** **build** when it is started is particularly useful for **microservices**. This still follows the **immutable** **server** **pattern**, in that **any** **change** to the server’s **configuration** (such as a new version of the microservice) is carried out by **building a new server instance**. It shortens the turnaround time for changes to a microservice, because it **doesn't** **require** building a **new** **server** **template**. However, this practice arguably **weakens** the **testing** **benefits** from the **pure immutable model**. Ideally, a given **combination** of **server** **template** **version** and **microservice** **version** will have been **tested** through each stage of a change management **pipeline**. But there is some **risk** that the process of installing a microservice, or making other changes, when creating a server will behave slightly differently when done for different servers. This could cause unexpected behavior. So **this practice trades some of the consistency benefits of baking everything into a template and using it unchanged in every instance in order to speed up turnaround times for changes made in this way. In many cases, such as those involving frequent changes, this trade-off works quite well.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3099-3116). O'Reilly Media. Kindle Edition. > 7. **Containerized services** works by packaging applications and services in **lightweight containers** (as popularized by Docker). This **reduces coupling** between **server configuration** and the things that **run on** the **servers**. **So host servers tend to be very simple, with a lower rate of change.** One of the other change management **models** still needs to be **applied** to these **hosts**, but their implementation becomes much simpler and easier to maintain. **Most effort and attention goes into packaging, testing, distributing, and orchestrating the services and applications**, but this follows something similar to the immutable infrastructure model, which again is simpler than managing the configuration of full-blown virtual machines and servers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1617-1621). O'Reilly Media. Kindle Edition. > 8. **Declarative** **configuration** is **different** from **imperative** **configuration** , where you simply take a series of actions (e.g., apt-get install foo ) to modify the world. Years of production experience have taught us that maintaining a written **record** of the system's **desired** **state** leads to a more **manageable**, **reliable** system. Declarative configuration enables numerous **advantages**, including **code** **review** for configurations as well as **documenting** the **current** **state** of the world for distributed teams. Additionally, it is the **basis** for all of the **self-healing** behaviors in Kubernetes that keep applications running **without user action.**" Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 892-896). Kindle Edition. > 9. \[...] **OSI reference model** with up to seven layers, where each layer provides a different level of abstraction and performs a set of well-defined functions. These seven layers are as follows. **1. Physical layer:** These protocols employ methods for bit transmission over physical media and include such typical functions as signal processing, timing, and encoding. **2. Data Link Control (DLC) layer:** Its protocols establish point-to-point communication over a physical or logical link, performing such functions as organization of bits in data units (frames) organization, error detection, and flow control. **3. Network layer:** These protocols deliver data units over a network composed of the links established through the DLC protocols of layer 2. Part of these protocols is identification of the route the data units will follow to reach their target. **4. Transport layer:** Transport protocols establish end-to-end communication between end systems over the network defined by a layer 3 protocol. Often, transport layer protocols provide reliability, which refers to complete and correct data transfer between end systems. Reliability can be achieved through mechanisms for end-to-end error detection, retransmissions, and flow control. **5. Session layer:** This layer enables and manages sessions for complete data exchange between end nodes. Sessions may consist of multiple transport layer connections. **6. Presentation layer:** This layer is responsible for the presentation of exchanged data in formats that can be consumed by the application layer. **7. Application layer**: The application layer includes protocols that implement or facilitate end-to-end distributed applications over the network. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (pp. 13-14). Elsevier Science. Kindle Edition. > 10. , “Packer is an open source tool for creating **identical machine images** for **multiple** **platforms** from a single source **configuration**. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, **Packer** is able to use tools like Chef or Puppet to **install** **software** onto the **image.** A ***machine image*** is a single static unit that contains a **pre-configured operating system** and **installed software** which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include [AMIs](https://en.wikipedia.org/wiki/Amazon_Machine_Image) for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.." # Cloud Native Principles The concepts of immutable infrastructure, declarative apis, and microservices each deserve individual treatment. The following statements are high level generalizations that can help lead towards an understanding of these three concepts before diving deeply into each of them individually. ***P1*** *- If a project is* ***cloud native*** \[1],\[2]*, it uses* ***immutable infrastructure*** \[3], ***declarative apis**, and* ***microservices**.* ***P2*** *- If infrastructure is* ***immutable**, it is easily* ***reproduced*** \[4],\[5]*,* ***consistent*** \[6]*,* ***disposable*** \[7],\[8]*, will have a* ***repeatable*** \[9] ***deployment process**, and will not have configuration or artifacts that are modifiable in place.* The process that implements immutable infrastructure needs to be reproducible (without any needing to ‘think’ each time provisioning occurs) and repeatable (automated). This process also needs to be consistent (infrastructure elements should be identical) and disposable (designed to be easily created, destroyed, replaced, resized, etc). A project’s immutable infrastructure, which can include everything from the physical hardware at the lowests levels up to the platforms that the application is installed on, has its configuration protected from change (not modifiable) after it is deployed into an environment. This configuration is stored in such a way that it can be used to recreate the infrastructure as needed. Furthermore, the process should be idempotent allowing a state to be applied multiple times and the same desired state still being achieved. ***P3*** *- If a project has an efficient and repeatable deployment process, its process is* ***versioned*** \[10], ***automated*** \[11], *and has* ***low overhead/coarse grained packaging*** \[12],\[13],\[14],\[15] The core of cloud native development rests in coarse-grained packaging such as that found in container technologies such as Docker. Any light weight / low overhead technology that satisfies the requirements for low overhead and coarse grained packaging (packaging all of the dependencies together with the application) can satisfy the deployment requirements for cloud native applications. Normal CI/CD best practices apply for the deployment practice itself. ***P4*** *- If a project’s deployment is* ***automated**,* ***configuration*** \[16], ***environment*** \[17], *and* ***artifacts*** \[18] *are completely managed by a* \_**pipeline**.\_ ***P5*** *- If a projects deployment is managed completely by a* ***pipeline**, the project’s* ***environment*** *is* ***protected*** \[19] Production environments should be only directly modified by the automated pipeline process and therefore not *directly* modifiable by anyone. This protects against snowflake configuration. ***P6*** *- If a project’s environment is protected, it provides* ***observability*** \[21] *of the project’s internal components.* In order to maintain, debug, and have insight into a protected environment, its infrastructure elements must have the property of being observable. This means these elements must externalize their internal states in some way that lends itself to metrics, tracing, and logging. ***P7*** *- If a project's uses* ***declarative APIs*** \[22], *its* ***configuration*** *is* ***declarative*** \[23],\[24] ***P8*** *- If a project’s configuration is* ***declarative*** \[25], *it designates* ***what*** *to do,* ***not how*** *to do it.* Declarative APIs for an immutable infrastructure are anything that configures the infrastructure element. This declaration can come in the form of a YAML file or a script, as long as the configuration designates the desired outcome, not how to achieve said outcome. ***P9*** *- If a project exists as a* ***microservice*** \[26],\[28],\[29]*, it is* ***not monolithic**, it is* ***resilient**, it follows* ***12-factor principles*** \[30], *and is* ***discoverable*** \[31]. When a service is monolithic, multiple business capabilities are tightly coupled, therefore requiring coordination with multiple groups within the organization that are developing the service. A microservice separates concerns based on business capability (features or groups of features). This allows for a more rapid deployment of services with a faster feedback loop. ***P10*** *- If a microservice is* ***resilient**, it is* ***self-healing*** *and* ***distributed*** \[32]. A microservice is also resilient, in that it is accompanied by some kind of strategy for healing itself. This includes strategies for restarting after failures and distributive scaling in response to load. A microservice scales out to handle load (more processes are spawned on more machines) instead of scaling up (increasing the capacity of the individual machines) ***P11*** *- If a microservice is* ***self-healing*** \[33], *it is compatible with* ***declarative configuration*** *and orchestration* \[34]*.* Once a microservice is coupled with a declarative strategy (a strategy that outlines what the system should look like), it can then be handed over to an orchestrator in order to implement that strategy. **LICENSE** This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications Bill Mulligan Loodse ## Endnotes 1. “**Cloud** **native** technologies empower organizations to build and run scalable applications in modern, **dynamic** environments such as public, private, and hybrid clouds. **Containers, service meshes**, **microservices**, **immutable infrastructure,** and **declarative APIs** exemplify this approach. These techniques enable loosely coupled systems that are **resilient**, **manageable**, and **observable**. Combined with robust **automation**, they allow engineers to make high-impact changes frequently and predictably with minimal toil. 2. 3. “**Immutable infrastructure** makes configuration changes by **completely** **replacing** **servers**. Changes are made by **building new server templates**, and then rebuilding relevant servers using those templates. This increases predictability, as there is **little** **variance** between servers as **tested**, and servers in **production**. It requires sophistication in **server template management**.” Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1611-1614). O'Reilly Media. Kindle Edition. 4. It should be possible to **effortlessly** and reliably rebuild any element of an infrastructure. Effortlessly means that there is **no need to make any significant decisions** about **how** to **rebuild** the thing. Decisions about which software and versions to install on a server, how to choose a hostname, and so on should be captured in the scripts and tooling that provision it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 349-352). O'Reilly Media. Kindle Edition. 5. When problems are discovered, fixes may not be rolled out to all of the systems that could be affected by them. Differences in versions and configurations across servers mean that software and scripts that work on some machines don’t work on others. This leads to **inconsistency** across the **servers**, called **configuration drift**. \[...] Even when servers are initially created and configured consistently, **differences** can creep in **over time**: \[...]. But **variations should be captured and managed in a way that makes it easy to reproduce and to rebuild servers and services.** Unmanaged variation between servers leads to **snowflake servers** and automation **fear**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 278-290). O'Reilly Media. Kindle Edition. . 6. Given **two infrastructure elements** providing a **similar service** for example, two application servers in a cluster the servers should be nearly **identical**. Their system software and configuration should be the same, except for those **bits** of **configuration** that differentiate them, like their **IP addresses**. Letting inconsistencies slip into an infrastructure keeps you from being able to trust your automation. If one file server has an 80 GB partition, while another has 100 GB, and a third has 200 GB, then you can’t rely on an action to work the same on all of them. This encourages doing special things for servers that don’t quite match, which leads to **unreliable** **automation**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 380-384). O'Reilly Media. Kindle Edition. 7. One of the **benefits** of **dynamic infrastructure** is that **resources** can be easily **created, destroyed, replaced, resized, and moved**. In order to take advantage of this, systems should be designed to **assume** that the infrastructure will **always** be **changing**. **Software** should **continue running** even when **servers** **disappear**, appear, and when they are resized. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 357-359). O'Reilly Media. Kindle Edition. 8. A popular expression is to “**treat your servers like cattle, not pets**.” ,Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 362-363). O'Reilly Media. Kindle Edition. 9. Building on the **reproducibility** principle, any action you carry out on your infrastructure should be **repeatable**. This is an obvious benefit of **using scripts and configuration management tools** **rather than** making changes **manually**, but it can be hard to stick to doing things this way, especially for experienced system administrators. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 393-395). O'Reilly Media. Kindle Edition. 10. “**Everything** you need to **build**, **deploy**, **test**, and **release** your application should be kept in some form of **versioned** storage. This includes requirement documents, test scripts, automated test cases, network configuration scripts, deployment scripts, database creation, upgrade, downgrade, and initialization scripts, application stack configuration scripts, libraries, toolchains, technical documentation, and so on. All of this stuff should be version-controlled, and the relevant **version** should be **identifiable** for any given **build**. That is, these change sets should have a single identifier, such as a **build** **number** or a version control changeset number, that references every piece.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 26). Pearson Education. Kindle Edition. 11. “In general, your **build** **process** should be automated up to the point where it needs specific human direction or decision making. This is also true of your **deployment process** and, in fact, your entire software **release process**. Acceptance tests can be automated. Database upgrades and downgrades can be automated too. Even network and firewall configuration can be automated. You should automate as much as you possibly can.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 25). Pearson Education. Kindle Edition. 12. **Containerized services** works by packaging applications and services in **lightweight containers** (as popularized by Docker). This **reduces coupling** between **server configuration** and the things that **run on** the **servers**. **So host servers tend to be very simple, with a lower rate of change.** One of the other change management **models** still needs to be **applied** to these **hosts**, but their implementation becomes much simpler and easier to maintain. **Most effort and attention goes into packaging, testing, distributing, and orchestrating the services and applications**, but this follows something similar to the immutable infrastructure model, which again is simpler than managing the configuration of full-blown virtual machines and servers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1617-1621). O'Reilly Media. Kindle Edition. 13. “The **value** of a **containerization** **system** is that it provides a **standard** **format** for **container** **images** and tools for **building**, **distributing**, and **running** those **images**. Before Docker, teams could isolate running processes using the same operating system features, but Docker and similar tools make the process much simpler.” Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1631-1633). O'Reilly Media. Kindle Edition. 14. “**Configuration** **management** refers to the process by which all **artifacts** relevant to your project, and the **relationships** between them, are **stored**, **retrieved**, uniquely **identified**, and **modified**.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 31). Pearson Education. Kindle Edition. 15. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 25-26. “**Containers** leverage modern **Linux kernel primitives** such as **control** **groups** (cgroups) and **namespaces** to provide similar resource allocation and **isolation** features as those provided by virtual machines with much **less** **overhead** and much greater **portability**.” 16. “... we consider it **bad practice** to **inject** **configuration** **information** at **build** or **packaging** time. This follows from the principle that you should be able to **deploy** the **same** **binaries** to **every environment** so you can ensure that the **thing** that you **release** is the **same** thing that you **tested**. The corollary of this is that anything that **changes** **between deployments** needs to be **captured** as **configuration**, and **not baked** in when the application is **compiled** or **packaged**.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (pp. 41-42). Pearson Education. Kindle Edition. 17. “An **environment** is **all** of the **resources** that your **application** needs to **work** and their **configuration**.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 277). Pearson Education. Kindle Edition. 18. “The key **characteristic** of **binaries** is that you should be able to **copy** them onto a **new** **machine** and, given an appropriately **configured** **environment** and the correct **configuration** for the **application** in that environment, start your application—**without** relying on any part of your **development** **toolchain** being installed on that machine.”Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 134). Pearson Education. Kindle Edition. 19. “**Don’t Make Changes Directly on the Production Environment**: Most downtime in production environments is caused by uncontrolled changes. Production environments should be completely **locked** **down**, so that **only** your **deployment** **pipeline** can make **changes** to it. That includes everything from the **configuration** of the environment to the **applications** deployed on it and their **data**.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 273). Pearson Education. Kindle Edition. 20. \-- 21. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 27–28. “**Visibility**: Our **architectures** must provide us with the **tools** **necessary** to **see** **failure** when it happens. We need the ability to **measure** everything, establish a **profile** for “what’s **normal**,” **detect** **deviations** from the norm (including absolute values and rate of change), and **identify** the **components** contributing to those **deviations**. Feature-rich **metrics**, **monitoring**, **alerting**, and data **visualization** frameworks and tools are at the heart of all cloud-native application architecture”: 22. “**Declarative** **configuration** is **different** from **imperative** **configuration** , where you simply take a series of actions (e.g., apt-get install foo ) to modify the world. Years of production experience have taught us that maintaining a written **record** of the system’s **desired** **state** leads to a more **manageable**, **reliable** system. Declarative configuration enables numerous **advantages**, including **code** **review** for configurations as well as **documenting** the **current** **state** of the world for distributed teams. Additionally, it is the **basis** for all of the **self-healing** behaviors in Kubernetes that keep applications running **without user action.**” Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 892-896). Kindle Edition. 23. “To understand these **two approaches**, consider the task of producing three replicas of a piece of software. With an **imperative** approach, the configuration would say: “**run A, run B, and run C.**” The corresponding **declarative** configuration would be “**replicas equals three**.” Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 181-183). Kindle Edition. 24. “The **combination** of **declarative** **state** stored in a **version** control system and Kubernetes’s ability to make **reality** **match** this declarative **state** makes **rollback** of a change trivially **easy**. It is simply restating the previous declarative state of the system. With **imperative** **systems** this is usually **impossible**, since while the **imperative** **instructions** describe how to get you from point A to point B, they **rarely** **include** the **reverse** instructions that can get you back. “Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 186-190). Kindle Edition. 25. “Because it describes the state of the world, **declarative** **configuration** does **not** have to be **executed** to be **understood**. Its impact is concretely declared. Since the effects of declarative configuration can be understood before they are executed, declarative configuration is far **less error-prone**. Further, the traditional tools of software development, such as **source control, code review, and unit testing**, can be used in **declarative** configuration in ways that are **impossible** for **imperative** instructions. “ Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 183-186). Kindle Edition. 26. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 16.. “As we **decouple** the **business domain** into independently deployable **bounded contexts** of **capabilities**, we also **decouple** the associated **change** **cycles**. As long as the changes are restricted to a single bounded context, and the service continues to **fulfill** its existing **contracts**, those changes can be made and **deployed** **independent** of any **coordination** with the rest of the business. The result is enablement of **more** frequent and rapid **deployments**, allowing for a continuous flow of value.” 27. \-- 28. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 27–28. “**Adoption** of new technology can be **accelerated**. Large **monolithic** application architectures are typically associated with **long-term commitments** to technical **stacks**. These commitments exist to **mitigate** the **risk** of adopting new technology by simply not doing it. Technology **adoption** **mistakes** are more **expensive** in a **monolithic** architecture, as those mistakes can pollute the entire enterprise architecture. If we adopt new technology within the scope of a single monolith, we isolate and **minimize** the **risk** in much the same way that we isolate and minimize the risk of runtime failure.” 29. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 27–28. “**Microservices** offer independent, **efficient** **scaling** of services. **Monolithic** architectures can scale, but **require** us to **scale** **all** **components**, not simply those that are under heavy load. Microservices can be scaled if and only if their associated load requires it.” 30. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 10–11 “***Codebase*** Each deployable app is **tracked** as one codebase tracked in **revision** control. It may have many deployed instances across multiple environments. ***Dependencies*** An app explicitly declares and **isolates dependencie**s via appropriate tooling (e.g., Maven, Bundler, NPM) rather than depending on implicitly realized dependencies in its deployment environment. ***Config*** Configuration, or **anything** that is likely to **differ** between deployment **environments** (e.g., development, staging, production) is **injected** via operating system-level **environment** **variables**. ***Backing services*** Backing services, such as **databases** or message brokers, are treated as **attached resources** and consumed **identically** across all environments. ***Build, release, run*** The **stages** of building a **deployable** app artifact, **combining** that **artifact** with **configuration**, and **starting** one or more **processes** from that artifact/configuration combination, are strictly **separated**. ***Processes*** The app executes as one or more **stateless** **processes** (e.g., master/workers) that **share** **nothing**. Any necessary state is externalized to **backing** **services** (cache, object store, etc.). ***Port binding*** The app is self-contained and **exports** any/all **services** via **port binding** (including HTTP). ***Concurrency*** Concurrency is usually accomplished by **scaling out app processes horizontally** (though processes may also multiplex work via internally managed threads if desired). ***Disposability*** Robustness is maximized via **processes** that **start up** quickly and **shut down gracefully**. These aspects allow for **rapid elastic scaling**, deployment of changes, and **recovery** from crashes. ***Dev/prod parity*** Continuous delivery and deployment are enabled by **keeping** **development**, **staging**, and **production** environments as **similar** as possible. ***Logs*** Rather than managing logfiles, **treat logs as event streams**, allowing the execution environment to **collect**, **aggregate**, **index**, and **analyze** the **events** via **centralized** services. ***Admin processes*** Administrative or **management tasks**, such as database migrations, are executed as **one-off processes** in environments identical to the app’s long-running processes.” 31. “**Service discovery** tools help solve the problem of **finding** which **processes** are listening at which **addresses** for which **services**. A good service discovery system will enable users to **resolve** this information **quickly** and **reliably**. A good system is also **low-latency**; clients are updated soon after the information associated with a service change. Finally, a good service discovery system can **store** a **richer** **definition** of what that **service** is. For example, perhaps there are multiple ports associated with the service.” Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 1423-1426). Kindle Edition. 32. “In many cases **decoupling** **state** from **applications** and building your **microservices** to be as **stateless** as possible results in **maximally reliable, manageable systems**. However, nearly **every** **system** that has any complexity has **state** in the system somewhere, from the records in a **database** to the index shards that serve results for a web search engine. At some point you have to **have data stored somewhere. Integrating** this **data** with containers and container orchestration solutions is often the most **complicated** aspect of building a distributed system. This complexity largely stems from the fact that the move to containerized architectures is also a move toward decoupled, immutable, and declarative application development. These patterns are relatively easy to apply to stateless web applications, but even “cloud-native” storage solutions like Cassandra or MongoDB involve some sort of **manual or imperative steps to set up a reliable, replicated solution**. “ Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 2908-2915). Kindle Edition. 33. “A **self-healing** infrastructure is an inherently **smart deployment** that is **automated** to **respond** to known and **common failures**. Depending on the failure, the architecture is inherently **resilient** and takes appropriate measures to **remediate** the error.” Laszewski, Tom. Cloud Native Architectures: Design high-availability and cost-effective applications for the cloud (pp. 131-132). Packt Publishing. Kindle Edition. 34. Container **orchestration** tools have emerged following the rise of containerization systems like Docker. Most of these run agents on a pool of container hosts and are able to **automatically** **select** **hosts** to run new **container** **instances**, **replace** **failed** instances, and **scale** numbers of instances **up** and **down**. Some tools also handle **service** **discovery**, **network** **routing**, **storage**, scheduled **jobs**, and other capabilities. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2063-2066). O'Reilly Media. Kindle Edition. # Cloud Native Microservice Principles ## How cloud native organizations deliver software using pipelines **Cloud Native Microservices Writ Large** ***P1*** *- If an organization or a set of organizations deliver* ***cloud native software**, the software’s features will be delivered as* ***microservice software (a microservice, set of microservices, or component of a microservice)**,* ***declarative APIs**, and* ***immutable infrastructure**.* ***P2*** *- If an organization develops microservice software, the features of the microservice software will be constrained by the* ***organization's business capabilities*** \[1],\[2],\[3],\[4] *and* ***structure*** \[5]. Conway’s law predicts that a system will represent the organizational structure that created that system. The various groups within an organization have different rates of change and concerns with respect to business capability. The delivery of microservice software harnesses the existing group boundaries within an organization and works with, not against, the different rates of change and business capabilities residing within those boundaries. ***P3*** *- If an organization develops microservice software, the* ***responsibility*** *of the microservice software from* ***inception to delivery*** *will be with that organization.* ***P4*** *- If an organization has all of the responsibilities for the microservice software, that organization has the structure of a* ***product team*** \[6]. Microservice teams are product delivery teams. These teams are responsible for all parts of features delivery spanning from requirements gathering to production deployment. This allows the team to deploy based on business capability and to be sensitive to those capabilities’ rate of change. ***P5*** *- If a product team has responsibility for microservice software, the* ***rate of change*** \[7],\[8],\[9],\[10],\[11],\[12]*,* ***cycle time*** \[13],\[14]*, and* ***pipeline*** \[15],\[16],\[17],\[18] *for that microservice software will be driven by that* *product* *team.* Similar to how a building’s components have different **rates of change** (foundation, plumbing, exterior, etc), software components and services also have different rates of change. When we split up services based on business capability the **responsibility** for **changes** and **actual rate of change** are **coupled**. At the same time **conflicting** **agendas**, road maps, and concerns are **decoupled**. When the new freedom given with respect to rate of change microservice teams can then adopt techniques that are sensitive to **cycle time** and **MTTR** (mean time to recovery). This leads to using a software delivery methods and best practices that are compatible with deployment pipelines. ***P6*** *- If the microservice has dependencies, the* ***dependencies*** *constrain the* ***relationship*** *structure between* ***multiple organizations*** \[19],\[20] Organizations and product teams that deliver software require varying levels of coordination with one another. Teams that have higher levels of coordination with other teams need to coordinate deployment pipelines and integration testing. ***P7*** *- If the microservice software has a dependency, it will be delivered from a* ***provider*** \[21] *to a* ***consumer*** *in the form of a* ***library*** \[22],\[23] *or a* ***service instance*** \[24],\[25],\[26] *via a pipeline.* The rate of change between providers of microservice software and consumers of that software needs to be managed. When software is delivered as a library, it has a release number that can be referenced in the pipeline of the consumer. When software is delivered as a service instance, it can either be self service (and therefore can be referenced via release number in the consumer’s pipeline) or it can be hosted. If microservice software is hosted, there needs to be a way to reference a test instance of that service for the consumers pipeline. The license registration process for service instances should be automated, flexible, and should avoid impeding the development of a deployment pipeline. ***P8*** *- If a microservice is deployed, the microservice will be* ***deployed*** *with* ***all of its library dependencies*** \[27],\[28],\[29],\[30],\[31],\[32] The microservice has all of its dependencies deployed with it during the deployment phase of the pipeline. These dependencies are decoupled from the infrastructure environment (e.g. a node) so the rate of change of the environment is separate from the microservices it hosts. ***P9*** *- If a microservice is* ***deployed,*** \[33],\[34],\[35] *the pipeline artifacts and configuration for the microservice will be* ***versioned*** *and associated with the* ***stack*** \[36],\[37] *of infrastructure elements that were* ***provisioned*** *with* *it.* The provisioning of infrastructure and the deployment of a microservice are related. The deployment of a microservice must know the version of the infrastructure that it was deployed and tested with. **Cloud Native in the Small: Microservices and Networking** Cloud native network functions follow the same principles as cloud native microservices with few exceptions. ***P10*** *- If an organization or a set organizations deliver* ***cloud native network functions**, the software’s features will be delivered as* ***microservice software (a microservice, set of microservices, or component of a microservice)**.* ***P11*** *- If a pipeline provisions network infrastructure\[38] (physical or virtual layer 1 and layer 2* \[39] *networking functions), it will be* ***provisioned*** \[40] *using* ***declarative configuration**.* Network infrastructure (the platform that the cloud native network functions will be deployed into) is provisioned (instances are made available for use to consumers) using declarative configuration. Configuration should designate what the outcome is, while the tools that provision that network infrastructure should create that outcome. ***P12*** *- If a pipeline* ***provisions network infrastructure**, it will be* ***provisioned*** *immutably.* ***P13*** *- If a* ***provider*** *for network infrastructure delivers* ***software*** *or* ***hardware**, it will be delivered to the* ***consumer*** *as a* ***library dependency*** *or* ***service instance*** *(whether self service or hosted).* ***P14*** *- If the provider of networking software delivers* ***cloud native service chains**, the service chains will be* ***composed*** *of* ***immutable microservices*** *with* ***declarative APIs***,\[41],\[42],\[43],\[44]*.* Cloud native network functions can be composed with one another. During this composition their configuration is not modified after deployment (immutable), designates the outcome of the network that is wanted (declarative), and not steps of how to get to that outcome (imperative). ***P15*** *- If an* ***application developer*** *consumes a cloud native networking function, it will be* ***consumed*** *using a declarative API.* A cloud native network function exposes its configuration using a declarative API, such as a yaml file. An application developer has the ability to reference cloud native functions at a higher level, using elements that were provided by operators. ***P16*** *- If an* ***operator*** *combines cloud native network functions into a service chain, they will* ***combined*** ***using*** *a declarative API and will be* ***exposed*** *as a declarative API.* Operators compose fine grained cloud native functions and provide them in as a coarse grained element to consumers (e.g. application developers) via a declarative API. ***P17*** *- If a* ***cloud native network function developer*** *creates networking software, it will* ***expose*** *a declarative API.* The cloud native network functions themselves are developed in such a way as to expose a way to configure them declaratively. **LICENSE** \ This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications ## Endnotes 1. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 16.. “**Microservices represent the decomposition of monolithic business systems into independently deployable services that do “one thing well.”** That one thing usually represents a business capability, or the smallest, “atomic” unit of service that delivers business value.” 2. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 16.. “As we **decouple** the **business domain** into independently deployable **bounded contexts** of **capabilities**, we also **decouple** the associated **change** **cycles**. As long as the changes are restricted to a single bounded context, and the service continues to **fulfill** its existing **contracts**, those changes can be made and **deployed** **independent** of any **coordination** with the rest of the business. The result is enablement of **more** frequent and rapid **deployments**, allowing for a continuous flow of value.” 3. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 16–17.Development can be accelerated by scaling the development organization itself. It’s very **difficult** to build software faster by **adding more peopl**e due to the overhead of **communication** and coordination. Fred Brooks taught us years ago that **adding** more **people** to a **late** software project **makes it later**. However, rather than placing all of the developers in a single sandbox, we can create **parallel work streams** by building more **sandboxes** through **bounded contexts**. 4. Stine, Matt. Migrating to Cloud-Native Application Architecture, O'reilly, 2015, pp. 17 The new developers that we add to each sandbox can **ramp** **up** and become productive more **rapidly** due to the **reduced** **cognitive** **load** of learning the business domain and the existing code, and building relationships within a **smaller** **team**. 5. **Conway’s** **law** describes the relationship between the **structure** of an **organization** and its systems: Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a **design** whose **structure** is a **copy** of the **organization’s communication structure**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4914-4922). O'Reilly Media. Kindle Edition. 6. **Cross-functional teams** put **all** of the **people** **responsible** for building and running an aspect of a system **together**. This may include testers, project managers, analysts, and a commercial or product owner, as well as different types of engineers. These **teams** should be **small**; Amazon uses the term “two-pizza teams,” meaning the team is small enough that two pizzas is enough to feed everyone. The advantage of this approach is that **people** are dedicated to a **single**, **focused** **service** or **small set of services**, avoiding the need to multitask between projects. Teams formed of a consistent set of people work far more effectively than those whose membership changes from day to day. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 6457-6462). O'Reilly Media. Kindle Edition. 7. The peculiarity of buildings that turned Architectural Digest into a contradiction of itself is that different parts of **buildings change at different rates**. Brand, Stewart. How Buildings Learn (p. 21). Penguin Publishing Group. Kindle Edition. 8. “Our basic argument is that there isn’t such a thing as a building,” says Duffy. “A **building** properly conceived is **several** **layers** of **longevity** of built **components**.” He distinguishes four layers, which he calls Shell, Services, Scenery, and Set. **Shell** is the **structure**, which lasts the **lifetime** of the building (fifty years in Britain, closer to thirty-five in North America). **Services** are the cabling, plumbing, air conditioning, and elevators (“lifts”), which have to be replaced every **fifteen years** or so. **Scenery** is the layout of partitions, dropped ceilings, etc., which **changes** every **five to seven years**. **Set** is the shifting of furniture by the occupants, often a matter of **months or weeks**.Brand, Stewart. How Buildings Learn (pp. 21-22). Penguin Publishing Group. Kindle Edition. 9. I’ve taken the liberty of expanding Duffy’s “four S’s”—which are oriented toward interior work in commercial buildings—into a slightly revised, general-purpose **“six S’s**”: • **SITE** - This is the geographical setting, the urban location, and the legally defined lot, whose boundaries and context outlast **generations** of ephemeral buildings. “Site is eternal,” Duffy agrees. • **STRUCTURE** - The foundation and load-bearing elements are perilous and expensive to change, so people don’t. These are the building. Structural life ranges from **30 to 300 years** (but few buildings make it past 60, for other reasons). • **SKIN** - Exterior surfaces now change every **20 years** or so, to keep up with fashion or technology, or for wholesale repair. Recent focus on energy costs has led to re-engineered Skins that are air-tight and better-insulated. • **SERVICES** - These are the working guts of a building: communications wiring, electrical wiring, plumbing, sprinkler system, HVAC (heating, ventilating, and air conditioning), and moving parts like elevators and escalators. They wear out or obsolesce every **7 to 15 years**. Many buildings are demolished early if their outdated systems are too deeply embedded to replace easily. • **SPACE PLAN** - The interior layout—where walls, ceilings, floors, and doors go. Turbulent commercial space can change every **3 years** or so; exceptionally quiet homes might wait **30 years**. • **STUFF** - Chairs, desks, phones, pictures; kitchen appliances, lamps, hair brushes; all the things that twitch around **daily** to monthly. Furniture is called mobilia in Italian for good reason.Brand, Stewart. How Buildings Learn (pp. 24-25). Penguin Publishing Group. Kindle Edition. 10. Frank Duffy: “**Thinking** about buildings in this **time-laden** way is very **practical**. As a **designer** you **avoid** such classic mistakes as **solving a five- minute problem with a fifty-year solution**, or vice versa. It **legitimizes** the existence of **different** **design** **skills**—architects, service engineers, space planners, interior designers—all with their **different agendas** defined by this **time** **scale**. It means you **invent** building **forms** which are very **adaptive**.” Brand, Stewart. How Buildings Learn (p. 32). Penguin Publishing Group. Kindle Edition. 11. The **layering** also **defines** how a **building** **relates** to **people**. **Organizational** **levels** of **responsibility** **match** the **pace** **levels**. The building interacts with individuals at the level of Stuff; with the tenant organization (or family) at the Space plan level; with the landlord via the Services (and slower levels) which must be maintained; with the public via the Skin and entry; and with the whole community through city or county decisions about the footprint and volume of the Structure and restrictions on the Site. The community does not tell you where to put your desk or your bed; you do not tell the community where the building will go on the Site (unless you’re way out in the country). Brand, Stewart. How Buildings Learn (pp. 32-33). Penguin Publishing Group. Kindle Edition. 12. O’Neill’s A **Hierarchical** **Concept** of **Ecosystems**. O’Neill and his co-authors noted that **ecosystems** could be **better understood** by **observing** the **rates of change** of different **components**. Hummingbirds and flowers are quick, redwood trees slow, and whole redwood forests even slower. Most **interaction** is **within** the same **pace** **level**—hummingbirds and flowers pay attention to each other, oblivious to redwoods, who are oblivious to them. Meanwhile the forest is attentive to climate change but not to the hasty fate of individual trees. Brand, Stewart. How Buildings Learn (p. 33). Penguin Publishing Group. Kindle Edition. 13. The most effective **measurement** of a change management **pipeline** is the **cycle** **time**. **Cycle** **time** is the **time** between **deciding** on the **need** for a **change** to seeing that **change** in **production** use. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4867-4868). O'Reilly Media. Kindle Edition. 14. **Metrics** are best used by the team to help itself, and should be continually reviewed to decide whether they are still providing value. Some common metrics used by infrastructure teams include: **Cycle time** The time taken from a **need** being **identified** to **fulfilling** it. This is a measure of the efficiency and speed of change management. Cycle time is discussed in more detail later in this chapter. **Mean time to recover (MTTR)** The **time** taken from an **availability** **problem** (which includes critically degraded performance or functionality) being identified to a **resolution**, even where it’s a workaround. This is a measure of the efficiency and speed of problem resolution. **Mean time between failures (MTBF)** The **time** **between** critical **availability** **issues**. This is a measure of the stability of the system, and the quality of the change management process. Although it’s a valuable metric, **over-optimizing for MTBF** is a common cause of **poor performance** on **other** **metrics**. **Availability** The percentage of time that the **system** is **available**, usually excluding time the system is offline for planned maintenance. This is another measurement of system stability. It is often used as an SLA in service contracts. True **availability** The percentage of time that the system is **available**, **not excluding planned maintenance.** 15. **Continuous** **delivery** for software is implemented using a **deployment** **pipeline**. A deployment pipeline is an **automated** **manifestation** of a **release** **process**. It **builds** the application code, and **deploys** and **tests** it on a **series** of **environments** **before** allowing it to be deployed to **production**. The **same** concept is applied to **infrastructure** changes. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3823-3826). O'Reilly Media. Kindle Edition. 16. The **point** of CD and the software deployment **pipeline** is to allow **changes** to be delivered in a **continuous** **flow, rather than in large batches**. Changes can be validated more thoroughly, not only because they are applied with an automated process, but also because **changes** are **tested** when they are **small**, and because they are **tested** **immediately** after being committed. The result, when done well, is that **changes** can be **made** **more** **frequently**, more **rapidly**, and more **reliably**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4525-4529). O'Reilly Media. Kindle Edition. 17. Teams who embrace the **pipeline** as the way to manage changes to their infrastructure find a number of **benefits**: Their infrastructure management tooling and **codebase** is **always** **production** **ready**. There is **never** a situation where **extra** **work** is needed (e.g., **merging, regression testing, and “hardening”**) to take work **live**. Delivering changes is nearly painless. Once a change has passed the technical validation stages of the pipeline, it **shouldn’t need technical attention** to carry through to production unless there is a problem. There is no need to make technical decisions about how to apply a change to production, as those decisions have been made, implemented, and tested in earlier stages. It’s easier to make changes through the pipeline than any other way. **Hacking** a change **manually** other than to bring up a system that is down is **more work,** and **scarier**, than just pushing it through the **pipeline**. **Compliance** and **governance** are easy. The **scripts**, tools, and **configuration** for making changes are **transparent** to **reviewers**. Logs can be **audited** to prove what changes were made, when, and by whom. With an automated change management pipeline, a team can **prove** what **process** was followed for each and every **change**. This tends to be **stronger** than taking someone’s word that **documented manual processes** are always followed. Change management processes can be more lightweight. People who might otherwise need to discuss and inspect each change can build their requirements into the automated tooling and tests. They can periodically review the pipeline implementation and logs, and make improvements as needed. Their time and attention goes to the process and tooling, rather than inspecting each change one by one. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4551-4563). O'Reilly Media. Kindle Edition. 18. The **design** of your change management **pipelines** is a **manifestation** of your system’s **architecture**. Both of these are a **manifestation** of your **team structure.** Conway’s law describes the relationship between the structure of an organization and its systems: Any **organization** that **designs a system (**defined more broadly here than just information systems) will inevitably **produce a design** whose **structure** is a **copy** of the **organization’s communication structure.** Organizations can take advantage of this to shape their teams, systems, and pipeline to optimize for the outcomes they want. This is sometimes called the **Inverse Conway Maneuver .** Ensure that the people needed to deliver a given change through to production are all a part of the same team. This may involve restructuring the team but may also be done by changing the system’s design. It can often be achieved by changing the service model, which is the goal of **self-service** **systems**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4914-4922). O'Reilly Media. Kindle Edition. 19. ***Integration Models*** The design and implementation of **pipelines** for testing how **systems** and infrastructure elements **integrate** **depends** on the **relationships** between them, and the relationships between the **teams** responsible for them. There are several typical situations: ***Single team*** One **team** owns all of the elements of the system and is **fully responsible** for managing changes to them. In this case, a single **pipeline, with fan-in** as needed, is often sufficient. ***Group of teams*** A group of teams works together on a **single system** with **multiple services** and/or infrastructure elements. Different teams own different parts of the system, which all integrate together. In this case, a **single fan-in pipeline** may work up to a point, but as the size of the group and its system **grows**, **decoupling** may become **necessary**. ***Separate teams with high coordination*** Each team (which may itself be a group of teams) **owns a system**, which **integrates** with **systems** owned by **other teams.** A given system may integrate with multiple systems. Each **team** will have its **own pipeline** and manage its releases independently. But they may have a close enough relationship that one **team** is willing to **customize** its **systems** and releases to **support** another **team’s requirements.** This is often seen with different groups within a large company and with **close vendor** relationships. ***Separate teams with low coordination*** As with the previous situation, except one of the teams is a **vendor** with **many other customers.** Their release process is designed to meet the requirements of many teams, with little or **no** **customizations** to the requirements of individual customer teams. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4892-4907). O'Reilly Media. Kindle Edition. 20. **Practice: Decouple Pipelines** When **separate** **teams** build **different** **components** of a system, such as **microservices**, joining pipeline branches for these components together with the **fan-in pattern** can create a **bottleneck**. The teams need to spend more **effort** on **coordinating** the way they handle **releases**, testing, and fixing. This may be fine for a small number of teams who work closely together, but the overhead grows exponentially as the number of teams grows. Decoupling pipelines involves structuring the **pipelines** so that a **change** to each **component** can be **released** **independently**. The components may still have dependencies between each other, so they may need integration testing. But rather than requiring all of the components to be released to production together in a “big bang” deployment, a **change** to **one component** could go ahead to **production** before changes to the second component are released. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4883-4889). O'Reilly Media. Kindle Edition. 21. Given **two integrated components**, one **provides** a service, and the other **consumes** it. The **provider** component needs to **test** that it is providing the service correctly for its consumers. And the **consumer** needs to **test** that it is consuming the provider service correctly. For example, one team may manage a monitoring service, which is used by multiple application teams. The monitoring team is the provider, and the application teams are the consumers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4930-4933). O'Reilly Media. Kindle Edition. 22. **Pattern: Library Dependency** One way that one component can provide a capability to another is to work like a **library**. The **consumer** pulls a **version** of the **provider** and **incorporates** it into its own **artifact**, usually in the **build** stage of a **pipeline**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4937-4939). O'Reilly Media. Kindle Edition. 23. The important characteristic is that the **library** component is **versioned**, and the **consumer** can **choose** which **version** to use. If a newer version of the library is released, the consumer may opt to immediately pull it in, and then run tests on it. However, it has the option to “**pin**” to an **older version** of the library. This gives the **consumer** team the **flexibility** to release changes even if they **haven’t yet incorporated new,** incompatible changes to their provider library. But it creates the **risk** that important changes, such as **security** patches, **aren’t integrated** in a **timely** way. This is a major source of security vulnerability in IT systems. For the **provider**, this pattern gives **freedom** to **release** new changes without having to **wait** for all **consumer** teams to update their components. But it can result in having **many** different **versions** of the component in **production**, which increases the time and hassle of support. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4941-4947). O'Reilly Media. Kindle Edition. 24. **Pattern: Self-Provisioned Service Instance** The library pattern can be adapted for full-blown services. A well-known example of this is AWS’s Relational Database Service, or RDS , offered by AWS. A team can provision complete working database instances for itself, which it can use in a pipeline for a component that uses a database. As a provider, Amazon releases new database versions, while still making older versions available as “previous generation DB instances” . This has the **same** effect as the **library pattern**, in that the **provider** can **release new versions** **without waiting** for **consumer teams** to **upgrade** their own components. Being a **service** **rather** than a **library**, the provider is able to **transparently** **release** **minor updates** to the service. Amazon can apply **security** patches to its RDS offering, and **new instances** created by consumer teams will automatically **use the updated version**. The key is for the **provider** to keep close track of the **interface** **contract**, to make sure the service behaves as expected after updates have been applied. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4963-4973). 25. Prefer Products with **Cloud-Compatible Licensing** Licensing can make dynamic infrastructure difficult with some products. Some examples of **licensing** approaches that work **poorly** include: A **manual** process to **register** each new instance, agent, node, etc., for licensing. Clearly, this defeats automated provisioning. If a product’s license does require registering infrastructure elements, there needs to be an **automatable** **process** for adding and removing them. **Inflexible licensing periods.** Some products require customers to buy a **fixed** set of **licenses** for a **long** **period**. For example, a monitoring tool may have licensing based on the maximum number of nodes that can be monitored. The licenses may need to be purchased on a **monthly** cycle. This forces the customer to pay for the maximum number of nodes they might use during a given month, even when they only run that number of nodes for a fraction of the time. This **cloud-unfriendly** **pricing** model **discourages** customers from taking advantage of the ability to **scale capacity up and down with demand.** Vendors pricing for cloud **charge by the hour at most.** Heavyweight purchasing process to increase capacity. This is closely related to the licensing period. When an organization is hit with an unexpected surge in business, they **shouldn’t** **need** to spend **days or weeks** to **purchase the extra capacity** they need to meet the demand. It’s common for vendors to have limits in place to protect customers against accidentally over-provisioning, but it should be possible to raise these limits quickly. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1880-1892). O'Reilly Media. Kindle Edition. 26. **Providing Test Instances of a Service to Consumers** The **provider** of a **hosted** service needs to **provide** **support** for **consumers** to **develop** and **test** their **integration** with the **service**. This is useful to consumers for a number of purposes: To learn how to correctly integrate to the service. To **test** that integration still works after **changes** to the **consumer** system. To **test** that the consumer system still works **after** **changes** to the **provider**. To **test** and develop against **new** **provider** **functionality** **before** it is **released**. To **reproduce** **production** **issues** for troubleshooting. To run **demonstrations** of the consumer system without affecting production data. An effective way for a provider to support these is to provide self-provisioned service instances. If **consumers** can **create** and **configure** **instances** **on-demand**, then they can easily handle their own testing and demonstration needs. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4992-4999). O'Reilly Media. Kindle Edition. 27. The benefits of **decoupling** **runtime** **requirements** from the **host** **system** are particularly powerful for infrastructure management. It creates a clean **separation** of concerns between **infrastructure** and **applications**. The host system **only** needs to have the **container** **runtime** **software** installed, and then it can run nearly any container image. Applications, services, and jobs are packaged into containers along with all of their dependencies \[...]. These dependencies can include operating system packages, language runtimes, libraries, and system files. **Different** **containers** may have different, even **conflicting** **dependencies**, but still run on the **same** **host** without issues. **Changes** to the **dependencies** can be made **without** any **changes** to the **host** system. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1652-1658). O'Reilly Media. Kindle Edition. 28. The important thing is **how the artifact is treated**, conceptually. A **configuration** **artifact** is an **atomic**, **versioned** collection of materials that **provision and/or configure a system component**. An **artifact** is: ***Atomic*** A given **set** of **materials** is **assembled**, **tested**, and **applied** together as a unit. ***Portable*** It can be **progressed** through the **pipeline**, and different versions can be applied to **different environments** or instances. It can be reliably and **repeatably applied to any environment,** and so any given environment has an unambiguous version of the component. ***Complete*** A given artifact should have **everything needed** to **provision** or **configure** the relevant **component**. It should **not assume** that **previous versions** of the **component artifacts** have been **applied** **before**. ***Consistent*** **Applying** the artifact to any two component instances should have the **same results**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4669-4678). O'Reilly Media. Kindle Edition. 29. The best way to think of a **container** is as a **method** to **package** a **service**, application, or job. It’s an RPM on steroids, taking the application and adding in its dependencies, as well as providing a standard way for its **host** system to **manage** its **runtime** environment . Rather than a single container running multiple processes, aim for **multiple** **containers**, each running **one** **process**. These processes then become **independent**, **loosely** **coupled** entities. This makes containers a nice match for microservice application architectures. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1708-1711). O'Reilly Media. Kindle Edition. 30. The **benefits** of **containerization** include: **Decoupling** the **runtime** **requirements** of specific applications from the **host** **server** that the container runs on. Repeatably create **consistent** **runtime** **environments** by having a **container** **image** that can be distributed and run on **any** **host** **server** that supports the runtime. Defining **containers** as **code** (e.g.,in a **Dockerfile**) that can be **managed** in a **VCS**, used to trigger **automated** **testing**, and generally having all of the characteristics for infrastructure as code. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1633-1637). O'Reilly Media. Kindle Edition. 31.The **immutable server pattern** mentioned in “Server Change Management Models” **doesn’t make configuration updates to existing servers**. Instead, changes are made by **building a new server** with the new configuration. With **immutable servers**, **configuration** is **usually** **baked** into the **server template**. When the configuration is updated, a new template is **packaged**. **New instances** of **existing servers** are built from the **new template** and used to **replace** the **older servers**. This approach **treats** **server templates** like **software artifacts**. Each build is versioned and tested before being deployed for production use. This creates a high level of confidence in the consistency of the server configuration between testing and production. **Advocates** of **immutable server**s view making a **change** to the **configuration** of a **production** **server** as **bad** practice, no better than modifying the source code of software directly on a production server. Immutable servers can also **simplify configuration** management, by **reducing** the area of the server that **needs** to be managed by **definition files**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2239-2247). O'Reilly Media. Kindle Edition. 1. Using the term “**immutable**” to describe this pattern can be misleading. “Immutable” means that a thing can’t be changed, so a **truly immutable server would be useless**. As soon as a server boots, its **runtime** **state** **changes** **processes** run, entries are written to logfiles, and **application data** is added, updated, and removed. It’s more **useful** to think of the term “**immutable**” as applying to the **server’s configuration,** rather than to the server as a whole. This creates a clear **line** between **configuration** and **data**. It forces teams to explicitly **define** which elements of a server they will **manage** deterministically as **configuration** and which elements will be treated as **data**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2918-2926). O'Reilly Media. Kindle Edition. 2. **Blue-green** **replacement** is the most straightforward pattern to replace an infrastructure element **without** **downtime**. This is the blue-green deployment pattern for software 4 applied to infrastructure. It requires running two instances of the affected infrastructure, **keeping** one of them **live** at any point in time. Changes and **upgrades** are made to the **offline** **instance**, which can be **thoroughly** **tested** before **switching** usage over to it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5681-5685). O'Reilly Media. Kindle Edition. 3. **Phoenix** **replacement** is the natural progression from blue-green using dynamic infrastructure. Rather than keeping an idle instance around between changes, a **new** **instance** can be created each time a **change** is needed. As with blue-green, the change is **tested** on the new instance before putting it into use. The previous instance can be **kept** **up** for a **short** **time**, until the new instance has been **proven** in use. But then the **previous** **instance** is **destroyed**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5694-5697). O'Reilly Media. Kindle Edition. 4. The **canary** **pattern** involves deploying the **new** **version** of an element alongside the old one, and then **routing** some **portion** of usage to the new elements. For example, with version A of an application running on 20 servers, version B may be deployed to two servers. A subset of traffic, perhaps flagged by IP address or by randomly setting a cookie, is sent to the servers for version B. The behavior, performance, and resource usage of the new element can be monitored to validate that it’s ready for wider use. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5724-5728). O'Reilly Media. Kindle Edition. 5. A **stack** is a **collection** of **infrastructure elements** that are **defined** as a **unit** (the inspiration for choosing the term stack comes mainly from the term’s use by AWS CloudFormation). A stack can be any size. It could be a single server. It could be a pool of servers with their networking and storage. It could be all of the servers and other infrastructure involved in a given application. Or it could be everything in an entire data center. What makes a set of infrastructure elements a **stack** isn’t the size, but **whether it’s defined and changed as a unit.** The concept of a stack **hasn’t** been commonly used with **manually managed infrastructures.** Elements are added organically, and **networking boundaries** are naturally used to think about infrastructure groupings. But automation tools force more explicit groupings of infrastructure elements. It’s certainly possible to put everything into one large group. And it’s also possible to structure stacks by following **network boundaries.** But these **aren’t the only ways to organize stacks**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3227-3230). O'Reilly Media. Kindle Edition. 6. “I Heard Calico Is Suggesting Layer 2: I Thought You Were Layer 3! What’s Happening?” Project Calico Documentation, docs.projectcalico.org/v3.5/usage/troubleshooting/faq#i-heard-calico-is-suggesting-layer-2-i-thought-you-were-layer-3-whats-happening. It’s important to distinguish what Calico provides to the workloads hosted in a data center (a purely layer 3 network) with what the Calico project recommends operators use to build their **underlying network fabric**. Calico’s core principle is that **applications** and **workloads** overwhelmingly **need only IP connectivity** to communicate. For this reason we build an **IP-forwarded network** to **connect** the tenant **applications** and **workloads** to **each other**, and the broader world. However, **the underlying physical fabric obviously needs to be set up too**. Here, Calico has discussed how both a layer 2 (see here) or a layer 3 (see here) fabric could be integrated with Calico. This is one of the great strengths of the Calico model: it allows the **infrastructure** to be **decoupled** from what we show to the **tenant applications** and **workloads**. We have some thoughts on different interconnect approaches (as noted above), but just because we say that there are **layer 2** and **layer 3** ways of **building the fabric**, and that those decisions may have an impact on **route scale**, does not mean that Calico is “going back to Ethernet” or that we’re recommending layer 2 for tenant applications. In all cases we forward on IP packets, no matter what architecture is used to build the fabric. 7. “Concerns over Ethernet at scale” Calico over an Ethernet interconnect fabric, . It has been acknowledged by the industry for years that, beyond a certain size, **classical Ethernet networks** are **unsuitable** for **production** deployment. Although there have been [multiple](https://en.wikipedia.org/wiki/Provider_Backbone_Bridge_Traffic_Engineering) [attempts](https://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_14-3/143_trill.html) [to address](https://en.wikipedia.org/wiki/Virtual_Private_LAN_Service) these issues, the scale-out networking community has, largely abandoned Ethernet for anything other than providing physical point-to-point links in the networking fabric. The principal reasons for **Ethernet** **failures** at **large scale** are: 1. **Large numbers of end points** [1](https://docs.projectcalico.org/v3.5/reference/private-cloud/l2-interconnect-fabric#fn:1). Each **switch** in an Ethernet network must **learn** the **path** to **all Ethernet endpoints** that are connected to the Ethernet network. Learning this amount of state can become a **substantial** task when we are talking about **hundreds of thousands of end points**. 2. **High rate** of **churn** or change in the network. With that many end points, most of them being **ephemeral** (such as virtual machines or containers), there is a large amount of churn in the network. That load of **re-learning** paths can be a **substantial** burden on the **control plane** processor of **most Ethernet switches**. 3. High volumes of **broadcast** **traffic**. As each node on the **Ethernet** network **must** use **Broadcast packets** to **locate peers**, and many use broadcast for other purposes, the resultant packet replication to each and every end point can lead to broadcast storms in large Ethernet networks, effectively consuming most, if not all resources in the network and the attached end points. 4. Spanning tree. **Spanning tree** is the protocol used to **keep** an Ethernet network **from** forming **loops**. The protocol was designed in the era of smaller, simpler networks, and it has not aged well. As the number of links and interconnects in an Ethernet network goes up, many implementations of spanning tree become more **fragile**. Unfortunately, **when** spanning tree **fails** in an Ethernet network, the effect is a **catastrophic** loop or partition (or both) in the network, and, in most cases, difficult to troubleshoot or resolve. While many of these issues are **crippling** at **VM scale** (tens of thousands of end points that live for hours, days, weeks), they will be absolutely **lethal** at **container** **scale** (**hundreds of thousands of end points that live for seconds, minutes, days**). 8. “**Provisioning”** is a term that can be used to mean somewhat different things \[...] provisioning is used to mean **making** an **infrastructure element** such as a **server** or **network device** **ready for use**. Depending on what is being provisioned, this can involve: Assigning **resources** to the element. **Instantiating** the element. **Installing** software onto the element. **Configuring** the element. **Registering** the element with infrastructure services. At the end of the provisioning process, the element is **fully ready for use.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1206-1208). O'Reilly Media. Kindle Edition. 9. “**Declarative** **configuration** is **different** from **imperative** **configuration** , where you simply take a series of actions (e.g., apt-get install foo ) to modify the world. Years of production experience have taught us that maintaining a written **record** of the system’s **desired** **state** leads to a more **manageable**, **reliable** system. Declarative configuration enables numerous **advantages**, including **code** **review** for configurations as well as **documenting** the **current** **state** of the world for distributed teams. Additionally, it is the **basis** for all of the **self-healing** behaviors in Kubernetes that keep applications running **without user action.**” Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 892-896). Kindle Edition. 10. “The **combination** of **declarative** **state** stored in a **version** control system and Kubernetes’s ability to make **reality** **match** this declarative **state** makes **rollback** of a change trivially **easy**. It is simply restating the previous declarative state of the system. With **imperative** **systems** this is usually **impossible**, since while the **imperative** **instructions** describe how to get you from point A to point B, they **rarely** **include** the **reverse** instructions that can get you back. “Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 186-190). Kindle Edition. 11. “Because it describes the state of the world, **declarative** **configuration** does **not** have to be **executed** to be **understood**. Its impact is concretely declared. Since the effects of declarative configuration can be understood before they are executed, declarative configuration is far **less error-prone**. Further, the traditional tools of software development, such as **source control, code review, and unit testing**, can be used in **declarative** configuration in ways that are **impossible** for **imperative** instructions. “ Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 183-186). Kindle Edition. 12. So **declarative** definitions lend themselves to running **idempotently**. You can **safely** **apply** your **definitions** **over and over again**, without thinking about it too much. If something is changed to a system outside of the tool, **applying** the definition will bring it back into line, **eliminating** sources of **configuration** **drift**. When you need to make a change, you simply modify the definition, and then let the tooling work out what to do. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1275-1278). O'Reilly Media. Kindle Edition. # Cloud Native Immutable Infrastructure Principles ***P1*** *- If infrastructure is* ***immutable**, it is easily* ***reproduced*** \[1],\[2]*,* ***consistent*** \[3], ***disposable*** \[4],\[5]*, will have a* ***repeatable*** \[6] ***provisioning process**, and will not have configuration or artifacts that are modifiable in place.* Many qualities characterize an immutable process. Reproducibility, consistency, disposability and repeatability are mandatory attributes in any well designed infrastructure process, including an immutable one. ***P2*** *- If the provisioning of cloud native infrastructure is dynamic, it will use unattended code execution* \[7]\[8]\[9] *with declarative configuration* \[10],\[11],\[12],\[13],\[14]*.* The gold standard for cloud infrastructure is for it be able to be provisioned without any assistance. The tools that provision that infrastructure should accept declarative configuration as inputs. ***P3*** *- If a cloud native* ***infrastructure element*** \[15],\[16] *(**compute, storage, or network**) is* ***provisioned*** \[17],\[18],\[19],\[20],\[21],\[22]*, that element is* ***ready for use**.* ***P4*** *- If a cloud native infrastructure element is* ***provisioned**, it will be provisioned* ***immutably**.* Once immutable infrastructure \\(the orchestrator and all of the software and hardware that it depends on\\) is provisioned, the infrastructure is not changed after it is made ready for use. New changes to the infrastructure are rolled out as new instances of infrastructure. ***P5*** *- If an infrastructure element’s provisioning is* ***immutable**, its* ***base configuration*** \[23],\[24],\[25] *will* ***not*** *be* ***changed.*** In the immutable change management model \[38], immutable infrastructure elements are built from scratch (or from artifacts and configuration with a known state) as a new instance of the element. The artifacts of an immutable infrastructure are composed of scripts, binaries, containers, images, and server templates while the configuration is the declaration of what that infrastructure should look like after it is instantiated. If the infrastructure element is hardware, such as a physical layer 1 networking device, it should be ‘flashed’ (a complete replacement of its software) for its artifact updates. Conversely changes to virtual infrastructure should be treated the same way, with a new virtual instance being deployed based on a current artifact. The \_configuration\_ for such a device should be managed with an atomic application of a versioned configuration file, which replaces all of the configuration on the device at once. This configuration should only apply to non base configuration and not the fundamental configuration used to define the provisioning of such a device itself. The choice of \*\*change management model\*\* is \*\*separate\*\* from a \*\*deployment strategy\*\*, such as a phoenix \[46] or canary \[47] deployment strategy. Any deployment strategy that supports immutable infrastructure (i.e. when infrastructure such as a server needs a configuration or artifact change, a brand new instance of that infrastructure is created) can be used. ***P6*** *- If an infrastructure element is* ***immutable*** \[26],\[27],\[28],\[29],\[30],\[31]*, its* ***base*** ***configuration*** *is stored as a* ***template*** \[32],\[33],\[34],\[35],\[36],\[37]*.* An infrastructure \*\*image\*\* resides at the lowest level and usually includes an operating system, but may also include an \*\*orchestrator\*\* for the higher level applications or any other dependencies that have a low rate of change but are needed for applications. This \*\*image\*\* is managed using a \*\*template\*\* \*\*system\*\* with \*\*versioning\*\* (e.g. a versioned image of an operating system) and minimizes the MTTR\[44] (mean time to recovery) and deployment time of the infrastructure. ***P7*** *- If there is* ***configuration*** *outside of an infrastructure element’s template, it is* ***versioned*** *and stored in* ***source control*** \[41],\[43]*.* Any configuration that is applied after an infrastructure’s base image/template has been created (also called \*\*bootstrapping\*\*) will be applied \*\*before\*\* the infrastructure is considered ready for use. After the element is in use, no more configuration is allowed. ***P8*** *- If an infrastructure element is* ***immutable**, the* ***dependencies*** *of the applications that run on that infrastructure element will be* ***decoupled*** \[48],\[49],\[50],\[51]\[52],\[53] *from that* ***infrastructure element*** An infrastructure element such as a generic host server, a network device, or a storage device, has a different rate of change with respect to, and is separate from, the applications that are deployed on or in relation to those elements. For instance, a generic host server \[42] can act as a network router, but should still have separation between the dependencies that make it ready for use to the network applications deployed on that server. A microservice has all of its dependencies deployed with it during its deployment phase of the pipeline, which is separate from the infrastructure’s pipeline. These dependencies are decoupled from the infrastructure environment (e.g. a node) so the rate of change of the environment is separate from the microservices it hosts. ***P9*** *- If an infrastructure element is* ***immutable**, the applications that run on that infrastructure element will run in an* ***unprivileged mode*** \[54],\[55] Applications should not require any elevated level of security permissions on the underlying infrastructure as they should have no need to make modifications to it. ***P10*** *- If an application and its dependencies are* ***decoupled*** *from the infrastructure element that it runs on, the* ***application*** *will be* ***orchestrated*** \[56]. All application components that run on the infrastructure, except data \[57],\[58],\[59],\[60],\[61], will be orchestrated and built from versioned artifacts and configuration templates during each deploy. ***P11*** *- If an infrastructure element provides a* ***service*** \[62],\[63],\[64],\[65]*, that service will be available via* ***service discovery.*** ***P12*** *- If an infrastructure is* ***immutable**, its changes will be managed via a* ***change management pipeline*** \[66\[,\[67],\[68] ***P13*** *- If* ***artifacts*** \[69],\[70] *are* ***provisioned*** *as new* ***immutable*** ***infrastructure*** ***elements**, they are deployed using* ***templates**.* The pipeline for low level infrastructure artifacts should combine them into templates and/or images which reduces the provisioning time for everything dependent on them. ***P14*** *- If an immutable infrastructure has a change management pipeline, the* ***change management pipeline*** *applies* ***tests*** \[71],\[72],\[73],\[74],\[75],\[76] *to the codebase with increasing levels of complexity.* ***P15*** *- If the immutable or idempotent infrastructure has dependencies, the* ***dependencies*** ***constrain the relationship structure*** \[77] *between* ***multiple organizations*** ***P16*** *- If the immutable or idempotent infrastructure has a dependency, that* ***dependency*** *will be* ***delivered*** \[78] *from a* ***provider*** *to a* ***consumer*** \[79] *in the form of a* ***library*** \[80]\[81] *or a* ***service instance*** \[82] *via a* ***pipeline**.* All software components delivered by a provider need to have the ability to be integration \*\*tested\*\* \[85],\[86] using the consumer’s pipeline \[95] in a non-production environment. This should be via the consumer’s own pipeline via a library or a self-serviced instance, or a hosted environment \[83] given by the provider. The software should resist \*\*breaking\*\* the \*\*contract\*\* formed \*\*between\*\* the \*\*provider\*\* of that software and the \*\*consumer\*\* of that software. This means paying attention to forward and backward compatibility \[87],\[88],\[89],\[90],\[91],\[92] with respect to interfaces between the provider and consumer. Any breaks in compatibility should force a major version \[93],\[94] change. **LICENSE** This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications ## Endnotes 1. It should be possible to **effortlessly** and reliably rebuild any element of an infrastructure. Effortlessly means that there is **no need to make any significant decisions** about **how** to **rebuild** the thing. Decisions about which software and versions to install on a server, how to choose a hostname, and so on should be captured in the scripts and tooling that provision it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 349-352). O'Reilly Media. Kindle Edition. 2. When problems are discovered, fixes may not be rolled out to all of the systems that could be affected by them. Differences in versions and configurations across servers mean that software and scripts that work on some machines don’t work on others. This leads to **inconsistency** across the **servers**, called **configuration drift**. \[...] Even when servers are initially created and configured consistently, **differences** can creep in **over time**: \[...]. But **variations should be captured and managed in a way that makes it easy to reproduce and to rebuild servers and services.** Unmanaged variation between servers leads to **snowflake servers** and automation **fear**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 278-290). O'Reilly Media. Kindle Edition. 3. Given **two infrastructure elements** providing a **similar service** for example, two application servers in a cluster the servers should be nearly **identical**. Their system software and configuration should be the same, except for those **bits** of **configuration** that differentiate them, like their **IP addresses**. Letting inconsistencies slip into an infrastructure keeps you from being able to trust your automation. If one file server has an 80 GB partition, while another has 100 GB, and a third has 200 GB, then you can’t rely on an action to work the same on all of them. This encourages doing special things for servers that don’t quite match, which leads to **unreliable** **automation**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 380-384). O'Reilly Media. Kindle Edition. 4. One of the **benefits** of **dynamic infrastructure** is that **resources** can be easily **created, destroyed, replaced, resized, and moved**. In order to take advantage of this, systems should be designed to **assume** that the infrastructure will **always** be **changing**. **Software** should **continue running** even when **servers** **disappear**, appear, and when they are resized. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 357-359). O'Reilly Media. Kindle Edition. 5. A popular expression is to “**treat your servers like cattle, not pets**.” ,Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 362-363). O'Reilly Media. Kindle Edition. 6. Building on the **reproducibility** principle, any action you carry out on your infrastructure should be **repeatable**. This is an obvious benefit of **using scripts and configuration management tools** **rather than** making changes **manually**, but it can be hard to stick to doing things this way, especially for experienced system administrators. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 393-395). O'Reilly Media. Kindle Edition. 7. These are some characteristics of scripts and tasks that help to support reliable unattended execution: **Idempotent** It should be possible to execute the same script or task multiple times without bad effects. **Pre-checks** A task should validate its starting conditions are correct, and fail with a visible and useful error if not, leaving things in a usable state. **Post-checks** A task should check that it has succeeded in making the changes. This isn’t just a matter of checking return codes on commands, but proving that the end result is there. For example, checking that a virtual host has been added to a web server could involve making an HTTP request to the web server. **Visible failure** When a task fails to execute correctly, it should be visible to the team. This may involve an information radiator and/or integration with monitoring services (covered in “What Is An Information Radiator?” and “Alerting: Tell Me When Something Is Wrong” ). **Parameterized** Tasks should be applicable to multiple operations of a similar type. For example, a single script can be used to configure multiple virtual hosts, even ones with different characteristics. The script will need a way to find the parameters for a particular virtual host, and some conditional logic or templating to configure it for the specific situation. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1071-1086). O'Reilly Media. Kindle Edition. 8. In order to allow a **tool** to **repeatedly run unattended**, it needs to be **idempotent**. This means that the result of running the tool should be the same no matter how many times it’s run. **Idempotent scripts** and tools can be set to **run continuously** (for example, at a fixed time interval), which helps to prevent configuration drift and improve confidence in automation. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1090-1092). O'Reilly Media. Kindle Edition. 9. If you apply the Terraform definition five times, it will only create one web server. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Location 1291). O'Reilly Media. Kindle Edition. 10. A good **domain-specific language (DSL)** for server configuration works by having you **define** the **state** you **want** something to be in, and then doing whatever is needed to **bring it into that state**. This should happen without side effects from being applied to the same server many times. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1096-1097). O'Reilly Media. Kindle Edition. 11. So **declarative definitions** lend themselves to running **idempotently**. You can safely **apply** your definitions **over and over** again, without thinking about it too much. If something is changed to a system outside of the tool, applying the definition will bring it back into line, eliminating sources of configuration drift. When you need to make a change, you simply modify the definition, and then let the tooling work out what to do. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1275-1278). O'Reilly Media. Kindle Edition. 12. “**Configuration definition file**” is a generic term for the tool-specific **files** used to **drive infrastructure automation tools**. Most tools seem to have their own names: playbooks, cookbooks, manifests, templates, and so on. A configuration definition could be any one of these, or even a configuration file or script. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1156-1158). O'Reilly Media. Kindle Edition. 13. A **configuration registry** is a directory of information about the elements of an infrastructure. It provides a **means** for scripts, **tools**, applications, and services to **find the information** they need in order to **manage** and integrate with **infrastructure**. This is particularly useful with dynamic infrastructure because this information changes continuously as elements are added and removed. 14. There are many **configuration registry** products. Some examples include **Zookeeper** , **Consul** , and **etcd** . Many server configuration tool vendors provide their own configuration registry for example, Chef Server, PuppetDB, and Ansible Tower. These products are designed to integrate easily with the configuration tool itself, and often with other elements such as a dashboard. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1335-1339). O'Reilly Media. Kindle Edition. 15. A **stack** is a **collection** of **infrastructure elements** that are **defined** as a **unit** (the inspiration for choosing the term stack comes mainly from the term’s use by AWS CloudFormation). A stack can be any size. It could be a single server. It could be a pool of servers with their networking and storage. It could be all of the servers and other infrastructure involved in a given application. Or it could be everything in an entire data center. What makes a set of infrastructure elements a **stack** isn’t the size, but **whether it’s defined and changed as a unit.** The concept of a stack **hasn’t** been commonly used with **manually managed infrastructures.** Elements are added organically, and **networking boundaries** are naturally used to think about infrastructure groupings. But automation tools force more explicit groupings of infrastructure elements. It’s certainly possible to put everything into one large group. And it’s also possible to structure stacks by following **network boundaries.** But these **aren’t the only ways to organize stacks.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3227-3230). O'Reilly Media. Kindle Edition. 16. The difficulty of a **monolithic definition** is that it becomes **cumbersome** to change. With most definition tools, the file can be organized into separate files. But if making a change involves **running the tool** against the **entire infrastructure stack**, things become dicey: It’s **easy** for a **small change** to **break many things**. It’s hard to avoid tight coupling between the parts of the infrastructure. Each **instance** of the environment is **large** and **expensive**. Developing and testing changes requires **testing an entire stack at once**, which is **cumbersome**. If **many people** can make changes to the infrastructure, there is a high **risk** someone will **break** something. On the other hand, if **changes** are **limited** to a **small group** to minimize the risk, then there are likely to be **long delays** waiting for changes to be made. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3357-3363). O'Reilly Media. Kindle Edition. 17. “**Provisioning”** is a term that can be used to mean somewhat different things \[...] provisioning is used to mean **making** an **infrastructure element** such as a **server** or **network device** **ready for use**. Depending on what is being provisioned, this can involve: Assigning **resources** to the element. **Instantiating** the element. **Installing** software onto the element. **Configuring** the element. **Registering** the element with infrastructure services. At the end of the provisioning process, the element is **fully ready for use.** 18. “**Provisioning**” is sometimes used to refer to a more **narrow** part of this process. For instance, **Terraform** and **Vagrant** both use it to define the callout to a server configuration tool like Chef or Puppet to **configure** a **server** **after** it has been **created**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1218-1220). O'Reilly Media. Kindle Edition. 19. The **infrastructure element** could be a **server**; a **part** of a **server**, such as a user account; **network** **configuration**, such as a **load balancer rule**; or many other things. Different tools have different terms for this: for example, playbooks (Ansible), recipes (Chef), or manifests (Puppet). The term “**configuration definition file”** is used \[...] as a generic term for these. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 421-424). O'Reilly Media. Kindle Edition. 20. An **infrastructure definition tool** will **create servers** but **isn’t** responsible for what’s **on the server** itself. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1304-1305). O'Reilly Media. Kindle Edition. 21. The term “**environment**” is typically used when there are **multiple stacks** that are actually **different instances** of the **same** **service** or set of services. The most common use of environments is for testing. An application or service may have “development,” “test,” “preproduction,” and “production” environments. The infrastructure for these should generally be the same, with perhaps some variations due to scale. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3227-3230). O'Reilly Media. Kindle Edition. 22. This dependency on **manual effort** to keep multiple files up to date and consistent is the **weakness** of **per-environment** **definition** **files**. There is no way to ensure that changes are made consistently. Also, because the change is made manually to each environment, it isn’t possible to use an automated pipeline to test and promote the change from one environment to the next. 23. \[...] server configuration should result in the following : A new **server** can be **completely provisioned on demand**, without waiting more than a few minutes. A new server can be completely provisioned **without human involvemen**t for example, in response to events. When a server configuration **change** is defined, it is **applied** to servers **without human involvement**. Each **change is applied to all the servers** it is relevant to, and is reflected in all new servers provisioned after the change has been made. The processes for provisioning and for applying changes to servers are **repeatable, consistent, self-documented, and transparent**. It is **easy and safe to make changes** to the processes used to provision servers and change their configuration. **Automated tests are run every time a change is made** to a server configuration definition, and to any process involved in provisioning and modifying servers. **Changes** to configuration, and changes to the processes that carry out tasks on an infrastructure, are **versioned** and applied to different environments, in order to support controlled testing and staged release strategies. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1440-1449). O'Reilly Media. Kindle Edition. 24. A **new server** is created by the dynamic infrastructure platform using an **infrastructure definition tool** \[...] The server is created from a **server template**, which is a **base image** of some kind. This might be in a VM image format specific to the infrastructure platform (e.g., an AWS AMI image or VMware VM template), or it could be an OS installation disk image from a vendor (e.g., an ISO image of the Red Hat installation DVD). Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1460-1465). O'Reilly Media. Kindle Edition. 25. There are many **use cases** where **new servers** are created: A member of the infrastructure team needs to build a new server of a standard type for **example**, **adding a new file server to a cluster**. They **change** an **infrastructure definition file** to specify the new server. A user wants to set up a new instance of a standard application for example, a bug-tracking application. They use a **self-service portal**, which builds an application server with the bug-tracking software installed. A web server VM crashes because of a hardware issue. The **monitoring service** **detects** the **failure** and **triggers** the creation of a **new VM to replace it**. User **traffic** grows beyond the capacity of the existing application server pool, so the infrastructure platform’s **autoscaling** functionality creates new application servers and adds them to the pool to meet the demand. A **developer commits a change** to the software they are working on. The CI software (e.g., Jenkins or GoCD) **automatically provisions** an application server in a test environment with the new build of the software so it can run an automated test suite against it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1467-1475). O'Reilly Media. Kindle Edition. 26. The **immutable server pattern** mentioned in “Server Change Management Models” **doesn’t make configuration updates to existing servers**. Instead, changes are made by **building a new server** with the new configuration. With **immutable servers**, **configuration** is **usually** **baked** into the **server template**. When the configuration is updated, a new template is **packaged**. **New instances** of **existing servers** are built from the **new template** and used to **replace** the **older servers**. This approach **treats** **server templates** like **software artifacts**. Each build is versioned and tested before being deployed for production use. This creates a high level of confidence in the consistency of the server configuration between testing and production. **Advocates** of **immutable server**s view making a **change** to the **configuration** of a **production** **server** as **bad** practice, no better than modifying the source code of software directly on a production server. Immutable servers can also **simplify configuration** management, by **reducing** the area of the server that **needs** to be managed by **definition files**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2239-2247). O'Reilly Media. Kindle Edition. 27. Using the term “**immutable**” to describe this pattern can be misleading. “Immutable” means that a thing can’t be changed, so a **truly immutable server would be useless**. As soon as a server boots, its **runtime** **state** **changes** **processes** run, entries are written to logfiles, and **application data** is added, updated, and removed. It’s more **useful** to think of the term “**immutable**” as applying to the **server’s configuration,** rather than to the server as a whole. This creates a clear **line** between **configuration** and **data**. It forces teams to explicitly **define** which elements of a server they will **manage** deterministically as **configuration** and which elements will be treated as **data**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2918-2926). O'Reilly Media. Kindle Edition. 28. **Antipattern: Handcrafted Server** ... **manually building** servers leads almost immediately to **configuration drift** and **snowflake** servers. Server creation is **not traceable, versionable, or testable, and is certainly not self-documenting.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2376-2391). O'Reilly Media. Kindle Edition. 29. **Antipattern: Hot Cloned Serve**r ... A server that has been **cloned** from a **running server** is **not** **reproducible**. You can’t create a third server from the same starting point, because both the original and the new server have moved on: they’ve been in use, so various things on them will have changed. **Cloned servers** also suffer because they have **runtime data** from the **original server.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2412-2423). O'Reilly Media. Kindle Edition. 30. **Antipattern: Snowflake Factory** Many organizations adopt automated tools to provision servers, but a person still creates each one by **running the tool** and **choosing options** for the particular server. This typically happens when processes that were used for **manual server provisioning** are simply carried over to the automated tools. The result is that it may be quicker to build servers, but that servers are still inconsistent, which can make it difficult to automate the process of keeping them patched and updated. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2436-2441). O'Reilly Media. Kindle Edition. 31. The phrase **lift and shift** describes installing **software** that was **written** for **static infrastructure** (i.e., most software designed with legacy, pre-cloud assumptions) **onto dynamic infrastructure**. Although the software ends up running on a dynamic infrastructure platform, the infrastructure must be **managed statically** to avoid breaking anything. These applications are **unlikely** to be able to take advantage of advanced infrastructure capabilities such as **automatic** **scaling** and **recovery**, or the creation of ad hoc instances. Some **characteristics** of non-cloud-native software that require “lift and shift” migrations : **Stateful sessions**. Storing **data on the local filesystem**. **Slow-running startup routines**. **Static configuration** of **infrastructure** **parameters** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5628-5634). O'Reilly Media. Kindle Edition. 32. In many cases, new servers can be built using off-the-shelf server template images. Infrastructure platforms, such as **IaaS clouds**, often provide **template images** for common **operating systems**. Many also offer libraries of **templates** built by vendors and third parties, who may provide images that have been preinstalled and configured for particular purposes, such as **application servers**. But many infrastructure teams find it useful to **build their own server templates**. They can pre-configure them with their team’s preferred tools, software, and configuration. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1506-1512). O'Reilly Media. Kindle Edition. 33. **Packaging** **common elements** onto a template makes it **faster** to **provision** new servers. Some teams take this further by creating server templates for particular **roles** such as web servers and application servers. One of the key **trade-offs** is that, as **more elements** are **managed** by packaging them into server **templates**, the **templates** need to be **updated** **more** often. This then requires more sophisticated processes and tooling to build and manage templates. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1510-1515). O'Reilly Media. Kindle Edition. 34. Keeping **templates** **minimal** makes **sense** when there is a **lot of variation** in what may be installed on a server. For example, if people create servers by self-service, choosing from a large menu of configuration options, it makes sense to provision dynamically when the server is created. Otherwise, the **library** of **prebuilt** templates would need to be **huge** to include **all of the variations** that a user might select. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2564-2567). O'Reilly Media. Kindle Edition. 35. At the other end of the provisioning spectrum is **putting** nearly **everything** into the **server template**. **Building** new servers then becomes very **quick** and simple, just a matter of selecting a template and applying instance-specific configuration such as the hostname. This can be **useful** for infrastructures where new instances **need** to be spun up very **quickly** for **example**, to support **automated scaling**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2574-2577). O'Reilly Media. Kindle Edition. 36. With **immutable servers**, **templates** are treated like a **software** **artifact** in a **continuous delivery pipeline**. Any change to a server’s configuration is made by building a **new version of the template**. Each new template version is automatically **tested** before it is rolled out to production environments. This ensures that every production server’s configuration has been thoroughly and reliably tested. There is **no opportunity** to **introduce** an **untested configuration** change to production. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2587-2590). O'Reilly Media. Kindle Edition. 37. It may be useful to **create a separate server template** for each **role**, with the relevant **software** and **configuration** baked into it. This requires more **sophisticated** (i.e., automated) processes for **building** and managing **templates** but makes **creating** **servers** **faster** and simpler. Different server templates may be needed for reasons other than the functional role of a server. Different operating systems and distributions will each need their own server template, as will significant versions. For example, a team could have separate server templates for CentOS 6.5.x, CentOS 7.0.x, Windows Server 2012 R2, and Windows Server 2016. In other cases, it could **make** **sense** to have server **templates** **tuned** for different **purposes**. Database server nodes could be built from one template that has been tuned for high-performance file access, while web servers may be tuned for network I/O throughput. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2796-2802). O'Reilly Media. Kindle Edition. 38. The **four models** for making **configuration changes** servers are **ad hoc,** **configuration synchronization**, **immutable servers**, and **containerized servers**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2887-2888). O'Reilly Media. Kindle Edition. 39. **Ad** **hoc** change management makes **changes** to **servers** only **when** a specific **change** is **needed**. This was the traditional approach before the automated server configuration tools became mainstream, and is still the most commonly used approach. It is **vulnerable** to **configuration** **drift**, **snowflakes**, and all of the evils described \[...]. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1599-1602). O'Reilly Media. Kindle Edition. 40. **Configuration synchronization** **repeatedly applies configuration** definitions to servers, for example, by running a Puppet or Chef agent on an hourly schedule. This ensures that any **changes** to parts of the system managed by these definitions are **kept in line**. Configuration synchronization is the mainstream approach for infrastructure as code, and most server configuration tools are designed with this approach in mind. The main limitation of this approach is that **many areas** of a server are **left unmanaged**, leaving them vulnerable to **configuration drift**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1605-1609). O'Reilly Media. Kindle Edition. 41. **Immutable infrastructure** makes configuration changes by **completely replacing servers**. Changes are made by **building new server templates**, and then rebuilding relevant servers using those templates. This increases predictability, as there is little variance between servers as tested, and servers in production. It requires **sophistication in server template managemen**t. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1611-1614). O'Reilly Media. Kindle Edition. 42. **Containerized services** works by packaging applications and services in **lightweight containers** (as popularized by Docker). This **reduces coupling** between **server configuration** and the things that **run on** the **servers**. **So host servers tend to be very simple, with a lower rate of change.** One of the other change management **models** still needs to be **applied** to these **hosts**, but their implementation becomes much simpler and easier to maintain. **Most effort and attention goes into packaging, testing, distributing, and orchestrating the services and applications**, but this follows something similar to the immutable infrastructure model, which again is simpler than managing the configuration of full-blown virtual machines and servers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1617-1621). O'Reilly Media. Kindle Edition. 43. **Bootstrap Configuration** with **Immutable Servers**: The **purest use** of **immutable servers** is to **bake** everything onto the server **template** and **change nothing**, even when creating server instances from the template. **But some teams have found that for certain types of changes, the turnaround time needed to build a new template is too slow.** **An emerging practice is to put almost everything into the server template, but add one or two elements when bootstrapping a new server.** This might be a **configuration setting that is only known when the server is created, or it might be a frequently changing element such as an application build for testing**. A small development team using continuous integration (CI) or continuous delivery (CD) is likely to deploy dozens of builds of their application a day, so **building a new server template for every build may be unacceptably slow**. Having a **standard server template image** that can **pull** **in** and **start** a **specified** **application** **build** when it is started is particularly useful for **microservices**. This still follows the **immutable** **server** **pattern**, in that **any** **change** to the server’s **configuration** (such as a new version of the microservice) is carried out by **building a new server instance**. It shortens the turnaround time for changes to a microservice, because it **doesn’t** **require** building a **new** **server** **template**. However, this practice arguably **weakens** the **testing** **benefits** from the **pure immutable model**. Ideally, a given **combination** of **server** **template** **version** and **microservice** **version** will have been **tested** through each stage of a change management **pipeline**. But there is some **risk** that the process of installing a microservice, or making other changes, when creating a server will behave slightly differently when done for different servers. This could cause unexpected behavior. So **this practice trades some of the consistency benefits of baking everything into a template and using it unchanged in every instance in order to speed up turnaround times for changes made in this way. In many cases, such as those involving frequent changes, this trade-off works quite well.** Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3099-3116). O'Reilly Media. Kindle Edition. 44. The most **effective** **measurement** of a change management pipeline is the **cycle** **time**. Cycle time is the time between deciding on the need for a change to seeing that change in production use. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4867-4868). O'Reilly Media. Kindle Edition. 45. **Blue-green** **replacement** is the most straightforward pattern to replace an infrastructure element **without** **downtime**. This is the blue-green deployment pattern for software 4 applied to infrastructure. It requires running two instances of the affected infrastructure, **keeping** one of them **live** at any point in time. Changes and **upgrades** are made to the **offline** **instance**, which can be **thoroughly** **tested** before **switching** usage over to it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5681-5685). O'Reilly Media. Kindle Edition. 46. **Phoenix** **replacement** is the natural progression from blue-green using dynamic infrastructure. Rather than keeping an idle instance around between changes, a **new** **instance** can be created each time a **change** is needed. As with blue-green, the change is **tested** on the new instance before putting it into use. The previous instance can be **kept** **up** for a **short** **time**, until the new instance has been **proven** in use. But then the **previous** **instance** is **destroyed**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5694-5697). O'Reilly Media. Kindle Edition. 47. The **canary** **pattern** involves deploying the **new** **version** of an element alongside the old one, and then **routing** some **portion** of usage to the new elements. For example, with version A of an application running on 20 servers, version B may be deployed to two servers. A subset of traffic, perhaps flagged by IP address or by randomly setting a cookie, is sent to the servers for version B. The behavior, performance, and resource usage of the new element can be monitored to validate that it’s ready for wider use. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5724-5728). O'Reilly Media. Kindle Edition. 48. The **benefits** of **containerization** include: **Decoupling** the **runtime** **requirements** of specific applications from the **host** **server** that the container runs on. Repeatably create **consistent** **runtime** **environments** by having a **container** **image** that can be distributed and run on **any** **host** **server** that supports the runtime. Defining **containers** as **code** (e.g.,in a **Dockerfile**) that can be **managed** in a **VCS**, used to trigger **automated** **testing**, and generally having all of the characteristics for infrastructure as code. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1633-1637). O'Reilly Media. Kindle Edition. 49. The benefits of **decoupling** **runtime** **requirements** from the **host** **system** are particularly powerful for infrastructure management. It creates a clean **separation** of concerns between **infrastructure** and **applications**. The host system **only** needs to have the **container** **runtime** **software** installed, and then it can run nearly any container image. Applications, services, and jobs are packaged into containers along with all of their dependencies \[...]. These dependencies can include operating system packages, language runtimes, libraries, and system files. **Different** **containers** may have different, even **conflicting** **dependencies**, but still run on the **same** **host** without issues. **Changes** to the **dependencies** can be made **without** any **changes** to the **host** system. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1652-1658). O'Reilly Media. Kindle Edition. 50. Sharing the **OS** **kernel** means a container has **less** **overhead** than a hardware **virtual** **machine**. A **container** image can be much **smaller** than a **VM** image, because it doesn’t need to include the entire OS. It can start up in seconds, as it **doesn’t** **need** to **boot** a **kernel** from scratch. And it consumes **fewer** system **resources**, because it **doesn’t** need to run its own **kernel**. So a given **host** can run **more** **container** processes **than** full **VMs**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1701-1703). O'Reilly Media. Kindle Edition. 51. The best way to think of a **container** is as a **method** to **package** a **service**, application, or job. It’s an RPM on steroids, taking the application and adding in its dependencies, as well as providing a standard way for its **host** system to **manage** its **runtime** environment . Rather than a single container running multiple processes, aim for **multiple** **containers**, each running **one** **process**. These processes then become **independent**, **loosely** **coupled** entities. This makes containers a nice match for microservice application architectures. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1708-1711). O'Reilly Media. Kindle Edition. 52. **Containerization** has the potential to create a clean **separation** between layers of **infrastructure** and the **services** and **applications** that **run** **on** it. **Host** servers that run containers can be kept very **simple**, without needing to be tailored to the requirements of specific applications, and without imposing constraints on the applications beyond those imposed by containerization and supporting services like logging and monitoring. So **the infrastructure that runs containers consists of generic container hosts**. These can be stripped down to a bare minimum, including only the minimum toolsets to run containers, and potentially a few agents for monitoring and other administrative tasks. This **simplifies** management of these **hosts**, as they **change** **less** often and have fewer things that can break or need updating. It also reduces the surface area for security exploits. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1723-1729). O'Reilly Media. Kindle Edition. 53. **Containerization** offers a different **model** for managing **server** **processes**. **Processes** are **packaged** so that they can be **run** on servers that **haven’t** been **specifically** **built** for the purpose. A pool of **generic container hosts** can be available to run a variety of different containerized processes or jobs. Assigning containerized processes to hosts is flexible and quick. The **number** of container **hosts** can be **adjusted** **automatically** based on the **aggregated** **demand** across many different types of services. However, this approach requires a **scheduler** to start and manage container instances. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2051-2055). O'Reilly Media. Kindle Edition. 54. “**Don’t Make Changes Directly on the Production Environment**: Most downtime in production environments is caused by uncontrolled changes. Production environments should be completely **locked** **down**, so that **only** your **deployment** **pipeline** can make **changes** to it. That includes everything from the **configuration** of the environment to the **applications** deployed on it and their **data**.” Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (p. 273). Pearson Education. Kindle Edition. 55. “The **CNF** should **run** **without** **privileges**. **Privileged** actions should be **managed** by the **scheduler** and environment”, x-factor-cnfs, Fred Kautz, To achieve process isolation, X-factor CNFs are designed to run in process containers without privileges. **Privileged** **actions** may be **requested** by the container and performed by **privileged** **delegate**. Running **unprivileged** promotes **loose** **coupling** between environments, **reduces** the overall **attack** **surface**, and gives the scheduler the ability to **clean** **up** after the pod in the case of the **pod** **failing**. The X-factor CNF methodology recognizes the need for hardware which requires additional kernel modules. When possible, **kernel** **modules** must follow standard Linux kernel device driver standards \[...] and do **not** affect the **kernel's** **runtime** **environment** **beyond** **enabling the device**. These devices must also **not** be **bound** directly from the **CNF**. Instead, they are listed as an **interface** **mechanism** and **injected** into the **container** **runtime** by the **orchestrator**. The existence of a hardware device should not affect other CNFs. Some **kernel** **modifications** may be acceptable, e.g. **DPDK** or drivers. This should be immutable infrastructure with a **clean** **interface** for pods. In short, **pods** should **not** be allowed to **modify** their **infrastructure**. 56. Container **orchestration** tools have emerged following the rise of containerization systems like Docker. Most of these run agents on a pool of container hosts and are able to **automatically** **select** **hosts** to run new **container** **instances**, **replace** **failed** instances, and **scale** numbers of instances **up** and **down**. Some tools also handle **service** **discovery**, **network** **routing**, **storage**, scheduled **jobs**, and other capabilities. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2063-2066). O'Reilly Media. Kindle Edition. 57. **Data**: **Files generated and updated by the system**, applications, and so on. It may change frequently. The **infrastructure** may have some **responsibility** for this **data** (e.g., **distributing** it, **backing** it **up**, **replicating** it, etc.). But the infrastructure will normally **treat** the **contents** of the **files** as a **black** **box**, not caring about what’s in the files. Database data files and logs files are examples of data in this sense. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2306-2309). O'Reilly Media. Kindle Edition. 58. The key **difference** between **configuration** and **data** is whether automation **tools** will automatically **manage** what’s **inside** the **file**. So even though some infrastructure tools do care about what’s in system logfiles, they’re normally treated as data files. 59. **Data** creates a particular **challenge** for **zero-downtime** change strategies. All of the patterns described involve running multiple versions of a component simultaneously, with the option to roll back and keep the first version of the element if there is a problem with the new one. However, if the components use read/write data storage, this can be a problem. The **problem** comes when the **new** **version** of the **component** involves a **change** to **data** **formats** so that it’s **not** **possible** to have **both** **versions** share the **same data storage** **without** **issues**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5766-5770). O'Reilly Media. Kindle Edition. 60. An effective way to approach **data** for **zero-downtime** deployments is to **decouple** **data** **format** **changes** **from** **software** **releases**. This requires the **software** to be written so that it can **work** with **two** **different data formats**, the **original** version and the **new** version. It is first **deployed** and **validated** with the data in the **original** **format**. The data is then **migrated** as a background task **while** the **software** is **running**. The **software** is able to **cope** with **whichever** **data** **format** a given record is in. Once the **data** **migration** is complete, **compatibility** with the **old** **data** format should be **removed** from the **next** **release** of the software, to keep the code clean. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5776-5780). O'Reilly Media. Kindle Edition. 61. Any **mechanism** that efficiently **replicates** **data** across distributed storage will replicate **corrupted** data just as happily as good data. A data availability strategy needs to address how to **preserve** and **restore** **previous** **versions** of data. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5823-5826). O'Reilly Media. Kindle Edition. 62. \[...] **core infrastructure resources: compute, networking, and storage**. These provide the basic building blocks for infrastructure as code. However, most infrastructures will need a variety of other supporting services and tools. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1821-1822). O'Reilly Media. Kindle Edition. 63. \[...] principles of infrastructure as code for services can be summarized as: The **service** can be **easily** **rebuilt** or **reproduced**. The elements of the service are **disposable**. The infrastructure **elements** **managed** by the **service** are **disposable**. The infrastructure elements managed by the service are always changing. Instances of the service are configured consistently. Processes for **managing** and using the **service** are **repeatable**. Routine requests are fulfilled quickly, with little effort, preferably through **self-service** or automatically. Complex changes can be made easily and safely. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1832-1837). O'Reilly Media. Kindle Edition. 64. Some of the specific practices include: Use **externalized definition files. Self-document** systems and processes. Version all the things. Continuously **test** systems and processes. **Make small changes rather than batches of them**. Keep services available continuously. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1838-1840). O'Reilly Media. Kindle Edition. 65. Prefer Products with **Cloud-Compatible Licensing** Licensing can make dynamic infrastructure difficult with some products. Some examples of **licensing** approaches that work **poorly** include: A **manual** process to **register** each new instance, agent, node, etc., for licensing. Clearly, this defeats automated provisioning. If a product’s license does require registering infrastructure elements, there needs to be an **automatable** **process** for adding and removing them. **Inflexible licensing periods.** Some products require customers to buy a **fixed** set of **licenses** for a **long** **period**. For example, a monitoring tool may have licensing based on the maximum number of nodes that can be monitored. The licenses may need to be purchased on a **monthly** cycle. This forces the customer to pay for the maximum number of nodes they might use during a given month, even when they only run that number of nodes for a fraction of the time. This **cloud-unfriendly** **pricing** model **discourages** customers from taking advantage of the ability to **scale capacity up and down with demand.** Vendors pricing for cloud **charge by the hour at most.** Heavyweight purchasing process to increase capacity. This is closely related to the licensing period. When an organization is hit with an unexpected surge in business, they **shouldn’t** **need** to spend **days or weeks** to **purchase the extra capacity** they need to meet the demand. It’s common for vendors to have limits in place to protect customers against accidentally over-provisioning, but it should be possible to raise these limits quickly. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1880-1892). O'Reilly Media. Kindle Edition. 66. Continuous delivery for software is implemented using a **deployment** **pipeline**. A deployment pipeline is an **automated** manifestation of a **release** **process**. It **builds** the application code, and **deploys** and **tests** it on a series of **environments** **before** allowing it to be deployed to **production**. The **same** concept is applied to **infrastructure** **changes**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3823-3826). O'Reilly Media. Kindle Edition. 67. The **point** of **CD** and the software deployment pipeline is to allow changes to be delivered in a **continuous** **flow**, **rather** than in **large** **batches**. Changes can be **validated** more **thoroughly**, not only because they are applied with an automated process, but also because **changes** are **tested** when they are **small**, and because they are tested immediately after being committed. The result, when done well, is that changes can be made **more** **frequently**, more **rapidly**, and more **reliably**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4525-4529). O'Reilly Media. Kindle Edition. 68. Teams who embrace the **pipeline** as the way to manage **changes** to their **infrastructure** find a number of benefits: Their infrastructure management **tooling** and **codebase** is always **production** **ready**. There is **never** a situation where **extra** **work** is needed (e.g., merging, regression testing, and “hardening”) to take work live. Delivering changes is nearly painless. Once a change has passed the technical validation stages of the pipeline, it **shouldn’t** **need** **technical** **attention** to carry through to **production** unless there is a problem. There is **no** **need** to make technical **decisions** about how to apply a change to **production**, as those decisions have been made, implemented, and tested in earlier stages. It’s easier to make changes through the pipeline than any other way. Hacking a change manually other than to bring up a system that is down is more work, and scarier, than just pushing it through the pipeline. **Compliance** and **governance** are easy. The **scripts**, **tools**, and **configuration** for making changes are **transparent** to **reviewers**. **Logs** can be **audited** to prove what **changes** were made, when, and by whom. With an automated change management pipeline, a team can prove what process was followed for each and every change. This tends to be **stronger** than taking someone’s word that **documented manual processes** are always followed. Change management processes can be more lightweight. People who might otherwise need to discuss and inspect each change can build their requirements into the automated tooling and tests. They can periodically review the pipeline implementation and logs, and make improvements as needed. Their time and attention goes to the process and tooling, rather than inspecting each change one by one. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4551-4563). O'Reilly Media. Kindle Edition. 69. The important thing is **how the artifact is treated**, conceptually. A **configuration** **artifact** is an **atomic**, **versioned** collection of materials that **provision and/or configure a system component**. An **artifact** is: ***Atomic*** A given **set** of **materials** is **assembled**, **tested**, and **applied** together as a unit. ***Portable*** It can be **progressed** through the **pipeline**, and different versions can be applied to **different environments** or instances. It can be reliably and **repeatably applied to any environment,** and so any given environment has an unambiguous version of the component. ***Complete*** A given artifact should have **everything needed** to **provision** or **configure** the relevant **component**. It should **not assume** that **previous versions** of the **component artifacts** have been **applied** **before**. ***Consistent*** **Applying** the artifact to any two component instances should have the **same results**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4669-4678). O'Reilly Media. Kindle Edition. 70. With immutable servers \[...], the **artifact** is the **server template image**. The **build** stage checks a server **template definition file,** such as a **Packer template,** or script out from **VCS**, **builds** the server **template** **image**, and **publishes it** by making it available in the **infrastructure platform**. With AWS, for example, this results in an AMI image. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4697-4701). O'Reilly Media. Kindle Edition. 71. A **pipeline** typically applies **tests** with **increasing levels** of **complexity**. The **earlier** **stages** focus on faster and **simpler** tests, such as unit tests, and testing a single service. **Later** **stages** cover **broader** sections of a system, and often replicate more of the complexities of production, such as **integration** with other services. It’s important that the environments, tooling, and processes involved in applying changes and deploying software are consistent across all of the stages of the pipeline. This ensures that **problems** that might appear in production are **discovered early**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 3827-3831). O'Reilly Media. Kindle Edition. 72. A **top-heavy test suite** is **difficult** to **maintain**, **slow** to run, and doesn’t pinpoint errors as well as a more balanced suite. **High-level** tests tend to be **brittle**. One change in the system can break a large number of tests, which can be more work to fix than the original change. This leads to the **test** **suite** falling **behind** **development**, which means it **can’t be run continuously.** Higher-level tests are also slower to run than the more focused lower-level tests, which makes it impractical to run the full suite frequently. And because higher-level tests cover a broad scope of code and components, when a test fails it may take a **while** to **track** **down** and **fix** the cause. This usually comes about when a team puts a UI-based test automation tool at the core of their test automation strategy. This in turn often happens when testing is treated as a separate function from building. **Testers** who **aren’t** **involved** in **building** the system **don’t** have the **visibility** or involvement with the different layers of the stack. This **prevents** them from **developing** **lower-level** **tests** and incorporating them into the build and change process. For someone who only interacts with the system as a black box, the UI is the easiest way to interact with it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4025-4034). O'Reilly Media. Kindle Edition. 73. So it’s good sense to **sanity** **check** each new **server** when it’s **created**. Automated server **smoke** **testing** scripts can check the basic things you expect for all of your servers, things specific to the server’s role, and general compliance. For example: Is the server running and accessible? Is the monitoring agent running? Has the server appeared in DNS, monitoring, and other network services? Are all of the necessary services (web, app, database, etc.) running? Are required user accounts in place? Are there any ports open that shouldn’t be? Are any user accounts enabled that shouldn’t be? Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2511-2515). O'Reilly Media. Kindle Edition. 74. People managing projects to develop and deploy software have a bucket of requirements they call **non-functional** **requirements**, or NFRs ; these are also sometimes referred to as cross-functional requirements (CFRs). **Performance**, **availability**, and **security** tend to be swept into this bucket. NFRs related to infrastructure can be labeled **operational** **qualities** for convenience. These are things that can’t easily be described using functional terms: take an action or see a result. Operational requirements are only apparent to users and stakeholders when they go wrong. If the system is slow, flaky, or compromised by attackers, people notice. **Automated** **testing** is **essential** to ensuring **operational** **requirements**. Every time a **change** is made to a system or its infrastructure, it’s important to **prove** that the change **won’t** cause **operational** **problems**. The team should have targets and thresholds defined which it can write tests against. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4242-4249). O'Reilly Media. Kindle Edition. 75. **Metrics** are best used by the team to help itself, and should be continually reviewed to decide whether they are still providing **value**. Some common metrics used by infrastructure teams include: ***Cycle time*** The **time** taken from a **need** being **identified** to **fulfilling** it. This is a measure of the **efficiency** and **speed** of **change management**. \[...] ***Mean time to recover*** (MTTR) The time taken from an **availability problem** (which includes critically degraded performance or functionality) being **identified** to a **resolution**, even where it’s a workaround. This is a measure of the **efficiency** and **speed** of **problem resolution.** ***Mean time between failures*** (MTBF) The **time** **between** **critical** availability **issues**. This is a measure of the stability of the system, and the quality of the change management process. Although it’s a valuable metric, **over-optimizing for MTBF is a common cause of poor performance on other metrics**. ***Availability*** The **percentage** of **time** that the **system** is **available**, usually excluding time the system is offline for planned maintenance. This is another measurement of system stability. It is often used as an SLA in service contracts. True availability The percentage of time that the system is available, not excluding planned maintenance. 76. **Layered** **templates** fit well, conceptually at least, with a change management pipeline (see Chapter 12 ), because changes to the base image can then **ripple out** to **automatically** **build** the **role-specific** **templates**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 2805-2807). O'Reilly Media. Kindle Edition. 77. **Integration Models** The design and implementation of pipelines for testing how systems and infrastructure elements integrate depends on the relationships between them, and the relationships between the teams responsible for them. There are several typical situations: ***Single team*** One team owns all of the elements of the system and is fully responsible for managing changes to them. In this case, a **single pipeline,** with **fan-in** as needed, is often sufficient. ***Group of teams*** A group of teams works together on a single system with multiple services and/or infrastructure elements. Different teams own different parts of the system, which all integrate together. In this case, a **single fan-in pipeline may work** up to a point, but as the **size** of the group and its system **grows**, **decoupling** may become necessary. ***Separate teams with high coordination*** Each team (which may itself be a group of teams) **owns** a **system**, which **integrates** with **systems** **owned** by **other teams.** A given system may integrate with multiple systems. Each **team** will have its **own pipeline** and manage its **releases independently.** But they may have a close enough relationship that one team is willing to **customize** its **systems** and releases to **support another team’s requirements.** This is often seen with different **groups within a large company** and with close vendor relationships. ***Separate teams with low coordination*** As with the previous situation, except **one** of the **teams** is a **vendor** with many other customers. Their release process is designed to **meet the requirements of many teams,** with **little** or **no customizations** to the requirements of individual customer teams. “X as a Service” vendors, providing logging, infrastructure, web analytics, and so on, tend to use this model. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4892-4907). O'Reilly Media. Kindle Edition. 78. **Managing** the configuration of **hardware network devices** to support a dynamic infrastructure is a particular **challenge**. For example, you may need to add and remove servers from a load balancer if they are created and destroyed automatically in response to demand. **Network** **devices** tend to be difficult to **automatically** **configure**, although many are able to load configuration files over the network for example, from a TFTP server. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 859-861). O'Reilly Media. Kindle Edition. 79. Given two **integrated** **components**, one **provides** a **service**, and the other **consumes** it. The **provider** component needs to **test** that it is providing the service correctly for its consumers. And the consumer needs to **test** that it is consuming the provider service correctly. For example, one team may manage a monitoring service, which is used by multiple application teams. The monitoring team is the provider, and the application teams are the consumers. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4930-4933). O'Reilly Media. Kindle Edition. 80. **Pattern: Library Dependency** One way that one component can provide a capability to another is to work like a **library**. The **consumer** pulls a **version** of the **provider** and **incorporates** it into its own **artifact**, usually in the **build** stage of a **pipeline**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4937-4939). O'Reilly Media. Kindle Edition. 81. The important characteristic is that the **library** component is **versioned**, and the **consumer** can **choose** which **version** to use. If a **newer version** of the library is released, the **consumer** may **opt** to immediately pull it in, and then run **tests** on it. However, it has the option to **“pin” to an older version** of the library. This gives the **consumer** team the flexibility to **release changes** even if they **haven’t** yet **incorporated** **new**, incompatible changes to their **provider library**. But it creates the **risk** that important changes, such as **security** patches, aren’t integrated in a timely way. This is a major source of security vulnerability in IT systems. For the **provider**, this pattern gives freedom to **release new changes** **without** having to wait for all **consumer** teams to update their components. But it can result in having **many different versions** of the **component** in **production**, which increases the time and **hassle** of **support**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4941-4947). O'Reilly Media. Kindle Edition. 82. **Pattern: Self-Provisioned Service Instance** The library pattern can be adapted for full-blown services. A well-known example of this is AWS’s Relational Database Service, or RDS , offered by AWS. A team can provision complete working database instances for itself, which it can use in a pipeline for a component that uses a database. As a provider, Amazon releases new database versions, while still making older versions available as “previous generation DB instances” . This has the **same** effect as the **library pattern**, in that the **provider** can **release new versions** **without waiting** for **consumer teams** to **upgrade** their own components. Being a **service** **rather** than a **library**, the provider is able to **transparently** **release** **minor updates** to the service. Amazon can apply **security** patches to its RDS offering, and **new instances** created by consumer teams will automatically **use the updated version**. The key is for the **provider** to keep close track of the **interface** **contract**, to make sure the service behaves as expected after updates have been applied. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4963-4973). O'Reilly Media. Kindle Edition. 83. **Providing Test Instances of a Service to Consumers** The **provider** of a **hosted** service needs to **provide** **support** for **consumers** to **develop** and **test** their **integration** with the **service**. This is useful to consumers for a number of purposes: To learn how to correctly integrate to the service. To **test** that integration still works after **changes** to the **consumer** system. To **test** that the consumer system still works **after** **changes** to the **provider**. To **test** and develop against **new** **provider** **functionality** **before** it is **released**. To **reproduce** **production** **issues** for troubleshooting. To run **demonstrations** of the consumer system without affecting production data. An effective way for a provider to support these is to provide self-provisioned service instances. If **consumers** can **create** and **configure** **instances** **on-demand**, then they can easily handle their own testing and demonstration needs. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4992-4999). O'Reilly Media. Kindle Edition. 84. \[...] a common technique is to **deploy** **changes** into **production** **without** necessarily **releasing** them to end users. New **versions** of components are put into the production environment and integrated with other components in ways that won’t impact normal operations. This allows them to be **tested** in true **production** conditions, **before** the **switch is flipped** to put them into active use. They can even be put into use in a drip-feed fashion, measuring their performance and impact before rolling them out to the full user base. Techniques for doing this include those used to hide unfinished changes in the codebase, such as **feature** **toggles**, as well as zero-downtime replacement patterns. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5062-5067). O'Reilly Media. Kindle Edition. 85. **Contract tests** are automated tests that check whether a provider **interface** **behaves** as **consumers** **expect**. This is a much smaller set of tests than full functional tests, purely **focused on the API** that the service has committed to provide to its consumers. By running contract tests in their own **pipeline**, the **provider** team will be alerted if they accidentally make a change that **breaks** the **contract**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5092-5095). O'Reilly Media. Kindle Edition. 86. **Practice: Run Consumer-Driven Contract (CDC) Tests** A variation on these previous practices is for a provider to run **tests** **provided by consumer** teams. These tests are written to formalize the expectations the consumer has for the provider’s interface. The **provider** runs the **tests** in a stage of its own **pipeline**, and fails any build that fails to pass these tests. A failure of a CDC test tells the provider they need to investigate the nature of the failed expectation. In some cases, the provider will realize they have made an error, so they can correct it. In others, they **may see** that the **consumer’s** **expectations** are **incorrect**, so they can let them know to change their own code. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5112-5118). O'Reilly Media. Kindle Edition. 87. **Practice: Ensure Backward Compatibility of Interfaces** Providers should work hard to **avoid** making changes that **break** existing **interfaces**. Once a release is published and in use, **new releases** should not **change** the **interfaces** used by consumers. It’s normally easy to **add** new functionality **without** **breaking** interfaces. A command-line tool can add a new argument without changing the behavior of existing arguments. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5045-5048). O'Reilly Media. Kindle Edition. 88. If there is a need to make **drastic changes** to an existing interface, it’s often better to **create** a **new** **interface**, leaving the old one in place. The old interface can be deprecated, with warnings to users that they should move to using the new version. This **gives consumers** **time** to **develop** and test their use of the new interface before they switch, which in turn gives the provider the flexibility to release and iterate on the new functionality without having to wait for all of their consumers to switch. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5052-5055). O'Reilly Media. Kindle Edition. 89. **Backward-compatible** changes are those that **allow** any **future** **implementations** to **consume** **older** **versions** of the design. For example, if a **client** that **supports** one or more of the **new** extensions receives a representation from a **server** that does **not** **support** these **new** extensions, the **client** implementation will continue to **function** **successfully**. Usually, this means the **new** extensions do **not** create a **required** **dependency** that causes implementations to break if that extension is missing. Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node (p. 147). Kindle Edition. 90. Often **backward-compatible** changes mean that **implementations** need to be **prepared** to **handle** representations that are **missing expected elements**. This may mean that some functionality or feature is not available to an implementation and that implementation must work around that missing element or fall back to a mode that supports an older design of the media type. Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node (p. 147). Kindle Edition. 91. An **extension** can be considered “**forward** **compatible**” if the **changes** do **not** **break** **existing** **implementations**. This means existing client applications will continue to **work** **successfully** when they **receive** a **representation** that contains the **new** **features**. Servers will continue to function properly when they receive a representation that contains the new features. Usually, this means the **new features/functionality can be safely ignored by implementations that do not recognize them**. Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node (p. 146). Kindle Edition. 92. The flipside of maintaining backward compatibility for providers is for consumers to ensure version tolerance. A consumer team should ensure that a single build of their system can easily work with different versions of a provider that they integrate with, if it’s a likely situation. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5070-5071). O'Reilly Media. Kindle Edition. 93. In order to support both forward and backward compatibility, there are some general guidelines that should be followed when making changes to media type designs. **Existing design elements cannot be removed** \[...] **The meaning or processing of existing elements cannot be changed** \[...] **New design elements must be treated as optional** Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node (pp. 147-148). Kindle Edition. 94. **Versioning** a media type means **making** **changes** to the media type that will likely cause **existing** **implementations** of the original media type to “**break**” or misbehave in some significant way. \[...] Versioning should be seen as a last resort. Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node (p. 148). Kindle Edition. 95. **Practice: Decouple Pipelines** When separate teams build different components of a system, such as microservices, **joining** **pipeline** branches for these components together with the **fan-in pattern** can create a **bottleneck**. The **teams** need to spend more **effort** on **coordinating** the way they handle releases, testing, and fixing. This may be **fine** for a **small number** of teams who work closely together, but the overhead grows exponentially as the number of teams grows. Decoupling pipelines involves **structuring** the **pipelines** so that a **change** to each **component** can be **released** **independently**. The components may still have dependencies between each other, so they may need integration testing. But rather than requiring all of the components to be released to production together in a “big bang” deployment, a **change** to one **component** could **go** ahead to **production** **before** changes to the **second** **component** are released. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 4883-4889). O'Reilly Media. Kindle Edition. # Cloud Native Declarative OSI Principles The best practices of cloud native infrastructure writ large should be applied to networking infrastructure in the small. When describing network infrastructure in a cloud native manner, the specific challenges for provisioning each OSI \[1],\[2],\[3],\[4],\[5],\[6] layer need to be addressed in order to minimize the usual toil associated with provisioning, deploying functionality into, and managing complex networks. **Cloud native networking** addresses **provisioning** the **fabric** at layers 1 (physical layer) \[7] and some of layer 2 (data-link layer: physical & virtual layer 2 switches) \[8],\[9], and providing **cloud native network functions (CNFs)** that are the orchestrated implementation of layers 2 (data-link layer: data units (frames) organization, error detection, and flow control), 3 \[10] (network layer \[11], including data planes \[12],\[13] and control planes \[14]), 4 (transport layer) \[15],\[16], and the applications layer(s) \[17],\[18] 5 (session layer) \[19], 6 (presentation layer) \[20], and 7 (application layer) \[21]. ***P1*** *- If a pipeline provisions network infrastructure* \[22]*, it will be* ***provisioned*** *and* ***managed*** *using* ***declarative configuration*** \[23],\[24],\[25],\[26]*.* Network infrastructure can be separated into the underlying network fabric (underlay) and the application or workload network (overlay). The establishment of an underlay network consists of the provisioning and configuration that resides at the lower OSI layers, such as the implementation of the physical or virtual OSI layer 1 (physical media, interconnects \[27] such as buses \[28] and layer 1 switches \[29],\[30],\[31],\[32], network adapters \[33],\[34], and other mechanisms \[35],\[36]) or physical or virtual layer 2 \[37] (layer 2 switches, Bridges \[38] etc). The application (overlay) network functionality is deployed onto the underlying infrastructure network. ***P2*** *- If a pipeline provisions* ***physical network layer 1 or layer 2 infrastructure**, it will be provisioned* ***immutably**.* Configuration that relates layer 1 and layer 2 tend to be operating system boot options, machine specific bios settings (e.g. SR-IOV bios settings), or a configuration file for traditional layer 1 devices. These settings can be for physical hardware or virtual hardware. In order to provision physical hardware immutably it must be taken offline, reset to a known state (e.g. networking device should get a ‘flashed’ or 'pushed', complete replacement for its artifact updates), have any of its patches applied sequentially. At this point new configuration (that is stored in a templated format and is maintained in a version control system) can be applied with the network infrastructure element then being ready for use. Both artifacts and configuration should be maintained independently of the physical or virtual device. ***P3*** *- If a* ***cloud native network*** *is provisioned, it will encompass the provisioning of an* ***infrastructure (underlay) network*** \[39] *and an* ***application/workload (overlay) network*** \[40]. Some decisions, such as how to manage (or avoid) the ethernet churn of hundreds of thousands of endpoints, must be made with respect to the configuration of underlying networking hardware and topology of cloud native networks. This infrastructure (underlay) network serves as the foundation which supports the higher level (overlay) networks that will be provided to applications. The components of the underlying infrastructure network (whether it be physical or virtual layer 1 and layer 2) operate at a different rate of change to, have different concerns from, and must not interfere (e.g. degrade performance or quality of service) with, the use cases of the application/workload network. Another way of stating this, is that underlay network must be provisioned and managed in a way that alterations to its deployment do not conflict with overlay networks being consumed by applications. ***P4*** *- If a CNF has specified a set of* ***preferred local mechanisms*** \[36],\[46], *the infrastructure will provide those mechanisms to the CNF in the order of preference specified should the infrastructure support the requested mechanism.* Some CNFs may need to declare the mechanisms (Linux interface, memif, etc) that they support so that the orchestrator can decide the most efficient way to implement the CNF. This may include the selection of a mechanism based on affinity (e.g. the availability of an interface type between two endpoints that reside in the same host). The CNF's preference for a specific type of local mechanism does not supersede the principle of immutability. Mechanisms of any type should be considered as any other resource type. If said resource is not available, then the CNF should not be scheduled. ***P5*** *- Regardless of whether a CNF is* ***location dependent, affinity aware, or location agnostic,*** *it should be* ***deployed*** *using either* *the* ***phoenix*** \[41]***, or canary*** \[42] *deployment patterns.* Some layer 1 and layer 2 cloud native network functions may need location specific information in order to be provisioned (i.e. they can’t be configured to use service discovery). When this is the case, the design of that cloud native network function should support the phoenix or canary deployment patterns in order to do a phased rollout of the equipment with the new changes. The blue-green \[43] deployment pattern should not be used as it implies non-immutability. ***P6*** *- If a CNF has an API defined, the* ***API*** *will be defined using the* ***most*** *declarative part of the* ***declarative spectrum*** \[44] *as is possible* When using declarative configuration, the overall outcome is defined. There is a sense in which location is imperative (designating how instead of what) because it encompasses ‘how’ to get to a destination (e.g. hardcoded IPs or subnets). To a lesser degree, affinity (the property a component that must be ‘close’ to another component, such as a special type of network card) is imperative as well. A declarative spectrum for configuration emerges where there is no location specific information on one side (the most declarative), and hard coded subnets on the other side (the least declarative) of the spectrum. This is not to say that technologies such as affinity/anti-infinity cannot be declared as a desired end-state, just that the CNFs API should not be specifying to the underlaying infrastructure how to achieve these ends. When designing cloud native network functions, the configuration should be as declarative as possible. ***P7*** *- If an infrastructure element that is part of a CNF is* ***virtual layer 1**, it will be immutable* Virtualized cloud native networking infrastructure components that are part of the physical layer 1, such as virtual network cards, should have configuration that is immutable. ***P8*** *- If an infrastructure element that is part of a CNF is* ***virtual layer 2 or higher**, it will be* ***immutable and orchestrated.*** Virtual layer 2 and higher network functions, such as layer 2 MPLS VPNs, should be provisioned immutably. Configuration for said network services should be captured in a template, stored with an associated version, and 'pushed' via the higher level orchestration construct in an atomic fashion. ***P9*** *- If a CNF is* ***virtual layer 2 or higher**, it will expose itself using* ***service discovery.*** ***P10*** *- If a CNF is* ***virtual layer 1**, its* ***provisioning*** *will use a* ***server template*** The infrastructure elements of the lowest level virtual underlay network (e.g. networking components of a hypervisor that map to the physical components of its node), should have its configuration baked into an artifact that is versioned and managed with an artifact management system. Whatever configuration that is not on the image should be applied after the initial artifact is deployed (Day 2), via an orchestrated and versioned process, before the infrastructure element is considered ready for use. ***P11*** *- If an infrastructure element that is part of a CNF is* ***virtual layer 2, 3, or higher**, its* ***deployment*** *will be within a* ***microservice*** *and will be* ***orchestrated.*** Virtual layer 2 (e.g. layer 2 MPLS VPNs), layer 3 (e.g. software data planes and control planes) and above should be deployed using coarse grained packaging (such as containers), orchestrated, and deployed onto a generic host \[45] infrastructure element. ***P12*** *- If an* ***application developer*** *consumes a* ***cloud native networking function**, it should be* ***consumed*** *using a declarative API.* ***P13*** *- If an* ***operator*** *combines cloud native network functions into a* ***service chain**, they will be* ***combined*** *using a declarative API and will be* ***exposed*** *as a declarative API.* ***P14*** *- If a* ***cloud native network function developer*** *creates networking software, it will* ***expose*** *a declarative API.* **LICENSE** \ This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). **LIST OF CONTRIBUTORS** If you would like credit for helping with these documents (for either this document or any of the other four documents linked above), please add your name to the list of contributors. W Watson Vulk Coop Taylor Carpenter Vulk Coop Denver Williams Vulk Coop Jeffrey Saelens Charter Communications Bill Mulligan Loodse ## Endnotes 1. An important property of the OSI Reference Model is that it enables standardization of the protocols used in the protocol stacks, leading to the specification of interfaces between layers. Furthermore, an important feature of the model is the distinction it makes between **specification (layers)** and **implementation (protocols)**, thus leading to openness and flexibility. Openness is the ability to develop new protocols for a particular layer and independently of other layers as network technologies evolve. Openness enables competition, leading to low-cost products. Flexibility is the ability to combine different protocols in stacks, enabling the interchange of protocols in stacks as necessary. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 14). Elsevier Science. Kindle Edition. 2. The vast number of protocols developed for communication at different levels and for meeting requirements of different environments led to the need to organize protocols and their functionalities methodologically. In addition to this structuring, the need to enable free competition in the development of network systems that execute protocols led to development of a standardized reference model for protocols. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 6). Elsevier Science. Kindle Edition. 3. The technological progress of physical media, transmission methods, and communication needs over a long period has led to a rich and complex landscape of network architectures and network systems. The different engineering approaches to the problem of networking, the diverse application areas, and the quest for proprietary solutions have resulted in a large number of complex network designs that differ significantly among them. In order to reduce complexity in network design, most networks are organized in layers, where each **layer** represents a **level of abstraction** focusing on the **communication/networking services it provides**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 11). Elsevier Science. Kindle Edition. 4. Each **layer** **provides** particular **communication** **functionalities** while drawing on the functionalities **provided by the layer below**. The architectures of network systems reflect this layered protocol architecture. The **layer** at which a network system operates (i.e., its placement within the network architecture) **determines** what **functionalities** need to be built into the system. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 6). Elsevier Science. Kindle Edition. 5. The purpose of the OSI reference model has been to specify layers of protocols employed by network nodes to communicate successfully. Thus, two communicating end systems need to have implemented at least one common protocol per corresponding layer. However, **communicating systems do not need to implement full seven-layer protocol stacks**, as described later. The number of layers implemented in communicating system stacks is influenced by the functionality of the systems, that is, the level of abstraction they provide, depending on their goals. For **example**, systems that target to deliver packets between **two networks do not** need to implement **end-to-end reliable transmission** or application layer protocols because of their **specified and intended functionality**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 13). Elsevier Science. Kindle Edition. 6. A layered reference model for protocols enables the **interconnection of heterogeneous networks**, that is, end systems and networks that use different technology, through network systems, \[...] reliable end-to-end connectivity is typically achieved at the transport layer (layer 4), while interconnection of networks can be established at lower layers. \[...] , an end system transmits data packets to a receiving end system traversing two different networks. The **networks** are **interconnected** through a system that implements **two protocol stacks**, one per network, and delivers packets of lower layer protocols between the networks. This is a typical configuration, following the layered OSI Reference model where different DLC protocols are used to establish two logical links and the network system enables the interconnection of the two links into a single network at layer 3 (network layer). Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (pp. 15-16). Elsevier Science. Kindle Edition. 7. **Physical layer**: These protocols employ methods for **bit transmission over physical media** and include such typical functions as **signal processing, timing, and encoding**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 12). Elsevier Science. Kindle Edition. 8. **Data Link Control (DLC) layer**: Its protocols establish **point-to-point communication** over a **physical or logical link**, performing such functions as organization of bits in **data units (frames) organization, error detection,** and **flow control**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 12). Elsevier Science. Kindle Edition. 9. \[...] network systems operating at the **data link layer** can **switch** **frames** at **two levels**: (i) **MAC** or (ii) **LLC**. The ability to switch at the MAC level seems like a natural choice, as all MAC protocols of the IEEE 802. x family—the predominant family of bridged networks—operate under the same standard 802.2 protocol. However, standardized protocols of the 802. x family present significant differences between them in many parameters, such as frame length, priorities, routing methods, etc. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 90). Elsevier Science. Kindle Edition. 10. **Network layer**: These protocols deliver **data units** over a network composed of the **links** **established** through the DLC protocols of **layer 2**. Part of these protocols is **identification** of the **route** **the data units** will follow to reach their target. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 12). Elsevier Science. Kindle Edition. 11. \[...] **link layer systems** can interconnect end systems at the **scale** of a **local area network**. However, scaling a network built of bridges and switches to global scale is not feasible. The filtering database would be very large, broadcast storms would limit the operation efficiency, and routing would be inefficient due to the spanning tree algorithm. Therefore, it is necessary to use **systems** that are specifically designed to achieve **global connectivity**. These network layers systems (or “**routers**”) **overcome** the **limitations** of **link layer systems**. **Routers interconnect local area networks**, and the resulting network of networks is an Internet that spans the globe. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 111). Elsevier Science. Kindle Edition. 12. One important aspect of **router systems** is that there is a **distinction** between the **data plane** and the **control plane**. \[...] Most routers have a dedicated control processor that manages routing computations and error handling. This processor is connected to the switch fabric and thus can be reached by any port. The data path is the data flow and the corresponding sequence of operations that are encountered by a “normal” packet that is simply forwarded from an input port to an output port. The **control plane** handles the **data flow** and **operations** that are performed for **traffic** that contains **routing updates, triggers error handling**, etc. Because the vast majority of packets encountered by the system are conventional data packets, router designs are optimized to handle these packets very efficiently. The **control plane** is typically **more complex** and **not** as **performance** **critical**. When a port encounters a packet that needs to be handled by the control processor, it simply forwards it through the switch fabric to the dedicated control processor. 13. The **data plane** of a router implements a **sequence of operations** that are **performed** for **typical network traffic**. As discussed earlier, these steps include **IP processing** of the arriving packet, **transmission** through the **switch fabric** to the **output port**, and **scheduling** for **outgoing transmission**. One of the key operations in the data plane is to determine to **which output port to send the packet**. This process is known as **route lookup** \[...] Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 117). Elsevier Science. Kindle Edition. 14. The **control plane** of a router handles functions that are **not** directly related to **traffic forwarding**, but that are necessary to ensure correct operation. Typical control plane operations include: • **Exchange of routing messages** and **routing algorithms** • Handling of **error conditions** Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 129). Elsevier Science. Kindle Edition. 15. **Transport layer**: Transport protocols **establish** **end-to-end communication** between end systems over the network **defined** by a **layer 3 protocol**. Often, transport layer protocols provide **reliability**, which refers to complete and correct data transfer between end systems. Reliability can be achieved through mechanisms for **end-to-end error detection**, **retransmissions**, and **flow control**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 12). Elsevier Science. Kindle Edition. 16. The main functionality of the **transport layer** is to provide a **connection** between **processes** on **end hosts**. Communication between processes is the basis of any distributed application. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 141). Elsevier Science. Kindle Edition. 17. The **application layer** is responsible for implementing **distributed applications** and their **protocols**. This layer implements **functionality** accessed by **end users**. When considering distributed applications that use the network for communication, numerous examples come to mind: electronic mail, access to Web documents, interactive audio, streaming video, real-time gaming, etc. 18. The **application layer** can be viewed as consisting of several **sublayers**: **session layer**, **presentation layer**, and **application** **layer**. In the OSI layered protocol model, these sublayers are numbered layers 5–7, respectively. However, in Internet architecture, they are combined into a single application layer. The reason that they are not treated independently is that these layers often provide functionality that is tuned to higher layers. For example, mechanisms implemented to maintain sessions in layer 5 are often specific to the application used in layer 7. Therefore, it can be justified that these **three layers** are treated as a **single application layer**. Note that in some cases this combined application layer is referred to as layer 7, layer 5, or layers 5–7. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 161). Elsevier Science. Kindle Edition. 19. **Session layer**: This layer enables and manages **sessions** for complete data exchange between end nodes. Sessions may consist of **multiple transport layer connections**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 12). Elsevier Science. Kindle Edition. 20. The **presentation layer**, which corresponds to OSI layer 6, handles the **representation of information** used in the communication between end-system applications. **Data** can be **encoded** in a **number of different ways**, and the presentation layer ensures that they are **translated appropriately** for transmission on the network and to be useful to the end-system application. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 162). Elsevier Science. Kindle Edition. 21. **Application layer**: The application layer includes **protocols** that **implement** or **facilitate** **end-to-end distributed applications** over the network. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 13). Elsevier Science. Kindle Edition. 22. “I Heard Calico Is Suggesting Layer 2: I Thought You Were Layer 3! What’s Happening?” Project Calico Documentation, docs.projectcalico.org/v3.5/usage/troubleshooting/faq#i-heard-calico-is-suggesting-layer-2-i-thought-you-were-layer-3-whats-happening. It’s important to distinguish what Calico provides to the workloads hosted in a data center (a purely layer 3 network) with what the Calico project recommends operators use to build their **underlying network fabric**. Calico’s core principle is that **applications** and **workloads** overwhelmingly **need only IP connectivity** to communicate. For this reason we build an **IP-forwarded network** to **connect** the tenant **applications** and **workloads** to **each other**, and the broader world. However, **the underlying physical fabric obviously needs to be set up too**. Here, Calico has discussed how both a layer 2 (see here) or a layer 3 (see here) fabric could be integrated with Calico. This is one of the great strengths of the Calico model: it allows the **infrastructure** to be **decoupled** from what we show to the **tenant applications** and **workloads**. We have some thoughts on different interconnect approaches (as noted above), but just because we say that there are l**ayer 2** and **layer 3** ways of **building the fabric**, and that those decisions may have an impact on **route scale**, does not mean that Calico is “going back to Ethernet” or that we’re recommending layer 2 for tenant applications. In all cases we forward on IP packets, no matter what architecture is used to build the fabric. 23. “Declarative configuration is different from imperative configuration , where you simply take a series of actions (e.g., apt-get install foo ) to modify the world. Years of production experience have taught us that maintaining a written record of the system’s desired state leads to a more manageable, reliable system. **Declarative configuration** enables numerous advantages, including **code review** for **configurations** as well as documenting the **current state of the world** for distributed teams. Additionally, it is the **basis** for all of the **self-healing behaviors** in Kubernetes that keep applications running without user action.” Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 892-896). Kindle Edition. 24. “The combination of **declarative state** stored in a **version control** system and Kubernetes’s **ability** to make **reality** **match** this **declarative state** makes rollback of a change trivially easy. It is simply restating the previous declarative state of the system. With imperative systems this is usually impossible, since while the **imperative instructions** describe how to get you from point A to point B, they **rarely** include the **reverse instructions** that can get you back. “Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 186-190). Kindle Edition. 25. “Because it describes the state of the world, **declarative configuration** does **not** have to be **executed** to be **understood**. Its impact is concretely declared. Since the effects of declarative configuration can be understood before they are executed, **declarative configuration** is far **less error-prone**. Further, the traditional tools of software development, such as source control, code review, and unit testing, can be used in declarative configuration in ways that are impossible for imperative instructions. “ Hightower, Kelsey; Burns, Brendan; Beda, Joe. Kubernetes: Up and Running: Dive into the Future of Infrastructure (Kindle Locations 183-186). Kindle Edition. 26. So **declarative definitions** lend themselves to running **idempotently**. You can safely apply your definitions over and over again, without thinking about it too much. If something is changed to a system outside of the tool, **applying** the **definition** will bring it **back** into line, **eliminating** sources of **configuration drift**. When you need to make a change, you simply modify the definition, and then let the tooling work out what to do. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1275-1278). O'Reilly Media. Kindle Edition. 27. Network systems and computing systems employ **interconnections** to **deliver** **data** among their **components**. In **computing** systems, an **interconnection** is necessary to enable **data transfe**r among the **processor**, the **memory** system, and input and output **(I/O)** subsystems. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 35). Elsevier Science. Kindle Edition. 28. The most **typical** **interconnection** for component communication is the well-known **bus**, which is composed of a set of wires delivering data, address information, and control information (e.g., timing, arbitration). Busses are **shared interconnections** among a number of attached **components**, implementing a **point-to-point communication path** between any two components. The typical operation of a bus is as follows: **components** that need to transmit information to another component **request access** to the **bus**, an **arbiter** **selects** the **component** that will transmit (in case of several requests), and then the **selected** **component** **transmits** its data. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 35). Elsevier Science. Kindle Edition. 29. **Switches** and networks of switches constitute **alternative** **interconnections** to **busses**, implementing **parallel**, nonconflicting **paths** among communicating components and systems. \[...] a **switch** with N inputs and N outputs, employing a typical architecture to implement input-to-output connections dynamically. \[...] **switch**, able to implement **any combination of N parallel**, nonconflicting input-to-output **connections**, is called a **crossbar switch** and constitutes the building block of several switch-based networks. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 36). Elsevier Science. Kindle Edition. 30. \[...] new network technologies that employ **switches** were developed for various environments and applications; these technologies and related protocols include **ATM** \[ 103 ], **Fiber Channel** \[ 89 ], and **InfiniBand** \[ 164 ]. Importantly, switches emerged not only for networks but for **intersystem** **interconnection** as well. For example, the evolution of multicore processors led to the employment of interconnection networks (multiple data paths) of various types, such as switch interconnects, HyperTransport \[ 179 ], and multiple networks, such as the EiB of the Cell BE \[ 27 ]. Furthermore, **switched backplanes** are introduced for network systems, such as **routers**. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 37). Elsevier Science. Kindle Edition. 31. The basic **crossbar switch** was originally developed for interconnection networks of multiprocessors. It is a **2×2 buffer-less switch**, which was named crossbar because it could be in one of **two states**, **cross** or **bar**, as shown in Figure 4-3(a) . The concept of crossbar switching was extended to switches of larger sizes as well, where **switches implement any input-to-output permutation with more inputs and outputs**. The design of a crossbar switch is simple but **expensive**, in terms of resources, as it has to implement all potential permutations of inputs to outputs. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (pp. 38-39). Elsevier Science. Kindle Edition. 32. **Scheduling** is necessary in **switches** because **high load** and **routing conflicts** lead to **contention** for **resources**. In switches that employ input queuing, scheduling is necessary to choose the input queues that will be served at every clock cycle; in switches that employ output queuing, packets contend for output queues and need to be serialized for buffering and transmission over a link. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 43). Elsevier Science. Kindle Edition. 33. **Network adapters** are used to enable **connectivity** on a **single network link** and typically implement a single protocol stack. Network adapters provide the **system implementation** where that **protocol stack** is **executed**. The dependency of the adapter on the physical medium of the attached network usually influences the specification and naming of the adapter in the market. For example, off-the-shelf adapters are known as Ethernet adapters, Wi-Fi adapters, etc. Importantly, because adapters implement single protocol stacks, they are often considered and used as building blocks for multistack systems, such as bridges, routers, and gateways, implementing stacks of appropriate sizes, Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 63). Elsevier Science. Kindle Edition. 34. The **lower layers of the stack**, including the physical layer, which requires **specialized hardware**, constitute the portion mapped on the **adapter**, while the **higher layers** may be mapped on the **end system**. Consider, for example, the configuration of a typical personal computer (PC) with an Ethernet adapter. In the general case, the **PC** with the **adapter** implements at least a **four-layer protocol stack** with Ethernet physical and Media Access Control (MAC) protocols as well as Logical Link Control (LLC), Internet Protocol (IP), and Transmission Control Protocol (TCP), from lower to higher layers. However, the **protocol stack** is implemented **partly** on the **adapter** (e.g., the **Ethernet** physical and **MAC**) and **partly** on the **PC** (e.g., **LLC, IP, and TCP as part of the PC's operating system**). Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (pp. 63-64). Elsevier Science. Kindle Edition. 35. “The **CNF** should **run** **without** **privileges**. **Privileged actions** should be **managed** by the **scheduler** and **environment**”, x-factor-cnfs, Fred Kautz, To achieve process isolation, X-factor CNFs are designed to run in process containers without privileges. Privileged actions may be requested by the container and performed by privileged delegate. Running unprivileged promotes loose coupling between environments, reduces the overall attack surface, and gives the scheduler the ability to clean up after the pod in the case of the pod failing. The X-factor CNF methodology recognizes the need for hardware which requires additional kernel modules. When possible, kernel modules must follow standard Linux kernel device driver standards \[...] and do not affect the kernel's runtime environment beyond enabling the device. These devices must also not be bound directly from the CNF. Instead, they are listed as an interface mechanism and injected into the container runtime by the orchestrator. The existence of a hardware device should not affect other CNFs.Some kernel modifications may be acceptable, e.g. DPDK or drivers. This should be immutable infrastructure with a clean interface for pods. In short, pods should not be allowed to modify their infrastructure. 36. “List mechanisms supported in order of preference”, x-factor-cnfs, Fred Kautz, A given X-factor CNF **lists** in **order of preference** what types of **interface mechanisms** are **supported** for both its **terminating** and **initiating** **interfaces**. **An interface mechanism is any serial/block device, file or socket that is used to transport data in and out of the container**. The most common type of interface mechanism is the Linux interface. Other common mechanisms include SR-IOV, vhost-user, shmem, unix sockets, or serial/block devices. An X-factor CNF may list multiple preferences of what types of interface mechanisms it supports. However, only one mechanism will be wired in for the connection it terminates and only one will be wired in for the connection it initiates. By listing these mechanisms explicitly, the orchestrator can coordinate with both the CNF and data plane to determine what the most fitting interface for the CNF should be. Likewise, the operator may choose to disable certain types of interface mechanisms administratively for a given CNF to preserve resources for other CNFs which are in higher need when resources are scare, such as hardware devices. 37. Considering the need for autonomous operation and high performance, **layer 2 switches** perform all operations that typical **bridges** do. However, due to their focus on performance for dedicated segments, they employ **specialized hardware for frame forwarding**, and some of them even employ cut-through routing techniques instead of the typical store-and-forward technique used in common bridges. Thus, their main **difference** from **bridges** is typically the **technology** used to implement **frame forwarding**, which is mostly **hardware-based**, in contrast to typical **bridges**, which generally are **more programmable** and accommodate a wider range of heterogeneous LANs. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 110). Elsevier Science. Kindle Edition. 38. **Layer 2 switches** can be considered a **special implementation of bridges** and thus can be viewed as a **subset** of **bridging** systems. Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p. 89). Elsevier Science. Kindle Edition. 39. “Concerns over Ethernet at scale” Calico over an Ethernet interconnect fabric, . It has been acknowledged by the industry for years that, beyond a certain size, **classical Ethernet networks** are **unsuitable** for **production** deployment. Although there have been [multiple](https://en.wikipedia.org/wiki/Provider_Backbone_Bridge_Traffic_Engineering) [attempts](https://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_14-3/143_trill.html) [to address](https://en.wikipedia.org/wiki/Virtual_Private_LAN_Service) these issues, the scale-out networking community has, largely abandoned Ethernet for anything other than providing physical point-to-point links in the networking fabric. The principal reasons for **Ethernet** **failures** at **large scale** are: 1. **Large numbers of end points** [1](https://docs.projectcalico.org/v3.5/reference/private-cloud/l2-interconnect-fabric#fn:1). Each **switch** in an Ethernet network must **learn** the **path** to **all Ethernet endpoints** that are connected to the Ethernet network. Learning this amount of state can become a **substantial** task when we are talking about **hundreds of thousands of end points**. 2. **High rate** of **churn** or change in the network. With that many end points, most of them being **ephemeral** (such as virtual machines or containers), there is a large amount of churn in the network. That load of **re-learning** paths can be a **substantial** burden on the **control plane** processor of **most Ethernet switches**. 3. High volumes of **broadcast** **traffic**. As each node on the **Ethernet** network **must** use **Broadcast packets** to **locate peers**, and many use broadcast for other purposes, the resultant packet replication to each and every end point can lead to broadcast storms in large Ethernet networks, effectively consuming most, if not all resources in the network and the attached end points. 4. Spanning tree. **Spanning tree** is the protocol used to **keep** an Ethernet network **from** forming **loops**. The protocol was designed in the era of smaller, simpler networks, and it has not aged well. As the number of links and interconnects in an Ethernet network goes up, many implementations of spanning tree become more **fragile**. Unfortunately, **when** spanning tree **fails** in an Ethernet network, the effect is a **catastrophic** loop or partition (or both) in the network, and, in most cases, difficult to troubleshoot or resolve. While many of these issues are **crippling** at **VM scale** (tens of thousands of end points that live for hours, days, weeks), they will be absolutely **lethal** at **container** **scale** (**hundreds of thousands of end points that live for seconds, minutes, days**). 40. “Introduction” x-factor-cnfs,, Fred Kautz, , X-CNFs also have additional properties not common in 12 Factor Apps which enable their use as a CNF: **State** their **payload type** for easy **service function chaining orchestration**; **List** their supported **mechanisms** supported mechanisms in **order of preference** to facilitate wiring to a **data plane**; **Connect** to Cloud-Native **Microservices** over their **default orchestration-managed network interface**; 41. **Phoenix replacement** is the natural **progression** from **blue-green** using dynamic infrastructure. Rather than keeping an idle instance around between changes, a **new instance** can be **created each time a change is needed**. As with blue-green, the change is tested on the new instance before putting it into use. The **previous instance** can be **kept** up for a **short time**, until the **new instance** has been **proven** in use. But then the **previous instance** is **destroyed**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5694-5697). O'Reilly Media. Kindle Edition. 42. The **canary pattern** involves deploying the **new version** of an **element** **alongside the old one**, and then **routing** some **portion** of usage to the new elements. For example, with version A of an application running on 20 servers, version B may be deployed to two servers. A subset of traffic, perhaps flagged by IP address or by randomly setting a cookie, is sent to the servers for version B. The behavior, performance, and resource usage of the new element can be monitored to validate that it’s ready for wider use. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5724-5728). O'Reilly Media. Kindle Edition. 43. **Blue-green replacement** is the most straightforward pattern to replace an infrastructure element without downtime. This is the blue-green deployment pattern for software applied to infrastructure. It requires **running two instances** of the affected infrastructure, **keeping one of them live** at any point in time. Changes and **upgrades** are **made** to the **offline** **instance**, which can be thoroughly tested before switching usage over to it. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 5681-5685). O'Reilly Media. Kindle Edition. 44. Given **two infrastructure elements** providing a **similar service** for example, two application servers in a cluster the servers **should** be **nearly** **identical**. Their system **software** and **configuration** should be the **same**, except for those bits of configuration that **differentiate** them, like their **IP addresses**. Letting inconsistencies slip into an infrastructure keeps you from being able to trust your automation. If one file server has an 80 GB partition, while another has 100 GB, and a third has 200 GB, then you can’t rely on an action to work the same on all of them. This encourages doing special things for servers that don’t quite match, which leads to **unreliable automation**. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 380-384). O'Reilly Media. Kindle Edition. 45. **Containerization** has the potential to create a **clean separation** between layers of **infrastructure** and the **services** and applications that **run on it**. **Host servers** that run **containers** can be **kept very simple**, without needing to be tailored to the requirements of specific applications, and without imposing constraints on the applications beyond those imposed by containerization and supporting services like logging and monitoring. So **the infrastructure that runs containers consists of generic container hosts**. These can be **stripped** down to a bare **minimum**, including only the minimum toolsets to run containers, and potentially a few agents for monitoring and other administrative tasks. This **simplifies management** of these **hosts**, as they change less often and have fewer things that can break or need updating. It also reduces the surface area for security exploits. Morris, Kief. Infrastructure as Code: Managing Servers in the Cloud (Kindle Locations 1723-1729). O'Reilly Media. Kindle Edition. 46. \[...] communication paradigms and requirements influence **network** **protocols** as well as the systems that execute them. It is important to **differentiate**, however, **protocols** from the **systems** that **execute** them, for several reasons. **Protocols** **define** **communication** **methods**, as explained previously, while network systems execute these protocols. In general, protocols include **mechanisms** that accommodate systems with different **performance** and **reliability** **characteristics**, with methods that regulate traffic flow among systems and mechanisms to detect transmission errors and lead to data retransmission. Thus, the activity of **protocol** **development** and specification does **not** take into account any **specifics** about the system that will execute a protocol and does **not place any specific requirements on it**. This characteristic of protocols not only **enables** the **definition** of **communication** methods **independently** of **technology** to a large degree, but also enables the **development** of economically **scalable** **network** systems, where manufacturers can develop systems that execute the same protocol on **different platforms with different performance, dependability characteristics, and cost.** Serpanos, Dimitrios,Wolf, Tilman. Architecture of Network Systems (The Morgan Kaufmann Series in Computer Architecture and Design) (p.5). Elsevier Science. Kindle Edition. # Contributing and Fundamentals To talk about cloud native applications and systems, we need to define cloud native. CNCF provides a high-level overview of what it means to be cloud native and lists some implementations and the benefits they provide in the [CNCF Cloud Native Definition v1.0](https://github.com/cncf/toc/blob/master/DEFINITION.md). The [cloud native principles](https://github.com/cloud-native-principles/cloud-native-principles) seek to clarify further the underlying fundamental concepts of what it means to be cloud native. NOTE: This is a work in progress. ## Fundamental Concepts * Service discovery * Loose coupling * Microservices * Low overhead/coarse-grained packaging * examples: containers, unikernel * Declarative consumption model * includes APIs and configuration * applies to all layers: infra, workloads, platform, and components * Designed for automation * applies to testing, deployment, application and platform autoscaling, infra and network provisioning * Immutable infrastructure Each of these concepts is covered in more depth in [cloud-native-principles.md](https://github.com/cloud-native-principles/cloud-native-principles/blob/master/cloud-native-principles.md). When these principles are implemented, they lead to benefits such as: * Increased interoperability * Increased resilience * Increased scalability * Lowered risk Defining these concepts help in applying cloud native methodologies to different technology domains. For instance, in the networking and communication service provider spaces, the [Cloud Native Networking papers](/cloud-native-networking-preamble-1) illustrate this. ## Contributing to and Using this content **CONTRIBUTING** We encourage civil discourse and debate about the principles. It's not only acceptable but desirable for the points to be discussed and adapted to where the community as a whole feels we are. In other words, it is ok to disagree with a principle and propose an alternative. How to contribute: * [Add ideas and comments to existing issues](https://github.com/cloud-native-principles/cloud-native-principles/issues) * [create a new issue](https://github.com/cloud-native-principles/cloud-native-principles/issues/new) * Make changes to documents by: * [fork the repo](https://github.com/cloud-native-principles/cloud-native-principles/fork) * make your change in your fork (+branch) * [create a PR](https://github.com/cloud-native-principles/cloud-native-principles/pulls) **LICENSE** This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). You are free to: * Share — copy and redistribute the material in any medium or format * Adapt — remix, transform, and build upon the material for any purpose, even commercially. Under the following terms: * Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. * No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.