Skip to content

Term Project

The goal of the term project is to run your application in a hybrid cloud environment.

CSYE 6225

CSYE 7374

Tech Stack Components

The goal of the term project is to run your application in a hybrid cloud environment. In addition to the application components that you have implemented and are running right now in your stack, you will add few more. Below is list of components

  1. Kubernetes Cluster (version 1.13 or later)
  2. FluentD Logging DaemonSet
  3. Prometheus Metrics Stack
  4. Helm
  5. Main Web Application
  6. AWS RDS
  7. AWS SNS
  8. AWS Lambda
  9. AWS SES
  10. AWS CloudWatch
  11. AWS S3
  12. AWS DynamoDB
  13. AWS VPC (and networking components)
  14. AWS Route53
  15. AWS ELB (for LoadBalancer Services)
  16. AWS EC2
  17. Notifier Web Application
  18. Apache Kafka (& Apache Zookeeper)
  19. CI/CD using Jenkins
  20. GitHub Repositories

Setup GitHub Repository

Term Project Infrastructure

  1. Create a GitHub repository for term project. This must be a private repository that only your team, TAs and instructor can access. Make sure to create empty repository.
  2. GitHub repository name must be csye7374-spring2019-project-infra.
  3. Add your TA and me to your GitHub repository as collaborators. Our emails and GitHub IDs can be found on home page.
  4. Add README.md in your repository.
  5. All Kubernetes resource configurations (yaml files) for infrastructure components such as Jenkins, Helm, Kafka (& Zookeeper), Fluentd, Prometheus, etc. should reside in deploy directory in your repository.
  6. Any manual setups needed to configure one or more component must be documented in the README.md file.

Main Web Application Repository

  1. Create a GitHub repository for term project. This must be a private repository that only your team, TAs and instructor can access. Make sure to create empty repository.
  2. GitHub repository name must be csye7374-spring2019-project-webapp.
  3. Add your TA and me to your GitHub repository as collaborators. Our emails and GitHub IDs can be found on home page.
  4. Add README.md in your repository.
  5. All Kubernetes resource configurations (yaml files) for main web application should reside in deploy directory in your repository.
  6. Dockerfile for each containers must at the root level of your repository.
    1. Dockerfile.webapp should build your container with web application.
    2. Dockerfile.init.db is used to build init container that will bootstrap your RDS database
    3. Dockerfile.init.kafka is used to build init container that will verify and configure Kafka topics as needed before the application starts.

Notifier Web Application Repository

  1. Create a GitHub repository for term project. This must be a private repository that only your team, TAs and instructor can access. Make sure to create empty repository.
  2. GitHub repository name must be csye7374-spring2019-project-notifier.
  3. Add your TA and me to your GitHub repository as collaborators. Our emails and GitHub IDs can be found on home page.
  4. Add README.md in your repository.
  5. All Kubernetes resource configurations (yaml files) for notifier web application should reside in deploy directory in your repository.
  6. Dockerfile for each containers must at the root level of your repository.
    1. Dockerfile.webapp should build your container with web application.
    2. Dockerfile.init.db is used to build init container that will bootstrap your RDS database
    3. Dockerfile.init.kafka is used to build init container that will verify and configure Kafka topics as needed before the application starts.

Lambda Function Repository

  1. Create a GitHub repository for term project. This must be a private repository that only your team, TAs and instructor can access. Make sure to create empty repository.
  2. GitHub repository name must be csye7374-spring2019-project-serverless.
  3. Add your TA and me to your GitHub repository as collaborators. Our emails and GitHub IDs can be found on home page.
  4. Add README.md in your repository.

Readme File

README.md in your repository must contain team member information and instructions on running your playbooks, scripts, etc. to setup Kubernetes cluster and tech stack for your web application.

Ansible Roles & Playbooks

Create Ansible roles & playbooks need to setup Kubernetes cluster and tech stack for your web application.

Kubernetes Cluster Requirements

Ansible playbook should be used to setup Kubernetes cluster using KOPS. You cluster must be setup in HA mode with 3 master nodes with each node in a separate availability zone. You will run minimum of 3 worker nodes in your cluster. Node instance size should provide enough resources for all of the required components to run and allow for autoscaling of your web application.

AWS VPC Setup

Your technical stack components will run in multiple VPCs. One VPC will run your Kubernetes cluster while second VPC will contain all your cloud resources such as RDS, Lambda, etc. You may create both VPCs in same AWS account or use organizations and setup AWS VPCs in different account. Once both VPCs are setup, you will manually configure VPC peering for them.

Docker Registry

A separate AWS ECR docker registry must be created for each application and init containers.

Main Web Application

For password reset endpoint that you had implemented in CSYE 6225, you will now send the message to Kafka topic instead of SNS. You can send the message in same JSON format that you were using with SNS. This application will be the Kafka producer.

Notifier Web Application

Notifier web application will be the Kafka consumer. It should consume messages from all partitions and forward the message to SNS. To keep Notifier web application simple it does not need to track consumer offset. It also does not need to track if message from Kafka topic has been successfully delivered to SNS topic. You can implement this application in any language.

Service Discovery

Apache Kafka (& Apache Zookeeper) setup using Helm charts will setup services for Kafka & Zookeeper.

Main web application will use Kafka message queue to send messages to Notifier web application. Notifier web application will then forward the message to the AWS SNS topic.

Both web applications will “discover” Kafka & Zookeeper endpoints using service discovery.

Configuration Data

All configuration data for both web applications must be passed using ConfigMap.

Application Secrets

All secrets for both web applications must be passed using Secret.

CI/CD Pipeline

Implement CI/CD pipeline for all web applications using Jenkins.

Kafka Topic

Info

For any parameters not specified, you may use default values.

Parameter Value
Topic Name csye7374
Replication Factor 3
Partitions 2

Web Application Deployment Init Containers

Note

All Init containers must be idempotent.

Init Containers to Boostrap RDS

Create init container that will check if RDS is setup with your database schema. If schema exists, init container must do nothing. If schema does not exist, it will create the schema.

Init Containers to Create Kafka Topics

Create init container that will check if Kafka topic exists. Topic will be created if it does not exist.

Main Web Application Endpoints

Endpoints for the main web application must be exposed to the users externally under your domain name. These endpoints must be secured using SSL/TLS certificates. You can use AWS Certificate Manager for SSL certificates. See this blog post for details.