Module: Big Data Platforms

How do we handle large data sets?

Learning Outcomes

Understand how to process very large data sets

You understand

what are the problems caused by very large data sets
what are some of the technical approaches to processing large data sets
what are some of the platforms and tools for processing large data sets
how does virtualization help with processing large data sets
the differences between virtual machines and containers
what is a compute cluster and how does it work
how to setup a computer cluster
what are some of the programming paradigms for cluster computing

Readings

Hadoop

Slides

Spark

Slides

Spark Cluster Overview

Spark Documentation

A Beginner-Friendly Introduction to Containers, VMs and Docker

Virtualization, VMs, Containers, & Docker

Apache Yarn

Overview of Yarn.

Spark: Cluster Computing with Working Sets

Hotcloud 2010 Paper

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-memory Cluster Computing

NSDI 2012 Paper

A Tale of Three Apache Spark APIs: RDDs, DataFrames and Datasets

Web Article

Who uses Spark

Notable users of Spark

Spark SQL, DataFrames and Datasets Guide

Spark Documentation

Machine Learning Library (MLlib) Guide

Spark Documentation

Experiential Learning

Ex5. Containers

Learn how to use Docker

Docker Hands-on

Ex6. Hadoop

Learn how to install & use Hadoop

Hadoop Hands-on