Learning Outcomes

Understand how to virtualization works

You understand

• what is virtualization
• how virtualization works
• what is a hypervisor
• what are virtual machines
• what are containers and images
• what is the difference between VMs and containers

You are able to

• install Docker
• create and use docker containers
• create docker images
• map ports between host and containers

Referencing modules: Virtualization

Understand how to program using sockets

You understand

• basic network communications using TCP and UDP
• client-server paradigm

You are able to

• design programs using the client-server paradigm
• write programs using the sockets API
• serialize/deserialize data to be sent over sockets

Referencing modules: Sockets & Client-Server Programming

Understand how parallel & distributed databases work

You understand

• the need for parallelism in data processing
• the different types of parallelism
• the different parallel processing architectures
• how to measure the performance of parallel processing
• the different data partitioning methods
• basic parallel query processing
• basic parallel programming using threads

Referencing modules: Parallel Databases

Understand how data storage is organized

You understand

• how to use an array of disks to store more data
• how to deal with disk failures
• how to scale storage systems to handle more and more data
• the different ways of connecting disk arrays to processing units
• how database records are organized and stored on disks

Referencing modules: Storage

Understand data-intensive application architecture

You understand

• what are data-intensive applications
• components of data-intensive applications
• what are the three key requirements of data intensive applications
• common architecture of data-intensive applications
• what is reliability
• what is scalability
• what is maintainability
• different data models
• query languages associated with different data models

Referencing modules: Data-Intensive Application Design

Understand query processing in parallel databases

You understand

• the query processing process including query optimization
• how query execution plans are represented
• how query execution plans are manipulated algebraically to handle parallelism
• how different query operators work especially in a parallel environment
• how to analyze query execution plans and estimate their costs

Referencing modules: Query Processing

Understand concurrency control in parallel databases

You understand

• transaction semantics in a parallel/distributed setting
• theoretical model for transactions
• the different types of concurrency control algorithms
• locking-based concurrency control
• timestamp-based concurrency control

Referencing modules: Transactions & Concurrency Control

Understand reliability in parallel databases

You understand

• Reliability and availability concepts
• Failures and recovery
• Local reliability protocols
• Distributed reliability protocols

Referencing modules: Reliability, Consistency & Consensus

Understand database partitioning and replication

You understand

• why is replication needed
• consistency requirements for replication
• replication protocols for maintaining consistency
• synchronous versus asynchronous replication
• leader-based versus distributed/leaderless replication
• other factors in replication such as transparency, distribution of master copy etc.
• how replication is done using different techniques (WAL,change capture)
• problems associated with replication lag
• conflict resolution in replication systems
• how causal ordering of operations is used for conflict resolution
• how to use data partitioning and replication together
• different ways of partitioning data (hash and range)
• the problem of skew in partitions
• how partitioning affect indexing
• how to rebalance partitions

Referencing modules: Replication & Partitioning

Understand Basic Peer-to-peer Concepts

You understand

• basic peer-to-peer concepts
• how distributed hash tables work
• how peer-to-peer concepts are applied in applications like bittorrent and bitcoin

Referencing modules: P2P Systems & Blockchain

Understand NoSQL systems

You understand

• alternatives to the relational data model
• what are document-based data models
• what are key-value data models (e.g. JSON)
• what are graph data models
• what are the query languages for non-relational data models
• what are NoSQL systems
• the difference between SQL and NoSQL systems
• the implications of NoSQL to application development

Referencing modules: Non-SQL Data Models & Systems

Understand log analytis problems and technology

You understand

• sources of log data
• the problems that log analysis can address
• the technolog and systems for ingesting, indexing, querying, and visualizing log data

Referencing modules: Log Analytics

Understand Business Intelligence and Data Warehousing

You understand

• business intelligence, data integration & data warehousing
• star and snowflake schema used in data warehousing
• OLAP queries and their application to reports and dashboards
• the process of setting up a BI system
• how to integrate data from diverse data sources into a data warehouse.

Referencing modules: Data Warehousing

Understand How to Engineer Big Data Systems using Lambda Architecture

You understand

• the limitations of incremental architectures
• the desirable properties of big data systems
• what the lambda architecture is and how it relates to big data systems
• the different trade-offs
• the assumptions made on the data and the application state
• the data model used in lambda architecture

Referencing modules: Lambda Architecture

Understand How to Engineer Batch Processing Systems

You understand

• the requirements of the batch layer in lambda architecture
• how distributed file systems work
• how map-reduce works
• the trade-offs between alternatives
• how to model queries using pipe diagrams
• how to translate pipe diagrams to Map-Reduce jobs.
• how to model queries using JCascalog
• the relationship between queries and batch views

Referencing modules: Batch Layer : Map-Reduce

Understand How to Engineer the Serving Layer

You understand

• the requirements of the serving layer in lambda architecture
• the trade-offs between alternatives

Referencing modules: Serving Layer

Understand How to Engineer the Speed Layer

You understand

• the requirements of the speed layer in lambda architecture
• the trade-offs between alternatives

Referencing modules: Speed Layer

Understand the key ethical questions on Big Data

You understand

• what is ethics
• the different approaches to values to deal with ethical questions
• what are the ethical questions in Big Data

Referencing modules: Big Data Ethics

Apply parallel data processing concepts

You are able to

• install, configure and use data management tools
• extract, transform, load large data sets
• query and update large data sets to extract relevant information
• compare different data management solutions quantitatively

Referencing modules: Project