Hadoop

1. Test Hadoop Standalone Execution on Bare Metal

Follow the instructions to download, install and run hadoop for Hadoop Single Cluster up to the standalone operation mode.

Plan the number of nodes in your cluster and the associated networking architecture (see Creating Bridges and/or Docker container networking)
Create/run the docker containers for the cluster (see Primer below)
Check connectivity of each node
Create user accounts on each node
Enable passwordless ssh on each node

Test Hadoop Standalone on one docker container (make sure all dependencies are met)
Follow the instructions for Cluster Setup
Copy some input data into hdfs
Run a hadoop job (use the examples)

docker images = list all images
docker image {ls, rm, build}
docker rmi = remove image
docker run imagename
docker run -it imagename
docker run -it –name=containername –hostname=myhostname image
docker run -p 127.0.0.1:80:8080 ubuntu bash = map port 8080 of the container to 127.0.0.1:80 of the host
docker ps -a = list all containers including stopped ones
docker rm imagename = remove container
docker start imagename = start a stopped container
docker build -t imagetag imagedir
docker network ls
docker network inspect networkname
docker network create -d bridge my_bridge
docker run -d –net=my_bridge –name db training/postgres