hadoop/dev-support/docker

Docker images for building Hadoop

This folder contains the Dockerfiles for building Hadoop on various platforms.

Dependency management

The mode of installation of the dependencies needed for building Hadoop varies from one platform to the other. Different platforms have different toolchains. Some packages tend to be polymorphic across platforms and most commonly, a package that's readily available in one platform's toolchain isn't available on another. We thus, resort to building and installing the package from source, causing duplication of code since this needs to be done for all the Dockerfiles pertaining to all the platforms. We need a system to track a dependency - for a package - for a platform. Thus, there's a lot of diversity that needs to be handled for managing package dependencies and pkg-resolver caters to that.

Supported platforms

pkg-resolver/platforms.json contains a list of the supported platforms for dependency management.

Package dependencies

pkg-resolver/packages.json maps a dependency to a given platform. Here's the schema of this JSON.

{
  "dependency_1": {
    "platform_1": "package_1",
    "platform_2": [
      "package_1",
      "package_2"
    ]
  },
  "dependency_2": {
    "platform_1": [
      "package_1",
      "package_2",
      "package_3"
    ]
  }
}

The root JSON element contains unique dependency children. This in turn contains the name of the _ platforms_ and the list of packages to be installed for that platform. Just to give an example of how to interpret the above JSON -

1. For dependency_1, package_1 needs to be installed for platform_1.
2. For dependency_2, package_1 and package_2 needs to be installed for platform_2.
3. For dependency_2, package_1, package_3 and package_3 needs to be installed for platform_1.

Standalone packages

Most commonly, some packages are not available across the toolchains in various platforms. Thus, we would need to build and install them. Since we need to do this across all the Dockerfiles for all the platforms, it could lead to code duplication and managing them becomes a hassle. Thus, we put the build steps in a pkg-resolver/install-<package>.sh and invoke this in all the Dockerfiles.