1c0b2edde9
Reviewed-by: Inigo Goiri <inigoiri@apache.org> |
||
---|---|---|
.. | ||
pkg-resolver | ||
Dockerfile | ||
Dockerfile_aarch64 | ||
Dockerfile_centos_7 | ||
Dockerfile_centos_8 | ||
hadoop_env_checks.sh | ||
README.md |
Docker images for building Hadoop
This folder contains the Dockerfiles for building Hadoop on various platforms.
Dependency management
The mode of installation of the dependencies needed for building Hadoop varies from one platform to
the other. Different platforms have different toolchains. Some packages tend to be polymorphic
across platforms and most commonly, a package that's readily available in one platform's toolchain
isn't available on another. We thus, resort to building and installing the package from source,
causing duplication of code since this needs to be done for all the Dockerfiles pertaining to all
the platforms. We need a system to track a dependency - for a package - for a platform. Thus,
there's a lot of diversity that needs to be handled for managing package dependencies and
pkg-resolver
caters to that.
Supported platforms
pkg-resolver/platforms.json
contains a list of the supported platforms for dependency management.
Package dependencies
pkg-resolver/packages.json
maps a dependency to a given platform. Here's the schema of this JSON.
{
"dependency_1": {
"platform_1": "package_1",
"platform_2": [
"package_1",
"package_2"
]
},
"dependency_2": {
"platform_1": [
"package_1",
"package_2",
"package_3"
]
}
}
The root JSON element contains unique dependency children. This in turn contains the name of the _ platforms_ and the list of packages to be installed for that platform. Just to give an example of how to interpret the above JSON -
- For
dependency_1
,package_1
needs to be installed forplatform_1
. - For
dependency_2
,package_1
andpackage_2
needs to be installed forplatform_2
. - For
dependency_2
,package_1
,package_3
andpackage_3
needs to be installed forplatform_1
.
Standalone packages
Most commonly, some packages are not available across the toolchains in various platforms. Thus, we
would need to build and install them. Since we need to do this across all the Dockerfiles for all
the platforms, it could lead to code duplication and managing them becomes a hassle. Thus, we put
the build steps in a pkg-resolver/install-<package>.sh
and invoke this in all the Dockerfiles.