big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,123 Commits 2 Branches 2 Tags 264 MiB
38f10c657e
Commit Graph

79 Commits

Author SHA1 Message Date
slfan1989
4c3d4e6a57
HADOOP-19038. Improve create-release RUN script. (#6448) Contributed by Shilun Fan. 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-18 19:12:12 +08:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows (#5943) 2023-11-01 09:10:15 -07:00
Gautham B A
a80e3dba3b
HADOOP-18734. Create qbt.sh symlink on Windows (#5626) 2023-05-08 09:55:15 -07:00
Gautham B A
0d06fd77de
HADOOP-18134. Setup Jenkins nightly CI for Windows 10 (#5062) 
This PR gets Yetus to run on Windows 10
against the Hadoop codebase. It introduces
the following changes to allow us to setup
the nightly CI on Jenkins for Hadoop on
Windows 10.
* Hadoop personality changes for Yetus.
  Additional arguments have been passed,
  which are necessary to build and test
  Hadoop on Windows 10.
* Docker image for building Hadoop on
  Windows 10.
  Installs the necessary tools that are
  needed to run Yetus.
* dev-support/jenkins.sh file.
  Passing of some flags are handled here
  which are needed for the nightly CI.
 2023-05-03 22:44:54 +05:30
Ashutosh Gupta
84b0455cf8
HADOOP-18240. Upgrade Yetus to 0.14.0 (#4328) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-25 17:31:18 +09:00
Steve Loughran
aa917b1cd4
HADOOP-18198. add -mvnargs option to create-release command line (#4239) 
This allows for builds to be run with options like
--mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false"

Contributed by Ayush Saxena.

Change-Id: I396e82d0915d679657d063a948f865041bcdde29
 2022-05-03 14:08:39 +01:00
Steve Loughran
d7fd61d616
HADOOP-18202. create-release fails fatal: unsafe repository (#4188) 
Since April 2022/CVE-2022-24765, git refuses to work in directories
whose owner != the current user, unless explicitly told to trust it.

This patches the create-release script to trust the /build/source
dir mounted from the hosting OS, whose userid is inevitably different
from that of the account in the container running git.

Contributed by: Steve Loughran, Ayush Saxena and the new git error messages
 2022-04-18 19:24:35 +01:00
Liang-Chi Hsieh
a424878e0e
HADOOP-17891. Fix compilation error under skipShade (ADDENDUM) (#3441) 2021-09-16 09:58:28 -07:00
Liang-Chi Hsieh
b8f7c7527a
HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries (#3385) 2021-09-14 11:17:18 -07:00
Gautham B A
1c0b2edde9
HADOOP-17727. Modularize docker images (#3043) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-06-07 20:11:08 -07:00
Wei-Chiu Chuang
110cda3de6
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-17 20:21:54 -07:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Akira Ajisaka
63aa142b77
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 2021-03-09 11:10:10 +09:00
Akira Ajisaka
b9b49ed956
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (#1738) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
 2021-02-20 17:18:23 +09:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Masatake Iwasaki
9aefe72ed6
HDFS-15743. Fix -Pdist build failure of hadoop-hdfs-native-client. (#2569) 2020-12-22 07:19:44 +09:00
Akira Ajisaka
34e2abd811
HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. (#2433) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-11-10 13:58:27 +09:00
Liang-Chi Hsieh
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 
This switches the SnappyCodec to use the java-snappy codec, rather than the native one. 

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh
 2020-10-06 17:07:54 +01:00
Chao Sun
fefacf2578
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-18 12:08:58 +09:00
Akira Ajisaka
9c290c08db
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.
 2020-06-03 18:01:37 +09:00
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Vinayakumar B
4d24d99e85
HADOOP-16985. Handle release package related issues (#1957) 2020-04-15 20:38:19 +05:30
Vinayakumar B
52b360a928
HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. (#1801) 2020-01-13 10:40:29 +05:30
Masatake Iwasaki
ccca5f4a40 HADOOP-16441. if use -Dbundle.openssl=true, bundled with unnecessary libk5crypto.*. Contributed by KWON BYUNGCHANG. 2019-12-04 14:25:46 +09:00
Akira Ajisaka
567091aa9b
HADOOP-15958. Revisiting LICENSE and NOTICE files. 
This closes #1307

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-27 13:47:12 +09:00
Akira Ajisaka
5e156b9dde
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298
 2019-08-22 13:01:11 +09:00
Akira Ajisaka
34dd9ee366
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy. 
This closes #1243
 2019-08-22 11:01:02 +09:00
Erik Krogen
ab0b180ddb HDFS-12345 Add Dynamometer to hadoop-tools, a tool for scale testing the HDFS NameNode with real metadata and workloads. Contributed by Erik Krogen. 2019-06-25 08:07:39 -07:00
Sammi Chen
d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Wanqiang Ji
d23e8e9232
HADOOP-16334. Fix yetus-wrapper not working when HADOOP_YETUS_VERSION >= 0.9.0 
This closes #864

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-05-30 15:11:05 +09:00
Akira Ajisaka
af5958f7b6
HADOOP-15925. The config and log of gpg-agent are removed in create-release script. Contributed by Dinesh Chitlangia. 2018-11-16 13:16:59 +09:00
Akira Ajisaka
df5e863fee
HADOOP-15930. Exclude MD5 checksum files from release artifact. 2018-11-15 10:45:34 +09:00
Akira Ajisaka
703b2860a4
HADOOP-15923. create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agent. 2018-11-13 13:40:43 +09:00
Eric Yang
e2a9fa8448 HADOOP-15821. Move YARN Registry to Hadoop Registry. 
Contributed by Íñigo Goiri
 2018-10-19 19:46:48 -04:00
Bharat Viswanadham
62f817d32e HDDS-447. Separate ozone-dist and hadoop-dist projects with real classpath separation. Contributed by Elek Marton. 2018-09-24 10:10:11 -07:00
Márton Elek
b6ad84ebb1 HDDS-468. Add version number to datanode plugin and ozone file system jar. Contributed by Bharat Viswanadham. 
(cherry picked from commit a71aea732315ffc5c99c011369adfbeddb9a8c01)
 2018-09-18 11:01:52 +02:00
Bharat Viswanadham
8b2f5e60fa HDDS-352. Separate install and testing phases in acceptance tests. Contributed by Elek Marton. 2018-09-17 17:13:52 -07:00
Bharat Viswanadham
3d89c3e73e HDDS-463. Fix the release packaging of the ozone distribution. Contributed by Elek Marton. 2018-09-17 11:49:09 -07:00
Anu Engineer
8af8453589 HDDS-435. Enhance the existing ozone documentation. 
Contributed by Elek, Marton.
 2018-09-17 10:46:28 -07:00
Bharat Viswanadham
b95aa56062 HDDS-446. Provide shaded artifact to start ozone service as a datanode plugin. Contributed by Elek Marton 2018-09-14 14:12:03 -07:00
Anu Engineer
7b5886bf78 HDDS-222. Remove hdfs command line from ozone distribution. 
Contributed by Elek, Marton.
 2018-09-11 10:19:50 -07:00
Allen Wittenauer
4b8c2b1c1c
HADOOP-14671. Upgrade Apache Yetus to 0.8.0 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-09-11 08:55:34 -07:00
Mukul Kumar Singh
692736f7cf HDDS-280. Support ozone dist-start-stitching on openbsd/osx. Contributed by Elek, Marton. 2018-08-30 02:21:24 +05:30
Xiaoyu Yao
9dd5d5ba71 HDDS-218. add existing docker-compose files to the ozone release artifact. Contributed by Elek Marton. 2018-08-17 07:58:24 -07:00
Arpit Agarwal
061b168529 HDDS-167. Rename KeySpaceManager to OzoneManager. Contributed by Arpit Agarwal. 2018-07-06 12:09:05 -07:00
Anu Engineer
e16e5b307d HDDS-177. Create a releasable ozonefs artifact 
Contributed by Marton, Elek.
 2018-06-24 01:05:04 -07:00
Mukul Kumar Singh
63fc5873ce HDDS-84. The root directory of ozone.tar.gz should contain the version string. Contributed by Elek, Marton. 2018-05-23 21:07:37 +05:30
Anu Engineer
481bfdb94f HDDS-89. Create ozone specific inline documentation as part of the build. 
Contributed by Elek, Marton.
 2018-05-22 10:49:10 -07:00