big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,886 Commits 2 Branches 2 Tags 264 MiB
0a65883c50
Commit Graph

25886 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ashutosh Gupta
9c3330c22f
MAPREDUCE-7377. Remove unused imports in MapReduce project (#4299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:34:19 +09:00
Jian Chen
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110) 
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:11:42 +09:00
Szilard Nemeth
eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Hui Fei
dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen
680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
Szilard Nemeth
5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
singer-bin
d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Lei Yang
6a95c3a039
HADOOP-18193:Support nested mount points in INodeTree 
Fixes #4181

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-11 17:01:21 -07:00
Gautham B A
1350539f2d
HDFS-16465. Remove redundant strings.h inclusions (#4279) 
* Remove redundant strings.h inclusions

* strings.h was included in a bunch of
  C/C++ files and were redundant.
* Also, strings.h is not available on
  Windows and thus isn't cross-platform
  compatible.

* Build for all platforms in CI

* Revert "Build for all platforms in CI"

This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb.

* Debug failure on Centos 8

* Skipping pipeline run on
  Centos 7 to debug the
  failure on Centos 8.

* Revert "Debug failure on Centos 8"

This reverts commit e365e34d6fab9df88f4df622910ddb28a8c8796f.
 2022-05-11 23:04:22 +05:30
Szilard Nemeth
9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth
f143e99428 MAPREDUCE-7379. RMContainerRequestor#makeRemoteRequest has confusing log message. Contributed by Ashutosh Gupta 2022-05-11 16:55:19 +02:00
Szilard Nemeth
eeedc0c7e4 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-11 14:39:42 +02:00
Szilard Nemeth
b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989
09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Viraj Jasani
6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
hchaverr
99a83fd4bd
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times 
Fixes #4266

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-10 13:58:39 -07:00
daimin
d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
Ayush Saxena
665ada6d21
MAPREDUCE-7376. AggregateWordCount fetches wrong results. (#4257). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2022-05-09 22:56:14 +05:30
Gautham B A
a394c2b031
HDFS-16572. Fix typo in readme of hadoop-project-dist 2022-05-08 23:47:13 +05:30
Ashutosh Gupta
40a8b9a6a5
HADOOP-17479. Fix the examples of hadoop config prefix (#4197) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:09:24 +09:00
Ashutosh Gupta
01027e52a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:05:34 +09:00
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
daimin
29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Gautham B A
d346be9bd2
HDFS-16564. Use uint32_t for hdfs_find (#4245) 
* hdfs_find uses u_int32_t type for
  storing the value for the max-depth
  command line argument.
* The type u_int32_t isn't standard,
  isn't available on Windows and thus
  breaks cross-platform compatibility.
  We need to replace this with uint32_t
  which is available on all platforms since
  it's part of the C++ standard.
 2022-05-04 20:19:08 +05:30
Daniel Carl Jones
4230162a76
HADOOP-18168. Fix S3A ITestMarkerTool use of purged public bucket. (#4140) 
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones

Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
 2022-05-03 14:28:08 +01:00
Steve Loughran
6ec39d45c9 Revert "HADOOP-18168. . (#4140)" 
This reverts commit 6ab7b72cd6.
 2022-05-03 14:27:52 +01:00
Daniel Carl Jones
6ab7b72cd6
HADOOP-18168. . (#4140) 
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated  test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones
 2022-05-03 14:26:52 +01:00
Steve Loughran
aa917b1cd4
HADOOP-18198. add -mvnargs option to create-release command line (#4239) 
This allows for builds to be run with options like
--mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false"

Contributed by Ayush Saxena.

Change-Id: I396e82d0915d679657d063a948f865041bcdde29
 2022-05-03 14:08:39 +01:00
Murali Krishna
9807dfd25c
HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240) 2022-05-03 17:42:45 +05:30
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
jteagles
d4a91bd0c0
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:14:26 +09:00
Ashutosh Gupta
a74acc755e
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:08:30 +09:00
Akira Ajisaka
4653fcd704
HADOOP-18219. Fix shaded client test failure (#4254) 
This patch MUST be cherrypicked into all branches with
HADOOP-15983 Use jersey-json that is built to use jackson2

Contributed by Akira Ajisaka
 2022-05-02 12:25:09 +01:00
Ashutosh Gupta
23e1511cd0
HDFS-16256. Minor fix in HDFS Fedbalance document (#4192) 2022-05-02 08:08:12 +08:00
Gautham B A
88155cebe9
HDFS-16468. Define ssize_t for Windows (#4228) 
* Some C/C++ files use ssize_t data type.
  This isn't available for Windows and we
  need to define an alias for this and set it
  to an appropriate type to make it cross
  platform compatible.
 2022-04-29 22:25:09 +05:30
smarthan
7bd7725532
HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211). Contributed by Chengwei Wang. 2022-04-29 22:59:23 +08:00
Viraj Jasani
b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack
9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
 2022-04-28 10:28:01 -07:00
stack
4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)" 
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
 2022-04-28 10:26:46 -07:00
huaxiangsun
bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
 2022-04-28 10:24:37 -07:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
Felix Nguyen
b4ff49a394
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168) 2022-04-27 14:42:30 +08:00
smarthan
f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
liever18
2632d49282
HADOOP-18216. io.file.buffer.size must be positive (#4220) 
Documents that the buffer size must be > 0

Contributed by Jingxuan Fu.
 2022-04-26 19:53:11 +01:00
hchaverri
d60262fe00
HADOOP-18167. Add metrics to track delegation token secret manager op… (#4092) 
* HADOOP-18167. Add metrics to track delegation token secret manager operations
 2022-04-26 16:20:11 +00:00
litao
f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Ashutosh Gupta
006767eb94
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md (#4221) 2022-04-26 09:51:11 +08:00
Ashutosh Gupta
fb13c1e4a8
MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". (#4223) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:29:35 +09:00
Ashutosh Gupta
89279f0b90
HDFS-16255. RBF: Fix dead link to fedbalance document (#4193) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:11:41 +09:00
Steve Loughran
44ae2fa8e5
HADOOP-16202. Enhanced openFile(): hadoop-azure changes. (#2584/4) 
Stops the abfs connector warning if openFile().withFileStatus()
is invoked with a FileStatus is not an abfs VersionedFileStatus.

Contributed by Steve Loughran.

Change-Id: I85076b365eb30aaef2ed35139fa8714efd4d048e
 2022-04-24 17:33:05 +01:00