big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,822 Commits 2 Branches 2 Tags 264 MiB
911e9e0c01
Commit Graph

26822 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
Viraj Jasani
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points (#5554). Contributed by Viraj Jasani. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-11 08:45:34 +05:30
zhtttylz
5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API (#5602). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 23:10:19 +05:30
slfan1989
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00
cxzl25
be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist (#5565) 2023-05-10 15:16:33 +08:00
slfan1989
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. (#5363) 2023-05-09 16:17:23 -07:00
slfan1989
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
zhangshuyan
03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). (#5622). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-09 10:05:28 +08:00
slfan1989
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. (#5403) 2023-05-08 10:13:09 -07:00
Gautham B A
a80e3dba3b
HADOOP-18734. Create qbt.sh symlink on Windows (#5626) 2023-05-08 09:55:15 -07:00
ZanderXu
4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. (#5200). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-06 08:50:21 +05:30
slfan1989
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616) 2023-05-05 13:43:40 -07:00
slfan1989
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. (#5609) 2023-05-05 13:41:38 -07:00
Gautham B A
c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 (#5618) 2023-05-05 13:08:58 -07:00
Chris
372631c566
HADOOP-18706. Improve S3ABlockOutputStream recovery (#5563) 
Contributed by Chris Bevard
 2023-05-05 11:57:42 +01:00
Szilard Nemeth
c7699d3dcd YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta 2023-05-04 22:16:18 -04:00
Viraj Jasani
ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity (#5615) 2023-05-04 09:45:40 -07:00
Dongjoon Hyun
27776ac45e
HADOOP-18727. Fix WriteOperations.listMultipartUploads function description (#5613) 
Contributed by Dongjoon Hyun
 2023-05-04 13:03:48 +01:00
Peter Szucs
bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum (#5614) 2023-05-04 12:27:25 +02:00
Gautham B A
0d06fd77de
HADOOP-18134. Setup Jenkins nightly CI for Windows 10 (#5062) 
This PR gets Yetus to run on Windows 10
against the Hadoop codebase. It introduces
the following changes to allow us to setup
the nightly CI on Jenkins for Hadoop on
Windows 10.
* Hadoop personality changes for Yetus.
  Additional arguments have been passed,
  which are necessary to build and test
  Hadoop on Windows 10.
* Docker image for building Hadoop on
  Windows 10.
  Installs the necessary tools that are
  needed to run Yetus.
* dev-support/jenkins.sh file.
  Passing of some flags are handled here
  which are needed for the nightly CI.
 2023-05-03 22:44:54 +05:30
slfan1989
476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. (#5587) 2023-05-03 09:21:56 -07:00
slfan1989
c1d10f3872
YARN-9049. Add application submit data to state store. (#5606) 2023-05-03 09:19:54 -07:00
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
slfan1989
668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. (#5596) 2023-05-02 13:46:06 -07:00
zhangshuyan
fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. (#5612). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-03 00:09:36 +08:00
Viraj Jasani
bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile (#5580) 
Contributed by Viraj Jasani
 2023-05-02 15:21:46 +01:00
Szilard Nemeth
73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase (#5455) 2023-05-02 15:52:57 +02:00
slfan1989
87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. (#5598) 2023-05-01 16:01:50 -07:00
Gautham B A
5147106b59
HADOOP-18725. Avoid cross-platform build for irrelevant Dockerfile changes (#5610) 2023-05-01 09:35:50 -07:00
SevenAddSix
1079890ae3
HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus (#4665). Contributed by Jiale Qi. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-01 21:22:26 +05:30
Pralabh Kumar
d75c6d9d57
HADOOP-18715. Add debug log for getting details of tokenKindMap (#5608). Contributed by Pralabh Kumar. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:28:49 +05:30
fanluoo
1a2cd965a7
HDFS-16897. Fix abundant Broken pipe exception in BlockSender (#5329). Contributed by fanluo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:24:56 +05:30
Hexiaoqiao
70c0aa342e
YARN-11482. Fix bug of DRF comparision DominantResourceFairnessComparator2 in fair scheduler. (#5607). Contributed by Xiaoqiao He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2023-04-29 11:18:42 +08:00
slfan1989
5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. (#5570) 
Co-authored-by: slfan1989 <louj1988@@>
 2023-04-28 14:11:13 -07:00
wangzhaohui
0e63152218
HDFS-16995. Remove unused parameters at NameNodeHttpServer#initWebHdfs (#5601). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-28 05:25:01 +05:30
Tsz-Wo Nicholas Sze
d9576bb9ee
HDFS-16972. Delete a snapshot may deleteCurrentFile. (#5532) 2023-04-27 09:17:47 -07:00
Riya Khandelwal
60a7e8acaa
YARN-11459 Changed label called "max resource" on UIv1 and UIv2 (#5527) 2023-04-27 15:25:25 +02:00
yl09099
245fde17d7
YARN-11474.The yarn queue list is displayed on the CLI (#5577) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-27 18:05:48 +08:00
Steve Loughran
eb749ddd4d
HADOOP-18695. S3A: reject multipart copy requests when disabled (#5548) 
Contributed by Steve Loughran.
 2023-04-27 10:59:46 +01:00
slfan1989
55eebcf277
YARN-11378. [Federation] Support checkForDecommissioningNodes、refreshClusterMaxPriority API's for Federation. (#5551) 2023-04-25 14:12:38 -07:00
slfan1989
a716459cdf
YARN-11437. [Federation] SQLFederationStateStore Support Version. (#5589) 2023-04-25 14:09:45 -07:00
cxzl25
5af0845076
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-26 02:11:11 +05:30
Steve Loughran
b6b9bd67bb
MAPREDUCE-7437. MR Fetcher class to use an AtomicInteger to generate IDs. (#5579) 
...as until now it wasn't thread safe

Contributed by Steve Loughran
 2023-04-25 19:53:40 +01:00
Sebastian Baunsgaard
6aac6cb212
HADOOP-18660. Filesystem Spelling Mistake (#5475). Contributed by Sebastian Baunsgaard. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 21:44:04 +05:30
cxzl25
2f66f0b83a
HADOOP-18694. Client.Connection#updateAddress needs to ensure that address is resolved before updating (#5542). Contributed by dzcxzl. 
Reviewed-by: Steve Vaughan <email@stevevaughan.me>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-25 03:52:49 +05:30
zhtttylz
c9e0af9961
HDFS-16981. Support getFileLinkStatus API in WebHDFS (#5572). Contributed by Hualong Zhang. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 03:30:56 +05:30
Tsz-Wo Nicholas Sze
dc78849f27
HDFS-16975. FileWithSnapshotFeature.isCurrentFileDeleted is not reloaded from FSImage. (#5546) 2023-04-24 09:04:28 -07:00
Tamas Domok
05e6dc19ea
HADOOP-18705. ABFS should exclude incompatible credential providers. (#5560) 
Contributed by Tamas Domok.
 2023-04-24 15:46:40 +01:00
zhangshuyan
6a23c376c9
HDFS-16986. EC: Fix locationBudget in getListing(). (#5582). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-24 18:53:25 +08:00
Doroszlai, Attila
5b23224970
HADOOP-18714. Wrong StringUtils.join() called in AbstractContractRootDirectoryTest (#5578) 2023-04-24 09:17:12 +02:00