big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,539 Commits 2 Branches 2 Tags 264 MiB
422bf3b24c
Commit Graph

26539 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pranav Saxena
c67c2b7569
HADOOP-18546. ABFS. disable purging list of in progress reads in abfs stream close() (#5176) 
This addresses HADOOP-18521, "ABFS ReadBufferManager buffer sharing
across concurrent HTTP requests" by not trying to cancel
in progress reads.

It supercedes HADOOP-18528, which disables the prefetching.
If that patch is applied *after* this one, prefetching
will be disabled.

As well as changing the default value in the code,
core-default.xml is updated to set
fs.azure.enable.readahead = true

As a result, if Configuration.get("fs.azure.enable.readahead")
returns a non-null value, then it can be inferred that
it was set in or core-default.xml (the fix is present)
or in core-site.xml (someone asked for it).

Contributed by Pranav Saxena.
 2022-12-07 20:15:45 +00:00
Murali Krishna
2e88096266
HADOOP-18538. Upgrade kafka to 2.8.2 (#5164) 
Signed-off-by: Brahma Reddy Battula <brahma@apache.org>
 2022-12-06 22:27:46 +05:30
slfan1989
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. (#5146) 2022-12-06 08:17:05 -08:00
Akshat Bordia
86ac1ad9e5
YARN-10978. Fix ApplicationClassLoader to Correctly Expand Glob for Windows Path (#3558) 2022-12-06 16:39:49 +05:30
Gautham B A
dadd3d9138
YARN-11386. Fix issue with classpath resolution (#5183) 
* This PR ensures that all the special notations such as
  <CPS> are resolved before getting added to classpath.
 2022-12-06 16:32:26 +05:30
Steve Loughran
b666075a41
HADOOP-18560. AvroFSInput opens a stream twice and discards the second one without closing (#5186) 
This is needed for branches with  the hadoop-common changes of
HADOOP-16202. Enhanced openFile()
 2022-12-06 09:58:51 +00:00
Steve Loughran
84b33b897c
HADOOP-18470. index.md update for 3.3.5 release 2022-12-05 16:13:24 +00:00
ZanderXu
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces (#5123) 2022-12-05 16:15:47 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
slfan1989
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5179) 2022-12-02 10:56:17 -08:00
slfan1989
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. (#5104) 2022-12-01 13:20:21 -08:00
Szilard Nemeth
5440c75c4a YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs 2022-12-01 15:11:58 +01:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Anmol Asrani
7786600744
HADOOP-18457. ABFS: Support account level throttling (#5034) 
This allows  abfs request throttling to be shared across all 
abfs connections talking to containers belonging to the same abfs storage
account -as that is the level at which IO throttling is applied.

The option is enabled/disabled in the configuration option 
"fs.azure.account.throttling.enabled";
The default is "true"

Contributed by Anmol Asrani
 2022-11-30 13:05:31 +00:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
Owen O'Malley
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 22:12:35 +00:00
HarshitGupta11
0ef572abed
HADOOP-18530. ChecksumFileSystem::readVectored might return byte buffers not positioned at 0 (#5168) 
Contributed by Harshit Gupta
 2022-11-29 14:51:22 +00:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. (#5142) 2022-11-29 00:49:10 +00:00
Simbarashe Dzinamarira
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-29 00:47:01 +00:00
slfan1989
f93167e678
YARN-11380. Fix hadoop-yarn-api module Java Doc Errors. (#5152). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-28 18:54:32 +05:30
sreeb-msft
1a7acc403b
HADOOP-18498. ABFS: Remove unwanted ? prefix from SAS Tokens (#5136) 
This commit parses SAS Tokens and removes the unwanted prefix of '?' from them, if present.

At present, SAS Tokens are provided to the driver through customer implementations of the SASTokenProvider interface. The SAS token providers should not assume that the token will be the first query parameter in the URIs that communicate with the backend. However, it was observed that certain public interfaces provided by Storage to generate SAS can include the '?' as the first character of the SAS Token, which would ideally be the case when it is the first query parameter. Thus, tokens that contain this prefix will lead to an error in the driver due to a clash of query parameters.

To avoid failures for use of such SAS tokens, after receiving the SAS Token from the provider, the code checks for whether any ? prefix is present or not. If yes, it is removed before further usage of the token. This way, users would not have to manually remove the prefix before passing it on as a configuration.

Contributed by Sree Bhattacharya
 2022-11-28 11:38:13 +00:00
PJ Fanning
e09e81abe4
HADOOP-18496: remove unused okhttp.version (#5140). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-27 18:59:40 +05:30
slfan1989
1ddc9091f6
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5153). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-26 21:01:07 +05:30
ZanderXu
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md (#4922) 2022-11-25 09:26:28 +08:00
ZanderXu
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request (#5086) 2022-11-25 09:23:33 +08:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
ZanderXu
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap (#5125) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:07:27 +08:00
huhaiyang
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB (#5129) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:00:27 +08:00
Ashutosh Gupta
2c1158e858
HADOOP-18531. Fix assertion failure in ITestS3APrefetchingInputStream (#5149) 
This patch MUST be applied to all branches containing HADOOP-18378
so as to ensure reliable test runs.

Contributed by Ashutosh Gupta
 2022-11-23 17:47:39 +00:00
huhaiyang
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate (#5063) 2022-11-23 09:50:57 +08:00
slfan1989
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. (#5130) 2022-11-22 14:38:24 -08:00
Szilard Nemeth
3c37a01654 YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta 2022-11-22 13:37:55 +01:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
GuoPhilipse
069bd973d8
HADOOP-18532. Update command usage in FileSystemShell.md (#5141) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 15:55:46 +09:00
Ashutosh Gupta
696d042054
HADOOP-8728. Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. (#5010) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:54:50 +09:00
Ashutosh Gupta
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common (#4717) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:40:02 +09:00
Ashutosh Gupta
dcde414570
MAPREDUCE-7422. Upgrade Junit 4 to 5 in hadoop-mapreduce-examples (#5029) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:36:35 +09:00
Owen O'Malley
c71a68ca80
HDFS-16844: Adds resilancy when StateStore gets exceptions. (#5138) 
Allows the StateStore to stay up when there are errors reading the data.
 2022-11-18 17:24:08 +00:00
Owen O'Malley
1ea5db52dd
HADOOP-18324. Interrupting RPC Client calls can lead to thread exhaustion. (#4527) 
* Exactly 1 sending thread per an RPC connection.
* If the calling thread is interrupted before the socket write, it will be skipped instead of sending it anyways.
* If the calling thread is interrupted during the socket write, the write will finish.
* RPC requests will be written to the socket in the order received.
* Sending thread is only started by the receiving thread.
* The sending thread periodically checks the shouldCloseConnection flag.
 2022-11-18 16:24:45 +00:00
Hu Xinqiu
7d39abd799
HADOOP-18429. fix infinite loop in MutableGaugeFloat#incr(float) (#4823) 2022-11-17 17:50:39 +08:00
slfan1989
eccd2d0492
YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs. (#5057) 2022-11-16 18:00:38 -08:00
Szilard Nemeth
142df247ed YARN-11369. Commons.compress throws an IllegalArgumentException with large uids after 1.21. Contributed by Benjamin Teke 2022-11-16 13:07:05 +01:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
Mehakmeet Singh
69e50c7b44
HADOOP-18528. Disable abfs prefetching by default (#5134) 
Disables block prefetching on ABFS InputStreams, by setting
fs.azure.enable.readahead to false in core-default.xml and
the matching java constant.

This prevents
HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests.

Once a fix for that is committed, this change can be reverted.

Contributed by Mehakmeet Singh.
 2022-11-15 14:28:41 +00:00
Ashutosh Gupta
a48e8c9beb
MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb (#5014) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-14 11:07:40 +09:00
slfan1989
04b31d7ecf
MAPREDUCE-7390. Remove WhiteBox in mapreduce module. (#4462) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-14 10:45:20 +09:00
ZanderXu
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. (#4945). Contributed by ZanderXu. 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-11-13 22:01:35 +08:00
Szilard Nemeth
22c9f28f4d YARN-10005. Code improvements in MutableCSConfigurationProvider. Contributed by Peter Szucs 2022-11-12 18:35:49 +01:00