big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,572 Commits 2 Branches 2 Tags 264 MiB
e9ba4f4591
Commit Graph

549 Commits

Author SHA1 Message Date
Ashvin
fabd41fa48 HDFS-14856. Fetch file ACLs while mounting external store. (#1478) 2019-10-14 09:44:56 -07:00
Chen Liang
1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Inigo Goiri
3f223bebfa HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun. 2019-09-20 11:45:01 -07:00
Inigo Goiri
f580a87079 HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun. 2019-09-17 14:55:34 -07:00
Stephen O'Donnell
06ad6540b3 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-13 10:36:45 -07:00
Surendra Singh Lilhore
fa7f03fc56 HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C. 2019-09-06 17:55:30 +05:30
Inigo Goiri
72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Surendra Singh Lilhore
5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
leosunli
915271245b Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-08-29 18:35:26 -07:00
Arpit Agarwal
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 2019-08-26 15:57:55 -07:00
Stephen O'Donnell
b67812ea21 HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index (#1028). Contributed by Stephen O'Donnell. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-08-22 17:09:57 -07:00
Stephen O'Donnell
93daf69f90 HDFS-14675. Increase Balancer Defaults Further. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Gabor Bota <gabota@apache.org>
 2019-08-22 15:21:33 -07:00
Wei-Chiu Chuang
d873ddd656 HDFS-14127. Add a description about the observer read configuration. Contributed by xiangheng. 2019-08-17 04:39:22 -07:00
Ayush Saxena
e9b6b81de4 HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng. 2019-08-13 19:17:10 +05:30
Erik Krogen
827dbb11e2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 2019-08-07 09:25:58 -07:00
Erik Krogen
1f26cc8705 HDFS-13783. Add an option to the Balancer to make it run as a long-running service. Contributed by Chen Zhang. 2019-07-30 15:42:55 -07:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Weiwei Yang
e3172221ea HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 21:48:16 -07:00
Anu Engineer
101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
He Xiaoqiao
bd46bdf9f9 HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 17:21:52 -07:00
Rakesh Radhakrishnan
9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Puleya7
13907d8479 HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-04-17 21:39:06 +09:00
Chen Liang
626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Inigo Goiri
7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Uma Maheswara Rao G
35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Erik Krogen
1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Yongjun Zhang
f7a27cdee4 HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li. 2019-02-23 09:35:36 -08:00
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Erik Krogen
00e99c6594 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2018-12-24 09:33:59 -08:00
Erik Krogen
c81ac2ff02 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2018-12-24 09:33:59 -08:00
Vinayakumar B
c03024a530 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 2018-12-06 18:20:28 +05:30
Tsz Wo Nicholas Sze
0081b02e35 HDFS-14112. Avoid recursive call to external authorizer for getContentSummary. 2018-11-29 13:55:21 -08:00
Vinayakumar B
68d5dfdc78 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 2018-11-16 18:05:10 +05:30
Chen Liang
635786a511 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2018-10-23 14:53:45 -07:00
Inigo Goiri
88cce32551 HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin. 2018-10-19 10:38:04 -07:00
Kitti Nanasi
10185d9a77 HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-10 08:55:32 -07:00
Nanda kumar
b39b802422 HDFS-13949. Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml. Contributed by Toshihiro Suzuki. 2018-10-10 12:29:07 +05:30
Yiqun Lin
5689355783 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-02 09:43:14 +08:00
Yiqun Lin
0da49642fc HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu. 2018-09-10 13:57:36 +08:00
Hrishikesh Gadre
335a8139f5 HDFS-13820. Add an ability to disable CacheReplicationMonitor. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-07 14:59:06 -07:00
Yiqun Lin
b9b964d253 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-27 14:55:46 +08:00
Uma Maheswara Rao G
39ed3a66db HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
2acc50b826 HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Surendra Singh Lilhore
75ccc1396b HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Uma Maheswara Rao G
3b83110d5e HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R. 2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
c561cb316e HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:02 -07:00
Uma Maheswara Rao G
00eceed233 HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
bfd3f8bd8a HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
0e820f16af HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier feature. Contributed by Rakesh R. 2018-08-12 03:06:01 -07:00
Rakesh Radhakrishnan
422f870607 HDFS-11289. [SPS]: Make SPS movement monitor timeouts configurable. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:56 -07:00
Rakesh Radhakrishnan
5179d99b7e HDFS-11123. [SPS] Make storage policy satisfier daemon work on/off dynamically. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
cd5262aba0 HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou 2018-08-12 03:05:55 -07:00
Chen Liang
5326a7906d HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-09 10:11:47 -07:00
Virajith Jalaparti
b1a59b1644 HDFS-13796. Allow verbosity of InMemoryLevelDBAliasMapServer to be configurable. 2018-08-07 10:15:28 -07:00
Brahma Reddy Battula
3108d27edd HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani 2018-07-30 15:50:04 +05:30
Wei-Chiu Chuang
e95c5e9f62 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 2018-07-25 11:04:18 -07:00
Andrew Wang
eecb5baaaa HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 2018-07-09 15:17:21 +02:00
Inigo Goiri
1804a31515 HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-07-02 10:48:20 -07:00
Wei-Chiu Chuang
c966a3837a HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth. 2018-06-15 13:36:46 -07:00
Inigo Goiri
2e5cfe6df3 HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 2018-06-11 11:20:29 -07:00
Arpit Agarwal
000a678396 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:12 -07:00
Aaron Fabbri
a732acd873
HDFS-5926 Documentation should clarify dfs.datanode.du.reserved impact from reserved disk capacity (Gabor Bota) 2018-05-04 16:38:30 -07:00
Inigo Goiri
fc074a359c HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 2018-04-30 13:28:33 -07:00
Owen O'Malley
2adda92de1
HDFS-7240 git commit fixup 
Signed-off-by: Owen O'Malley <omalley@apache.org>
 2018-04-26 13:52:17 -07:00
Anu Engineer
4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects. 
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
 2018-04-26 05:36:04 -07:00
Anu Engineer
5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key. 
Contributed by Mukul Kumar Singh.
 2018-04-26 05:36:04 -07:00
Nanda kumar
1094af072c HDFS-11699. Ozone:SCM: Add support for close containers in SCM. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
377b31ffa1 HDFS-13022. Block Storage: Kubernetes dynamic persistent volume provisioner. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
3cf07b43bc HDFS-13127. Fix TestContainerStateManager and TestOzoneConfigurationFields. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
fb09d75379 HDFS-13018. Block Storage: make the iscsi target addres configurable for discovery. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
94c0346f35 HDFS-13024. Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
1fb1ce107f HDFS-13012. TestOzoneConfigurationFields fails due to missing configs in ozone-default.xml. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
98d62e55c4 Revert "HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee." 
This reverts commit 6ce5ec676164b84a9e2f8dc65b5f2199a141506d.
 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
98eeabbdb3 HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
0380dbb928 HDFS-12941. Ozone: ConfServlet does not trim values during the description parsing. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
fd09c2ce5b HDFS-12890 . Ozone: XceiverClient should have upper bound on async requests. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
2f3068bc72 HDFS-12698. Ozone: Use time units in the Ozone configuration values. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
c0c87dea9b HDFS-12626. Ozone : delete open key entries that will no longer be closed. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
85ec48a646 HDFS-12867. Ozone: TestOzoneConfigurationFields fails consistently. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer
8023ba50b5 HDFS-12807. Ozone: Expose RockDB stats via JMX for Ozone metadata stores. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
86802a461c HDFS-12851. Ozone: Upgrade to latest ratis build. Contributed by Mukul Kumar Singh 2018-04-26 05:36:04 -07:00
Yiqun Lin
cbe4f314b3 HDFS-12731. Ozone: Fix inconsistent default value in ozone settings. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
cec96b296f HDFS-12549. Ozone: OzoneClient: Support for REST protocol. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
709d56fdc9 HDFS-12474. Ozone: SCM: Handling container report with key count and container usage. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
b687902590 HDFS-12718. Ozone: fix thread number calculation in CBlockManager. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
6291ca1e32 HDFS-12663. Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
1b56a73a4f HDFS-12680. Ozone: SCM: Lease support for container creation. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
a467017068 HDFS-12411. Ozone: Add container usage information to DN container report. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Nandakumar
c3ef381011 HDFS-12543. Ozone : allow create key without specifying size. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer
c17521b1bd HDFS-12387. Ozone: Support Ratis as a first class replication mechanism. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Anu Engineer
6a754caca0 HDFS-12568. Ozone: Cleanup the ozone-default.xml. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Anu Engineer
aaec2689f4 HDFS-12511. Ozone: Add tags to config. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
a1a3ba6529 HDFS-12454. Ozone : the sample ozone-site.xml in OzoneGettingStarted does not work. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer
2804435386 HDFS-12017. Ozone: Container: Move IPC port to 98xx range. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Chen Liang
9ff136bb02 HDFS-12210. Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang
dddded0016 Revert "Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh." to fix commit message. 
This reverts commit 087c69ba2434813044089f733581607d172efbd1.
 2018-04-26 05:36:04 -07:00
Chen Liang
8052374e79 Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Weiwei Yang
3b290fe3bf HDFS-12466. Ozone: KSM: Make ozone.ksm.address as mandatory property for client. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
77d4b18700 HDFS-12385. Ozone: OzoneClient: Refactoring OzoneClient API. Contributed by Nadakumar. 2018-04-26 05:36:04 -07:00