big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,575 Commits 2 Branches 2 Tags 264 MiB
bf5c948995
Commit Graph

3900 Commits

Author SHA1 Message Date
Konstantin V Shvachko
60fd0d7fd7 HDFS-12638. Delete copy-on-truncate block along with the original block, when deleting a file being truncated. Contributed by Konstantin Shvachko. 2017-11-30 18:18:28 -08:00
Tsz-Wo Nicholas Sze
b1c7654ee4 HDFS-12594. snapshotDiff fails if the report exceeds the RPC response limit. Contributed by Shashikant Banerjee 2017-11-30 12:18:29 -08:00
Chris Douglas
0e560f3b8d HDFS-12681. Make HdfsLocatedFileStatus a subtype of LocatedFileStatus 2017-11-29 20:28:06 -08:00
Kihwal Lee
53509f295b HDFS-11754. Make FsServerDefaults cache configurable. Contributed by Mikhail Erofeev. 2017-11-29 15:11:14 -06:00
Surendra Singh Lilhore
30941d99c9 HDFS-12857. StoragePolicyAdmin should support schema based path. Contributed by Surendra Singh Lilhore. 2017-11-28 23:57:03 +05:30
Kihwal Lee
738d1a206a HDFS-12754. Lease renewal can hit a deadlock. Contributed by Kuhu Shukla. 2017-11-22 13:01:55 -06:00
Lei Xu
785732c13e HDFS-12847. Regenerate editsStored and editsStored.xml in HDFS tests. Contributed by Lei (Eddy) Xu. 2017-11-22 10:22:32 -08:00
Chen Liang
60fc2a1388 HDFS-12804. Use slf4j instead of log4j in FSEditLog. Contributed by Mukul Kumar Singh. 2017-11-20 12:49:53 -08:00
Manoj Govindassamy
9fb4effd2c HDFS-12730. Verify open files captured in the snapshots across config disable and enable. 2017-11-20 10:28:12 -08:00
Chris Douglas
675e9a8f57 Revert "HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus." 
This reverts commit b85603e3f8.
 2017-11-15 19:20:37 -08:00
Arpit Agarwal
4908a8970e HDFS-12705. WebHdfsFileSystem exceptions should retain the caused by exception. Contributed by Hanisha Koneru. 2017-11-13 11:30:39 -08:00
Arpit Agarwal
6d201f77c7 HDFS-12498. Journal Syncer is not started in Federated + HA cluster. Contributed by Bharat Viswanadham. 2017-11-10 16:30:38 -08:00
Arpit Agarwal
1d6f8bebe9 HADOOP-14929. Cleanup usage of decodecomponent and use QueryStringDecoder from netty. Contributed by Bharat Viswanadham. 2017-11-10 16:28:12 -08:00
Chris Douglas
b85603e3f8 HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus. 2017-11-03 14:30:57 -07:00
Kihwal Lee
4d2dce40bb HDFS-12771. Add genstamp and block size to metasave Corrupt blocks list. Contributed by Kuhu Shukla. 2017-11-03 15:16:46 -05:00
Xiao Chen
299d38295d HDFS-11467. Support ErasureCoding section in OIV XML/ReverseXML. Contributed by Huafeng Wang. 2017-11-03 12:05:45 -07:00
Xiao Chen
b00f828d84 HDFS-12725. BlockPlacementPolicyRackFaultTolerant fails with very uneven racks. 2017-11-02 21:53:13 -07:00
Xiao Chen
e565b5277d HDFS-12682. ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED. 2017-11-02 21:27:35 -07:00
Lei Xu
9367c25dbd HDFS-12482. Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery. (lei) 2017-10-31 21:58:14 -07:00
Andrew Wang
5f681fa821 Revert "HDFS-12499. dfs.namenode.shared.edits.dir property is currently namenode specific key. Contributed by Bharat Viswanadham." 
This reverts commit b922ba7393.
 2017-10-31 10:46:10 -07:00
Arpit Agarwal
b922ba7393 HDFS-12499. dfs.namenode.shared.edits.dir property is currently namenode specific key. Contributed by Bharat Viswanadham. 2017-10-31 10:23:00 -07:00
Inigo Goiri
982bd2a5bf HDFS-12699. TestMountTable fails with Java 7. Contributed by Inigo Goiri. 2017-10-31 10:21:42 -07:00
Chris Douglas
d015e0bbd5 HDFS-7878. API - expose a unique file identifier. 2017-10-31 09:44:01 -07:00
Sean Mackrory
81225430d0 HDFS-206. Support for head in FSShell. Contributed by Gabor Bota. 2017-10-31 08:34:01 -06:00
Chris Douglas
d55a84951a HDFS-12582. Replace HdfsFileStatus constructor with a builder pattern. Contributed by Bharat Viswanadham 2017-10-27 15:36:03 -07:00
Manoj Govindassamy
075dd45a24 HDFS-12544. SnapshotDiff - support diff generation on any snapshot root descendant directory. 2017-10-25 10:54:40 -07:00
Zhe Zhang
17cd8d0c17 Revert "HDFS-12502. nntop should support a category based on FilesInGetListingOps." 
This reverts commit 60bfee270e.
 2017-10-24 15:06:50 -07:00
Wei-Chiu Chuang
0f2a69127a HDFS-12249. dfsadmin -metaSave to output maintenance mode blocks. Contributed by Wellington Chevreuil. 2017-10-23 15:24:34 -07:00
Arpit Agarwal
96be795656 HDFS-12650. Use slf4j instead of log4j in LeaseManager. Contributed by Ajay Kumar. 2017-10-23 13:20:46 -07:00
Xiao Chen
248d9b6fff HDFS-12518. Re-encryption should handle task cancellation and progress better. 2017-10-20 16:25:04 -07:00
Andrew Wang
0477eff8be HDFS-12497. Re-enable TestDFSStripedOutputStreamWithFailure tests. Contributed by Huafeng Wang. 2017-10-20 13:27:21 -07:00
Kai Zheng
ce7cf66e5e HDFS-12448. Make sure user defined erasure coding policy ID will not overflow. Contributed by Huafeng Wang 2017-10-20 09:42:04 +08:00
Inigo Goiri
0f1c037618 HDFS-12620. Backporting HDFS-10467 to branch-2. Contributed by Inigo Goiri. 2017-10-19 18:08:45 -07:00
Zhe Zhang
60bfee270e HDFS-12502. nntop should support a category based on FilesInGetListingOps. 2017-10-18 23:51:24 -07:00
Lei Xu
f27a4ad032 HDFS-12612. DFSStripedOutputStream.close will throw if called a second time with a failed streamer. (Lei (Eddy) Xu) 2017-10-17 15:52:09 -07:00
Lei Xu
31ebccc962 HDFS-12613. Native EC coder should implement release() as idempotent function. (Lei (Eddy) Xu) 2017-10-16 19:44:30 -07:00
Manoj Govindassamy
b406d8e375 HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. 2017-10-16 17:42:41 -07:00
Wei-Chiu Chuang
8dbc8909c9 HDFS-12485. expunge may fail to remove trash from encryption zone. Contributed by Wei-Chiu Chuang. 2017-10-16 12:57:48 -07:00
Xiao Chen
7bd700941d HDFS-12637. Extend TestDistributedFileSystemWithECFile with a random EC policy. Contributed by Takanobu Asanuma. 2017-10-16 09:55:22 -07:00
Xiao Chen
035c6ee587 HDFS-12603. Enable async edit logging by default. Contributed by Andrew Wang. 2017-10-16 09:51:10 -07:00
Xiao Chen
20575ececb HDFS-12659. Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval. Contributed by Ajay Kumar. 2017-10-15 19:28:22 -07:00
Manoj Govindassamy
133d7ca76e HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 2017-10-13 17:43:21 -07:00
Arpit Agarwal
8dd1eeb94f HDFS-12553. Add nameServiceId to QJournalProtocol. Contributed by Bharat Viswanadham 2017-10-13 14:22:21 -07:00
Arpit Agarwal
bb0a742aac HDFS-12627. Fix typo in DFSAdmin command output. Contributed by Ajay Kumar. 2017-10-11 12:26:14 -07:00
Andrew Wang
a297fb0886 HDFS-12547. Extend TestQuotaWithStripedBlocks with a random EC policy. Contributed by Takanobu Asanuma. 2017-10-10 17:35:49 -07:00
Manoj Govindassamy
78af6cdc53 HDFS-12573. Divide the total blocks metrics into replicated and erasure coded. Contributed by Takanobu Asanuma. 2017-10-10 14:23:29 -07:00
Andrew Wang
82cd85d868 Revert "HDFS-12603. Enable async edit logging by default." 
This reverts commit afb42aeabf.
 2017-10-09 17:53:56 -07:00
Andrew Wang
7938208233 HDFS-12541. Extend TestSafeModeWithStripedFile with a random EC policy. Contributed by Takanobu Asanuma. 2017-10-09 13:03:32 -07:00
Andrew Wang
afb42aeabf HDFS-12603. Enable async edit logging by default. 2017-10-09 11:20:00 -07:00
Arpit Agarwal
09ad848b64 HDFS-12599. Remove Mockito dependency from DataNodeTestUtils. Contributed by Ted Yu. 2017-10-09 10:29:01 -07:00
Lei Xu
46644319e1 HDFS-12606. When using native decoder, DFSStripedStream.close crashes JVM after being called multiple times. (Lei (Eddy) Xu) 2017-10-09 10:08:30 -07:00
Inigo Goiri
81601dac8e HDFS-12273. Federation UI. Contributed by Inigo Goiri. 
(cherry picked from commit adbb2e00c7b85524fd43bd68895d49814c16680a)
 2017-10-06 18:50:52 -07:00
Inigo Goiri
67785fe006 HDFS-12450. Fixing TestNamenodeHeartbeat and support non-HA. Contributed by Inigo Goiri. 
(cherry picked from commit 928d1e87f9dbe64f89b858ccc1780723f3af58e7)
 2017-10-06 18:50:50 -07:00
Inigo Goiri
bc9e588a19 HDFS-12335. Federation Metrics. Contributed by Inigo Goiri. 
(cherry picked from commit 3b19e77752afce87936f5c0d1e6d272fba798d7b)
 2017-10-06 18:50:50 -07:00
Inigo Goiri
7cb6bdf09e HDFS-10631. Federation State Store ZooKeeper implementation. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 23c4ddee11ab1300325a6361124ee8ad6f68d7a4)
 2017-10-06 18:50:49 -07:00
Inigo Goiri
b3e6bd22e3 HDFS-10646. Federation admin tool. Contributed by Inigo Goiri. 
(cherry picked from commit ae27e31fbcf546481db0b0345772db2e9132372e)
 2017-10-06 18:50:48 -07:00
Inigo Goiri
6f0de27318 HDFS-10880. Federation Mount Table State Store internal API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 58b97df661441150d35abd44b3a8606206b46441)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
d8c8107332 HDFS-11826. Federation Namenode Heartbeat. Contributed by Inigo Goiri. 
(cherry picked from commit 928f8dab52191e733984d37f47b69719ccf11313)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
55da7fd7eb HDFS-10687. Federation Membership State Store internal API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 95cae08849d23cf12c5d280c29f90908e6a90d40)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
ca4f209b49 HDFS-11546. Federation Router RPC server. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 8a9cdebebf26841a0f1e99fb08135f4597f2eba2)
 2017-10-06 18:50:46 -07:00
Inigo Goiri
4bf877b03f HDFS-10630. Federation State Store FS Implementation. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit c6e0bd640cdaf83a660fa050809cad6f1d4c6f4d)
 2017-10-06 18:50:45 -07:00
Inigo
2761bbc91a HDFS-10629. Federation Roter. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 6821e801724ac38e9737538b2164c9ae88792282)
 2017-10-06 18:50:44 -07:00
Xiao Chen
25f31d9fc4 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 2017-10-05 19:46:47 -07:00
Andrew Wang
644c2f6924 HDFS-12567. BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes. 2017-10-05 16:58:43 -07:00
Arpit Agarwal
b6942cbe9b HDFS-12420. Add an option to disallow 'namenode format -force'. Contributed by Ajay Kumar. 2017-10-05 15:26:52 -07:00
Xiaoyu Yao
107c177782 HDFS-12455. WebHDFS - Adding "snapshot enabled" status to ListStatus query result. Contributed by Ajay Kumar. 2017-10-03 13:02:12 -07:00
Arpit Agarwal
b91305119b HDFS-11968. ViewFS: StoragePolicies commands fail with HDFS federation. Contributed by Mukul Kumar Singh. 2017-10-03 11:23:40 -07:00
Lei Xu
27ffd43b64 HDFS-12569. Unset EC policy logs empty payload in edit log. (Lei (Eddy) Xu) 2017-10-02 15:35:49 -07:00
Chen Liang
5a68c2cef3 HDFS-12552. Use slf4j instead of log4j in FSNamesystem. Contributed by Ajay Kumar. 2017-10-02 11:26:53 -07:00
Lei Xu
f40dbc170e HDFS-12453. TestDataNodeHotSwapVolumes fails in trunk Jenkins runs. (Lei (Eddy) Xu) 2017-09-29 10:46:17 -07:00
Wei-Chiu Chuang
7c34ceaf4f HDFS-12458. TestReencryptionWithKMS fails regularly. Contributed by Xiao Chen. 2017-09-28 05:10:26 -07:00
Daryn Sharp
0da29cbeea [HDFS-12386] Add fsserver defaults call to WebhdfsFileSystem. (Rushabh Shah via daryn) 2017-09-26 15:15:24 -05:00
Brahma Reddy Battula
9d3e4cccf9 HDFS-5040.Audit log for admin commands/ logging output of all DFS admin commands. Contributed by Kuhu Shukla. 2017-09-26 21:59:22 +05:30
Yiqun Lin
e9b790db9d HDFS-12495. TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently. Contributed by Eric Badger. 2017-09-26 17:21:42 +08:00
Anu Engineer
d0b2c5850b HDFS-12516. Suppress the fsnamesystem lock warning on nn startup. Contributed by Ajay Kumar. 2017-09-24 21:03:06 -07:00
Anu Engineer
36fb90c939 HDFS-12064. Reuse object mapper in HDFS. Contributed by Hanisha Koneru. 2017-09-23 10:11:00 -07:00
Jitendra Pandey
cda3378659 HDFS-12486. GetConf to get journalnodeslist. Contributed by Bharat Viswanadham. 2017-09-22 16:52:47 -07:00
Andrew Wang
a12f09ba3c HDFS-12447. Rename AddECPolicyResponse to AddErasureCodingPolicyResponse. Contributed by SammiChen. 2017-09-20 11:51:17 -07:00
Ming Ma
230b85d586 HDFS-12473. Change hosts JSON file format. 2017-09-20 09:03:59 -07:00
Andrew Wang
12d9d7bc50 HDFS-12437. Fix test setup in TestLeaseRecoveryStriped. 2017-09-19 16:42:20 -07:00
Andrew Wang
51edaacd09 HDFS-12445. Correct spellings of choosen to chosen. Contributed by hu xiaodong. 2017-09-19 13:48:23 -07:00
Andrew Wang
59830ca772 HDFS-12444. Reduce runtime of TestWriteReadStripedFile. Contributed by Huafeng Wang and Andrew Wang. 2017-09-19 13:44:42 -07:00
Lei Xu
7bbeacb75e HDFS-12449. TestReconstructStripedFile.testNNSendsErasureCodingTasks randomly cannot finish in 60s. (SammiChen via lei) 2017-09-19 11:50:01 -07:00
Brahma Reddy Battula
fda1221c55 HDFS-11799. Introduce a config to allow setting up write pipeline with fewer nodes than replication factor. Contributed by Brahma Reddy Battula 2017-09-19 11:25:45 +05:30
Brahma Reddy Battula
31b58406ac HDFS-12480. TestNameNodeMetrics#testTransactionAndCheckpointMetrics Fails in trunk. Contributed by Hanisha Koneru 2017-09-19 10:37:07 +05:30
Arpit Agarwal
a2dcba1853 HDFS-12470. DiskBalancer: Some tests create plan files under system directory. Contributed by Hanisha Koneru. 2017-09-18 09:53:24 -07:00
Kai Zheng
0f9af246e8 HDFS-12460. Make addErasureCodingPolicy an idempotent operation. Contributed by Sammi Chen 2017-09-18 18:07:12 +08:00
Arpit Agarwal
8d7cc22ac2 HDFS-12472. Add JUNIT timeout to TestBlockStatsMXBean. Contributed by Bharat Viswanadham. 2017-09-16 10:09:27 -07:00
Andrew Wang
ef8cd5dc56 HDFS-10701. TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails. Contributed by SammiChen. 2017-09-15 16:20:36 -07:00
Erik Krogen
90894c7262 HDFS-12323. NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout. Contributed by Erik Krogen. 2017-09-15 13:56:27 -07:00
Lei Xu
fbe06b5880 HDFS-12349. Improve log message when it could not alloc enough blocks for EC. (Lei (Eddy) Xu) 2017-09-15 12:12:42 -07:00
Arpit Agarwal
50764ef3e4 HDFS-12456. TestNamenodeMetrics.testSyncAndBlockReportMetric fails. Contributed by Bharat Viswanadham. 2017-09-14 22:23:36 -07:00
Kai Zheng
08d996d3e9 HDFS-12395. Support erasure coding policy operations in namenode edit log. Contributed by Sammi Chen 2017-09-15 09:43:39 +08:00
Kai Zheng
ae8f55b932 HDFS-7859. Erasure Coding: Persist erasure coding policies in NameNode. Contributed by Sammi Chen 2017-09-15 09:08:18 +08:00
Andrew Wang
390c2b5df0 Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang." 
This reverts commit 7fcc73fc0d.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2017-09-14 15:12:26 -07:00
John Zhuge
7ee02d1065 HADOOP-14864. FSDataInputStream#unbuffer UOE should include stream class name. Contributed by Bharat Viswanadham. 2017-09-14 11:26:49 -07:00
Arpit Agarwal
65a941008d Revert "HDFS-10391. Always enable NameNode service RPC port. Contributed by Gergely Novak." 
This reverts commit aa4b6fbe75.
 2017-09-14 11:17:08 -07:00
Kai Zheng
e12f3e85bd HDFS-12438. Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.threads. Contributed by Andrew Wang 2017-09-14 15:39:10 +08:00
ChenSammi
e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
Lei Xu
73aed34dff HDFS-12409. Add metrics of execution time of different stages in EC recovery task. (Lei (Eddy) Xu) 2017-09-13 17:14:13 -07:00