big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,535 Commits 2 Branches 2 Tags 264 MiB
826507c41b
Commit Graph

4662 Commits

Author SHA1 Message Date
Kihwal Lee
9b90e52f1e HDFS-11641. Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus. Contributed by Daryn Sharp. 2017-05-16 11:28:46 -05:00
Vinayakumar B
1411612aa4 HDFS-11674. reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered (Contributed by Vinayakumar B) 2017-05-12 07:38:18 +05:30
Andrew Wang
83dd14aa84 HDFS-11757. Query StreamCapabilities when creating balancer's lock file. Contributed by SammiChen. 2017-05-11 14:09:49 -07:00
Chris Douglas
51b671ef18 HDFS-11681. DatanodeStorageInfo#getBlockIterator() should return an iterator to an unmodifiable set Contributed by Virajith Jalaparti 2017-05-10 22:25:28 -07:00
Kihwal Lee
ad1e3e4d9f HDFS-11755. Underconstruction blocks can be considered missing. Contributed by Nathan Roberts. 2017-05-10 14:15:57 -05:00
Andrew Wang
543aac9f28 HADOOP-14386. Rewind trunk from Guava 21.0 back to Guava 11.0.2. 2017-05-09 09:22:53 -07:00
Chris Douglas
a3954ccab1 HDFS-9807. Add an optional StorageID to writes. Contributed by Ewan Higgs 2017-05-05 12:01:26 -07:00
Yiqun Lin
97c2e576c9 HDFS-11530. Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault. Contributed by Yiqun Lin and Chen Liang. 2017-05-05 11:54:50 +08:00
Arpit Agarwal
07761af357 HDFS-11448. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-04 15:57:44 -07:00
Andrew Wang
c2a52ef9c2 HDFS-11643. Add shouldReplicate option to create builder. Contributed by SammiChen. 2017-05-04 11:39:14 -07:00
Arpit Agarwal
81092b1f11 HDFS-11722. Change Datanode file IO profiling sampling to percentage. Contributed by Hanisha Koneru. 2017-05-03 16:29:30 -07:00
Arpit Agarwal
30cd265134 Revert "HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru." 
This reverts commit 08fb82d6d1.
 2017-05-03 16:28:47 -07:00
Arpit Agarwal
08fb82d6d1 HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-03 15:46:08 -07:00
Erik Krogen
d9014bda93 HDFS-11717. Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 2017-05-02 18:34:11 -07:00
Xiaoyu Yao
1058b40843 HDFS-11593. Change SimpleHttpProxyHandler#exceptionCaught log level from info to debug. Contributed by Xiaobing Zhou. 2017-05-01 16:12:51 -07:00
Kihwal Lee
4cfc866436 HDFS-11714. Newly added NN storage directory won't get initialized and cause space exhaustion. Contributed by Kihwal Lee. 2017-05-01 17:29:25 -05:00
Kihwal Lee
07b98e7830 HDFS-11609. Some blocks can be permanently lost if nodes are decommissioned while dead. Contributed by Kihwal Lee. 2017-05-01 14:19:02 -05:00
Kai Zheng
cb672a45a0 HDFS-11605. Allow user to customize new erasure code policies. Contributed by Huafeng Wang 2017-04-28 13:18:50 +08:00
Zhe Zhang
d8a3309830 HDFS-11709. StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 2017-04-27 09:50:30 -07:00
Konstantin V Shvachko
28eb2aabeb HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko. 2017-04-26 17:28:49 -07:00
Akira Ajisaka
93fa48fcf2
HDFS-11417. Add datanode admin command to get the storage info. Contributed by Surendra Singh Lilhore. 2017-04-26 17:43:29 +09:00
Chris Douglas
2f73396b59 HDFS-6708. StorageType should be encoded in the block token. Contributed by Ewan Higgs 2017-04-25 23:57:00 -07:00
Xiaoyu Yao
4ea2778acb HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. Contributed by Surendra Singh Lihore. 2017-04-25 18:45:56 -07:00
Jason Lowe
e4321ec843 HDFS-11691. Add a proper scheme to the datanode links in NN web UI. Contributed by Kihwal Lee 2017-04-25 17:30:42 -05:00
Yongjun Zhang
20e3ae260b HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN files. (Manoj Govindassamy via Yongjun Zhang) 2017-04-21 20:35:48 -07:00
Kai Zheng
b0803388fc HDFS-11604. Define and parse erasure code policies. Contributed by Lin Zeng 2017-04-21 13:33:33 +08:00
Andrew Wang
74a723852d HDFS-11660. TestFsDatasetCache#testPageRounder fails intermittently with AssertionError. 2017-04-19 18:10:04 -07:00
Konstantin V Shvachko
8dfcd95d58 HDFS-11634. Optimize BlockIterator when interating starts in the middle. Contributed by Konstantin V Shvachko. 2017-04-17 16:56:24 -07:00
Zhe Zhang
ad49098eb3 HDFS-11615. FSNamesystemLock metrics can be inaccurate due to millisecond precision. Contributed by Erik Krogen. 2017-04-17 16:22:33 -07:00
Andrew Wang
a7312715a6 HDFS-10996. Ability to specify per-file EC policy at create time. Contributed by SammiChen. 2017-04-12 12:27:34 -07:00
Anu Engineer
abce613356 HDFS-11645. DataXceiver thread should log the actual error when getting InvalidMagicNumberException. Contributed by Chen Liang. 2017-04-12 11:40:58 -07:00
Kihwal Lee
d4c01dde49 HDFS-11648. Lazy construct the IIP pathname. Contributed by Daryn Sharp. 2017-04-12 13:29:24 -05:00
Chris Nauroth
23b1a7bdf1 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore. 2017-04-11 22:03:09 -07:00
Mingliang Liu
3a91376707 HDFS-11558. BPServiceActor thread name is too long. Contributed by Xiaobing Zhou 2017-04-11 15:29:38 -07:00
Chris Douglas
a5e57df3c5 HADOOP-14225. Remove xmlenc dependency 2017-04-10 11:48:40 -07:00
Andrew Wang
e8bdad7385 HDFS-11623. Move system erasure coding policies into hadoop-hdfs-client. 2017-04-07 16:46:28 -07:00
Andrew Wang
203edc026c HDFS-11633. FSImage failover disables all erasure coding policies. Contributed by Wei-Chiu Chuang. 2017-04-07 15:46:11 -07:00
Allen Wittenauer
96cbb4fce2 HADOOP-14202. fix jsvc/secure user var inconsistencies 
Signed-off-by: John Zhuge <jzhuge@apache.org>
 2017-04-07 08:59:21 -07:00
Xiaoyu Yao
a49fac5302 HDFS-11362. StorageDirectory should initialize a non-null default StorageDirType. Contribute by Hanisha Koneru. 2017-04-06 14:33:16 -07:00
Arpit Agarwal
8c57aeb5b4 HDFS-11131. TestThrottledAsyncChecker#testCancellation is flaky. 2017-04-05 17:54:30 -07:00
Andrew Wang
9e0e430f18 HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang. 2017-04-04 23:05:24 -07:00
Akira Ajisaka
8cb7325527
HDFS-9651. Add missing robots.txt files. 2017-04-05 12:47:45 +09:00
Andrew Wang
18432130a7 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-04 13:38:25 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Wei-Chiu Chuang
bc7aff7cec HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang. 2017-04-03 07:35:09 -07:00
Hanisha Koneru
73835c73e2 HDFS-11560. Expose slow disks via NameNode JMX. Contributed by Hanisha Koneru. 2017-03-31 13:50:29 -07:00
Arpit Agarwal
5485d93bda HDFS-11603. Improve slow mirror/disk warnings in BlockReceiver. 2017-03-31 12:10:27 -07:00
Hanisha Koneru
28cdc5a8dc HDFS-11551. Handle SlowDiskReport from DataNode at the NameNode. Contributed by Hanisha Koneru. 2017-03-30 22:41:26 -07:00
Mingliang Liu
e5f0622a6f HDFS-11592. Closing a file has a wasteful preconditions in NameNode. Contributed by Eric Badger 2017-03-30 15:44:21 -07:00
Daniel Templeton
15e3873dc3 HDFS-11571. Typo in DataStorage exception message (Contributed by Anna Budai via Daniel Templeton) 2017-03-29 10:56:36 -07:00
Rakesh Radhakrishnan
84d787b9d5 HDFS-11541. Call RawErasureEncoder and RawErasureDecoder release() methods. Contributed by SammiChen. 2017-03-29 11:41:48 +05:30
Yiqun Lin
6b09336438 HDFS-11577. Combine the old and the new chooseRandom for better performance. Contributed by Chen Liang. 2017-03-28 23:02:07 +08:00
Wei-Chiu Chuang
46d37a65cf HDFS-10506. Addendum patch to include missing changes. Contributed by Akira Ajisaka. 2017-03-27 05:23:45 -07:00
Yiqun Lin
96e2ab8144 HDFS-11555. Fix typos in class OfflineImageReconstructor. Contributed by Yiqun Lin. 2017-03-27 19:23:37 +08:00
Tsuyoshi Ozawa
84ddedc0b2 HADOOP-10101. Update guava dependency to the latest version. (ozawa) 2017-03-25 10:06:52 +09:00
Xiaobing Zhou
1168ece596 HDFS-11534. Add counters for number of blocks in pending IBR. Contributed by Xiaobing Zhou. 2017-03-24 14:33:44 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Ravi Prakash
d4f73e7e27 HDFS-11574. Spelling mistakes in the Java source. Contributed by Hu Xiaodong. 2017-03-24 09:38:17 -07:00
Yiqun Lin
3b908f71c5 HDFS-11358. DiskBalancer: Report command supports reading nodes from host file. Contributed by Yiqun Lin. 2017-03-21 17:44:17 +08:00
Arpit Agarwal
e7c8da614c HDFS-11545. Propagate DataNode's slow disks info to the NameNode via Heartbeats. Contributed by Hanisha Koneru. 2017-03-20 21:54:58 -07:00
Chris Douglas
b104f3a282 HDFS-6648. Order of namenodes in ConfiguredFailoverProxyProvider is undefined. Contributed by Inigo Goiri 2017-03-20 17:15:13 -07:00
Arpit Agarwal
ffa160ddb8 HDFS-11547. Add logs for slow BlockReceiver while writing data to disk. Contributed by Xiaobing Zhou. 2017-03-17 19:02:24 -07:00
Hanisha Koneru
7f8e928400 HDFS-11517. Expose slow disks via DataNode JMX. Contributed by Hanisha Koneru 2017-03-17 15:42:25 -07:00
Andrew Wang
92ea6d74ec HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki. 2017-03-17 11:26:02 -07:00
Andrew Wang
4812518b23 HDFS-10530. BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy. Contributed by Manoj Govindassamy and Rui Gao. 2017-03-16 15:07:38 -07:00
Kihwal Lee
ba62b50eba HDFS-10601. Improve log message to include hostname when the NameNode is in safemode. Contributed by Kuhu Shukla. 2017-03-16 10:54:59 -05:00
Hanisha Koneru
d69a82c89c HDFS-11511. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-03-15 18:01:45 -07:00
Chen Liang
615ac09499 HDFS-11419. DFSTopologyNodeImpl#chooseRandom optimizations. Contributed by Chen Liang. 2017-03-15 12:28:22 -07:00
Yiqun Lin
4416a07c9c HDFS-11420. Edit file should not be processed by the same type processor in OfflineEditsViewer. Contributed by Yiqun Lin. 2017-03-15 18:05:03 +08:00
Andrew Wang
cc1292e73a HDFS-9705. Refine the behaviour of getFileChecksum when length = 0. Contributed by Kai Zheng and SammiChen. 2017-03-14 16:41:10 -07:00
Andrew Wang
7515e75103 HDFS-11505. Do not enable any erasure coding policies by default. Contributed by Manoj Govindassamy. 2017-03-14 11:47:25 -07:00
Yiqun Lin
023b941e3b HDFS-11526. Fix confusing block recovery message. Contributed by Yiqun Lin. 2017-03-14 17:49:48 +08:00
Chen Liang
9832ae0ed8 HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-03-13 17:30:10 -07:00
Jing Zhao
55796a0946 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. 2017-03-13 14:24:51 -07:00
Lei Xu
6d356b6b4d HDFS-11340. DataNode reconfigure for disks doesn't remove the failed volumes. (Manoj Govindassamy via lei) 2017-03-10 14:37:13 -08:00
Andrew Wang
819808a016 HDFS-11506. Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code. Contributed by Manoj Govindassamy. 2017-03-09 17:29:11 -08:00
Masatake Iwasaki
385d2cb777 HDFS-11499. Decommissioning stuck because of failing recovery. Contributed by Lukas Majercak and Manoj Govindassamy. 2017-03-09 21:13:50 +09:00
Andrew Wang
33a38a5341 HDFS-11314. Enforce set of enabled EC policies on the NameNode. 2017-03-08 16:41:44 -08:00
Andrew Wang
5ca6ef0c26 HDFS-10983. OIV tool should make an EC file explicit. Contributed by Manoj Govindassamy. 2017-03-08 15:36:19 -08:00
Andrew Wang
5addacb1e3 HDFS-11152. Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors. Contributed by SammiChen. 2017-03-08 08:47:38 -08:00
Arpit Agarwal
959940b0ab HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-07 10:12:35 -08:00
Arpit Agarwal
b5adc5c301 HDFS-10838. Last full block report received time for each DN should be easily discoverable. Contributed by Surendra Singh Lilhore. 2017-03-06 16:39:53 -08:00
Andrew Wang
5e74196ede HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0. 2017-03-06 15:04:13 -08:00
Mingliang Liu
0f336bab9c HDFS-11494. Log message when DN is not selected for block replication. Contributed by Yiqun Lin 2017-03-03 18:47:13 -08:00
Jing Zhao
ac5ae0065a HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou. 2017-03-03 13:31:20 -08:00
Andrew Wang
3085a60430 HDFS-8112. Relax permission checking for EC related operations. 2017-03-03 13:00:22 -08:00
Rakesh Radhakrishnan
3749152b66 HDFS-11416. Refactor out system default erasure coding policy. Contributed by Andrew Wang. 2017-03-03 09:28:03 +05:30
Andrew Wang
8f4817f2c5 Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru." 
This reverts commit e61491d476.
 2017-03-02 16:20:48 -08:00
Arpit Agarwal
e61491d476 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:25 -08:00
Arpit Agarwal
b3ec531f40 HDFS-11461. DataNode Disk Outlier Detection. Contributed by Hanisha Koneru. 2017-03-02 12:45:48 -08:00
Arpit Agarwal
eeca8b0c4e HDFS-11419. HDFS specific network topology classes with storage type info included. Contributed by Chen Liang. 2017-03-02 09:21:27 -08:00
Ming Ma
25c84d279b HDFS-11412. Maintenance minimum replication config value allowable range should be [0, DefaultReplication]. (Manoj Govindassamy via mingma) 2017-03-01 20:23:52 -08:00
Rakesh Radhakrishnan
82ef9accaf HDFS-11428. Change setErasureCodingPolicy to take a required string EC policy name. Contributed by Andrew Wang. 2017-03-01 16:06:02 +05:30
Brahma Reddy Battula
dcd03df9f9 HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula. 2017-03-01 10:45:56 +05:30
Zhe Zhang
d269b488a7 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang. 2017-02-28 09:00:49 -08:00
Andrew Wang
55c07bbed2 HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage. Contributed by Manoj Govindassamy. 2017-02-27 17:07:45 -08:00
Wei-Chiu Chuang
05391c1845 HDFS-10506. OIV's ReverseXML processor cannot reconstruct some snapshot details. Contributed by Akira Ajisaka. 2017-02-25 14:38:50 -08:00
Xiao Chen
e24ed47d9a HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-02-24 16:49:46 -08:00
Arpit Agarwal
d2b3ba9b8f HDFS-11295. Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete(). Contributed by Marton Elek. 2017-02-24 15:44:11 -08:00
Andrew Wang
132f758e3d HDFS-11426. Refactor EC CLI to be similar to storage policies CLI. 2017-02-23 16:00:11 -08:00
Jing Zhao
13d4bcfe35 HDFS-4025. QJM: Sychronize past log segments to JNs that missed them. Contributed by Hanisha Koneru. 2017-02-22 16:33:38 -08:00
Andrew Wang
d150f061f4 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu. 2017-02-22 15:16:09 -08:00
Andrew Wang
8ef7ebbc71 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu. 2017-02-21 14:54:20 -08:00
Andrew Wang
fc9ad3ce3a HDFS-11405. Rename "erasurecode" CLI subcommand to "ec". Contributed by Manoj Govindassamy. 2017-02-21 13:55:27 -08:00
Vinayakumar B
4804050630 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore) 2017-02-21 18:13:19 +05:30
Xiao Chen
02c549484a HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp. 2017-02-16 18:07:55 -08:00
Wei-Chiu Chuang
5690b51ef7 HDFS-11100. Recursively deleting file protected by sticky bit should fail. Contributed by John Zhuge. 2017-02-16 05:39:37 -08:00
Ming Ma
a136936d01 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma) 2017-02-15 20:24:07 -08:00
Mingliang Liu
0741dd3b9a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore 2017-02-15 11:37:26 -08:00
Wei-Chiu Chuang
859bd159ae HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang. 2017-02-15 02:50:35 -08:00
Akira Ajisaka
8acb376c9c HDFS-11238. Fix checkstyle warnings in NameNode#createNameNode. Contributed by Ethan Li. 2017-02-15 16:53:50 +09:00
Yongjun Zhang
8e53f2b9b0 HDFS-11391. Numeric usernames do no work with WebHDFS FS write access. (Pierre Villard via Yongjun Zhang) 2017-02-14 13:40:53 -08:00
Yiqun Lin
4164a2032a HDFS-11408. The config name of balance bandwidth is out of date. Contributed by Yiqun Lin. 2017-02-14 18:57:20 +08:00
Chris Douglas
4ed33e9ca3 HDFS-11026. Convert BlockTokenIdentifier to use Protobuf. Contributed by Ewan Higgs. 2017-02-13 11:29:18 -08:00
Yiqun Lin
663e683adf HDFS-10219. Change the default value for dfs.namenode.reconstruction.pending.timeout-sec from -1 to 300. Contributed by Yiqun Lin. 2017-02-06 19:19:30 +08:00
Yiqun Lin
9cbbd1eae8 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 2017-02-06 13:15:16 +08:00
Arpit Agarwal
6aa09dc28a HDFS-11112. Journal Nodes should refuse to format non-empty directories. Contributed by Yiqun Lin. 2017-02-01 16:51:58 -08:00
Jing Zhao
b6f290d5b6 HDFS-11370. Optimize NamenodeFsck#getReplicaInfo. Contributed Takanobu Asanuma. 2017-02-01 11:21:35 -08:00
Lei Xu
bec9b7aa1d HDFS-11335. Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf. (Manoj Govindassamy via lei) 2017-02-01 14:42:51 +08:00
Xiao Chen
88da9f6b67 Revert "HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang." 
This reverts commit ada233b7cd.
 2017-01-26 13:42:50 -08:00
Arpit Agarwal
7c1cc30b3c HDFS-11369. Change exception message in StorageLocationChecker. 2017-01-26 09:12:38 -08:00
Jing Zhao
b782bf2156 HDFS-11124. Report blockIds of internal blocks for EC files in Fsck. Contributed by Takanobu Asanuma. 2017-01-25 11:16:17 -08:00
Zhe Zhang
18e1d68209 HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki. 2017-01-25 09:58:39 -08:00
Arpit Agarwal
b57368b6f8 HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 16:58:20 -08:00
Wei-Chiu Chuang
355b907c09 HDFS-11121. Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping. Contributed by Takanobu Asanuma. 2017-01-21 05:56:22 +08:00
Lei Xu
78ae2aed8f HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-19 16:28:31 +08:00
Andrew Wang
a8f1c7f542 HDFS-110268. Correctly reconstruct erasure coding file from FSImage. Contributed by SammiChen. 2017-01-19 00:24:11 -08:00
Lei Xu
63320d1daa HDFS-11287. Storage class member storageDirs should be private to avoid unprotected access by derived classes. (Manoj Govindassamy via lei) 2017-01-19 15:38:20 +08:00
Allen Wittenauer
0eb4b513b7 HADOOP-13673. Update scripts to be smarter when running with privilege 
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
 2017-01-18 14:39:05 -08:00
Andrew Wang
a2a5d7b5bc HDFS-10759. Change fsimage bool isStriped from boolean to an enum. Contributed by Ewan Higgs. 2017-01-18 13:31:05 -08:00
Vinitha Reddy Gankidi
8a0fa0f7e8 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:46:32 -08:00
Akira Ajisaka
9130af3f74 HADOOP-13955. Replace deprecated HttpServer2 and SSLFactory constants. Contributed by John Zhuge. 2017-01-18 18:11:33 +09:00
Xiaoyu Yao
b1fce2b8b1 HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. Contributed by Xiaoyu Yao. 2017-01-17 14:33:26 -08:00
Arpit Agarwal
89bb05d92b HDFS-11339. Fix import. 2017-01-16 23:05:28 -08:00
Arpit Agarwal
79e939d0b8 HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. Contributed by Hanisha Koneru. 2017-01-16 14:43:29 -08:00
Arpit Agarwal
a853b4e1b5 HDFS-11342. Fix FileInputStream leak in loadLastPartialChunkChecksum. Contributed by Chen Liang. 2017-01-16 13:45:22 -08:00
Lei Xu
2f8e9b7e4b Revert "HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei)" 
This reverts commit c18590fce2.
 2017-01-16 15:11:53 +08:00
Anu Engineer
2604e82eba HDFS-11344. The default value of the setting dfs.disk.balancer.block.tolerance.percent is different. Contributed by Yiqun Lin. 2017-01-13 22:38:46 -08:00
Wei-Chiu Chuang
1cde954a4f HDFS-11306. Print remaining edit logs from buffer if edit log can't be rolled. Contributed by Wei-Chiu Chuang. 2017-01-13 11:46:30 -08:00
Brahma Reddy Battula
a5ec1e31a3 HDFS-9935. Remove LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD and unused import from HdfsServerConstants. Contributed Yiqun Lin. 2017-01-11 19:34:01 +05:30
Ming Ma
467f5f1735 HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj Govindassamy via mingma) 2017-01-10 20:12:42 -08:00
Andrew Wang
e69231658d HDFS-11072. Add ability to unset and change directory EC policy. Contributed by Sammi Chen. 2017-01-10 11:32:48 -08:00
Arpit Agarwal
4046794a53 HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-10 10:43:11 -08:00
Lei Xu
c18590fce2 HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-10 19:37:58 +08:00
Jing Zhao
7ec609b289 HDFS-11273. Move TransferFsImage#doGetUrl function to a Util class. Contributed by Hanisha Koneru. 2017-01-09 18:05:33 -08:00
Yongjun Zhang
603cbcd513 HDFS-11292. log lastWrittenTxId etc info in logSyncAll. Contributed by Yongjun Zhang. 2017-01-09 15:44:42 -08:00
Arpit Agarwal
91bf504440 HDFS-11301. Double wrapping over RandomAccessFile in LocalReplicaInPipeline#createStreams. Contributed by Hanisha Koneru. 2017-01-09 10:32:18 -08:00
Xiaoyu Yao
87bb1c49bb HDFS-11279. Cleanup unused DataNode#checkDiskErrorAsync(). Contributed by Hanisha Koneru 2017-01-03 18:25:46 -08:00
Andrew Wang
7fcc73fc0d HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 2017-01-03 09:58:00 -08:00
Xiao Chen
e769957556 HDFS-11275. Check groupEntryIndex and throw a helpful exception on failures when removing ACL. 2016-12-29 10:36:50 -08:00
Lei Xu
a4f66655ec HDFS-11267. Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei) 2016-12-29 16:57:40 +08:00
Lei Xu
e9f1396834 HDFS-11251. ConcurrentModificationException during DataNode#refreshVolumes. (Manoj Govindassamy via lei) 2016-12-29 15:11:25 +08:00
Xiaoyu Yao
603f3ef138 HDFS-11274. Datanode should only check the failed volume upon IO errors. Contributed by Xiaoyu Yao. 2016-12-28 22:08:13 -08:00
Daniel Templeton
e216e8e233 HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-27 13:03:16 -08:00
Mingliang Liu
c0e0ef2969 HDFS-11270. Document the missing options of NameNode bootstrap command. Contributed by Yiqun Lin 2016-12-27 11:21:10 -08:00
Akira Ajisaka
8f218ea284 HDFS-11250. Fix a typo in ReplicaUnderRecovery#setRecoveryID. Contributed by Yiqun Lin. 2016-12-26 16:51:29 +09:00
Akira Ajisaka
483cd06ad4 HDFS-11271. Typo in NameNode UI. Contributed by Wei-Chiu Chuang. 2016-12-26 16:36:54 +09:00
Xiaoyu Yao
4e9029653d HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2016-12-22 23:46:58 -08:00
Kai Zheng
56a13a6a59 HDFS-11216. Add remoteBytesRead counter metrics for erasure coding reconstruction task. Contributed by Sammi Chen 2016-12-22 14:18:54 +08:00
Haohui Mai
f6e80acd68 HDFS-11258. File mtime change could not save to editlog. Contributed by Jimmy Xiang. 2016-12-21 13:04:03 -08:00
Xiaoyu Yao
f678080dbd HDFS-11182. Update DataNode to use DatasetVolumeChecker. Contributed by Arpit Agarwal. 2016-12-20 13:53:32 -08:00
Xiaoyu Yao
5daa8d8631 HDFS-10913. Introduce fault injectors to simulate slow mirrors. Contributed by Xiaobing Zhou. 2016-12-20 13:17:52 -08:00
Xiao Chen
5b7acdd206 HDFS-11195. Return error when appending files by webhdfs rest api fails. Contributed by Yuanbo Liu. 2016-12-20 12:24:00 -08:00
Brahma Reddy Battula
1b401f6a73 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage compression. Contributed by Brahma Reddy Battula. 2016-12-20 20:54:03 +05:30
Xiaoyu Yao
fe4ff64a4a HDFS-10959. Adding per disk IO statistics and metrics in DataNode. Contributed by Xiaoyu Yao. 2016-12-19 22:42:54 -08:00
Akira Ajisaka
d2026ae9e6 HDFS-11262. Remove unused variables in FSImage.java. Contributed by Jagadesh Kiran N. 2016-12-20 14:34:35 +09:00
Anu Engineer
a95639068c HDFS-9911. TestDataNodeLifeline Fails intermittently. Contributed by Yiqun Lin 2016-12-16 09:46:21 -08:00
Xiao Chen
cee0c468b0 HDFS-11188. Change min supported DN and NN versions back to 2.x. Contributed by Andrew Wang. 2016-12-15 16:48:50 -08:00
Wei-Chiu Chuang
aebb9127ba HDFS-11160. VolumeScanner reports write-in-progress replicas as corrupt incorrectly. Contributed by Wei-Chiu Chuang and Yongjun Zhang. 2016-12-15 16:32:50 -08:00
Mingliang Liu
6fce191549 HDFS-11253. FileInputStream leak on failure path in BlockSender. Contributed by Arpit Agarwal 2016-12-15 15:39:15 -08:00
Mingliang Liu
ac0d402c6a HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-15 14:53:53 -08:00
Akira Ajisaka
36947f79ba HDFS-11249. Redundant toString() in DFSConfigKeys.java. Contributed by Jagadesh Kiran N. 2016-12-16 00:33:27 +09:00
Arpit Agarwal
6ba9587d37 HDFS-10958. Add instrumentation hooks around Datanode disk IO. 2016-12-14 11:18:58 -08:00
Kai Zheng
1f14f6d038 HDFS-8411. Add bytes count metrics to datanode for ECWorker. Contributed by Sammi Chen and Andrew Wang 2016-12-14 14:50:50 +08:00
Uma Maheswara Rao G
e24a923db5 HDFS-11164: Mover should avoid unnecessary retries if the block is pinned. Contributed by Rakesh R 2016-12-13 17:09:58 -08:00
Mingliang Liu
754f15bae6 HDFS-11226. cacheadmin, cryptoadmin and storagepolicyadmin should support generic options. Contributed by Brahma Reddy Battula 2016-12-12 18:21:15 -08:00
Akira Ajisaka
2d4731c067 HDFS-11233. Fix javac warnings related to the deprecated APIs after upgrading Jackson. Contributed by Yiqun Lin. 2016-12-13 11:11:15 +09:00
Wei-Chiu Chuang
2a28e8cf04 HDFS-11229. HDFS-11056 failed to close meta file. Contributed by Wei-Chiu Chuang. 2016-12-09 16:02:42 -08:00
Andrew Wang
5bd7dece92 HDFS-11217. Annotate NameNode and DataNode MXBean interfaces as Private/Stable. Contributed by Jagadesh Kiran N. 2016-12-09 12:05:42 -08:00
Vinayakumar B
d1d4aba71b HDFS-11224. Lifeline message should be ignored for dead nodes (Contributed by Vinayakumar B) 2016-12-09 14:53:50 +05:30
Kihwal Lee
13d8e552d2 Revert "HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from DatanodeProtocolProtos. Contributed by Tao Luo." 
This reverts commit 4c8db60092.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
 2016-12-08 15:41:49 -06:00
Kihwal Lee
9f8344db50 Revert "HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger" 
This reverts commit 8c4680852b.
 2016-12-08 15:36:08 -06:00
Xiao Chen
401c731872 HDFS-11197. Listing encryption zones fails when deleting a EZ that is on a snapshotted directory. Contributed by Wellington Chevreuil. 2016-12-08 12:40:20 -08:00
Akira Ajisaka
9ef89ede2f HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin. 2016-12-08 13:38:02 +09:00
Akira Ajisaka
deb4743b2c Revert "HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin." 
This reverts commit 0857641f62.
 2016-12-08 13:37:06 +09:00
Akira Ajisaka
0857641f62 HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin. 2016-12-08 13:34:45 +09:00
Andrew Wang
ea2895f4ed HDFS-8630. WebHDFS : Support get/set/unset StoragePolicy. Contributed by Surendra Singh Lilhore. 2016-12-07 15:52:16 -08:00
Ming Ma
c73e08a6da HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:26:09 -08:00
Andrew Wang
f53b481ffe HDFS-11172. Support an erasure coding policy using RS 10 + 4. Contributed by Wei Zhou. 2016-12-06 12:49:38 -08:00
Xiaoyu Yao
df983b524a HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2016-12-06 11:05:47 -08:00
Mingliang Liu
43cb1678cc HDFS-11211. Add a time unit to the DataNode client trace format. Contributed by Jagadesh Kiran N 2016-12-06 09:12:11 -08:00
Kihwal Lee
4dd4f3a4bb HDFS-11198. NN UI should link DN web address using hostnames. Contributed by Weiwei Yang. 2016-12-06 08:06:50 -06:00
Andrew Wang
08a7253bc0 Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang" 
This reverts commit c7ff34f8dc.
 2016-12-05 23:09:35 -08:00
Andrew Wang
8e63fa98ea HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are decomissioning. Contributed by Weiwei Yang. 2016-12-05 18:13:53 -08:00
Xiaoyu Yao
dcedb72af4 Revert "HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao." 
This reverts commit aeecfa24f4.
 2016-12-05 12:44:20 -08:00
Mingliang Liu
8c4680852b HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-05 11:48:58 -08:00
Wei-Chiu Chuang
1b5cceaffb Revert "HDFS-11201. Spelling errors in the logging, help, assertions and exception messages. Contributed by Grant Sohn." 
This reverts commit b9522e86a5.
 2016-12-05 10:54:43 -08:00
Wei-Chiu Chuang
b9522e86a5 HDFS-11201. Spelling errors in the logging, help, assertions and exception messages. Contributed by Grant Sohn. 2016-12-05 10:48:25 -08:00
Mingliang Liu
c7ff34f8dc HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang 2016-12-02 11:10:13 -08:00
Kihwal Lee
96c574927a HDFS-8674. Improve performance of postponed block scans. Contributed by Daryn Sharp. 2016-12-01 12:11:27 -06:00
Akira Ajisaka
e0fa49234f HDFS-11180. Intermittent deadlock in NameNode when failover happens. 2016-12-01 23:08:59 +09:00
Andrew Wang
7226a71b1f HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers and Andrew Wang. 2016-11-30 15:58:31 -08:00
Ravi Prakash
625df87c7b HDFS-8678. Bring back the feature to view chunks of files in the HDFS file browser. Contributed by Ivo Udelsmann. 2016-11-30 09:12:15 -08:00
Kai Zheng
51e6c1cc3f HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen 2016-11-30 15:52:56 +08:00
Kai Zheng
cfd8076f81 Revert due to an error "HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen" 
This reverts commit 5614f847b2ef2a5b70bd9a06edc4eba06174c6.
 2016-11-30 15:44:52 +08:00
Xiaoyu Yao
aeecfa24f4 HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2016-11-29 20:52:36 -08:00
Arpit Agarwal
eaaa32950c HDFS-11149. Support for parallel checking of FsVolumes. 2016-11-29 20:31:02 -08:00
Andrew Wang
b407d53195 Revert "HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers." 
This reverts commit 09451252fa.
 2016-11-29 10:50:10 -08:00
Andrew Wang
09451252fa HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers. 2016-11-29 00:12:35 -08:00
Robert Kanter
47ca9e26fb HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via rkanter) 2016-11-28 18:08:09 -08:00
Kai Zheng
5d5614f847 HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen 2016-11-28 14:34:44 +08:00
Andrew Wang
de4894936a HDFS-11368. Erasure Coding: Deprecate replication-related config keys. Contributed by Rakesh R. 2016-11-23 16:42:06 -08:00
Arpit Agarwal
613b902b98 HDFS-11148. Update DataNode to use StorageLocationChecker at startup. 2016-11-22 10:50:25 -08:00
Brahma Reddy Battula
afcf8d38e7 HADOOP-13646. Remove outdated overview.html. Contributed By Brahma Reddy Battula. 2016-11-22 19:45:45 +05:30
Kihwal Lee
49a09179e3 HDFS-10966. Enhance Dispatcher logic on deciding when to give up a source DataNode. Contributed by Mark Wagner and Zhe Zhang. 2016-11-21 10:13:24 -06:00
Akira Ajisaka
d232625f73 HDFS-11113. Document dfs.client.read.striped configuration in hdfs-default.xml. Contributed by Rakesh R. 2016-11-21 14:05:15 +09:00
Arpit Agarwal
04a024b683 HDFS-11147. Remove confusing log output in FsDatasetImpl#getInitialVolumeFailureInfos. Contributed by Chen Liang. 2016-11-16 14:47:34 -08:00
Zhe Zhang
ff0b99eafe HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 11:05:29 -08:00
Arpit Agarwal
3d26717777 HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup. 
This closes #155.
 2016-11-11 15:02:52 -08:00
Xiaoyu Yao
4484b48498 HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. Contributed by Chen Liang. 2016-11-11 14:31:23 -08:00
Brahma Reddy Battula
1ae57f0f75 HDFS-11128. CreateEditsLog throws NullPointerException. Contributed by Hanisha Koneru. 2016-11-11 20:08:15 +05:30
Vinayakumar B
ca68f9cb5b HDFS-9337. Validate required params for WebHDFS requests (Contributed by Jagadesh Kiran N) 2016-11-10 16:51:33 +05:30
Wei-Chiu Chuang
c619e9b43f HDFS-11056. Concurrent append and read operations lead to checksum error. Contributed by Wei-Chiu Chuang. 2016-11-09 09:16:50 -08:00
Brahma Reddy Battula
ed0bebabaa HDFS-9482. Replace DatanodeInfo constructors with a builder pattern. Contributed by Brahma Reddy Battula. 2016-11-08 18:17:07 -08:00
Arpit Agarwal
3fff158587 HDFS-11114. Support for running async disk checks in DataNode. 
This closes #153.
 2016-11-07 18:45:53 -08:00
Akira Ajisaka
049e7d27be HDFS-10970. Update jackson from 1.9.13 to 2.x in hadoop-hdfs. 2016-11-07 11:16:31 +09:00
Xiao Chen
d8bab3dcb6 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu. 2016-11-04 18:06:55 -07:00
Vinayakumar B
69dd5fa2d4 HDFS-11098. Datanode in tests cannot start in Windows after HDFS-10638 (Contributed by Vinayakumar B) 2016-11-04 08:31:43 +05:30
Andrew Wang
5cad93d5c7 HADOOP-11088. Quash unnecessary safemode WARN message during NameNode startup. Contributed by Yiqun Lin. 2016-11-03 15:31:07 -07:00
Arpit Agarwal
b71907b2ae HDFS-11097. Fix warnings for deprecated StorageReceivedDeletedBlocks constructor. Contributed by Yiqun Lin. 2016-11-03 10:27:52 -07:00
Anu Engineer
730cb0cff6 HDFS-10566. Submit plan request should throw exception if Datanode is in non-REGULAR status. Contributed by Xiaobing Zhou. 2016-11-02 17:46:56 -07:00
Lei Xu
b59206190e HDFS-10909. De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and DFSClient#initThreadsNumForStripedReads. (Manoj Govindassamy via lei) 2016-11-02 16:45:25 -07:00
Mingliang Liu
9b0c17f852 HDFS-11095. BlockManagerSafeMode should respect extension period default config value (30s). Contributed by Mingliang Liu 2016-11-02 14:58:24 -07:00
Andrew Wang
f2800f523d Revert "HADOOP-13717. Normalize daemonization behavior of the diskbalancer with balancer and mover." 
This reverts commit bd7f5911c7.
 2016-11-02 13:46:33 -07:00
Andrew Wang
bd7f5911c7 HADOOP-13717. Normalize daemonization behavior of the diskbalancer with balancer and mover. 2016-11-01 16:48:45 -07:00
Anu Engineer
34173a42d4 HDFS-10633. DiskBalancer : Add the description for the new setting dfs.disk.balancer.plan.threshold.percent in HDFSDiskbalancer.md. Contributed by Yiqun Lin. 2016-11-01 11:14:03 -07:00
Arpit Agarwal
a9d68d2e8e HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:39:30 -07:00
Brahma Reddy Battula
e9c7a97089 HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 2016-10-31 00:22:51 +05:30
Mingliang Liu
8a9388e5f6 HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger 2016-10-28 10:53:14 -07:00
Xiao Chen
57187fdb93 HDFS-11064. Mention the default NN rpc ports in hdfs-default.xml. Contributed by Yiqun Lin. 2016-10-27 18:13:06 -07:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Mingliang Liu
9e03ee5279 HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 2016-10-27 16:00:27 -07:00
Erik Krogen
f3ac1f41b8 HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen. 2016-10-27 15:58:25 -07:00
Kihwal Lee
ae48c496dc HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee 2016-10-27 14:17:16 -05:00
Kihwal Lee
db41965996 Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu" 
This reverts commit ac35ee9393.
 2016-10-27 14:07:28 -05:00
Brahma Reddy Battula
ac35ee9393 HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu 2016-10-27 20:20:56 +05:30
Vinayakumar B
1cf6ec4ad4 HDFS-8492. DN should notify NN when client requests a missing block (Contributed by Walter Su) 2016-10-27 16:44:00 +05:30
Vinayakumar B
77f2684dd3 HDFS-10769. BlockIdManager.clear doesn't reset the counter for blockGroupIdGenerator (Contributed by Yiqun Lin and Rakesh R) 2016-10-27 14:16:03 +05:30
Akira Ajisaka
9c680d0b05 HDFS-11049. The description of dfs.block.replicator.classname is not clear. Contributed by Yiqun Lin. 2016-10-27 16:30:57 +09:00
Vinayakumar B
a895c66255 HDFS-9929. Duplicate keys in NAMENODE_SPECIFIC_KEYS (Contributed by Akira Ajisaka) 2016-10-27 12:20:15 +05:30
Anu Engineer
9f32364d28 HDFS-11038. DiskBalancer: support running multiple commands in single test. Contributed by Xiaobing Zhou. 2016-10-26 19:39:02 -07:00
Lei Xu
f209e93566 HDFS-10638. Modifications to remove the assumption that StorageLocation is associated with java.io.File in Datanode. (Virajith Jalaparti via lei) 2016-10-26 10:32:35 -07:00
Kihwal Lee
e90af4a89b HDFS-11053. Unnecessary superuser check in versionRequest(). Contributed by Kihwal Lee. 2016-10-26 09:16:13 -05:00
Kai Zheng
287effff93 HDFS-10935. TestFileChecksum fails in some cases. Contributed by Sammi Chen 2016-10-26 09:40:31 +06:00
Arpit Agarwal
9dce9011ad HDFS-11054. Suppress verbose log message in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2016-10-25 19:50:56 -07:00
Zhe Zhang
f6367c5f44 HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee. 2016-10-25 10:19:13 -07:00
Akira Ajisaka
dc3272bfe8 HDFS-11046. Duplicate '-' in the daemon log name. 2016-10-25 10:07:43 +09:00
Kihwal Lee
9d175853b0 HDFS-10997. Reduce number of path resolving methods. Contributed by Daryn Sharp. 2016-10-24 17:14:51 -05:00
Wei-Chiu Chuang
df857f0d10 HDFS-10975. fsck -list-corruptfileblocks does not report corrupt EC files. Contributed by Takanobu Asanuma. 2016-10-21 15:43:12 -07:00
Andrew Wang
61e30cf83c HDFS-8410. Add computation time metrics to datanode for ECWorker. Contributed by SammiChen. 2016-10-21 13:12:41 -07:00
Chris Douglas
ae8bccd509 HADOOP-13702. Add instrumented ReadWriteLock. Contributed by Jingcheng Du 2016-10-21 12:59:54 -07:00
Wei-Chiu Chuang
5e83a21cb6 HDFS-10976. Report erasure coding policy of EC files in Fsck. Contributed by Wei-Chiu Chuang. 2016-10-20 13:06:43 -07:00
Wei-Chiu Chuang
6d2da38d16 HDFS-11018. Incorrect check and message in FsDatasetImpl#invalidate. Contributed by Yiqun Lin. 2016-10-20 10:56:39 -07:00
Kihwal Lee
5c0bffddc0 HDFS-10627. Volume Scanner marks a block as "suspect" even if the exception is network-related. Contributed by Rushabh S Shah. 2016-10-20 10:48:35 -05:00
Brahma Reddy Battula
4c73be135c HDFS-9480. Expose nonDfsUsed via StorageTypeStats. Contributed by Brahma Reddy Battula 2016-10-20 20:45:45 +05:30
Arpit Agarwal
b4564103e4 HDFS-10752. Several log refactoring/improvement suggestion in HDFS. Contributed by Hanisha Koneru. 2016-10-19 17:20:07 -07:00
Xiao Chen
c5573e6a75 HDFS-11009. Add a tool to reconstruct block meta file from CLI. 2016-10-18 22:42:28 -07:00
Ming Ma
b61fb267b9 HDFS-9390. Block management for maintenance states. 2016-10-17 17:45:41 -07:00
Brahma Reddy Battula
5f4ae85bd8 HDFS-11003. Expose XmitsInProgress through DataNodeMXBean. Contributed By Brahma Reddy Battula 2016-10-15 22:28:33 +05:30
Vinitha Reddy Gankidi
391ce535a7 HDFS-10301. Remove FBR tracking state to fix false zombie storage detection for interleaving block reports. Contributed by Vinitha Gankidi. 2016-10-14 18:13:54 -07:00
Anu Engineer
30bb1970cc HDFS-10558. DiskBalancer: Print the full path to plan file. Contributed by Xiaobing Zhou. 2016-10-14 17:07:59 -07:00
Jing Zhao
adb96e109f HDFS-10827. When there are unrecoverable ec block groups, Namenode Web UI doesn't show the block names. Contributed by Takanobu Asanuma. 2016-10-14 13:21:53 -07:00
Kihwal Lee
332a61fd74 HDFS-10987. Make Decommission less expensive when lot of blocks present. Contributed by Brahma Reddy Battula. 2016-10-13 13:52:49 -05:00
Brahma Reddy Battula
1291254042 HDFS-10986. DFSAdmin should log detailed error message if any. Contributed by MingLiang Liu 2016-10-13 22:05:00 +05:30
Kihwal Lee
9454dc5e80 HDFS-11000. webhdfs PUT does not work if requests are routed to call queue. Contributed by Kihwal Lee. 2016-10-13 08:47:15 -05:00
Kihwal Lee
85cd06f663 HDFS-10789. Route webhdfs through the RPC call queue. Contributed by Daryn Sharp and Rushabh S Shah. 2016-10-12 15:11:42 -05:00
Mingliang Liu
3c9a01062e HDFS-10903. Replace config key literal strings with config key names II: hadoop hdfs. Contributed by Chen Liang 2016-10-11 16:29:30 -07:00
Xiaoyu Yao
61f0490a73 HDFS-10984. Expose nntop output as metrics. Contributed by Siddharth Wagle. 2016-10-11 15:55:02 -07:00
Lei Xu
809cfd27a3 HDFS-10916. Switch from "raw" to "system" xattr namespace for erasure coding policy. (Andrew Wang via lei) 2016-10-11 10:04:46 -07:00
Lei Xu
96b12662ea HDFS-10637. Modifications to remove the assumption that FsVolumes are backed by java.io.File. (Virajith Jalaparti via lei) 2016-10-10 15:30:03 -07:00
Jing Zhao
4d106213c0 HDFS-10968. BlockManager#isInNewRack should consider decommissioning nodes. Contributed by Jing Zhao. 2016-10-07 22:44:54 -07:00
Xiao Chen
6a38d118d8 HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory. 2016-10-07 17:37:15 -07:00
Kihwal Lee
e57fa81d95 HDFS-10980. Optimize check for existence of parent directory. Contributed by Daryn Sharp. 2016-10-07 17:20:15 -05:00
Kihwal Lee
3565c9af17 HDFS-10979. Pass IIP for FSDirDeleteOp methods. Contributed by Daryn Sharp. 2016-10-07 14:15:59 -05:00
Brahma Reddy Battula
be3cb10f53 HDFS-10969. Fix typos in hdfs-default.xml Contributed by Yiqun Lin 2016-10-07 22:18:40 +05:30
Kihwal Lee
48b9d5fd2a HDFS-10955. Pass IIP for FSDirAttr methods. Contributed by Daryn Sharp. 2016-10-06 16:33:46 -05:00
Chris Nauroth
1d330fbaf6 HADOOP-13150. Avoid use of toString() in output of HDFS ACL shell commands. Contributed by Chris Nauroth. 2016-10-06 13:19:16 -07:00
Kihwal Lee
f32e9fc8f7 HDFS-10939. Reduce performance penalty of encryption zones. Contributed by Daryn sharp. 2016-10-06 15:11:14 -05:00
Brahma Reddy Battula
b90fc70d67 HDFS-10963. Reduce log level when network topology cannot find enough datanodes. Contributed by Xiao chen 2016-10-06 17:47:31 +05:30
Wei-Chiu Chuang
c5ca216915 HDFS-10683. Make class Token$PrivateToken private. Contributed by John Zhuge. 2016-10-05 17:36:50 -07:00
Andrew Wang
e68c7b96c7 HDFS-10962. TestRequestHedgingProxyProvider is flaky. 2016-10-05 15:08:56 -07:00
Jing Zhao
8867762256 HDFS-10826. Correctly report missing EC blocks in FSCK. Contributed by Takanobu Asanuma. 2016-10-05 10:52:50 -07:00
Kihwal Lee
44f48ee96e HDFS-10956. Remove rename/delete performance penalty when not using snapshots. Contributed by Daryn Sharp. 2016-10-04 15:05:09 -05:00
Brahma Reddy Battula
5ea0210c7f HDFS-10944. Correct the javadoc of dfsadmin#disallowSnapshot. Contributed by Jagadesh Kiran N 2016-10-04 18:33:22 +05:30
Mingliang Liu
8078a5efd0 HDFS-10810. setreplication removing block from underconstrcution temporarily. Contributed by Brahma Reddy Battula 2016-10-03 21:50:16 -04:00
Xiao Chen
853d65a157 HDFS-10918. Add a tool to get FileEncryptionInfo from CLI. Contributed by Xiao Chen. 2016-10-03 16:01:54 -07:00
Kihwal Lee
744208431f HDFS-10940. Reduce performance penalty of block caching when not used. Contributed by Daryn Sharp. 2016-10-03 11:27:23 -05:00
Kihwal Lee
90020624b0 HDFS-10619. Cache path in InodesInPath. Contributed by Daryn Sharp. 2016-10-03 09:13:04 -05:00
Arpit Agarwal
fe9ebe20ab Revert "HDFS-10923. Make InstrumentedLock require ReentrantLock." 
This reverts commit c7ce6fdc20.
 2016-09-30 23:11:51 -07:00
Arpit Agarwal
c7ce6fdc20 HDFS-10923. Make InstrumentedLock require ReentrantLock. 2016-09-30 23:00:06 -07:00
Zhe Zhang
434c5ea75d HDFS-10896. Move lock logging logic from FSNamesystem into FSNamesystemLock. Contributed by Erik Krogen. 2016-09-30 13:16:08 -07:00
Wei-Chiu Chuang
2ab1ef15c5 HDFS-10908. Improve StripedBlockReader#createBlockReader error logging. Contributed by Manoj Govindassamy. 2016-09-30 12:35:47 -07:00
Kihwal Lee
a0730aa5ce HDFS-10851. FSDirStatAndListingOp: stop passing path as string. Contributed by Daryn Sharp. 2016-09-30 13:03:24 -05:00
Kihwal Lee
0670149c88 HDFS-10850. getEZForPath should NOT throw FNF. Contributed by Andrew Wang. 2016-09-30 08:44:18 -05:00
Andrew Wang
92e5e91598 HDFS-10914. Move remnants of oah.hdfs.client to hadoop-hdfs-client. 2016-09-28 16:01:03 -07:00
Kihwal Lee
5f34402ada HDFS-10779. Rename does not need to re-solve destination. Contributed by Daryn Sharp. 2016-09-28 17:57:23 -05:00
Anu Engineer
03f519a757 HDFS-9850. DiskBalancer: Explore removing references to FsVolumeSpi. Contributed by Manoj Govindassamy. 2016-09-27 21:35:06 -07:00
Arpit Agarwal
8ae4729107 HDFS-10828. Fix usage of FsDatasetImpl object lock in ReplicaMap. (Arpit Agarwal) 2016-09-27 10:02:15 -07:00
Arpit Agarwal
059058f961 HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha Koneru. 2016-09-26 17:09:47 -07:00
Wei-Chiu Chuang
3ae652f821 HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. Contributed by Wei-Chiu Chuang. 2016-09-26 13:11:32 -07:00
Brahma Reddy Battula
3e37e243ee HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by Jagadesh Kiran N 2016-09-25 11:02:46 +05:30
Wei-Chiu Chuang
74b3dd514c HDFS-10876. Dispatcher#dispatch should log IOException stacktrace. Contributed by Manoj Govindassamy. 2016-09-23 13:26:57 -07:00
Konstantin V Shvachko
a5bb88c8e0 HDFS-10843. Update space quota when a UC block is completed rather than committed. Contributed by Erik Krogen. 2016-09-23 10:37:46 -07:00
Anu Engineer
d85d9b2e7b HDFS-10871. DiskBalancerWorkItem should not import jackson relocated by htrace. Contributed by Manoj Govindassamy. 2016-09-22 19:36:16 -07:00
Andrew Wang
40acacee08 HDFS-10877. Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers. Contributed by Sean Mackrory. 2016-09-22 11:43:11 -07:00
Xiao Chen
e52d6e7a46 HDFS-10875. Optimize du -x to cache intermediate result. Contributed by Xiao Chen. 2016-09-19 21:44:42 -07:00
Andrew Wang
c54f6ef30f HDFS-10868. Remove stray references to DFS_HDFS_BLOCKS_METADATA_ENABLED. 2016-09-19 11:17:03 -07:00
Xiao Chen
ea839bd48e HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen. 2016-09-17 22:25:39 -07:00
Akira Ajisaka
b09a03cd7d HDFS-10862. Typos in 4 log messages. Contributed by Mehran Hassani. 2016-09-16 16:08:47 +09:00
Arpit Agarwal
ec3ea18875 HDFS-9895. Remove unnecessary conf cache from DataNode. Contributed by Xiaobing Zhou. 2016-09-15 15:25:47 -07:00
Kai Zheng
ea0c2b8b05 HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng 2016-09-15 10:46:00 +08:00
Lei Xu
86c9862bec HDFS-10636. Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File. (Virajith Jalaparti via lei) 2016-09-13 12:54:14 -07:00
Andrew Wang
db6d243cf8 HDFS-10837. Standardize serializiation of WebHDFS DirectoryListing. 2016-09-13 11:02:36 -07:00
Anu Engineer
e3f7f58a5f HDFS-10599. DiskBalancer: Execute CLI via Shell. Contributed by Manoj Govindassamy. 2016-09-13 09:38:12 -07:00
Akira Ajisaka
f0876b8b60 HDFS-10856. Update the comment of BPServiceActor$Scheduler#scheduleNextBlockReport. Contributed by Yiqun Lin. 2016-09-13 14:50:03 +09:00
Jing Zhao
72dfb048a9 HDFS-10858. FBR processing may generate incorrect reportedBlock-blockGroup mapping. Contributed by Jing Zhao. 2016-09-12 16:40:11 -07:00
Anu Engineer
8a93f45a80 HDFS-10821. DiskBalancer: Report command support with multiple nodes. Contributed by Yiqun Lin. 2016-09-12 15:45:26 -07:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS 
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
 2016-09-12 11:10:00 -07:00
Xiao Chen
cc01ed7026 HDFS-10855. Fix typos in HDFS documents. Contributed by Yiqun Lin. 2016-09-10 23:24:49 -07:00
Arpit Agarwal
a99bf26a08 HDFS-10830. FsDatasetImpl#removeVolumes crashes with IllegalMonitorStateException when vol being removed is in use. (Arpit Agarwal and Manoj Govindassamy) 2016-09-10 18:22:25 -07:00
Anu Engineer
bee9f57f5c HDFS-10808. DiskBalancer does not execute multi-steps plan-redux. Contributed by Anu Engineer. 2016-09-09 15:00:39 -07:00
Anu Engineer
baab48922a HDFS-9849. DiskBalancer: reduce lock path in shutdown code. Contributed by Yuanbo Liu. 2016-09-08 20:00:42 -07:00
Anu Engineer
35c5943b8b HDFS-10553. DiskBalancer: Rename Tools/DiskBalancer class to Tools/DiskBalancerCLI. Contributed by Manoj Govindassamy. 2016-09-08 19:26:56 -07:00
Chris Douglas
011f3b24d4 HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang 2016-09-08 17:53:40 -07:00
Chris Douglas
b6d839a60c HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin 2016-09-08 13:29:30 -07:00
Akira Ajisaka
63f594892e HDFS-10778. Add -format option to make the output of FileDistribution processor human-readable in OfflineImageViewer. 2016-09-08 15:13:43 +09:00
Kai Zheng
d355573f56 Revert "HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng" 
This reverts commit 62a9667136
 2016-09-08 05:50:17 +08:00
Arpit Agarwal
5f23abfa30 HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:37:21 -07:00
Xiao Chen
f6c0b7543f HDFS-10841. Remove duplicate or unused variable in appendFile(). Contributed by Kihwal Lee. 2016-09-06 11:48:35 -07:00
Chris Nauroth
f0d5382ff3 HDFS-6962. ACL inheritance conflicts with umaskmode. Contributed by Chris Nauroth. 2016-09-06 11:02:39 -07:00
Chris Douglas
d37dc5d1b8 HDFS-9847. HDFS configuration should accept time units. Contributed by Yiqun Lin 2016-09-06 10:38:04 -07:00
Kai Zheng
62a9667136 HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng 2016-09-07 17:05:33 +08:00
Xiao Chen
07650bc37a HDFS-9781. FsDatasetImpl#getBlockReports can occasionally throw NullPointerException. Contributed by Manoj Govindassamy. 2016-09-02 15:33:11 -07:00
Zhe Zhang
6f4b0d33ca HDFS-10817. Add Logging for Long-held NN Read Locks. Contributed by Erik Krogen. 2016-08-31 15:40:09 -07:00
Andrew Wang
85bab5fb57 HDFS-10784. Implement WebHdfsFileSystem#listStatusIterator. 2016-08-31 14:29:37 -07:00
Kihwal Lee
01721dd88e HDFS-10729. Improve log message for edit loading failures caused by FS limit checks. Contributed by Wei-Chiu Chuang. 2016-08-31 14:02:37 -05:00
Anu Engineer
20ae1fa259 HDFS-10813. DiskBalancer: Add the getNodeList method in Command. Contributed by Yiqun Lin. 2016-08-30 18:42:55 -07:00
Ming Ma
9dcbdbdb5a HDFS-9392. Admins support for maintenance state. Contributed by Ming Ma. 2016-08-30 14:00:13 -07:00
Wei-Chiu Chuang
c4ee6915a1 HDFS-10760. DataXceiver#run() should not log InvalidToken exception as an error. Contributed by Pan Yuxuan. 2016-08-30 10:43:20 -07:00
Xiao Chen
cd5e10ccca HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge. 2016-08-29 17:48:08 -07:00