Commit Graph

4277 Commits

Author SHA1 Message Date
Masatake Iwasaki
06ffeb8d4d HADOOP-14405. Fix performance regression due to incorrect use of DataChecksum. Contributed by LiXin Ge. 2017-05-10 14:16:41 +09:00
Jason Lowe
f7faac8e90 HADOOP-14377. Increase Common test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 2017-05-09 16:08:34 -05:00
Chris Douglas
45382bf047 AuditLogger and TestAuditLogger are dead code. Contributed by Vrushali C 2017-05-09 10:31:52 -07:00
Lei Xu
149b903be3 HADOOP-14384. Reduce the visibility of FileSystem.newFSDataOutputStreamBuilder before the API becomes stable. (lei) 2017-05-09 10:27:17 -07:00
Andrew Wang
543aac9f28 HADOOP-14386. Rewind trunk from Guava 21.0 back to Guava 11.0.2. 2017-05-09 09:22:53 -07:00
Andrew Wang
54fd0e44b7 HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-08 21:59:49 -07:00
Haohui Mai
ff5ec3b841 HADOOP-14383. Implement FileSystem that reads from HTTP / HTTPS endpoints. 2017-05-08 15:28:45 -07:00
Chris Douglas
e4f34ecb04 HADOOP-14390. Correct spelling of 'succeed' and variants. Contributed by Dongtao Zhang 2017-05-05 12:10:50 -07:00
Steve Loughran
4e6bbd049d
HADOOP-14382 Remove usages of MoreObjects.toStringHelper.
Contributed by Andrew Wang
2017-05-05 13:03:09 +01:00
Andrew Wang
c2a52ef9c2 HDFS-11643. Add shouldReplicate option to create builder. Contributed by SammiChen. 2017-05-04 11:39:14 -07:00
Arpit Agarwal
81092b1f11 HDFS-11722. Change Datanode file IO profiling sampling to percentage. Contributed by Hanisha Koneru. 2017-05-03 16:29:30 -07:00
Arpit Agarwal
30cd265134 Revert "HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru."
This reverts commit 08fb82d6d1.
2017-05-03 16:28:47 -07:00
Arpit Agarwal
08fb82d6d1 HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-03 15:46:08 -07:00
Mingliang Liu
83dded556d HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran 2017-05-03 11:22:44 -07:00
Kihwal Lee
d4631e466b HADOOP-14372. TestSymlinkLocalFS timeouts are too low. Contributed by Eric Badger. 2017-05-03 09:47:17 -05:00
Robert Kanter
8b82317fab HADOOP-14352. Make some HttpServer2 SSL properties optional (jzhuge via rkanter) 2017-05-02 17:51:28 -07:00
Jason Lowe
cedaf4cab9 HADOOP-14306. TestLocalFileSystem tests have very low timeouts. Contributed by Eric Badger 2017-05-02 16:50:51 -05:00
Andrew Wang
dcc292d7de HADOOP-14369. NetworkTopology calls expensive toString() when logging. Contributed by Inigo Goiri. 2017-05-02 10:51:20 -07:00
Akira Ajisaka
20cde55274
HADOOP-14371. License error in TestLoadBalancingKMSClientProvider.java. Contributed by hu xiaodong. 2017-05-02 23:36:54 +09:00
Mingliang Liu
c735362df7 HADOOP-14363. Inconsistent default block location in FileSystem javadoc. Contributed by Chen Liang 2017-04-29 23:49:36 -07:00
Eric Payne
6fe11f6155 HADOOP-14320. TestIPC.testIpcWithReaderQueuing fails intermittently. Contributed by Eric Badger. 2017-04-28 14:45:26 -05:00
Junping Du
373bb4931f YARN-679. Add an entry point that can start any Yarn service. Contributed by Steve Loughran. 2017-04-28 10:45:02 -07:00
Kai Zheng
cb672a45a0 HDFS-11605. Allow user to customize new erasure code policies. Contributed by Huafeng Wang 2017-04-28 13:18:50 +08:00
Chris Douglas
689fb2fc66 HADOOP-14354. SysInfoWindows is not thread safe. Contributed by Inigo Goiri 2017-04-27 16:26:39 -07:00
Chris Douglas
31cf480d65 HADOOP-14309. Add PowerShell NodeFencer. Contributed by Inigo Goiri 2017-04-27 16:09:28 -07:00
Wei-Chiu Chuang
bbf8cac14d HADOOP-13200. Implement customizable and configurable erasure coders. Contributed by Tim Yao. 2017-04-27 12:39:47 -07:00
Wei-Chiu Chuang
ddaeb3e497 Revert "HADOOP-13200. Implement customizable and configurable erasure coders. Contributed by Tim Yao."
This reverts commit 872088c6e7.
2017-04-27 12:25:47 -07:00
Wei-Chiu Chuang
872088c6e7 HADOOP-13200. Implement customizable and configurable erasure coders. Contributed by Tim Yao. 2017-04-27 11:59:33 -07:00
Steve Loughran
61cda39e4e
HADOOP-14346. CryptoOutputStream throws IOException if stream is closed.
Contributed by Pierre Lacave

(cherry picked from commit 40b9c30d64ee1b07ade423533e627a7d4b099261)
2017-04-27 12:19:04 +01:00
John Zhuge
edd693833b HADOOP-14340. Enable KMS and HttpFS to exclude SSL ciphers. Contributed by John Zhuge. 2017-04-26 11:44:59 -07:00
John Zhuge
9ccb849eb6 HADOOP-14341. Support multi-line value for ssl.server.exclude.cipher.list. Contributed by John Zhuge. 2017-04-25 22:06:51 -07:00
Xiaoyu Yao
4ea2778acb HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. Contributed by Surendra Singh Lihore. 2017-04-25 18:45:56 -07:00
Mingliang Liu
d11d280b8b HADOOP-12856. FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to throw stricter IOEs; RawLocalFS contract tests to verify. Contributed by Steve Loughran 2017-04-25 18:33:41 -07:00
Andrew Wang
a22fe02fba HADOOP-14261. Some refactoring work for erasure coding raw coder. Contributed by Lin Zeng. 2017-04-21 11:35:36 -07:00
Mingliang Liu
667966c13c HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:13:36 -07:00
Daniel Templeton
f356f0f4cf YARN-6453. fairscheduler-statedump.log gets generated regardless of service
(Contributed by Yufei Gu via Daniel Templeton)
2017-04-20 14:13:07 -07:00
John Zhuge
0344bea3fd HADOOP-14241. Add ADLS sensitive config keys to default list. Contributed by John Zhuge. 2017-04-19 11:50:54 -07:00
Daniel Templeton
d4483a2fe4 HADOOP-13997. Typo in metrics docs (Contributed by Ana Krasteva via Daniel Templeton) 2017-04-19 09:41:09 -07:00
Kengo Seki
654372db85
HADOOP-14315. Python example in the rack awareness document doesn't work due to bad indentation
This closes #214

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-04-18 17:14:22 +09:00
Akira Ajisaka
84a8848aae
HADOOP-14318. Remove non-existent setfattr command option from FileSystemShell.md. Contributed by Doris Gu. 2017-04-18 16:43:50 +09:00
Andrew Wang
c0ca785dbb HDFS-11652. Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals. 2017-04-17 16:30:25 -07:00
Mingliang Liu
b053fdc547 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 2017-04-12 14:30:34 -07:00
Chris Nauroth
b8305e6d06 HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu. 2017-04-12 10:02:13 -07:00
Chris Nauroth
23b1a7bdf1 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore. 2017-04-11 22:03:09 -07:00
Wei-Chiu Chuang
f050afb578 HADOOP-13665. Erasure Coding codec should support fallback coder. Contributed by Kai Sasaki. 2017-04-11 07:32:13 -07:00
Jason Lowe
0e065f2ede YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. Contributed by Benson Qiu 2017-04-11 08:44:18 -05:00
Chris Douglas
a5e57df3c5 HADOOP-14225. Remove xmlenc dependency 2017-04-10 11:48:40 -07:00
Andrew Wang
be144117a8 HADOOP-14293. Initialize FakeTimer with a less trivial value. 2017-04-10 11:37:01 -07:00
Steve Loughran
2ed65540a3
HADOOP-14066 VersionInfo should be marked as public API 2017-04-08 13:41:10 +01:00
John Zhuge
56e81f2a20 HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge.
Signed-off-by: John Zhuge <jzhuge@apache.org>
2017-04-07 12:04:44 -07:00
Allen Wittenauer
96cbb4fce2 HADOOP-14202. fix jsvc/secure user var inconsistencies
Signed-off-by: John Zhuge <jzhuge@apache.org>
2017-04-07 08:59:21 -07:00
Zhe Zhang
95b7f1d29a HADOOP-14276. Add a nanosecond API to Time/Timer/FakeTimer. Contributed by Erik Krogen. 2017-04-06 16:59:21 -07:00
Xiaoyu Yao
32bb36b750 HDFS-11302. Improve Logging for SSLHostnameVerifier. Contributed by Chen Liang. 2017-04-05 17:28:00 -07:00
Andrew Wang
18432130a7 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-04 13:38:25 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Wei-Chiu Chuang
bbd68478d5 HDFS-11598. Improve -setrep for Erasure Coded files. Contributed by Yiqun Lin. 2017-04-03 07:57:28 -07:00
Jonathan Eagles
1309c585fb HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance (jeagles) 2017-03-30 14:14:43 -05:00
Wei-Chiu Chuang
8c591b8d19 HDFS-10974. Document replication factor for EC files. Contributed by Yiqun Lin. 2017-03-30 11:16:05 -07:00
Chris Douglas
7297e82cb4 HADOOP-14250. Correct spelling of 'separate' and variants. Contributed by Doris Gu 2017-03-30 10:17:11 -07:00
Andrew Wang
4966a6e26e HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy. 2017-03-29 14:37:21 -07:00
Mingliang Liu
13c766b62c HADOOP-14247. FileContextMainOperationsBaseTest should clean up test root path. Contributed by Mingliang Liu 2017-03-29 11:13:39 -07:00
Jonathan Eagles
523f467d93 HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles) 2017-03-29 10:12:02 -05:00
Yiqun Lin
6b09336438 HDFS-11577. Combine the old and the new chooseRandom for better performance. Contributed by Chen Liang. 2017-03-28 23:02:07 +08:00
John Zhuge
253e3e78ab HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 2017-03-27 23:05:57 -07:00
Akira Ajisaka
448ec81fd7
HADOOP-14218. Replace assertThat with assertTrue in MetricsAsserts. 2017-03-28 14:32:40 +09:00
Jonathan Eagles
db2adf356a Configuration#get return value optimization (jeagles) 2017-03-27 12:48:44 -05:00
Jonathan Eagles
858d597be0 Delay construction of PreCondition.check failure message in Configuration#set (jeagles) 2017-03-27 11:01:48 -05:00
Junping Du
6df029db36 Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 release. 2017-03-25 14:28:48 -07:00
Junping Du
d8e6d6df4c Add 2.8.0 RELEASENOTEs.md and CHANGES.md. 2017-03-25 14:02:25 -07:00
Tsuyoshi Ozawa
84ddedc0b2 HADOOP-10101. Update guava dependency to the latest version. (ozawa) 2017-03-25 10:06:52 +09:00
Andrew Wang
332a997e10 HDFS-11170. Add builder-based create API to FileSystem. Contributed by SammiChen and Wei Zhou. 2017-03-24 12:56:46 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Daniel Templeton
d49d1862a6 YARN-6360. Prevent FS state dump logger from cramming other log files
(Contributed by Yufei Gu via Daniel Templeton)
2017-03-24 11:31:04 -07:00
Andrew Wang
0e556a5ba6 HADOOP-14211. FilterFs and ChRootedFs are too aggressive about enforcing 'authorityNeeded'. Contributed by Erik Krogen. 2017-03-24 11:12:02 -07:00
John Zhuge
ab759e91b7 HDFS-10649. Remove unused PermissionStatus#applyUMask. Contributed by Chen Liang. 2017-03-23 23:31:27 -07:00
John Zhuge
128015584d HADOOP-14195. CredentialProviderFactory$getProviders is not thread-safe. Contributed by Vihang Karajgaonkar. 2017-03-23 13:57:08 -07:00
Mingliang Liu
d35e79abc2 HADOOP-14214. DomainSocketWatcher::add()/delete() should not self interrupt while looping await(). Contributed by Mingliang Liu 2017-03-23 13:34:03 -07:00
Ravi Prakash
595f62e362 HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to initialization. Contributed by Jonathan Eagles 2017-03-23 09:28:10 -07:00
Zhe Zhang
59d69257a8 HADOOP-9631. ViewFs should use underlying FileSystem's server side defaults. Contributed by Lohit Vijayarenu and Erik Krogen. 2017-03-22 22:22:22 -07:00
Erik Krogen
e1a99802fc HADOOP-14169. Implement listStatusIterator, listLocatedStatus for ViewFs. Contributed by Erik Krogen. 2017-03-17 17:12:49 -07:00
Andrew Wang
92ea6d74ec HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki. 2017-03-17 11:26:02 -07:00
Mingliang Liu
c04fb35c45 HADOOP-14191. Duplicate hadoop-minikdc dependency in hadoop-common module. Contributed by Xiaobing Zhou 2017-03-16 16:08:32 -07:00
Mingliang Liu
fa67a96d7b HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 14:38:21 -07:00
Mingliang Liu
e6cda5819b Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu"
This reverts commit b8c69557b7.
2017-03-14 13:03:42 -07:00
Chen Liang
9832ae0ed8 HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-03-13 17:30:10 -07:00
Jing Zhao
55796a0946 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. 2017-03-13 14:24:51 -07:00
Mingliang Liu
b8c69557b7 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-13 14:15:02 -07:00
Andrey Dyatlov
04a5f5a6dc
HADOOP-14156. Fix grammar error in ConfTest.java.
This closes #187

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-03-13 16:15:53 +09:00
Ray Chiang
4db9cc70d0 YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang) 2017-03-10 16:13:31 -08:00
Jian He
4478273e5f HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-10 09:25:58 -08:00
Mingliang Liu
fd26783aaf HADOOP-13946. Document how HDFS updates timestamps in the FS spec; compare with object stores. Contributed by Steve Loughran 2017-03-10 00:21:20 -08:00
Jian He
2be8947d12 Revert "HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand"
This reverts commit 241c1cc05b.
2017-03-08 13:20:01 -08:00
Jian He
241c1cc05b HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-08 10:48:27 -08:00
John Zhuge
98142d2f72 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
This reverts commit 53a12fa721.
2017-03-08 09:54:22 -08:00
Arpit Agarwal
959940b0ab HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-07 10:12:35 -08:00
Mingliang Liu
686823529b HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-06 17:16:36 -08:00
Mingliang Liu
52d7d5aa1a Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
This reverts commit 6b7cd62b8c.
2017-03-06 17:10:11 -08:00
Ray Chiang
d9dc444dc7 YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class property. (Yufei Gu via rchiang) 2017-03-06 14:02:49 -08:00
Akira Ajisaka
209ecd1a5c
HADOOP-14108. CLI MiniCluster: add an option to specify NameNode HTTP port. Contributed by Takanobu Asanuma. 2017-03-07 01:34:51 +09:00
Mingliang Liu
6b7cd62b8c HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:12 -08:00