Commit Graph

1618 Commits

Author SHA1 Message Date
Ewan Higgs
2ec97abb2e HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran. 2018-08-08 13:50:23 +02:00
Sean Mackrory
7862f1523f HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota) 2018-08-07 20:13:09 -06:00
Steve Loughran
48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
Inigo Goiri
9fea5c9ee7 YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi. 2018-07-31 09:36:34 -07:00
Sammi Chen
0857f116b7 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 2018-07-30 10:53:44 +08:00
Sean Mackrory
59adeb8d7f HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota. 2018-07-27 18:23:29 -06:00
bibinchundatt
3d3158cea4 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 2018-07-25 18:59:05 +05:30
Steve Loughran
45d9568aaa
HADOOP-15547/ WASB: improve listStatus performance.
Contributed by Thomas Marquardt.

(cherry picked from commit 749fff577ed9afb4ef8a54b8948f74be083cc620)
2018-07-19 12:31:19 -07:00
Wangda Tan
752dcce5f4 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda)
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
2018-07-16 10:54:41 -07:00
Sean Mackrory
a08812a1b1 HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota. 2018-07-12 17:24:01 +02:00
Sean Mackrory
d503f65b66 HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
connection in case of timeout.
2018-07-10 17:52:57 +02:00
Steve Loughran
ca8b80bf59
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan.
Contributed by Steve Loughran.
2018-07-10 10:43:59 +01:00
Aaron Fabbri
93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota) 2018-07-03 13:50:11 -07:00
Inigo Goiri
1804a31515 HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-07-02 10:48:20 -07:00
Inigo Goiri
73746c5da7 HDFS-13707. [PROVIDED Storage] Fix failing integration tests in ITestProvidedImplementation. Contributed by Virajith Jalaparti. 2018-06-29 09:56:13 -07:00
Akira Ajisaka
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma. 2018-06-28 14:37:22 +09:00
Todd Lipcon
7a3c6e9c3c HADOOP-15550. Avoid static initialization of ObjectMappers 2018-06-25 15:36:45 -07:00
Sean Mackrory
c687a6617d HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota. 2018-06-25 14:59:41 -06:00
Anu Engineer
e16e5b307d HDDS-177. Create a releasable ozonefs artifact
Contributed by Marton, Elek.
2018-06-24 01:05:04 -07:00
Anu Engineer
ca14fec02f HDDS-184. Upgrade common-langs version to 3.7 in hadoop-tools/hadoop-ozone.
Contributed by Takanobu Asanuma.
2018-06-23 15:49:44 -07:00
Sean Mackrory
55fad6a3de HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota. 2018-06-22 11:36:56 -06:00
Sean Mackrory
b089a06793 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 2018-06-20 16:45:08 -06:00
Chris Douglas
f34744603e HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 2018-06-17 23:12:18 -07:00
Chris Douglas
980031bb04 HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs 2018-06-17 11:54:26 -07:00
Jitendra Pandey
f4c7c91123 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-13 00:36:02 -07:00
Inigo Goiri
c35481594f YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 2018-06-12 10:59:50 -07:00
Steve Loughran
d901be6795
HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks.
Contributed by Esfandiar Manii.
2018-06-06 18:28:14 +01:00
Anu Engineer
3b34148c4f HDDS-88. Create separate message structure to represent ports in DatanodeDetails.
Contributed by Nanda Kumar.
2018-05-30 08:52:07 -07:00
Aaron Fabbri
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota) 2018-05-29 19:20:22 -07:00
Aaron Fabbri
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota. 2018-05-29 13:38:15 -07:00
Inigo Goiri
8fdc993a99 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 2018-05-28 16:45:42 -07:00
Johan Gustavsson
d14e26b31f
HADOOP-15477. Make unjar in RunJar overrideable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-28 17:29:59 +09:00
Steve Loughran
ba842847c9 HADOOP-15478. WASB: hflush() and hsync() regression.
Contributed by Thomas Marquardt.
2018-05-21 11:12:34 +01:00
Steve Loughran
cc3600aabd HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
Contributed by Steve Loughran.

(cherry picked from commit 170f1040d46f9b1a084e6637def91e9864446acc)
2018-05-17 19:08:55 +01:00
Steve Loughran
b670837469 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name.
Contributed by Sean Mackrory.
2018-05-15 16:13:56 +01:00
Steve Loughran
ba051b0686
HADOOP-15354. hadoop-aliyun & hadoop-azure modules to mark hadoop-common as provided
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-10 18:38:08 +09:00
Anu Engineer
46e0f27862 HDDS-18. Ozone Shell should use RestClient and RpcClient. Contributed by Lokesh Jain. 2018-05-09 23:00:13 -07:00
Sean Mackrory
1cfe7506f7 HADOOP-15356. Make HTTP timeout configurable in ADLS connector. Contributed by Atul Sikaria and Sean Mackrory. 2018-05-09 15:15:51 -06:00
Mukul Kumar Singh
343b51dcf4 HDDS-28. Ozone:Duplicate declaration in hadoop-tools/hadoop-ozone/pom.xml. Contributed by Tsz Wo Nicholas Sze. 2018-05-09 20:20:21 +05:30
Mukul Kumar Singh
208b97e969 HDDS-20. Ozone: Add support for rename key within a bucket for rpc client. Contributed by Lokesh Jain. 2018-05-09 19:06:07 +05:30
Aaron Fabbri
8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota) 2018-05-08 18:58:42 -07:00
Aaron Fabbri
69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota) 2018-05-08 15:29:54 -07:00
Steve Loughran
5b11b9fd41 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication.
Contributed by Thomas Marquardt
2018-05-07 11:54:08 +01:00
Aaron Fabbri
96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri) 2018-05-04 11:34:45 -07:00
Sean Mackrory
85381c7b60 HADOOP-15434. Upgrade to ADLS SDK that exposes current timeout. 2018-05-02 21:30:31 -06:00
Aaron Fabbri
919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota. 2018-04-30 16:02:57 -07:00
Weiwei Yang
ba2db39c65 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:36:21 +08:00
Aaron Fabbri
7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota) 2018-04-26 20:41:29 -07:00
Owen O'Malley
2adda92de1
HDFS-7240 git commit fixup
Signed-off-by: Owen O'Malley <omalley@apache.org>
2018-04-26 13:52:17 -07:00
Mukul Kumar Singh
ae8ac7f082 HDFS-13424. Ozone: Refactor MiniOzoneClassicCluster. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
025058f251 HDFS-13446. Ozone: Fix OzoneFileSystem contract test failures. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer
8b832f3c35 HDFS-13405. Ozone: Rename HDSL to HDDS.
Contributed by Ajay Kumar, Elek Marton, Mukul Kumar Singh, Shashikant Banerjee and Anu Engineer.
2018-04-26 05:36:04 -07:00
Nanda kumar
7ae9543a57 HDFS-13391. Ozone: Make dependency of internal sub-module scope as provided in maven. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
04e1287771 HDFS-13298. Ozone: Make ozone/hdsl/cblock modules turned off by default. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
61651dcf5c HDFS-13108. Ozone: OzoneFileSystem: Simplified url schema for Ozone File System. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
cd807c9890 HDFS-13133. Ozone: OzoneFileSystem: Calling delete with non-existing path shouldn't be logged on ERROR level. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
9272e1021d HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer
4460ac9c57 HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Anu Engineer
740a06cdd7 HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang
e395244e95 HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
2ec24a7542 HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Chen Liang
895a4cc60a HDFS-12632. Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer
d90ab3365d HDFS-12608. Ozone: Remove Warnings when building. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
9445a9267f HDFS-12572. Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer
0b3270a8de HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
5e4a6b686c HDFS-12425. Ozone: OzoneFileSystem: read/write/create/open/getFileInfo APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
f2d9360b9d HDFS-12199. Ozone: OzoneFileSystem: OzoneFileystem initialization code. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Akira Ajisaka
3e2ca23a53 HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Aaron Fabbri
989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota) 2018-04-23 15:26:53 -07:00
Inigo Goiri
fd24fd0ff7 YARN-8137. Parallelize node addition in SLS. Contributed by Abhishek Modi. 2018-04-20 09:09:12 -07:00
Yiqun Lin
1312f9ae4c HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 2018-04-18 16:04:00 +08:00
Inigo Goiri
78860372bd YARN-8134. Support specifying node resources in SLS. Contributed by Abhishek Modi. 2018-04-17 13:11:34 -07:00
Aaron Fabbri
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:55:38 -07:00
Xiao Chen
7c9cdad6d0 HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-10 21:31:48 -07:00
Aaron Fabbri
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota. 2018-04-05 20:23:17 -07:00
Steve Loughran
5a174f8ac6 HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException.
Contributed by Gabor Bota.
2018-04-03 16:31:34 +01:00
Sammi Chen
6542d17ea4 HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 2018-03-30 20:23:05 +08:00
Sean Mackrory
9d7a9031a5 HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 2018-03-29 10:13:40 -06:00
Chris Douglas
081c350188 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 2018-03-28 11:58:59 -07:00
Haibo Chen
fcea5a4d79 YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen) 2018-03-26 14:30:11 -07:00
Sean Mackrory
82665a7887 HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x. 2018-03-26 14:05:15 -06:00
Sean Mackrory
7ce6b41509 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-03-20 21:12:20 -07:00
Sammi Chen
d67a5e2dec HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 2018-03-19 15:02:37 +08:00
Wangda Tan
f47659fb97 MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda)
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
2018-03-15 13:26:01 -07:00
Steve Loughran
1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
fang zhenyi
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-15 16:12:07 +09:00
Chris Douglas
45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Chris Douglas
45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran
dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional.
Contributed by Steve Loughran.
2018-03-12 14:01:42 +00:00
Steve Loughran
7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms.
Contributed by Steve Loughran.
2018-03-08 11:24:06 +00:00
Steve Loughran
3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down.
Contributed by Virajith Jalaparti.
2018-03-08 11:15:46 +00:00
Steve Loughran
e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled.
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
2018-03-07 18:51:10 +00:00
Steve Loughran
2e1e049bd0 HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile.
Contributed by Steve Loughran.
2018-03-05 18:12:38 +00:00
Steve Loughran
8110d6a0d5 HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions.
Contributed by Aaron Fabbri.
2018-03-05 14:06:20 +00:00
Gergely Novák
e99a1e7767
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-02 16:15:04 -08:00
Ray Chiang
923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
fang zhenyi
29233c3c4a
HADOOP-15274. Move hadoop-openstack to slf4j
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-01 15:08:30 -08:00
Steve Loughran
edc9f1451b HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake.
Contributed by Steve Loughran
2018-02-28 14:42:25 +00:00
Weiwei Yang
7af4f34de5 YARN-7929. Support to set container execution type in SLS. (Jiandan Yang via Weiwei Yang) 2018-02-28 17:58:09 +08:00
Arpit Agarwal
2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Xuan Gong
68ce193efc MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 2018-02-23 14:37:26 -08:00
Carlo Curino
84cea0011f YARN-7732. Support Generic AM Simulator from SynthGenerator. (Contributed by Young Chen via curino) 2018-02-20 17:00:34 -08:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
Steve Loughran
7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key.
Contributed by Steve Loughran.
2018-02-16 16:37:06 +00:00
Steve Loughran
481d79fedc HADOOP-13972. ADLS to support per-store configuration.
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
2018-02-15 16:27:31 +00:00
Steve Loughran
9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client.
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)
2018-02-15 15:57:10 +00:00
Steve Loughran
b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document.
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6594c4e519ed39ef36669de2c5cee15)
2018-02-15 14:57:56 +00:00
Steve Loughran
c9a373fb14 HADOOP-15090. Add ADL troubleshooting doc.
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
2018-02-15 14:26:00 +00:00
Steve Loughran
f20dc0d577 HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions.
Contributed by Andras Bokor.
2018-02-14 16:20:14 +00:00
Steve Loughran
8cf88fcd1f HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK.
Contributed by Vishwajeet Dusane.

(cherry picked from commit bd5ab5912564d2d687651b01f552b8e4ca8c145a)
2018-02-12 15:13:00 +00:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
Anu Engineer
4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal
d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Steve Loughran
7fd287b4af HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line.
Contributed by Vishwajeet Dusane.

(cherry picked from commit 95a96b13e2a54e01ea6c6933045d912998477da3)
2018-01-29 09:48:14 -08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Yufei Gu
16be42d309 YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen. 2018-01-25 13:06:01 -08:00
Steve Loughran
f63d13f10d HADOOP-15185. Update adls connector to use the current version of ADLS SDK.
Contributed by Atul Sikaria.
2018-01-23 09:09:11 -08:00
Steve Loughran
1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
Contributed by Aaron Fabbri
2018-01-18 13:13:58 +00:00
Steve Loughran
f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits."
This reverts commit 35ad9b1dd2.
2018-01-18 12:35:57 +00:00
Aaron Fabbri
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Sammi Chen
9195a6e302 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 2018-01-17 15:55:59 +08:00
Steve Loughran
a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing
after OutputCommitter patch.
Contributed by Steve Loughran
2018-01-15 11:33:32 +00:00
Mukul Kumar Singh
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-15 15:45:51 +09:00
Steve Loughran
1a09da7400 HADOOP-15163. Fix S3ACommitter documentation
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
2018-01-10 15:37:07 +00:00
Steve Loughran
b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics
Contributed by Sean Mackrory
2018-01-09 18:46:52 +00:00
Steve Loughran
52babbb4a0 HADOOP-15086. NativeAzureFileSystem file rename is not atomic.
Contributed by Thomas Marquardt
2017-12-22 11:39:55 +00:00
Steve Loughran
c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls.
Contributed by Steve Loughran
2017-12-21 14:58:58 +00:00
Steve Loughran
ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance.
Contributed by Steve Loughran.
2017-12-21 14:15:53 +00:00
Akira Ajisaka
5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java. 2017-12-21 10:04:34 +09:00
Steve Loughran
1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran
35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
Contributed by Aaron Fabbri
2017-12-18 21:19:06 +00:00
Chris Douglas
4b3a785914 HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti 2017-12-15 17:51:42 -08:00
Chris Douglas
e515103a83 Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter"
This reverts commit c1bf2654b0e9118985b8518b0254eac4dd302a2f.
2017-12-15 17:51:42 -08:00
Virajith Jalaparti
8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
962b5e722b HDFS-12903. [READ] Fix closing streams in ImageWriter 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
9c35be86e1 HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
a027055dd2 HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
352f994b64 HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
4d59dabb7f HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
3d3be87e30 HDFS-12778. [READ] Report multiple locations for PROVIDED blocks 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
3b1d30301b HDFS-12775. [READ] Fix reporting of Provided volumes 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
6cd80b2521 HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
90d1b47a2a HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
87dc026bee HDFS-12789. [READ] Image generation tool does not close an opened stream 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
c293cc8e9b HDFS-12671. [READ] Test NameNode restarts when PROVIDED is configured 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
71d0a82571 HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
98f5ed5aa3 HDFS-11902. [READ] Merge BlockFormatProvider and FileRegionProvider. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
d6a9a89973 HDFS-12605. [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
17052c4aff HDFS-12584. [READ] Fix errors in image generation tool from latest rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
aca023b72c HDFS-12289. [READ] HDFS-12091 breaks the tests for provided block reads 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
546b95f484 HDFS-11673. [READ] Handle failures of Datanode with PROVIDED storage 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
4851f06bc2 HDFS-11791. [READ] Test for increasing replication of provided files. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
aa5ec85f7f HDFS-11663. [READ] Fix NullPointerException in ProvidedBlocksBuilder 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
d65df0f273 HDFS-11190. [READ] Namenode support for data stored in external stores. 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
8da3a6e314 HDFS-10706. [READ] Add tool generating FSImage from external store 2017-12-15 17:51:37 -08:00
Kai Zheng
f86c81d923 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-14 11:14:52 +08:00
Kai Zheng
28792b6b7f Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu"
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
2017-12-14 11:05:11 +08:00
Surendra Singh Lilhore
00129c5314 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-12 00:28:02 +05:30
Kai Zheng
ce04340ec7 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 2017-12-08 23:03:54 +08:00
Kai Zheng
73b86979d6 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-06 12:01:36 +08:00
Aaron Fabbri
6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory. 2017-12-05 11:06:32 -08:00
Steve Loughran
3150c019ae HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes.
Contributed by Steve Loughran
2017-12-05 15:05:41 +00:00
Steve Loughran
0ea182d0fa HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
Contributed by Rajesh Balamohan
2017-11-28 11:52:59 +00:00
Rohith Sharma K S
3cd75845da MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran. 2017-11-27 22:01:00 +05:30
Steve Loughran
de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
Contributed by Steve Loughran and Ryan Blue.
2017-11-22 15:28:12 +00:00
Steve Loughran
c326fc89b0 HADOOP-15024 Support user agent configuration and include that & Hadoop version information to oss server.
Contributed by Sammi Chen.
2017-11-20 18:56:42 +00:00
Akira Ajisaka
6903cf096e HADOOP-13514. Upgrade maven surefire plugin to 2.20.1
Signed-off-by: Allen Wittenauer <aw@apache.org>
2017-11-19 12:39:37 -08:00
Kai Zheng
18621af7ae HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu 2017-11-14 17:58:37 +08:00
Daniel Templeton
ba8136615a YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton)
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
2017-11-09 12:09:48 -08:00
John Zhuge
bf6a660232 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-11-09 10:16:12 -08:00
Subru Krishnan
f2df6b8983 HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 2017-11-08 18:07:12 -08:00
Arun Suresh
53c0fb7efe HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 2017-11-02 17:15:33 -07:00
Akira Ajisaka
cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 2017-11-02 18:32:24 +09:00
Chris Douglas
139cc752d2 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 2017-10-27 21:59:27 -07:00
Yufei Gu
99880d0a16 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 2017-10-27 14:41:37 -07:00
Subru Krishnan
625039ef20 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 2017-10-25 15:51:27 -07:00
Wangda Tan
281d83604d YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 2017-10-20 15:54:15 -07:00
Steve Loughran
f36cbc8475 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
Contributed by Andras Bokor.
2017-10-20 22:27:04 +01:00
Haibo Chen
46eb1033a8 YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen) 2017-10-17 16:04:19 -07:00
Steve Loughran
9fcc3a1fc8 HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
Contributed by Santhosh G Nayak
2017-10-16 16:01:47 +01:00
Steve Loughran
3de574413c HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
Contributed by Varada Hemeswari.
2017-10-11 18:06:43 +01:00
Steve Loughran
572cdb5463 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
Contributed by Kannapiran Srinivasan.
2017-10-06 17:43:38 +01:00
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Steve Loughran
9288206cb3
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
Contributed by Sivaguru Sankaridurg
2017-10-05 15:05:23 +01:00
Yufei Gu
79e37dc6db YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu) 2017-10-03 15:10:50 -07:00
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
John Zhuge
8aca46e53f HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge. 2017-09-29 12:15:06 -07:00
Steve Loughran
a530e7ab3b
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
Contributed by Varada Hemeswari
2017-09-28 19:52:56 +01:00
Aaron Fabbri
47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 15:59:38 -07:00
Jason Lowe
b133dc5700 MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 2017-09-22 15:21:42 -05:00
Steve Loughran
2d2d97fa7d
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure
Contributed by Steve Loughran
2017-09-15 17:03:01 +01:00
Aaron Fabbri
49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 14:10:48 -07:00
ChenSammi
e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
John Zhuge
a4661850c1 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 2017-09-08 11:51:03 -07:00
Steve Loughran
5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A.
Contributed by Aaron Fabbri
2017-09-08 12:07:02 +01:00
Steve Loughran
83449abb40
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter.
Contributed by Yonger.
2017-09-07 19:55:56 +01:00
Steve Loughran
13eda50003
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
2017-09-07 18:35:03 +01:00
Xiaoyu Yao
63720ef574 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-05 23:34:55 -07:00
John Zhuge
50506e90a8 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-05 23:26:57 -07:00
Steve Loughran
792eff9ea7
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS.
Contributed by Sivaguru Sankaridurg
2017-09-05 14:16:57 +01:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Allen Wittenauer
a3fee475f7
HADOOP-14414. Calling maven-site-plugin directly for docs profile is unnecessary (Andras Bokor via aw) 2017-09-01 08:38:30 -07:00
Steve Loughran
621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A.
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
2017-09-01 14:13:41 +01:00
John Zhuge
7ecc6dbed6 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria. 2017-08-31 21:13:22 -07:00
Allen Wittenauer
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-30 21:47:24 -07:00
Haibo Chen
d04f85f387 MAPREDUCE-6892. Issues with the count of failed/killed tasks in the jhist file. (Peter Bacsko via Haibo Chen) 2017-08-30 10:07:48 -07:00
Yufei Gu
26fafc3597 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-29 16:33:51 -07:00
Steve Loughran
9374f38820
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory
Contributed by Esfandiar Manii and Thomas Marquardt.
2017-08-29 19:03:44 +01:00
Steve Loughran
021974f4cb
HADOOP-14802. Add support for using container saskeys for all accesses.
Contributed by Sivaguru Sankaridurg
2017-08-29 19:02:43 +01:00
Andrew Wang
f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Arpit Agarwal
8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Steve Loughran
27ab5f7385
HADOOP-14787. AliyunOSS: Implement the createNonRecursive operator.
Contributed by Genmao Yu
2017-08-22 11:55:48 +01:00
Kai Zheng
267e19a09f HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu 2017-08-21 13:36:28 +08:00
Arun Suresh
8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Steve Loughran
c6b4e656b7
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted.
Contributed by Thomas Marquardt
2017-08-18 14:13:40 +01:00
Andrew Wang
dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Chris Douglas
645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
John Zhuge
7769e96149 HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 2017-08-13 00:22:34 -07:00
Allen Wittenauer
4222c97108
HADOOP-10392. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) (ajisakaa via aw) 2017-08-11 09:25:56 -07:00
Steve Loughran
54356b1e83
HADOOP-14183. Remove service loader config file for wasb fs.
Contributed by Esfandiar Manii.
2017-08-10 16:46:33 +01:00
Steve Loughran
f4e1aa0508
HADOOP-14715. TestWasbRemoteCallHelper failing.
Contributed by Esfandiar Manii.
2017-08-08 23:37:47 +01:00
Chris Douglas
55a181f845 HADOOP-14730. Support protobuf FileStatus in AdlFileSystem. 2017-08-07 21:31:28 -07:00
Steve Loughran
d91b7a8451
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek.
Contributed by Thomas Marquardt
2017-08-06 20:19:23 +01:00
Steve Loughran
7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws.
Contributed by Steve Loughran

(cherry picked from commit ced547d5f0dbea571cbc472c5f55fe89d5900a6f)
2017-08-04 11:09:08 +01:00
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Steve Loughran
778d4edd9a
HADOOP-14660 wasb: improve throughput by 34% when account limit exceeded.
Contributed by Thomas Marquardt.
2017-08-01 21:36:28 +01:00
John Zhuge
f735ad1b67 HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge. 2017-07-28 09:46:54 -07:00
Jitendra Pandey
a92bf39e23 HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:26:48 -07:00
Jitendra Pandey
f2921e51f0 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-24 23:01:01 -07:00
Ravi Prakash
94ca52ae9e HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed by Ravi Prakash. 2017-07-24 16:01:45 -07:00
Wei-Chiu Chuang
44350fdf49 HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default). Contributed by Bharat Viswanadham. 2017-07-20 18:23:13 -07:00
Haibo Chen
0ba8cda135 YARN-6685. Add job count in to SLS JSON input format. (Yufei Gu via Haibo Chen) 2017-07-20 08:15:46 -07:00
Jitendra Pandey
2843c688bc HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 00:13:06 -07:00
Jitendra Pandey
b0e78ae085 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 02:27:55 -07:00
Steve Loughran
7d272ea124
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
Contributed by Varada Hemeswari

(cherry picked from commit 1e69e5260351effc8077d1bdc397cec57cf1ff1b)
2017-07-12 10:37:39 +01:00
Steve Loughran
d670c3a4da
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
Contributed by Thomas Marquardt
2017-07-11 21:34:27 +01:00
Haibo Chen
147df300bf YARN-5067 Support specifying resources for AM containers in SLS. (Yufei Gu via Haibo Chen) 2017-06-30 17:03:44 -07:00
Mingliang Liu
38996fdcf0 HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak 2017-06-30 16:53:48 -07:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Mingliang Liu
72993b33b7 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
2017-06-29 17:07:52 -07:00
Mingliang Liu
b08cc97396 HADOOP-14601. Azure: Reuse ObjectMapper. Contributed by Mingliang Liu 2017-06-29 09:22:00 -07:00
Allen Wittenauer
20ba86d66a MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl
Signed-off-by: Ravi Prakash <raviprak@apache.org>
2017-06-28 17:33:44 -07:00
Mingliang Liu
990aa34de2 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran 2017-06-28 14:18:59 -07:00
Steve Loughran
ee243e5289
HADOOP-14190. Add more on S3 regions to the s3a documentation.
Contributed by Steve Loughran
2017-06-28 10:22:13 +01:00
Mingliang Liu
7e031c2c18 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas 2017-06-27 17:32:07 -07:00
Mingliang Liu
686a634f01 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran 2017-06-27 16:48:47 -07:00
Mingliang Liu
bc4dfe9c9c HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu 2017-06-27 13:19:14 -07:00
Steve Loughran
48f4a229a9
HADOOP-14461 Azure: handle failure gracefully in case of missing account access key.
Contributed by Mingliang Liu.
2017-06-26 13:39:47 +01:00
Akira Ajisaka
a594f96fb1
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he. 2017-06-26 16:41:00 +09:00
Steve Loughran
c6bd73c6c5
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
Contributed by Thomas.
2017-06-23 21:38:41 +01:00
Jason Lowe
dd65eea74b HADOOP-8143. Change distcp to have -pb on by default. Contributed by Mithun Radhakrishnan 2017-06-20 09:53:47 -05:00
Akira Ajisaka
3369540653
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:18:26 +09:00
Mingliang Liu
f214a9961f HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran 2017-06-15 11:16:16 -07:00
Mingliang Liu
325163f23f HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu 2017-06-09 13:07:11 -07:00
Mingliang Liu
df35e4cc3b HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan 2017-06-07 13:54:52 -07:00
Carlo Curino
b65100c14b YARN-6547. Enhance SLS-based tests leveraging invariant checker. 2017-06-06 17:19:15 -07:00
Mingliang Liu
536f057158 HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu 2017-06-06 11:09:28 -07:00
Mingliang Liu
6b5285bbcb HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu 2017-06-06 11:06:49 -07:00
Mingliang Liu
5fd9742c83 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan 2017-06-05 15:56:43 -07:00
Mingliang Liu
6aeda55bb8 HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 2017-06-05 11:26:56 -07:00
Mingliang Liu
ece33208b8 HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu 2017-06-01 11:52:11 -07:00
Akira Ajisaka
6e6e96583f
HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang. 2017-06-01 13:08:01 +09:00
Akira Ajisaka
07e60f85d8
HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu. 2017-05-30 15:11:10 +09:00
John Zhuge
6c6a7a5962 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 2017-05-28 22:25:00 -07:00
Mingliang Liu
89bb8bfe58 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari 2017-05-26 17:54:00 -07:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Akira Ajisaka
6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu. 2017-05-25 15:36:44 +09:00
Sunil G
b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Yufei Gu
fcbdecc54d YARN-6111. Rumen input does't work in SLS. Contributed by Yufei Gu. 2017-05-21 21:04:06 -07:00
Mingliang Liu
6672810eea HADOOP-14432. S3A copyFromLocalFile to be robust, tested. Contributed by Steve Loughran 2017-05-19 11:51:43 -07:00
Yongjun Zhang
b4adc8392c HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-18 15:35:22 -07:00
Sunil G
40e6a85d25 YARN-6560. SLS doesn't honor node total resource specified in sls-runner.xml. Contributed by Wangda Tan. 2017-05-18 22:49:42 +05:30
Steve Loughran
ba70225cf6
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
Contributed by Steve Loughran.

(cherry picked from commit 2ac5aab8d725f761a9f9723471a4426f6b5d78c4)
2017-05-18 15:44:39 +01:00
Akira Ajisaka
d4aa9e3c69
HADOOP-14415. Use java.lang.AssertionError instead of junit.framework.AssertionFailedError. Contributed by Chen Liang. 2017-05-17 15:38:28 -04:00
Robert Kanter
101852ca11 YARN-6535. Program needs to exit when SLS finishes. (yufeigu via rkanter) 2017-05-16 17:52:17 -07:00
Mingliang Liu
b415c6fe74 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg 2017-05-16 11:22:32 -07:00
Akira Ajisaka
0d5c8ed8e0
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor. 2017-05-11 16:37:32 -05:00
Mingliang Liu
eed731496f HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak 2017-05-10 15:05:11 -07:00
Carlo Curino
5cb6e3e082 YARN-6473. Create ReservationInvariantChecker to validate ReservationSystem + Scheduler operations. 2017-05-10 14:49:25 -07:00
Akira Ajisaka
2ba9903932
HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 2017-05-10 05:57:12 -05:00
Akira Ajisaka
c60164fb60
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong. 2017-05-09 05:37:43 -05:00
Andrew Wang
54fd0e44b7 HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-08 21:59:49 -07:00
Andrew Wang
8065129d87 HADOOP-14298. TestHadoopArchiveLogsRunner fails. Contribute dby Akira Ajisaka. 2017-05-07 13:59:15 -07:00
Steve Loughran
4e6bbd049d
HADOOP-14382 Remove usages of MoreObjects.toStringHelper.
Contributed by Andrew Wang
2017-05-05 13:03:09 +01:00
Robert Kanter
3082552b3b YARN-6522. Make SLS JSON input file format simple and scalable (yufeigu via rkanter) 2017-05-04 17:21:46 -07:00
Mingliang Liu
83dded556d HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran 2017-05-03 11:22:44 -07:00
Andrew Wang
b062b323b7 HADOOP-14281. Fix TestKafkaMetrics#testPutMetrics. Contributed by Alison Yu. 2017-05-02 11:49:19 -07:00
Akira Ajisaka
62579b69a0
HADOOP-11614. Remove httpclient dependency from hadoop-openstack. Contributed by Akira Ajisaka, Brahma Reddy Battula, and Steve Loughran. 2017-04-27 16:57:08 +09:00
Mingliang Liu
8b5f2c372e HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak 2017-04-26 13:47:18 -07:00
Mingliang Liu
e4c32bccde HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu 2017-04-25 20:54:56 -07:00
Robert Kanter
475f933b41 YARN-6423. Queue metrics doesn't work for Fair Scheduler in SLS (yufeigu via rkanter) 2017-04-25 16:26:09 -07:00
Steve Loughran
5f934f8386
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
Contributed by Steve Moist.
2017-04-24 20:33:19 +01:00
Wangda Tan
de69d6e811 YARN-6363. Extending SLS: Synthetic Load Generator. (Carlo Curino via wangda) 2017-04-20 21:54:30 -07:00
Mingliang Liu
667966c13c HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:13:36 -07:00
Andrew Wang
7e075a50e3 YARN-6365. Get static SLS html resources from classpath. Contributed by Yufei Gu. 2017-04-19 11:08:59 -07:00
Steve Loughran
6b015d00c9
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
Contributed by Steve Loughran
2017-04-19 10:21:44 +01:00
Mingliang Liu
0cab57223e HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg 2017-04-12 16:07:10 -07:00
Mingliang Liu
b053fdc547 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 2017-04-12 14:30:34 -07:00
Chris Nauroth
b8305e6d06 HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu. 2017-04-12 10:02:13 -07:00
John Zhuge
56e81f2a20 HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge.
Signed-off-by: John Zhuge <jzhuge@apache.org>
2017-04-07 12:04:44 -07:00
Andrew Wang
7eea8d83fa HADOOP-14280. Fix compilation of TestKafkaMetrics. 2017-04-05 15:12:03 -07:00
Mingliang Liu
56ab02eed9 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak 2017-04-04 11:10:28 -07:00
Mingliang Liu
27a44b60c1 HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak 2017-04-04 11:03:59 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Mingliang Liu
5faa949b78 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 2017-04-03 11:07:14 -07:00
Mingliang Liu
26172a94d6 HADOOP-14267. Make DistCpOptions immutable. Contributed by Mingliang Liu 2017-03-31 20:04:26 -07:00
Yongjun Zhang
bf3fb585aa HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-03-30 17:38:56 -07:00
Yongjun Zhang
144f1cf765 Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen."
This reverts commit 064c8b25ec.
2017-03-30 17:38:18 -07:00
Yongjun Zhang
064c8b25ec HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen. 2017-03-30 17:01:15 -07:00
Wei-Chiu Chuang
8c591b8d19 HDFS-10974. Document replication factor for EC files. Contributed by Yiqun Lin. 2017-03-30 11:16:05 -07:00
Akira Ajisaka
0d053eeb30
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 2017-03-30 18:11:50 +09:00
Robert Kanter
6a5516c238 YARN-5654. Not be able to run SLS with FairScheduler (yufeigu via rkanter) 2017-03-29 16:18:13 -07:00
Andrew Wang
4966a6e26e HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy. 2017-03-29 14:37:21 -07:00
Andrew Wang
0e6f8e4bc6 HDFS-10971. Distcp should not copy replication factor if source file is erasure coded. Contributed by Manoj Govindassamy. 2017-03-28 22:14:03 -07:00
John Zhuge
253e3e78ab HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 2017-03-27 23:05:57 -07:00
John Zhuge
d1b7439b48 HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge. 2017-03-24 13:37:04 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Chris Douglas
c55195588f HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge 2017-03-23 12:17:27 -07:00
Chris Douglas
03955612b7 HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria 2017-03-23 12:13:23 -07:00
Mingliang Liu
2e30aa72e0 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 2017-03-23 11:33:29 -07:00
Steve Loughran
a5a4867f3b
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject().
Contributed by Yuanbo Liu

(cherry picked from commit 20878d052cebc715c2494a97362fdff08885a77f)
2017-03-23 12:54:54 +00:00
Robert Kanter
cc938e99ec YARN-6326. Shouldn't use AppAttemptIds to fetch applications while AM Simulator tracks app in SLS (yufeigu via rkanter) 2017-03-21 15:21:11 -07:00
Steve Loughran
2841666f1f
HADOOP-14204 S3A multipart commit failing,
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9ddacd5fdb740ac1bf9bfeb0ffbbd6c0e)
2017-03-21 13:15:15 +00:00
Mingliang Liu
8e15e24059 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
2017-03-20 16:31:24 -07:00
Arpit Agarwal
6c399a88e9 HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:23 -07:00
Mingliang Liu
86035c1644 HADOOP-14192. Aliyun OSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu 2017-03-17 11:01:00 -07:00
Mingliang Liu
fa67a96d7b HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 14:38:21 -07:00
Mingliang Liu
e6cda5819b Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu"
This reverts commit b8c69557b7.
2017-03-14 13:03:42 -07:00
Mingliang Liu
b8c69557b7 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-13 14:15:02 -07:00
John Zhuge
5a40bafdae HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge.
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
2017-03-13 08:11:25 -07:00
Steve Loughran
092ec39fb9
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials.
Contributed by Yuanbo Liu
2017-03-10 17:43:22 +00:00
Mingliang Liu
881ec4d97b HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu 2017-03-10 00:16:09 -08:00
John Zhuge
c5ee7fded4 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge.
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
2017-03-09 18:30:17 -08:00
John Zhuge
98142d2f72 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
This reverts commit 53a12fa721.
2017-03-08 09:54:22 -08:00
Mingliang Liu
28daaf0eb2 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak 2017-03-07 14:55:52 -08:00
Robert Kanter
1598fd3b79 YARN-6275. Fail to show real-time tracking charts in SLS (yufeigu via rkanter) 2017-03-07 13:47:52 -08:00
Akira Ajisaka
f597f4c43e
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 2017-03-07 15:14:55 +09:00
Mingliang Liu
686823529b HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-06 17:16:36 -08:00
Mingliang Liu
52d7d5aa1a Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
This reverts commit 6b7cd62b8c.
2017-03-06 17:10:11 -08:00
Mingliang Liu
c571cda5c7 HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA 2017-03-06 17:00:13 -08:00
Mingliang Liu
6b7cd62b8c HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:12 -08:00
Akira Ajisaka
490abfb10f HADOOP-14057. Fix package.html to compile with Java 9. 2017-03-04 00:25:22 +09:00
Steve Loughran
a97833e0ed
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38bf40373d2855163fa6444821b0f994c)
2017-03-02 21:44:30 +00:00
Allen Wittenauer
989bd56b9f HADOOP-12556. KafkaSink jar files are created but not copied to target dist (Babak Behzad via aw) 2017-02-28 16:15:40 -08:00
Wangda Tan
eac6b4c35c YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 2017-02-28 10:35:50 -08:00
Akira Ajisaka
258342e76c HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. 2017-02-28 13:47:44 +09:00
Yongjun Zhang
d235dcdf0b HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 2017-02-27 20:42:13 -08:00
Steve Loughran
dab00da19f
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
Contributed by Steve Loughran.

(cherry picked from commit 29fe5af017b945d8750c074ca39031b5b777eddd)
2017-02-25 15:35:19 +00:00
Sunil G
b32ffa2753 YARN-4779. Fix AM container allocation logic in SLS. Contributed by Wangda Tan. 2017-02-24 21:39:25 +05:30
Steve Loughran
e60c6543d5
HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran 2017-02-24 13:24:59 +00:00
Steve Loughran
9c22a91662
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
(cherry picked from commit ff87ca84418a710c6dc884fe8c70947fcc6489d5)
2017-02-24 10:41:36 +00:00
Mingliang Liu
82607fce39 HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari. 2017-02-23 13:48:44 -08:00
Mingliang Liu
a4d4a23785 HADOOP-13321. Deprecate FileSystem APIs that promote inefficient call patterns. Contributed by Chris Nauroth and Mingliang Liu 2017-02-23 12:55:40 -08:00
Mingliang Liu
0013090fb4 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 2017-02-22 13:34:20 -08:00
Steve Loughran
4f4250fbcc HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:43:48 +00:00
Mingliang Liu
924def7854 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane 2017-02-21 13:44:42 -08:00
Steve Loughran
2158496f6b HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
2017-02-21 18:28:49 +00:00
Steve Loughran
8035749c26 HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:46 +00:00
Steve Loughran
3a2e30fa9f HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
(cherry picked from commit b1c1f05b1dc997906390d653dfafb4f0d7e193c4)
2017-02-18 18:17:11 +00:00
Mingliang Liu
658702efff HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 2017-02-16 16:51:03 -08:00
Mingliang Liu
bdad8b7b97 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 2017-02-16 16:41:31 -08:00
Mingliang Liu
f432999025 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane 2017-02-16 15:14:25 -08:00
Kai Zheng
cd3e59a3dc HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu 2017-02-15 16:34:30 +08:00
Akira Ajisaka
b9f8491252 HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 2017-02-15 01:45:56 +09:00
Lei Xu
71c23c9fc9 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei) 2017-02-13 13:33:13 -08:00
Kai Zheng
01be4503c3 HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui 2017-02-13 13:18:20 +08:00
Lei Xu
839b690ed5 HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 13:59:03 -08:00
Arpit Agarwal
2b7a7bbe0f HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus. 2017-02-10 10:09:31 -08:00
Kai Zheng
5b151290ae HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation deleteDirs. Contributed by Genmao Yu 2017-02-10 14:53:18 +08:00
Kai Zheng
a8a594b4c8 HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui 2017-02-09 18:00:22 +08:00
Akira Ajisaka
98c35bb1a4 HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows. 2017-01-21 17:31:20 +09:00
Lei Xu
ccf2d66244 HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) 2017-01-21 10:52:40 +08:00
Lei Xu
e015b56319 HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 2017-01-20 14:48:04 +08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Steve Loughran
ed33ce11dd HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
2017-01-19 11:25:40 +00:00
Steve Loughran
e648b6e138 HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:25:15 +00:00
Wangda Tan
287d3d6804 YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 2017-01-09 08:40:39 -08:00
Chris Nauroth
4c8f9e1302 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 2017-01-05 15:04:47 -08:00
Wangda Tan
0a55bd841e YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 2017-01-05 10:31:05 -08:00
Wangda Tan
5ed63e3e9d YARN-4899. Queue metrics of SLS capacity scheduler only activated after app submit to the queue. (Jonathan Hung via wangda) 2017-01-04 19:17:11 -08:00
Akira Ajisaka
01d31fe938 HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 2017-01-04 14:10:36 +09:00
Mingliang Liu
c6a5b689db HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki 2016-12-29 16:24:06 -08:00
Mingliang Liu
e92a77099b HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 2016-12-22 20:15:18 -08:00
Akira Ajisaka
ae401539ea YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 2016-12-22 14:32:24 +09:00
Wangda Tan
3bcfe3a9de YARN-5706. Fail to launch SLSRunner due to NPE. (Kai Sasaki via wangda) 2016-12-20 11:03:29 -08:00
Mingliang Liu
f92913c35b HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade 2016-12-15 12:35:08 -08:00
Mingliang Liu
ef34bf2bb9 HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane 2016-12-13 10:34:32 -08:00
Mingliang Liu
c6a3923245 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 2016-12-12 14:55:34 -08:00
Mingliang Liu
15dd1f3381 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth 2016-12-05 12:04:07 -08:00
Mingliang Liu
4113ec5fa5 HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane 2016-12-02 15:54:57 -08:00
Mingliang Liu
51211a7d7a HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 2016-12-02 14:01:42 -08:00
Mingliang Liu
2ff84a0040 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 2016-12-02 13:36:04 -08:00
Mingliang Liu
3fd844b99f HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 2016-11-30 13:01:19 -08:00
Mingliang Liu
d60a60be8a HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 2016-11-28 16:30:29 -08:00
Akira Ajisaka
209e805430 HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki. 2016-11-28 14:34:57 +09:00
Akira Ajisaka
9e173d2bac HADOOP-13816. Ambiguous plugin version warning from maven build. Contributed by Kai Sasaki. 2016-11-28 13:10:31 +09:00
Steve Loughran
0de0c32ddd HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:39 +00:00
Mingliang Liu
af791b774b HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:00:35 -08:00
Mingliang Liu
beb70fed4f HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
This closes #131
2016-11-22 13:12:23 -08:00
Wei-Chiu Chuang
c0b1a44f6c HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean Mackrory. 2016-11-17 22:48:35 -06:00
Mingliang Liu
5af572b644 HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 2016-11-15 10:57:00 -08:00
Chris Douglas
5c61ad2488 HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane 2016-11-11 11:15:07 -08:00
Mingliang Liu
89354f0475 HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth 2016-11-10 08:58:37 -08:00
Jason Lowe
3f93ac0733 YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 2016-11-08 22:01:26 +00:00
Xiao Chen
acd509dc57 HADOOP-13795. Skip testGlobStatusThrowsExceptionForUnreadableDir in TestFSMainOperationsSwift. Contributed by John Zhuge. 2016-11-07 09:21:01 -08:00
Akira Ajisaka
b970446b2c HADOOP-13798. TestHadoopArchives times out. 2016-11-07 19:53:43 +09:00
Akira Ajisaka
de0132748d MAPREDUCE-6796. Remove unused properties from JTConfig.java. Contributed by Haibo Chen. 2016-11-05 03:49:19 +09:00
Arun Suresh
0aafc122d4 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 2016-11-04 07:31:54 -07:00
Xiao Chen
20c4d8efab HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. Contributed by John Zhuge. 2016-11-03 11:16:07 -07:00
Steve Loughran
a1761a841e HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:54:46 +00:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Masatake Iwasaki
0bdd263d82 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 2016-10-27 15:46:59 +09:00
Chris Nauroth
1f8490a5ba HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 2016-10-26 08:55:42 -07:00
Chris Nauroth
9cad3e2350 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 2016-10-26 08:27:26 -07:00
Akira Ajisaka
d3bb69a667 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 2016-10-26 15:32:07 +09:00
Yongjun Zhang
0f0c15f7a5 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 2016-10-25 12:25:40 -07:00
Yongjun Zhang
3a60573039 Revert "Fix HDFS-11040"
This reverts commit 54c1815790.
2016-10-25 12:25:02 -07:00
Yongjun Zhang
54c1815790 Fix HDFS-11040 2016-10-25 12:19:34 -07:00
Chris Nauroth
309a43925c HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 2016-10-25 09:03:03 -07:00
Chris Nauroth
3372e94030 HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 2016-10-24 21:54:06 -07:00
Chris Nauroth
d8fa1cfa67 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 2016-10-24 21:22:34 -07:00
Chris Douglas
a1a0281e12 HADOOP-13626. Remove distcp dependency on FileStatus serialization 2016-10-24 12:46:54 -07:00
Anu Engineer
3fbf4cd5da HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:33:58 -07:00
Steve Loughran
9ae270af02 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:59 +01:00
Yongjun Zhang
8650cc84f2 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-19 17:37:54 -07:00
Xiao Chen
efdf810cf9 HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge. 2016-10-18 18:18:43 -07:00
Steve Loughran
6c348c5691 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 21:16:02 +01:00
Yongjun Zhang
0bc6d37f3c Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang."
This reverts commit 412c4c9a34.
2016-10-17 22:47:37 -07:00
Andrew Wang
987ee51141 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 2016-10-17 13:25:58 -07:00
Yongjun Zhang
412c4c9a34 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-17 11:04:42 -07:00
Mingliang Liu
d9f73f1b7c HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu 2016-10-13 22:33:55 -07:00
Jing Zhao
0a85d07983 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 2016-10-13 13:24:54 -07:00
Kai Zheng
c31b5e61b1 HADOOP-13701. AbstractContractRootDirectoryTest can fail when handling delete "/". Contributed by Genmao Yu 2016-10-08 17:35:59 +06:00
Kai Zheng
a57bba470b Merge branch 'trunk' into HADOOP-12756 2016-10-09 10:29:40 +08:00
Chris Nauroth
69620f9559 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 2016-10-07 11:41:19 -07:00
Steve Loughran
ebd4f39a39 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:51:40 +01:00
Steve Loughran
2cc841f16e HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka. 2016-10-06 16:31:00 +01:00
Steve Loughran
d6be1e75d8 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:02:11 +01:00
Akira Ajisaka
6cfe649375 HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. Contributed by Grant Henke.
This closes #133
2016-10-05 20:44:39 +09:00
Chris Nauroth
88b9444a81 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 2016-10-04 10:36:58 -07:00
Steve Loughran
ee0c722dc8 HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:01:00 +01:00
Chris Nauroth
47f80922dc HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 2016-09-28 16:19:32 -07:00
Kai Zheng
d1443988f8 Revert "HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back"
This reverts commit 5707f88d85
2016-09-29 09:18:27 +08:00
Brahma Reddy Battula
e17a4970be HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 2016-09-27 10:45:12 +05:30
Kai Zheng
5707f88d85 HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back.
HADOOP-12756 branch: Incorporate Aliyun OSS file system implementation. Contributors:
Mingfei Shi (mingfei.shi@intel.com)
Genmao Yu (genmao.ygm@alibaba-inc.com)
2016-09-26 20:42:22 +08:00
Steve Loughran
e5ef51e717 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:01:30 +01:00
Kai Zheng
26d5df390c HADOOP-13634. Some configuration in doc has been outdated. Contributed by Genmao Yu 2016-09-23 08:44:28 +08:00
Kai Zheng
a49b3be38e Merge branch 'trunk' into HADOOP-12756 2016-09-21 14:03:21 +08:00
Kai Zheng
22af6f8db3 HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu 2016-09-21 14:02:44 +08:00
Kai Zheng
08b37603d9 HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu 2016-09-20 15:12:02 +08:00
Kai Zheng
9cd4760257 HADOOP-13609. Refine credential provider related codes for AliyunOss integration. Contributed by Genmao Yu 2016-09-20 11:39:02 +08:00
Kai Zheng
005f4528c7 Merge branch 'trunk' into HADOOP-12756 2016-09-20 08:09:29 +08:00
Chris Nauroth
98bdb51397 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 2016-09-19 15:16:47 -07:00
Kai Zheng
a1940464a4 HADOOP-13610. Clean up AliyunOss integration tests. Contributed by Genmao Yu 2016-09-18 19:10:48 +08:00
Kai Zheng
dbb28eb1f2 Merge branch 'trunk' into HADOOP-12756 2016-09-18 19:03:39 +08:00
Mingliang Liu
96142efa2d HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu. 2016-09-17 22:07:46 -07:00
Steve Loughran
53a12fa721 HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:45:04 +01:00
Kai Zheng
60f66a9306 Merge branch 'trunk' into HADOOP-12756 2016-09-14 16:12:40 +08:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
2016-09-12 11:10:00 -07:00
Ravi Prakash
9faccd1046 HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic 2016-09-12 08:26:08 -07:00
Steve Loughran
4b6d795f28 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:55:32 +01:00
Steve Loughran
7fdfcd8a6c HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:25:23 +01:00
Mingfei
e671a0f52b HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu. 2016-09-07 11:17:43 +08:00
Mingfei
d33e928fbe HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
cdb77110e7 HADOOP-13498. The number of multi-part upload part should not bigger than 10000. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
6bb741b9f8 HADOOP-13499. Support session credentials for authenticating with Aliyun. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
bd2d97adee HADOOP-13483. File create should throw error rather than overwrite directories. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Mingfei
4d84c814fc HADOOP-13491. Fix several warnings from findbugs. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Kai Zheng
a5d5342228 HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou 2016-09-07 11:15:47 +08:00
Chris Nauroth
d152557cf7 HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 2016-09-06 09:36:21 -07:00
Arun Suresh
d6d9cff21b YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 2016-08-30 15:52:29 -07:00
Xiaoyu Yao
8b57be108f HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth.
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
(cherry picked from commit a65e159c59de41d26e12212dd1b7c28c28265cc3)
2016-08-29 15:30:49 -07:00
Chris Douglas
c37346d0e3 HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton. 2016-08-23 14:12:52 -07:00
Chris Nauroth
6f9c346e57 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:18:49 -07:00
Chris Nauroth
763f0497bb HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 2016-08-19 10:48:10 -07:00
Chris Nauroth
ae4db25443 HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 2016-08-18 10:44:36 -07:00
Junping Du
0da69c324d YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi.
(cherry picked from commit d464483bf7f0b3e3be3ba32cd6c3eee546747ab5)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
2016-08-18 07:27:23 -07:00
Steve Loughran
040c185d62 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:55 +01:00
Chris Nauroth
822d661b8f HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 2016-08-17 14:54:54 -07:00
Chris Nauroth
3808876c73 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 2016-08-16 17:05:52 -07:00
Chris Nauroth
2ed58c40e5 HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka. 2016-08-08 12:28:40 -07:00
Chris Nauroth
4ef1bc0bdb HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth. 2016-08-02 22:32:21 -07:00
Chris Nauroth
b43de80031 HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg. 2016-07-27 15:50:38 -07:00
Akira Ajisaka
54fe17a607 HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák. 2016-07-27 19:41:09 +09:00
Steve Loughran
86ae218893 HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:50:23 +01:00
Steve Loughran
be9e46b42d HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:45:03 +01:00
Steve Loughran
7052ca8804 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 18:00:08 +01:00
Chris Nauroth
c60a68ce1d HADOOP-13392. [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Contributed by Vishwajeet Dusane and Chris Douglas. 2016-07-22 09:16:38 -07:00
Chris Douglas
dca6335dbc HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS documentation 2016-07-21 23:21:47 -07:00
mattf-horton
12aa184479 HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 14:43:57 -07:00
Steve Loughran
37362c2f92 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:51 +01:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Jitendra Pandey
c2f9cd584c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:28:53 -07:00
Jian He
819224dcf9 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:36:20 -07:00
Andrew Wang
f292624bd8 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 2016-07-11 18:06:28 -07:00
Jitendra Pandey
6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Jitendra Pandey
aa42c7a6dd HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:34:03 -07:00
Steve Loughran
96fa0f848b HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu. 2016-06-29 16:04:50 +01:00
Akira Ajisaka
8d202f1258 HADOOP-13235. Use Date and Time API in KafkaSink. 2016-06-29 07:40:12 -07:00
Yongjun Zhang
8113855b3a HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 2016-06-28 23:15:13 -07:00
Allen Wittenauer
422c73a865 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 2016-06-28 07:21:04 -07:00
Steve Loughran
4ee3543625 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:45:25 +01:00
Chris Nauroth
b2c596cdda HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 2016-06-21 11:28:52 -07:00
Jitendra Pandey
b7c4cf7129 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:25:30 -07:00
Chris Nauroth
51d16e7b38 HADOOP-13242. Authenticate to Azure Data Lake using client ID and keys. Contributed by Atul Sikaria. 2016-06-16 23:35:20 -07:00
Chris Nauroth
127d2c7281 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 2016-06-16 11:18:02 -07:00
Ravi Prakash
4aefe119a0 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 11:13:35 -07:00
Chris Nauroth
c9e71382a5 HADOOP-12875. [Azure Data Lake] Support for contract test and unit test cases. Contributed by Vishwajeet Dusane. 2016-06-16 09:46:05 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Chris Nauroth
9581fb715c HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane. 2016-06-09 14:33:31 -07:00
Andrew Wang
58c3486850 HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris Douglas. 2016-06-09 13:49:52 -07:00
Steve Loughran
31ffaf76f2 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 21:00:47 +01:00
Steve Loughran
656c460c0e HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 17:28:49 +01:00
Steve Loughran
a3f78d8fa8 HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:42:36 +02:00
Chris Nauroth
c58a59f708 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:55:33 -07:00
Chris Nauroth
16b1cc7af9 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 2016-06-01 14:49:22 -07:00
Steve Loughran
5870611030 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:18:20 +01:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Yongjun Zhang
cfb860dee7 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 2016-05-26 23:30:31 -07:00
Steve Loughran
39ec1515a2 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 16:39:31 +01:00
Steve Loughran
757050ff35 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:15 +01:00
Steve Loughran
c918286b17 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:59 +01:00
Chris Nauroth
93258459fa HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 2016-05-19 22:00:21 -07:00
Jing Zhao
03788d3015 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 2016-05-17 15:46:30 -07:00
Steve Loughran
08ea07f1b8 HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:19:29 +01:00
Steve Loughran
c69a649257 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:00:18 +01:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
2016-05-16 17:54:45 -07:00
Chris Nauroth
b9685e85d5 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 2016-05-16 11:53:17 -07:00
Steve Loughran
dd3a8bed0a HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:47:12 +01:00
Andrew Wang
fa440a39b7 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 2016-05-12 23:58:18 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT."
This reverts commit 6b53802cba.
2016-05-12 15:32:45 -07:00
Steve Loughran
27c4e90efc HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:24:20 +01:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Steve Loughran
3f2816ab5a HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:35:57 +01:00
Steve Loughran
def2a6d385 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:35 +01:00
Chris Nauroth
27242f211e HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 2016-05-10 15:21:06 -07:00
Steve Loughran
025219b12f HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:37:22 +01:00
Jian He
996a210ab0 YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan 2016-05-09 14:49:41 -07:00
Allen Wittenauer
1c5bbf6499 HADOOP-12866. add a subcommand for gridmix (Kai Sasaki via aw) 2016-05-06 14:07:02 -07:00
Jason Lowe
d0da13229c YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-05-05 14:07:54 +00:00
Jing Zhao
af942585a1 HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2016-05-04 10:23:04 -07:00
Masatake Iwasaki
ef0870ad03 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 2016-04-29 20:25:52 +09:00
Arun Suresh
341888a0aa YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh) 2016-04-26 20:12:12 -07:00
Yongjun Zhang
959a28dd12 HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 2016-04-26 16:08:03 -07:00
Akira Ajisaka
02c51c27d9 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 2016-04-25 22:33:09 +09:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Steve Loughran
19f0f9608e HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:25:03 +01:00
Yongjun Zhang
a749ba0cea HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 2016-04-21 12:17:17 -07:00
Jing Zhao
404f57f328 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 2016-04-14 10:35:22 -07:00
Steve Loughran
df18b6e984 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:44:55 +01:00
Jason Lowe
2a5da97f81 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
This reverts commit 1cbcd4a491.
2016-04-11 15:51:01 +00:00
Steve Loughran
b9e3eff62a HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:25:31 +01:00
Chris Nauroth
843ee8d59d HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 2016-04-08 13:36:58 -07:00
Steve Loughran
ce5b4812f0 HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:10:00 +01:00
Akira Ajisaka
9c32f8785e HDFS-10266. Remove unused properties dfs.client.file-block-storage-locations.num-threads and dfs.client.file-block-storage-locations.timeout.millis. Contributed by Ray Chiang. 2016-04-08 15:32:13 +09:00
Jason Lowe
1cbcd4a491 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-04-05 13:40:19 +00:00
Steve Loughran
89c9347511 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:02:04 +01:00
Steve Loughran
0ecdd4cffa HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:40:19 +01:00
Harsh J
256c82fe29 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 2016-04-01 14:18:10 +05:30
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Akira Ajisaka
0050fa5f1c MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-29 18:17:52 +09:00
Akira Ajisaka
115be193df Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote."
This reverts commit 2c268cc936.
2016-03-28 16:05:35 +09:00
Akira Ajisaka
18c7e58283 HDFS-9640. Remove hsftp from DistCp in trunk. Contributed by Wei-Chiu Chuang. 2016-03-28 15:32:38 +09:00
Akira Ajisaka
2c268cc936 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-25 15:17:27 +09:00
Allen Wittenauer
738155063e HADOOP-12857. rework hadoop-tools (aw) 2016-03-23 13:46:38 -07:00
Chris Nauroth
c50aad0f85 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 2016-03-04 22:57:43 -08:00
Masatake Iwasaki
cbd31328a6 HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 2016-03-04 14:11:36 +09:00
Masatake Iwasaki
33a412e8a4 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 2016-03-03 18:57:23 +09:00
Chris Nauroth
f98dff329b HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 2016-03-01 13:18:42 -08:00
Chris Nauroth
d251e55415 HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 2016-02-29 21:59:52 -08:00
Masatake Iwasaki
809ebc0b14 HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 2016-02-29 18:20:14 +09:00
Chris Nauroth
7634d404b7 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 2016-02-28 11:22:55 -08:00
cnauroth
3369a4f691 HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 2016-02-23 21:37:50 -08:00
cnauroth
27b77751c1 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 2016-02-22 17:30:13 -08:00
cnauroth
d4f5fc23b2 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 2016-02-22 13:39:38 -08:00
Yongjun Zhang
ba1c9d484a HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 2016-02-19 10:17:37 -08:00
cnauroth
8ab7658025 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 2016-02-18 14:06:38 -08:00
cnauroth
76fab26c5c HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 2016-02-17 12:19:59 -08:00
cnauroth
91a96eaa53 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 2016-02-12 15:50:10 -08:00
Steve Loughran
29ae258013 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 15:05:16 +00:00
Akira Ajisaka
700a176fa7 HADOOP-12757. Findbug compilation fails for 'Kafka Library support'. (aajisaka) 2016-02-02 23:15:39 +09:00
Yongjun Zhang
eddd823cd6 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-29 12:11:55 -08:00
cnauroth
41da9a0773 MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 2016-01-28 14:46:01 -08:00
Xuan
4efdf3a979 YARN-4612. Fix rumen and scheduler load simulator handle killed tasks
properly. Contributed by Ming Ma.
2016-01-26 18:17:12 -08:00
Akira Ajisaka
736eb17a79 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-01-25 13:47:29 +09:00
Akira Ajisaka
0bae506c22 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 2016-01-22 21:08:31 +09:00
Xuan
1708a4cd23 YARN-4611. Fix scheduler load simulator to support multi-layer network
location. Contributed by Ming Ma.
2016-01-20 15:41:50 -08:00
Ravi Prakash
1acc509b45 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 2016-01-19 13:57:08 -08:00
Wangda Tan
89d1fd5dac HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 2016-01-19 21:27:38 +08:00
Wangda Tan
a44ce3f14f YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 2016-01-19 09:30:04 +08:00
Wangda Tan
150f5ae034 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)"
This reverts commit 3fe5728563.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2016-01-19 09:27:36 +08:00
cnauroth
8bc93db2e7 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 2016-01-18 09:08:58 -08:00
Wangda Tan
3fe5728563 YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)
(cherry picked from commit 805a9ed85e)
2016-01-18 17:06:05 +08:00
Wangda Tan
adf260a728 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)"
This reverts commit 805a9ed85e.
2016-01-18 16:50:45 +08:00
Wangda Tan
805a9ed85e YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 2016-01-18 11:04:25 +08:00
Yongjun Zhang
a9c69ebeb7 HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-15 10:03:09 -08:00
Colin Patrick Mccabe
b1ed28fa77 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe) 2016-01-14 11:02:34 -08:00
Colin Patrick Mccabe
8315582c4f HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) 2016-01-13 16:28:06 -08:00
Lei Xu
126705f67e HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 2016-01-12 12:19:53 -08:00
Xiaoyu Yao
c2e2e13455 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 2016-01-11 17:08:26 -08:00
Zhe Zhang
95f32015ad HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng.
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
2016-01-11 09:46:56 -08:00
cnauroth
0e76f1fcea HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 2016-01-09 22:18:11 -08:00
cnauroth
f0fa6d869b HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 2016-01-08 20:18:59 -08:00
cnauroth
978bbdfeb2 HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade. 2016-01-06 11:15:59 -08:00
Ravi Prakash
2d16f40dab HADOOP-12689. S3 filesystem operations stopped working correctly 2016-01-05 23:26:03 -08:00
Wangda Tan
a9594c61bb YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda)
(cherry picked from commit 6eefae1b33928486e7a8572e14eaf00b83f2acfc)
2015-12-30 15:39:27 -08:00
Wangda Tan
8310b2e9ff YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) 2015-12-30 15:30:12 -08:00
Wangda Tan
79c41b1d83 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 2015-12-16 13:18:19 -08:00
Robert Kanter
6d84cc16b3 MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter) 2015-11-25 17:12:40 -08:00
Akira Ajisaka
bab03f3ee6 HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang. 2015-11-26 01:21:20 +09:00
Karthik Kambatla
52948bb20b YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 2015-11-24 13:47:17 +05:30
Haohui Mai
dc46c46b91 HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux. 2015-11-22 18:10:08 -08:00
Haohui Mai
0d27917042 HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena. 2015-11-22 17:25:22 -08:00
Haohui Mai
176ff5ce90 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:54:41 -08:00
Akira Ajisaka
f94d892704 HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 2015-11-13 21:52:57 +09:00
Akira Ajisaka
3ae82863f0 HADOOP-12553. [JDK8] Fix javadoc error caused by illegal tag. (aajisaka) 2015-11-08 19:17:57 +09:00
Lei Xu
bff7c90a56 HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei) 2015-11-05 18:35:15 -08:00
Allen Wittenauer
19a0c2660c MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh (Simone Leo via aw) 2015-11-05 16:57:22 -08:00
Allen Wittenauer
fc7cd46faf MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (Joe Mudd via aw) 2015-11-05 16:53:32 -08:00
cnauroth
0fb1867fd6 HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown. Contributed by Gaurav Kanade. 2015-11-04 10:28:44 -08:00
Varun Vasudev
73b9c7b82b HADOOP-10787. Rename/remove non-HADOOP_*, etc from the shell scripts. Contributed by Allen Wittenauer. 2015-11-04 15:56:17 +05:30
cnauroth
9e7dcab185 HADOOP-12508. delete fails with exception when lease is held on blob. Contributed by Gaurav Kanade. 2015-11-02 10:21:39 -08:00
cnauroth
3ce0a6502e HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. Contributed by Dushyanth. 2015-11-02 10:17:41 -08:00
Kihwal Lee
b24fe06483 Addendum to MAPREDUCE-6451 2015-10-30 16:05:23 -05:00
Kihwal Lee
2868ca0328 MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 2015-10-30 14:56:41 -05:00
Gábor Lipták
18727c63da HADOOP-12133 Add schemas to Maven Assembly XMLs
Signed-off-by: Allen Wittenauer <aw@apache.org>
2015-10-30 11:36:52 -07:00
cnauroth
2d10cb8e00 HADOOP-12519. hadoop-azure tests should avoid creating a metrics configuration fiale in the module root directory. Contributed by Chris Nauroth. 2015-10-28 09:44:10 -07:00
cnauroth
1f7ecb0c84 HADOOP-11685. StorageException complaining " no lease ID" during HBase distributed log splitting. Contributed by Duo Xu. 2015-10-27 22:56:22 -07:00
cnauroth
73822de7c3 HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Contributed by Chris Nauroth. 2015-10-27 22:48:56 -07:00
Varun Vasudev
c28e16b40c YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt. Contributed by Bibin A Chundatt. 2015-10-27 20:56:00 +05:30
Akira Ajisaka
bcb2386e39 HADOOP-12515. Mockito dependency is missing in hadoop-kafka module. Contributed by Kai Zheng. 2015-10-27 20:17:16 +09:00
cnauroth
cb282d5b89 HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race Condition Scenarios. Contributed by Gaurav Kanade. 2015-10-22 14:29:57 -07:00
cnauroth
47641fcbc9 HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. Contributed by Gaurav Kanade. 2015-10-22 12:21:32 -07:00
Robert Kanter
0c4af0f998 MAPREDUCE-6495. Docs for archive-logs tool (rkanter) 2015-10-20 17:34:34 -07:00
Haohui Mai
da8441d0fe HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:24 -07:00
Haohui Mai
7e2c971fed Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu."
This reverts commit fa5a54cac7.
2015-10-10 03:24:59 -07:00
Haohui Mai
fa5a54cac7 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:06 -07:00
Robert Kanter
1bca1bb517 MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME (rkanter) 2015-10-06 16:49:50 -07:00
cnauroth
5f6edb30c2 HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and writes. Contributed by Dushyanth. 2015-10-05 20:11:41 -07:00
Allen Wittenauer
b59e434116 HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via aw) 2015-10-05 14:02:07 -07:00
Akira Ajisaka
ecbfd68974 HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). Contributed by Rahul Palamuttam and Kanaka Kumar Avvaru. 2015-10-01 18:25:31 +02:00
Robert Kanter
5db371f52f MAPREDUCE-6494. Permission issue when running archive-logs tool as different users (rkanter) 2015-09-30 17:33:53 -07:00
Kihwal Lee
3abbdc929b HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 2015-09-28 17:31:51 -05:00
Harsh J
66dad854c0 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 2015-09-28 13:12:43 +05:30
cnauroth
7fe521b1dd HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu. 2015-09-25 22:33:53 -07:00
Robert Kanter
d3c49e7662 MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) 2015-09-25 15:02:42 -07:00
Jian He
89cab1ba5f YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 2015-09-23 13:29:38 -07:00
Harsh J
cc2b473990 MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused. Contributed by Neelesh Srinivas Salian. 2015-09-22 21:38:06 +05:30
Tsuyoshi Ozawa
dfd807afab HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 2015-09-22 12:54:29 +09:00
Steve Loughran
7269906254 HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:26 +01:00
Karthik Kambatla
119cc75e7e MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) 2015-09-09 17:45:19 -07:00
Tsuyoshi Ozawa
355eaaa33d HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter error message. Contributed by Wei Yan. 2015-09-03 22:48:53 +09:00
Lei Xu
6ab2d19f5c HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu) 2015-08-29 09:59:30 -07:00
Rohith Sharma K S
feaf034994 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 2015-08-24 11:25:07 +05:30
Yongjun Zhang
0bc15cb6e6 HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 2015-08-20 08:02:54 -07:00
cnauroth
3da0bedaec HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Contributed by Chris Nauroth. 2015-08-05 14:56:41 -07:00
Lei Xu
d5403747b5 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 2015-08-04 18:51:52 -07:00
cnauroth
efa97243ec HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 2015-07-22 11:16:49 -07:00
Vinayakumar B
94c6a4aa85 HADOOP-12017. Hadoop archives command should use configurable replication factor when closing (Contributed by Bibin A Chundatt) 2015-07-22 10:25:49 +05:30
Allen Wittenauer
68d1f4bfe8 HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) 2015-07-21 11:19:29 -07:00
Wangda Tan
c39ca541f4 YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda) 2015-07-21 09:57:23 -07:00
Vinayakumar B
87f29c6b8a HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. (Contributed by Vinayakumar B) 2015-07-21 13:12:46 +05:30
Steve Loughran
05130e94c5 HADOOP-12235 hadoop-openstack junit & mockito dependencies should be "provided". (Ted Yu via stevel) 2015-07-20 11:22:39 +01:00
Chris Douglas
a431ed9075 HADOOP-12211. Collect disk usage on the node. Contributed by Robert Grandl 2015-07-13 15:36:11 -07:00
Ming Ma
08244264c0 YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) 2015-07-10 08:30:10 -07:00
Chris Douglas
1a0752d85a HADOOP-12210. Collect network usage on the node. Contributed by Robert Grandl 2015-07-09 17:48:43 -07:00
Robert Kanter
aaafa0b2ee HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) 2015-06-30 16:49:27 -07:00
cnauroth
460e98f7b3 HADOOP-12089. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 2015-06-29 13:48:02 -07:00
Colin Patrick Mccabe
5a27c3fd76 HADOOP-12104. Migrate Hadoop Pipes native build to new CMake framework (Alan Burlison via Colin P. McCabe) 2015-06-29 12:31:26 -07:00
Amareshwari Sriramadasu
8ef07f767f HADOOP-11203. Allow ditscp to accept bandwitdh in fraction MegaBytes. Contributed by Raju Bairishetti 2015-06-26 09:52:06 +05:30
cnauroth
c45784bc90 HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. 2015-06-08 22:42:14 -07:00
Andrew Wang
2dbc40e608 HADOOP-12059. S3Credentials should support use of CredentialProvider. Contributed by Sean Busbey. 2015-06-05 13:11:01 -07:00
Robert Kanter
cc70df98e7 MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high" (wilfreds via rkanter) 2015-06-03 18:41:45 -07:00
cnauroth
94e7d49a6d HADOOP-11959. WASB should configure client side socket timeout in storage client blob request options. Contributed by Ivan Mitic. 2015-05-28 12:31:06 -07:00
Colin Patrick Mccabe
7dba7005b7 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 2015-05-26 12:15:46 -07:00
Vinod Kumar Vavilapalli
500a1d9c76 YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. 2015-05-26 11:38:35 -07:00
Colin Patrick Mccabe
f34638399a HADOOP-11927. Fix "undefined reference to dlopen" error when compiling libhadooppipes (Xianyin Xin via Colin P. McCabe) 2015-05-22 12:51:03 -07:00
Akira Ajisaka
4aa730ce85 HADOOP-11698. Remove DistCpV1 and Logalyzer. Contributed by Brahma Reddy Battula. 2015-05-20 19:25:45 +09:00
Jing Zhao
0790275f05 HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. 2015-05-18 13:24:35 -07:00
Akira Ajisaka
f5c48238d5 HADOOP-11939. Deprecate DistCpV1 and Logalyzer. Contributed by Brahma Reddy Battula. 2015-05-18 16:13:16 +09:00
cnauroth
cb8e69a80c HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. 2015-05-14 22:22:24 -07:00
Akira Ajisaka
92c38e41e1 HADOOP-9723. Improve error message when hadoop archive output path already exists. Contributed by Jean-Baptiste Onofré and Yongjun Zhang. 2015-05-13 17:28:57 +09:00
Allen Wittenauer
ed0f4db0dd HADOOP-11775. Fix Javadoc typos in hadoop-openstack module (Yanjun Wang via aw) 2015-05-08 17:01:08 -07:00
Akira Ajisaka
c5afe444df HDFS-8349. Remove .xml and documentation references to dfs.webhdfs.enabled. Contributed by Ray Chiang. 2015-05-08 15:57:13 +09:00
Vinayakumar B
f33efbc827 HADOOP-11922. Misspelling of threshold in log4j.properties for tests in hadoop-tools (Contributed by Gabor Liptak) 2015-05-08 10:22:11 +05:30
Jonathan Eagles
2d7363b273 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) 2015-05-01 13:11:50 -05:00
Akira Ajisaka
f384a063a6 HADOOP-11821. Fix findbugs warnings in hadoop-sls. Contributed by Brahma Reddy Battula. 2015-04-30 19:34:41 +09:00
Allen Wittenauer
618ba707f0 MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) 2015-04-26 20:31:40 -07:00
Ravi Prakash
a287d2fb77 HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak) 2015-04-24 13:39:07 -07:00
Steve Loughran
19262d99eb HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 2015-04-23 21:41:51 +01:00