big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,273 Commits 2 Branches 2 Tags 264 MiB
e3c7300225
Commit Graph

732 Commits

Author SHA1 Message Date
Steve Loughran
040c185d62 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:55 +01:00
Chris Nauroth
822d661b8f HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 2016-08-17 14:54:54 -07:00
Chris Nauroth
3808876c73 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 2016-08-16 17:05:52 -07:00
Chris Nauroth
2ed58c40e5 HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka. 2016-08-08 12:28:40 -07:00
Chris Nauroth
4ef1bc0bdb HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth. 2016-08-02 22:32:21 -07:00
Chris Nauroth
b43de80031 HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg. 2016-07-27 15:50:38 -07:00
Akira Ajisaka
54fe17a607 HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák. 2016-07-27 19:41:09 +09:00
Steve Loughran
86ae218893 HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:50:23 +01:00
Steve Loughran
be9e46b42d HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:45:03 +01:00
Steve Loughran
7052ca8804 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 18:00:08 +01:00
Chris Nauroth
c60a68ce1d HADOOP-13392. [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Contributed by Vishwajeet Dusane and Chris Douglas. 2016-07-22 09:16:38 -07:00
Chris Douglas
dca6335dbc HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS documentation 2016-07-21 23:21:47 -07:00
mattf-horton
12aa184479 HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 14:43:57 -07:00
Steve Loughran
37362c2f92 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:51 +01:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Jitendra Pandey
c2f9cd584c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:28:53 -07:00
Jian He
819224dcf9 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:36:20 -07:00
Andrew Wang
f292624bd8 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 2016-07-11 18:06:28 -07:00
Jitendra Pandey
6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Jitendra Pandey
aa42c7a6dd HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:34:03 -07:00
Steve Loughran
96fa0f848b HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu. 2016-06-29 16:04:50 +01:00
Akira Ajisaka
8d202f1258 HADOOP-13235. Use Date and Time API in KafkaSink. 2016-06-29 07:40:12 -07:00
Yongjun Zhang
8113855b3a HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 2016-06-28 23:15:13 -07:00
Allen Wittenauer
422c73a865 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 2016-06-28 07:21:04 -07:00
Steve Loughran
4ee3543625 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:45:25 +01:00
Chris Nauroth
b2c596cdda HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 2016-06-21 11:28:52 -07:00
Jitendra Pandey
b7c4cf7129 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:25:30 -07:00
Chris Nauroth
51d16e7b38 HADOOP-13242. Authenticate to Azure Data Lake using client ID and keys. Contributed by Atul Sikaria. 2016-06-16 23:35:20 -07:00
Chris Nauroth
127d2c7281 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 2016-06-16 11:18:02 -07:00
Ravi Prakash
4aefe119a0 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 11:13:35 -07:00
Chris Nauroth
c9e71382a5 HADOOP-12875. [Azure Data Lake] Support for contract test and unit test cases. Contributed by Vishwajeet Dusane. 2016-06-16 09:46:05 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Chris Nauroth
9581fb715c HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane. 2016-06-09 14:33:31 -07:00
Andrew Wang
58c3486850 HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris Douglas. 2016-06-09 13:49:52 -07:00
Steve Loughran
31ffaf76f2 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 21:00:47 +01:00
Steve Loughran
656c460c0e HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 17:28:49 +01:00
Steve Loughran
a3f78d8fa8 HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:42:36 +02:00
Chris Nauroth
c58a59f708 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:55:33 -07:00
Chris Nauroth
16b1cc7af9 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 2016-06-01 14:49:22 -07:00
Steve Loughran
5870611030 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:18:20 +01:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Yongjun Zhang
cfb860dee7 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 2016-05-26 23:30:31 -07:00
Steve Loughran
39ec1515a2 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 16:39:31 +01:00
Steve Loughran
757050ff35 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:15 +01:00
Steve Loughran
c918286b17 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:59 +01:00
Chris Nauroth
93258459fa HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 2016-05-19 22:00:21 -07:00
Jing Zhao
03788d3015 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 2016-05-17 15:46:30 -07:00
Steve Loughran
08ea07f1b8 HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:19:29 +01:00
Steve Loughran
c69a649257 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:00:18 +01:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Chris Nauroth
b9685e85d5 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 2016-05-16 11:53:17 -07:00
Steve Loughran
dd3a8bed0a HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:47:12 +01:00
Andrew Wang
fa440a39b7 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 2016-05-12 23:58:18 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Steve Loughran
27c4e90efc HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:24:20 +01:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Steve Loughran
3f2816ab5a HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:35:57 +01:00
Steve Loughran
def2a6d385 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:35 +01:00
Chris Nauroth
27242f211e HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 2016-05-10 15:21:06 -07:00
Steve Loughran
025219b12f HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:37:22 +01:00
Jian He
996a210ab0 YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan 2016-05-09 14:49:41 -07:00
Allen Wittenauer
1c5bbf6499 HADOOP-12866. add a subcommand for gridmix (Kai Sasaki via aw) 2016-05-06 14:07:02 -07:00
Jason Lowe
d0da13229c YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-05-05 14:07:54 +00:00
Jing Zhao
af942585a1 HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2016-05-04 10:23:04 -07:00
Masatake Iwasaki
ef0870ad03 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 2016-04-29 20:25:52 +09:00
Arun Suresh
341888a0aa YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh) 2016-04-26 20:12:12 -07:00
Yongjun Zhang
959a28dd12 HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 2016-04-26 16:08:03 -07:00
Akira Ajisaka
02c51c27d9 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 2016-04-25 22:33:09 +09:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Steve Loughran
19f0f9608e HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:25:03 +01:00
Yongjun Zhang
a749ba0cea HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 2016-04-21 12:17:17 -07:00
Jing Zhao
404f57f328 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 2016-04-14 10:35:22 -07:00
Steve Loughran
df18b6e984 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:44:55 +01:00
Jason Lowe
2a5da97f81 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 1cbcd4a491.
 2016-04-11 15:51:01 +00:00
Steve Loughran
b9e3eff62a HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:25:31 +01:00
Chris Nauroth
843ee8d59d HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 2016-04-08 13:36:58 -07:00
Steve Loughran
ce5b4812f0 HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:10:00 +01:00
Akira Ajisaka
9c32f8785e HDFS-10266. Remove unused properties dfs.client.file-block-storage-locations.num-threads and dfs.client.file-block-storage-locations.timeout.millis. Contributed by Ray Chiang. 2016-04-08 15:32:13 +09:00
Jason Lowe
1cbcd4a491 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-04-05 13:40:19 +00:00
Steve Loughran
89c9347511 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:02:04 +01:00
Steve Loughran
0ecdd4cffa HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:40:19 +01:00
Harsh J
256c82fe29 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 2016-04-01 14:18:10 +05:30
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Akira Ajisaka
0050fa5f1c MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-29 18:17:52 +09:00
Akira Ajisaka
115be193df Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 2c268cc936.
 2016-03-28 16:05:35 +09:00
Akira Ajisaka
18c7e58283 HDFS-9640. Remove hsftp from DistCp in trunk. Contributed by Wei-Chiu Chuang. 2016-03-28 15:32:38 +09:00
Akira Ajisaka
2c268cc936 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-25 15:17:27 +09:00
Allen Wittenauer
738155063e HADOOP-12857. rework hadoop-tools (aw) 2016-03-23 13:46:38 -07:00
Chris Nauroth
c50aad0f85 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 2016-03-04 22:57:43 -08:00
Masatake Iwasaki
cbd31328a6 HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 2016-03-04 14:11:36 +09:00
Masatake Iwasaki
33a412e8a4 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 2016-03-03 18:57:23 +09:00
Chris Nauroth
f98dff329b HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 2016-03-01 13:18:42 -08:00
Chris Nauroth
d251e55415 HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 2016-02-29 21:59:52 -08:00
Masatake Iwasaki
809ebc0b14 HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 2016-02-29 18:20:14 +09:00
Chris Nauroth
7634d404b7 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 2016-02-28 11:22:55 -08:00
cnauroth
3369a4f691 HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 2016-02-23 21:37:50 -08:00
cnauroth
27b77751c1 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 2016-02-22 17:30:13 -08:00
cnauroth
d4f5fc23b2 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 2016-02-22 13:39:38 -08:00
Yongjun Zhang
ba1c9d484a HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 2016-02-19 10:17:37 -08:00
cnauroth
8ab7658025 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 2016-02-18 14:06:38 -08:00
cnauroth
76fab26c5c HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 2016-02-17 12:19:59 -08:00
cnauroth
91a96eaa53 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 2016-02-12 15:50:10 -08:00
Steve Loughran
29ae258013 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 15:05:16 +00:00
Akira Ajisaka
700a176fa7 HADOOP-12757. Findbug compilation fails for 'Kafka Library support'. (aajisaka) 2016-02-02 23:15:39 +09:00
Yongjun Zhang
eddd823cd6 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-29 12:11:55 -08:00
cnauroth
41da9a0773 MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 2016-01-28 14:46:01 -08:00
Xuan
4efdf3a979 YARN-4612. Fix rumen and scheduler load simulator handle killed tasks 
properly. Contributed by Ming Ma.
 2016-01-26 18:17:12 -08:00
Akira Ajisaka
736eb17a79 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-01-25 13:47:29 +09:00
Akira Ajisaka
0bae506c22 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 2016-01-22 21:08:31 +09:00
Xuan
1708a4cd23 YARN-4611. Fix scheduler load simulator to support multi-layer network 
location. Contributed by Ming Ma.
 2016-01-20 15:41:50 -08:00
Ravi Prakash
1acc509b45 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 2016-01-19 13:57:08 -08:00
Wangda Tan
89d1fd5dac HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 2016-01-19 21:27:38 +08:00
Wangda Tan
a44ce3f14f YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 2016-01-19 09:30:04 +08:00
Wangda Tan
150f5ae034 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe5728563.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2016-01-19 09:27:36 +08:00
cnauroth
8bc93db2e7 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 2016-01-18 09:08:58 -08:00
Wangda Tan
3fe5728563 YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85e)
 2016-01-18 17:06:05 +08:00
Wangda Tan
adf260a728 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 805a9ed85e.
 2016-01-18 16:50:45 +08:00
Wangda Tan
805a9ed85e YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 2016-01-18 11:04:25 +08:00
Yongjun Zhang
a9c69ebeb7 HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-15 10:03:09 -08:00
Colin Patrick Mccabe
b1ed28fa77 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe) 2016-01-14 11:02:34 -08:00
Colin Patrick Mccabe
8315582c4f HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) 2016-01-13 16:28:06 -08:00
Lei Xu
126705f67e HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 2016-01-12 12:19:53 -08:00
Xiaoyu Yao
c2e2e13455 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 2016-01-11 17:08:26 -08:00
Zhe Zhang
95f32015ad HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
 2016-01-11 09:46:56 -08:00
cnauroth
0e76f1fcea HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 2016-01-09 22:18:11 -08:00
cnauroth
f0fa6d869b HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 2016-01-08 20:18:59 -08:00
cnauroth
978bbdfeb2 HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade. 2016-01-06 11:15:59 -08:00
Ravi Prakash
2d16f40dab HADOOP-12689. S3 filesystem operations stopped working correctly 2016-01-05 23:26:03 -08:00
Wangda Tan
a9594c61bb YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) 
(cherry picked from commit 6eefae1b33928486e7a8572e14eaf00b83f2acfc)
 2015-12-30 15:39:27 -08:00
Wangda Tan
8310b2e9ff YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) 2015-12-30 15:30:12 -08:00
Wangda Tan
79c41b1d83 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 2015-12-16 13:18:19 -08:00
Robert Kanter
6d84cc16b3 MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter) 2015-11-25 17:12:40 -08:00
Akira Ajisaka
bab03f3ee6 HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang. 2015-11-26 01:21:20 +09:00
Karthik Kambatla
52948bb20b YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 2015-11-24 13:47:17 +05:30
Haohui Mai
dc46c46b91 HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux. 2015-11-22 18:10:08 -08:00
Haohui Mai
0d27917042 HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena. 2015-11-22 17:25:22 -08:00
Haohui Mai
176ff5ce90 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:54:41 -08:00
Akira Ajisaka
f94d892704 HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 2015-11-13 21:52:57 +09:00
Akira Ajisaka
3ae82863f0 HADOOP-12553. [JDK8] Fix javadoc error caused by illegal tag. (aajisaka) 2015-11-08 19:17:57 +09:00
Lei Xu
bff7c90a56 HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei) 2015-11-05 18:35:15 -08:00
Allen Wittenauer
19a0c2660c MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh (Simone Leo via aw) 2015-11-05 16:57:22 -08:00
Allen Wittenauer
fc7cd46faf MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (Joe Mudd via aw) 2015-11-05 16:53:32 -08:00
cnauroth
0fb1867fd6 HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown. Contributed by Gaurav Kanade. 2015-11-04 10:28:44 -08:00
Varun Vasudev
73b9c7b82b HADOOP-10787. Rename/remove non-HADOOP_*, etc from the shell scripts. Contributed by Allen Wittenauer. 2015-11-04 15:56:17 +05:30
cnauroth
9e7dcab185 HADOOP-12508. delete fails with exception when lease is held on blob. Contributed by Gaurav Kanade. 2015-11-02 10:21:39 -08:00
cnauroth
3ce0a6502e HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. Contributed by Dushyanth. 2015-11-02 10:17:41 -08:00
Kihwal Lee
b24fe06483 Addendum to MAPREDUCE-6451 2015-10-30 16:05:23 -05:00
Kihwal Lee
2868ca0328 MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 2015-10-30 14:56:41 -05:00
Gábor Lipták
18727c63da HADOOP-12133 Add schemas to Maven Assembly XMLs 
Signed-off-by: Allen Wittenauer <aw@apache.org>
 2015-10-30 11:36:52 -07:00
cnauroth
2d10cb8e00 HADOOP-12519. hadoop-azure tests should avoid creating a metrics configuration fiale in the module root directory. Contributed by Chris Nauroth. 2015-10-28 09:44:10 -07:00
cnauroth
1f7ecb0c84 HADOOP-11685. StorageException complaining " no lease ID" during HBase distributed log splitting. Contributed by Duo Xu. 2015-10-27 22:56:22 -07:00
cnauroth
73822de7c3 HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Contributed by Chris Nauroth. 2015-10-27 22:48:56 -07:00
Varun Vasudev
c28e16b40c YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt. Contributed by Bibin A Chundatt. 2015-10-27 20:56:00 +05:30
Akira Ajisaka
bcb2386e39 HADOOP-12515. Mockito dependency is missing in hadoop-kafka module. Contributed by Kai Zheng. 2015-10-27 20:17:16 +09:00
cnauroth
cb282d5b89 HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race Condition Scenarios. Contributed by Gaurav Kanade. 2015-10-22 14:29:57 -07:00
cnauroth
47641fcbc9 HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. Contributed by Gaurav Kanade. 2015-10-22 12:21:32 -07:00
Robert Kanter
0c4af0f998 MAPREDUCE-6495. Docs for archive-logs tool (rkanter) 2015-10-20 17:34:34 -07:00
Haohui Mai
da8441d0fe HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:24 -07:00
Haohui Mai
7e2c971fed Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu." 
This reverts commit fa5a54cac7.
 2015-10-10 03:24:59 -07:00
Haohui Mai
fa5a54cac7 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:06 -07:00
Robert Kanter
1bca1bb517 MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME (rkanter) 2015-10-06 16:49:50 -07:00
cnauroth
5f6edb30c2 HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and writes. Contributed by Dushyanth. 2015-10-05 20:11:41 -07:00
Allen Wittenauer
b59e434116 HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via aw) 2015-10-05 14:02:07 -07:00
Akira Ajisaka
ecbfd68974 HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). Contributed by Rahul Palamuttam and Kanaka Kumar Avvaru. 2015-10-01 18:25:31 +02:00
Robert Kanter
5db371f52f MAPREDUCE-6494. Permission issue when running archive-logs tool as different users (rkanter) 2015-09-30 17:33:53 -07:00
Kihwal Lee
3abbdc929b HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 2015-09-28 17:31:51 -05:00
Harsh J
66dad854c0 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 2015-09-28 13:12:43 +05:30
cnauroth
7fe521b1dd HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu. 2015-09-25 22:33:53 -07:00
Robert Kanter
d3c49e7662 MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) 2015-09-25 15:02:42 -07:00
Jian He
89cab1ba5f YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 2015-09-23 13:29:38 -07:00
Harsh J
cc2b473990 MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused. Contributed by Neelesh Srinivas Salian. 2015-09-22 21:38:06 +05:30
Tsuyoshi Ozawa
dfd807afab HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 2015-09-22 12:54:29 +09:00
Steve Loughran
7269906254 HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:26 +01:00
Karthik Kambatla
119cc75e7e MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) 2015-09-09 17:45:19 -07:00
Tsuyoshi Ozawa
355eaaa33d HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter error message. Contributed by Wei Yan. 2015-09-03 22:48:53 +09:00
Lei Xu
6ab2d19f5c HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu) 2015-08-29 09:59:30 -07:00
Rohith Sharma K S
feaf034994 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 2015-08-24 11:25:07 +05:30
Yongjun Zhang
0bc15cb6e6 HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 2015-08-20 08:02:54 -07:00
cnauroth
3da0bedaec HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Contributed by Chris Nauroth. 2015-08-05 14:56:41 -07:00
Lei Xu
d5403747b5 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 2015-08-04 18:51:52 -07:00
cnauroth
efa97243ec HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 2015-07-22 11:16:49 -07:00
Vinayakumar B
94c6a4aa85 HADOOP-12017. Hadoop archives command should use configurable replication factor when closing (Contributed by Bibin A Chundatt) 2015-07-22 10:25:49 +05:30
Allen Wittenauer
68d1f4bfe8 HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) 2015-07-21 11:19:29 -07:00
Wangda Tan
c39ca541f4 YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda) 2015-07-21 09:57:23 -07:00
Vinayakumar B
87f29c6b8a HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. (Contributed by Vinayakumar B) 2015-07-21 13:12:46 +05:30
Steve Loughran
05130e94c5 HADOOP-12235 hadoop-openstack junit & mockito dependencies should be "provided". (Ted Yu via stevel) 2015-07-20 11:22:39 +01:00
Chris Douglas
a431ed9075 HADOOP-12211. Collect disk usage on the node. Contributed by Robert Grandl 2015-07-13 15:36:11 -07:00
Ming Ma
08244264c0 YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) 2015-07-10 08:30:10 -07:00
Chris Douglas
1a0752d85a HADOOP-12210. Collect network usage on the node. Contributed by Robert Grandl 2015-07-09 17:48:43 -07:00
Robert Kanter
aaafa0b2ee HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) 2015-06-30 16:49:27 -07:00
cnauroth
460e98f7b3 HADOOP-12089. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 2015-06-29 13:48:02 -07:00
Colin Patrick Mccabe
5a27c3fd76 HADOOP-12104. Migrate Hadoop Pipes native build to new CMake framework (Alan Burlison via Colin P. McCabe) 2015-06-29 12:31:26 -07:00
Amareshwari Sriramadasu
8ef07f767f HADOOP-11203. Allow ditscp to accept bandwitdh in fraction MegaBytes. Contributed by Raju Bairishetti 2015-06-26 09:52:06 +05:30
cnauroth
c45784bc90 HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. 2015-06-08 22:42:14 -07:00
Andrew Wang
2dbc40e608 HADOOP-12059. S3Credentials should support use of CredentialProvider. Contributed by Sean Busbey. 2015-06-05 13:11:01 -07:00
Robert Kanter
cc70df98e7 MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high" (wilfreds via rkanter) 2015-06-03 18:41:45 -07:00
cnauroth
94e7d49a6d HADOOP-11959. WASB should configure client side socket timeout in storage client blob request options. Contributed by Ivan Mitic. 2015-05-28 12:31:06 -07:00
Colin Patrick Mccabe
7dba7005b7 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 2015-05-26 12:15:46 -07:00
Vinod Kumar Vavilapalli
500a1d9c76 YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. 2015-05-26 11:38:35 -07:00
Colin Patrick Mccabe
f34638399a HADOOP-11927. Fix "undefined reference to dlopen" error when compiling libhadooppipes (Xianyin Xin via Colin P. McCabe) 2015-05-22 12:51:03 -07:00
Akira Ajisaka
4aa730ce85 HADOOP-11698. Remove DistCpV1 and Logalyzer. Contributed by Brahma Reddy Battula. 2015-05-20 19:25:45 +09:00
Jing Zhao
0790275f05 HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. 2015-05-18 13:24:35 -07:00
Akira Ajisaka
f5c48238d5 HADOOP-11939. Deprecate DistCpV1 and Logalyzer. Contributed by Brahma Reddy Battula. 2015-05-18 16:13:16 +09:00
cnauroth
cb8e69a80c HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. 2015-05-14 22:22:24 -07:00
Akira Ajisaka
92c38e41e1 HADOOP-9723. Improve error message when hadoop archive output path already exists. Contributed by Jean-Baptiste Onofré and Yongjun Zhang. 2015-05-13 17:28:57 +09:00
Allen Wittenauer
ed0f4db0dd HADOOP-11775. Fix Javadoc typos in hadoop-openstack module (Yanjun Wang via aw) 2015-05-08 17:01:08 -07:00
Akira Ajisaka
c5afe444df HDFS-8349. Remove .xml and documentation references to dfs.webhdfs.enabled. Contributed by Ray Chiang. 2015-05-08 15:57:13 +09:00
Vinayakumar B
f33efbc827 HADOOP-11922. Misspelling of threshold in log4j.properties for tests in hadoop-tools (Contributed by Gabor Liptak) 2015-05-08 10:22:11 +05:30
Jonathan Eagles
2d7363b273 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) 2015-05-01 13:11:50 -05:00
Akira Ajisaka
f384a063a6 HADOOP-11821. Fix findbugs warnings in hadoop-sls. Contributed by Brahma Reddy Battula. 2015-04-30 19:34:41 +09:00
Allen Wittenauer
618ba707f0 MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml (Ray Chiang via aw) 2015-04-26 20:31:40 -07:00
Ravi Prakash
a287d2fb77 HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak) 2015-04-24 13:39:07 -07:00
Steve Loughran
19262d99eb HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 2015-04-23 21:41:51 +01:00
Ravi Prakash
cfba355052 HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak) 2015-04-21 16:49:37 -07:00
Jing Zhao
4c097e473b HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. 2015-04-15 12:37:20 -07:00
Akira Ajisaka
83979e61a4 MAPREDUCE-6307. Remove property mapreduce.tasktracker.taskmemorymanager.monitoringinterval. Contributed by J.Andreina. 2015-04-10 20:21:00 +09:00
Colin Patrick Mccabe
932730df7d HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) 2015-04-03 14:08:25 -07:00
Haohui Mai
75cb1d42ab HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao. 2015-04-01 16:50:59 -07:00
Harsh J
27d49e6714 MAPREDUCE-6291. Correct mapred queue usage command. Contributed by Brahma Reddy Battula. 2015-03-28 11:58:17 +05:30
Tsuyoshi Ozawa
e074952bd6 HADOOP-11760. Fix typo of javadoc in DistCp. Contributed by Brahma Reddy Battula. 2015-03-27 23:15:51 +09:00
Yongjun Zhang
44809b8081 HADOOP-11724. DistCp throws NPE when the target directory is root. (Lei Eddy Xu via Yongjun Zhang) 2015-03-25 15:45:45 -07:00
Harsh J
b6dea9776b MAPREDUCE-579. Streaming slowmatch documentation. 2015-03-25 14:39:00 +05:30
Harsh J
e5370477c2 MAPREDUCE-4653. TestRandomAlgorithm has an unused import statement. Contributed by Amir Sanjar. 2015-03-17 14:11:54 +05:30
Tsuyoshi Ozawa
ef9946cd52 HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal tags in hadoop-tools. Contributed by Akira AJISAKA. 2015-03-17 16:09:21 +09:00
Tsuyoshi Ozawa
79426f3334 HADOOP-11558. Fix dead links to doc of hadoop-tools. Contributed by Jean-Pierre Matsumoto. 2015-03-15 14:29:49 +09:00
Tsuyoshi Ozawa
b308a8d181 Revert "HADOOP-11558. Fix dead links to doc of hadoop-tools. Contributed by Masatake Iwasaki." 
This reverts commit 7da136ecca.
 2015-03-15 14:27:38 +09:00
Tsuyoshi Ozawa
7da136ecca HADOOP-11558. Fix dead links to doc of hadoop-tools. Contributed by Masatake Iwasaki. 2015-03-15 14:17:35 +09:00
cnauroth
7a346bcb4f HADOOP-11693. Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving. Contributed by Duo Xu. 2015-03-11 14:36:51 -07:00
Steve Loughran
64443490d7 HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) 2015-03-08 11:22:16 -07:00
cnauroth
608ebd52ba HADOOP-11642. Upgrade azure sdk version from 0.6.0 to 2.0.0. Contributed by Shashank Khandelwal and Ivan Mitic. 2015-03-06 15:25:01 -08:00
Jing Zhao
ed70fa142c HDFS-7535. Utilize Snapshot diff report for distcp. Contributed by Jing Zhao. 2015-03-04 10:30:53 -08:00
Vinod Kumar Vavilapalli
5af693fde2 MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public API in DistCp. Contributed by Jing Zhao. 2015-03-03 16:28:41 -08:00
Steve Loughran
15b7076ad5 HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel) 2015-03-03 16:18:51 -08:00
Tsuyoshi Ozawa
9ae7f9eb7b MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA. 2015-03-03 18:06:26 +09:00
Tsuyoshi Ozawa
d1c6accb6f HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-03-03 14:17:52 +09:00
Allen Wittenauer
039366e3b4 MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) 2015-02-28 22:53:38 -08:00
Allen Wittenauer
dce8b9c4d0 HADOOP-11637. bash location hard-coded in shell scripts (aw) 2015-02-26 09:29:16 -08:00
cnauroth
5731c0e0d0 HADOOP-11629. WASB filesystem should not start BandwidthGaugeUpdater if fs.azure.skip.metrics set to true. Contributed by Shanyu Zhao. 2015-02-25 09:08:55 -08:00
Akira Ajisaka
1a625b8158 HADOOP-11480. Typo in hadoop-aws/index.md uses wrong scheme for test.fs.s3.name. Contributed by Ted Yu. 2015-02-24 17:11:46 -08:00
Tsuyoshi Ozawa
9cedad11d8 Revert "HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)" 
This reverts commit 946456c6d8.

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
 2015-02-25 00:32:04 +09:00
Steve Loughran
709ff99cff HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) 2015-02-21 12:03:03 +00:00
Steve Loughran
aa1c437b6a HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel) 2015-02-20 20:51:24 +00:00
Tsuyoshi Ozawa
946456c6d8 HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-02-19 13:06:53 +09:00
Allen Wittenauer
b6fc1f3e43 HADOOP-11593. Convert site documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw) 2015-02-17 21:30:24 -10:00
Karthik Kambatla
409113d8f9 MAPREDUCE-6234. TestHighRamJob fails due to the change in MAPREDUCE-5785. (Masatake Iwasaki via kasha) 2015-02-17 14:38:00 -08:00
Steve Loughran
00b80958d8 HADOOP-11521. Make connection timeout configurable in s3a. (Thomas Demoor via stevel) 2015-02-17 20:06:27 +00:00
Steve Loughran
78a7e8d3a6 HADOOP-11522. Update S3A Documentation. (Thomas Demoor via stevel) 2015-02-17 18:15:02 +00:00
Steve Loughran
826267f789 HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). 2015-02-17 16:36:44 +00:00
Allen Wittenauer
7776847a59 HADOOP-11437. Remove the version and author information from distcp's README file (Brahma Reddy Battula via aw) 2015-02-11 15:47:36 -08:00
Steve Loughran
17165d3df9 HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) 2015-02-06 21:01:41 +00:00
Steve Loughran
4e7ad4f0a8 HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers. (Ted Yu via stevel) 2015-02-05 12:20:23 +00:00
Allen Wittenauer
9112f093cd HADOOP-10976. moving the source code of hadoop-tools docs to the directory under hadoop-tools (Masatake Iwasaki via aw) 2015-02-04 17:57:34 -08:00
Allen Wittenauer
f990e9d229 HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith via aw) 2015-02-04 17:27:45 -08:00
Akira Ajisaka
34fe11c987 MAPREDUCE-6243. Fix findbugs warnings in hadoop-rumen. Contributed by Masatake Iwasaki. 2015-02-04 09:25:44 -08:00
Tsuyoshi Ozawa
ba58da29f4 HADOOP-11546. Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck. (ozawa) 2015-02-05 00:42:14 +09:00
Akira Ajisaka
bd69fb2d44 MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors. (aajisaka) 2015-02-03 14:30:09 -08:00
Karthik Kambatla
c559df2219 YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha) 2015-02-03 13:20:41 -08:00
Haohui Mai
c89977f89c HADOOP-11529. Fix findbugs warnings in hadoop-archives. Contributed by Masatake Iwasaki. 2015-02-03 10:53:17 -08:00
cnauroth
1c09ca2ba4 HADOOP-11442. hadoop-azure: Create test jar. Contributed by Shashank Khandelwal. 2015-02-02 11:50:43 -08:00
Harsh J
ffc75d6ebe HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. 2015-02-01 00:17:04 +05:30
Allen Wittenauer
12e883007c MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw) 2015-01-30 11:08:20 -08:00
cnauroth
f2c91098c4 HADOOP-11523. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 2015-01-29 17:02:38 -08:00
Allen Wittenauer
3f982c5c26 MAPREDUCE-6150. Update document of Rumen (Masatake Iwasaki via aw) 2015-01-29 14:17:44 -08:00
cnauroth
ad55083f75 HADOOP-11441. Hadoop-azure: Change few methods scope to public. Contributed by Shashank Khandelwal. 2015-01-29 13:07:51 -08:00
Allen Wittenauer
0a05ae1782 HDFS-7566. Remove obsolete entries from hdfs-default.xml (Ray Chiang via aw) 2015-01-27 14:31:27 -08:00
Tsuyoshi Ozawa
7b82c4ab4e HADOOP-11450. Cleanup DistCpV1 not to use deprecated methods and fix javadocs. Contributed by Varun Saxena. 2015-01-26 12:58:38 +09:00
Allen Wittenauer
42715547f0 HADOOP-11256. Some site docs have inconsistent appearance (Masatake Iwasaki via aw) 2015-01-21 08:44:22 -08:00
Steve Loughran
2908fe4ec5 HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) 2015-01-17 18:40:50 +00:00
Steve Loughran
c6c0f4eb25 HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel) 2015-01-17 18:26:26 +00:00
Steve Loughran
000ca83ea3 HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) 2015-01-16 10:15:51 +00:00
Arpit Agarwal
c4cba6165a HADOOP-11465. Fix findbugs warnings in hadoop-gridmix. (Contributed by Varun Saxena) 2015-01-12 21:22:58 -08:00
Steve Loughran
27d8395867 HADOOP-11446 S3AOutputStream should use shared thread pool to avoid OutOfMemoryError 2015-01-05 13:00:01 +00:00
Karthik Kambatla
e7257acd8a YARN-2998. Abstract out scheduler independent PlanFollower components. (Anubhav Dhoot via kasha) 2014-12-30 19:55:24 -08:00
Tsuyoshi Ozawa
a164ce2ac9 HADOOP-11283. SequenceFile.Writer can leak file descriptors in DistCpV1#setup(). (Contributed by Varun Saxena via ozawa) 2014-12-25 23:30:45 +09:00
Jian He
fdf042dfff YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 2014-12-22 16:51:15 -08:00
Haohui Mai
02b21b7131 HADOOP-11429. Findbugs warnings in hadoop extras. Contributed by Varun Saxena. 2014-12-21 14:57:34 -08:00
arp
c1f857b0b4 HADOOP-11395. Add site documentation for Azure Storage FileSystem integration. (Contributed by Chris Nauroth) 2014-12-19 18:54:22 -08:00
Haohui Mai
efe6357748 HADOOP-11431. Clean up redundant maven-site-plugin configuration. Contributed by Herve Boutemy. 2014-12-18 16:23:35 -08:00
Steve Loughran
07bb0b0bbb HADOOP-11412 POMs mention "The Apache Software License" rather than "Apache License". (Herve Boutemy via stevel) 2014-12-16 11:04:36 +00:00
cnauroth
9458cd5bce HADOOP-11394. hadoop-aws documentation missing. Contributed by Chris Nauroth. 2014-12-12 23:29:11 -08:00
Haohui Mai
2e98ad34ce HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack. Contributed by Li Lu. 2014-12-09 20:45:21 -08:00
Haohui Mai
7bceb13ba9 HADOOP-11367. Fix warnings from findbugs 3.0 in hadoop-streaming. Contributed by Li Lu. 2014-12-09 10:41:35 -08:00
Jing Zhao
57cb43be50 HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. Contributed by Ted Yu. 2014-12-08 11:08:39 -08:00
cnauroth
79301e80d7 HADOOP-11201. Hadoop Archives should support globs resolving to files. Contributed by Gera Shegalov. 2014-11-18 17:05:48 -08:00
Aaron T. Myers
6ba52d88ec HADOOP-10714. AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call. Contributed by Juan Yu. 2014-11-05 17:17:04 -08:00
cnauroth
734eeb4f35 HADOOP-11248. Add hadoop configuration to disable Azure Filesystem metrics collection. Contributed by Shanyu Zhao. 2014-11-03 14:29:18 -08:00
Zhijie Shen
260ab6d5f4 HADOOP-11241. Fixed intermittent TestNMSimulator failure due to timing issue. Contributed by Varun Vasudev. 2014-10-31 18:02:39 -07:00
Haohui Mai
f1a149e916 HADOOP-11253. Hadoop streaming test TestStreamXmlMultipleRecords fails on Windows. Contributed by Varun Vasudev. 2014-10-31 11:54:33 -07:00
Kihwal Lee
b056048114 MAPREDUCE-6022. map_input_file is missing from streaming job 
environment. Contributed by Jason Lowe.
 2014-10-29 12:29:07 -05:00
Jian He
b8f7966c7a MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. Contributed by Junping Du 2014-10-22 10:43:20 -07:00
Vinod Kumar Vavilapalli
f2ea555ac6 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.
 2014-10-15 18:33:06 -07:00
cnauroth
f4b7e99f4e HADOOP-11188. hadoop-azure: automatically expand page blobs when they become full. Contributed by Eric Hanson. 2014-10-10 15:05:52 -07:00
cnauroth
2217e2f8ff HADOOP-10809. hadoop-azure: page blob support. Contributed by Dexter Bradshaw, Mostafa Elhemali, Eric Hanson, and Mike Liddell. 2014-10-08 14:20:23 -07:00
Kihwal Lee
c8617ff136 Missing files from the previous commit of HADOOP-11007 2014-10-07 16:30:53 -05:00
Kihwal Lee
99d7a452be HADOOP-11007. Reinstate building of ant tasks support. Contributed by 
Jason Lowe.
 2014-10-07 16:20:16 -05:00
Allen Wittenauer
83264cf457 HADOOP-10954. Adding site documents of hadoop-tools (Masatake Iwasaki via aw) 2014-09-26 11:59:45 -07:00
Andrew Wang
4ea77efa3a HADOOP-11140. hadoop-aws only need test-scoped dependency on hadoop-common's tests jar. Contributed by Juan Yu. 2014-09-25 21:56:26 -07:00
Allen Wittenauer
f614cb71d4 MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw) 2014-09-25 11:27:33 -07:00
Allen Wittenauer
3cde37c991 HADOOP-11009. Add Timestamp Preservation to DistCp (Gary Steelman via aw) 2014-09-24 15:38:30 -07:00