hadoop

Author	SHA1	Message	Date
Elixir Kook	a3f945fb84	HADOOP-17035. fixed typos (timeout, interruped) (#2007 ) Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>	2020-05-12 10:50:04 -05:00
S O'Donnell	29dddb8a14	HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.	2020-05-12 15:07:51 +01:00
Akira Ajisaka	328eae9a14	HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003 )	2020-05-11 14:44:18 +09:00
Akira Ajisaka	c784ba370e	HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001 )	2020-05-08 17:18:23 +09:00
Mingliang Liu	e9e1ead089	HADOOP-17027. Add tests for reading fair call queue capacity weight configs. Contributed by Fengnan Li	2020-05-07 16:50:23 -07:00
Uma Maheswara Rao G	9c8236d04d	HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.	2020-05-04 17:55:40 -07:00
Mingliang Liu	263c76b678	HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2020-04-30 14:15:28 -07:00
Ayush Saxena	6bdab3723e	HADOOP-16957. NodeBase.normalize doesn't removing all trailing slashes. Contributed by Ayush Saxena.	2020-04-30 19:55:20 +05:30
S O'Donnell	9ca6298a9a	HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.	2020-04-29 15:56:06 +01:00
lfengnan	4202750040	HADOOP-17010. Add queue capacity support for FairCallQueue (#1977 )	2020-04-28 16:14:55 -07:00
Mike	18d7dfbf35	HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952 ) Contributed by Mikhail Pryakhin.	2020-04-27 14:43:51 +01:00
Dhiraj	453c9c9a67	HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964 ) Contributed by Dhiraj Hegde. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Inigo Goiri <inigoiri@apache.org>	2020-04-25 13:04:32 -07:00
Ayush Saxena	ef9a6e775c	HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun.	2020-04-25 13:07:32 +05:30
Mingliang Liu	af85971a58	HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi	2020-04-22 12:36:33 -07:00
Steve Loughran	42711081e3	HADOOP-16986. S3A to not need wildfly on the classpath. (#1948 ) HADOOP-16986. S3A to not need wildfly JAR on its classpath. Contributed by Steve Loughran This is a successor to HADOOP-16346, which enabled the S3A connector to load the native openssl SSL libraries for better HTTPS performance. That patch required wildfly.jar to be on the classpath. This update: * Makes wildfly.jar optional except in the special case that "fs.s3a.ssl.channel.mode" is set to "openssl" * Retains the declaration of wildfly.jar as a compile-time dependency in the hadoop-aws POM. This means that unless explicitly excluded, applications importing that published maven artifact will, transitively, add the specified wildfly JAR into their classpath for compilation/testing/ distribution. This is done for packaging and to offer that optional speedup. It is not mandatory: applications importing the hadoop-aws POM can exclude it if they choose.	2020-04-20 14:32:13 +01:00
Ayush Saxena	79e03fb622	HADOOP-16971. TestFileContextResolveAfs#testFileContextResolveAfs creates dangling link and fails for subsequent runs. Contributed by Ctest.	2020-04-20 01:59:18 +05:30
Masatake Iwasaki	ac40daece1	HADOOP-16972. Ignore AuthenticationFilterInitializer for KMSWebServer. (#1961 )	2020-04-18 06:38:25 +09:00
belugabehr	eca05917d6	HADOOP-16951: Tidy Up Text and ByteWritables Classes. 1. Remove superfluous code 2. Remove superfluous comments 3. Checkstyle fixes 4. Remove methods that simply call super.method() 5. Use Java 8 facilities to streamline code where applicable 6. Simplify and unify some of the constructs between the two classes 7. Expanding of the arrays be 1.5x instead of 2x per expansion.	2020-04-17 08:16:12 -07:00
Ayush Saxena	3edbe8708a	HADOOP-16958. NPE when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL. Contributed by Ctest.	2020-04-14 02:13:42 +05:30
Ayush Saxena	2ea5adf2de	HADOOP-16967. TestSequenceFile#testRecursiveSeqFileCreate fails in subsequent run. Contributed by Ctest.	2020-04-12 11:31:07 +05:30
Ayush Saxena	5746533cde	HADOOP-16962. Making `getBoolean` log warning message for unrecognized value. Contributed by Ctest.	2020-04-09 23:30:28 +05:30
Ayush Saxena	1189af4746	HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena.	2020-04-08 15:55:14 +05:30
Takanobu Asanuma	b89d875f7b	HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina.	2020-04-06 11:03:10 +09:00
Masatake Iwasaki	8f8be6b92a	HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh Radhakrishnan.	2020-04-04 07:20:51 +09:00
He Xiaoqiao	d3b5951572	HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2020-04-01 16:36:46 -07:00
Brahma Reddy Battula	8914cf9167	Preparing for 3.4.0 development	2020-03-29 23:24:25 +05:30
Chao Sun	e3fbdcbc14	HADOOP-16912. Emit per priority RPC queue time and processing time from DecayRpcScheduler. Contributed by Fengnan Li.	2020-03-25 10:21:20 -07:00
Toshihiro Suzuki	d353b30baf	HDFS-15215. The Timestamp for longest write/read lock held log is wrong	2020-03-24 14:50:15 -07:00
Wei-Chiu Chuang	f197f05cff	HADOOP-16661. Support TLS 1.3 (#1880 )	2020-03-16 10:56:30 -07:00
Steve Loughran	c734d69a55	HADOOP-16898. Batch listing of multiple directories via an (unstable) interface Contributed by Steve Loughran. This moves the new API of HDFS-13616 into a interface which is implemented by HDFS RPC filesystem client (not WebHDFS or any other connector) This new interface, BatchListingOperations, is in hadoop-common, so applications do not need to be compiled with HDFS on the classpath. They must cast the FS into the interface. instanceof can probe the client for having the new interface -the patch also adds a new path capability to probe for this. The FileSystem implementation is cut; tests updated as appropriate. All new interfaces/classes/constants are marked as @unstable. Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681	2020-03-09 14:51:16 +00:00
Steve Loughran	d4d4c37810	HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden Contributed by Steve Loughran. Not all stores do complete validation here; in particular the S3A Connector does not: checking up the entire directory tree to see if a path matches is a file significantly slows things down. This check does take place in S3A mkdirs(), which walks backwards up the list of parent paths until it finds a directory (success) or a file (failure). In practice production applications invariably create destination directories before writing 1+ file into them -restricting check purely to the mkdirs() call deliver significant speed up while implicitly including the checks. Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0	2020-03-09 14:44:28 +00:00
Sebastian Nagel	18050bc583	HADOOP-16909 Typo in distcp counters. Contributed by Sebastian Nagel.	2020-03-09 14:37:08 +00:00
Steve Loughran	d0a7c790c6	HADOOP-16885. Fix hadoop-commons TestCopy failure Followup to HADOOP-16885: Encryption zone file copy failure leaks a temp file Moving the delete() call broke a mocking test, which slipped through the review process. Contributed by Steve Loughran. Change-Id: Ia13faf0f4fffb1c99ddd616d823e4f4d0b7b0cbb	2020-03-03 17:25:22 +00:00
cpugputpu	5678b19b01	HADOOP-16897. Sort fields in ReflectionUtils.java. Contributed by cpugputpu.	2020-03-02 17:53:38 +00:00
Xiaoyu Yao	0dd8956f2e	HADOOP-16885. Encryption zone file copy failure leaks a temp file Contributed by Xiaoyu Yao. Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails Copying file through the FsShell command into an HDFS encryption zone where the caller lacks permissions is leaks a temp ._COPYING file and potentially a wrapped stream unclosed. This is a convergence of a fix for S3 meeting an issue in HDFS. S3: a HEAD against a file can cache a 404, -you must not do any existence checks, including deleteOnExit(), until the file is written. Hence: HADOOP-16490, only register files for deletion the create worked and the upload is not direct. HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on create() failures. Which means that an entry is retained on the NN. -you need to register a file with deleteOnExit() even if the file wasn't created. This patch: * Moves the deleteOnExit to ensure the created file get deleted cleanly. * Fixes HDFS to close the wrapped stream on failures.	2020-03-02 13:22:00 +00:00
Kihwal Lee	352a4ec16d	HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein.	2020-02-26 09:33:29 -06:00
Takanobu Asanuma	5cbc4c5461	HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li.	2020-02-25 11:08:13 +09:00
Sahil Takiar	42dfd270a1	HADOOP-16859: ABFS: Add unbuffer support to ABFS connector. Contributed by Sahil Takiar	2020-02-24 16:28:00 +00:00
Ayush Saxena	b5698e0c33	HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun.	2020-02-24 00:07:18 +05:30
Wei-Chiu Chuang	cb3f3cca01	HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze.	2020-02-18 21:53:08 -08:00
Ayush Saxena	ac4b556e2d	HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena.	2020-02-19 08:20:59 +05:30
Arpit Agarwal	0cfff16ac0	HADOOP-16833. InstrumentedLock should log lock queue time. Contributed by Stephen O'Donnell. Change-Id: Idddff05051b6f642b88e51694b40c5bb1bef0026	2020-02-18 09:50:11 -08:00
Steve Loughran	a562942b05	HADOOP-16759. FileSystem Javadocs to list what breaks on API changes Followup to the main openFile().withStatus() patch. It turns out that this broke the hive builds, which was not well appreciated. This patch lists places to review in the hadoop codebase, and external projects where changes are likely to cause problems. Contributed by Steve Loughran Change-Id: Ifac815c65b74d083cd277764b780ac2b5b0f6b36	2020-02-17 22:14:39 +00:00
Ayush Saxena	84f7638840	HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri.	2020-02-18 00:43:33 +05:30
Akira Ajisaka	954930e9d9	HADOOP-16850. Support getting thread info from thread group for JvmMetrics to improve the performance. Contributed by Tao Yang.	2020-02-14 15:20:28 +09:00
Steve Loughran	56dee66770	HADOOP-16823. Large DeleteObject requests are their own Thundering Herd. Contributed by Steve Loughran. During S3A rename() and delete() calls, the list of objects delete is built up into batches of a thousand and then POSTed in a single large DeleteObjects request. But as the IO capacity allowed on an S3 partition may only be 3500 writes per second and each entry in that POST counts as a single write, then one of those posts alone can trigger throttling on an already loaded S3 directory tree. Which can trigger backoff and retry, with the same thousand entry post, and so recreate the exact same problem. Fixes * Page size for delete object requests is set in fs.s3a.bulk.delete.page.size; the default is 250. * The property fs.s3a.experimental.aws.s3.throttling (default=true) can be set to false to disable throttle retry logic in the AWS client SDK -it is all handled in the S3A client. This gives more visibility in to when operations are being throttled * Bulk delete throttling events are logged to the log org.apache.hadoop.fs.s3a.throttled log at INFO; if this appears often then choose a smaller page size. * The metric "store_io_throttled" adds the entire count of delete requests when a single DeleteObjects request is throttled. * A new quantile, "store_io_throttle_rate" can track throttling load over time. * DynamoDB metastore throttle resilience issues have also been identified and fixed. Note: the fs.s3a.experimental.aws.s3.throttling flag does not apply to DDB IO precisely because there may still be lurking issues there and it safest to rely on the DynamoDB client SDK. Change-Id: I00f85cdd94fc008864d060533f6bd4870263fd84	2020-02-13 19:09:49 +00:00
Stephen O'Donnell	d7c136b9ed	HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-02-11 08:00:15 -08:00
Jan Hentschel	cc8ae59104	HADOOP-16851. Removed unused import in Configuration Contributed by Jan Hentschel.	2020-02-11 11:51:45 +00:00
testfixer	d36cd37e60	HADOOP-16847. Test can fail if HashSet iterates in a different order. Contributed by Testfixer	2020-02-11 11:22:07 +00:00
Ayush Saxena	23787e4bdd	HDFS-15136. LOG flooding in secure mode when Cookies are not set in request header. Contributed by Renukaprasad C	2020-02-08 01:17:59 +05:30
Akira Ajisaka	3ebf505965	HADOOP-16834. Replace com.sun.istack.Nullable with javax.annotation.Nullable in DNS.java. Contributed by Xieming Li.	2020-02-07 19:30:06 +09:00
Vinayakumar B	7dac7e1d13	HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635 ). Contributed by Vinayakumar B.	2020-02-07 14:51:24 +05:30
Chen Liang	ce7b8b5634	HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.	2020-02-04 12:12:35 -08:00
belugabehr	1afd54fbbb	HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811 ). Contributed by David Mollitor.	2020-01-25 20:42:21 +05:30
Steve Loughran	839e6076c1	HADOOP-16827. TestHarFileSystem.testInheritedMethodsImplemented broken. (#1817 ) This is a regression caused by HADOOP-16759. The test TestHarFileSystem uses introspection to verify that HarFileSystem Does not implement methods to which there is a suitable implementation in the base FileSystem class. Because of the way it checks this, refactoring (protected) FileSystem methods in an IDE do not automatically change the probes in TestHarFileSystem. The changes in HADOOP-16759 did exactly that, and somehow managed to get through the build/test process without this being noticed. This patch fixes that failure. Caused by and fixed by Steve Loughran. Change-Id: If60d9c97058242871c02ad1addd424478f84f446 Signed-off-by: Mingliang Liu <liuml07@apache.org>	2020-01-24 10:50:16 -08:00
Mustafa Iman	839054754b	HADOOP-16792: Make S3 client request timeout configurable. Contributed by Mustafa Iman. This adds a new configuration option fs.s3a.connection.request.timeout to declare the time out on HTTP requests to the AWS service; 0 means no timeout. Measured in seconds; the usual time suffixes are all supported Important: this is the maximum duration of any AWS service call, including upload and copy operations. If non-zero, it must be larger than the time to upload multi-megabyte blocks to S3 from the client, and to rename many-GB files. Use with care. Change-Id: I407745341068b702bf8f401fb96450a9f987c51c	2020-01-24 13:37:07 +00:00
Mingliang Liu	6c1fa24ac0	HADOOP-16732. S3Guard to support encrypted DynamoDB table (#1752 ). Contributed by Mingliang Liu.	2020-01-23 14:21:42 +01:00
Zhankun Tang	d40d7cc4f9	HADOOP-16670. Stripping Submarine code from Hadoop codebase. Contributed by Zhankun Tang. Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-01-21 20:07:31 -08:00
Steve Loughran	5e2ce370a3	HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (#1761 ). Contributed by Steve Loughran * Enhanced builder + FS spec * s3a FS to use this to skip HEAD on open * and to use version/etag when opening the file works with S3AFileStatus FS and S3ALocatedFileStatus	2020-01-21 14:31:51 -08:00
Sahil Takiar	f206b736f0	HADOOP-16346. Stabilize S3A OpenSSL support. Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`. The new option is documented and marked as experimental. For details on how to use this, consult the peformance document in the s3a documentation. This patch is the successor to HADOOP-16050 "S3A SSL connections should use OpenSSL" -which was reverted because of incompatibilities between the wildfly OpenSSL client and the AWS HTTPS servers (HADOOP-16347). With the Wildfly release moved up to 1.0.7.Final (HADOOP-16405) everything should now work. Related issues: * HADOOP-15669. ABFS: Improve HTTPS Performance * HADOOP-16050: S3A SSL connections should use OpenSSL * HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8 * HADOOP-16405: Upgrade Wildfly Openssl version to 1.0.7.Final Contributed by Sahil Takiar Change-Id: I80a4bc5051519f186b7383b2c1cea140be42444e	2020-01-21 16:37:51 +00:00
sunlisheng	d887e49dd4	Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure() Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>	2020-01-21 18:22:44 +09:00
Akira Ajisaka	1defe3a65a	HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.	2020-01-21 10:58:32 +09:00
Vinayakumar B	edbbc03ce7	HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. (#1803 )	2020-01-16 23:27:50 +05:30
Chao Sun	d7c4f8ab21	HDFS-13616. Batch listing of multiple directories (#1725 )	2020-01-15 17:22:47 -08:00
David Mollitor	5d180463dc	HADOOP-16790. Add Write Convenience Methods. Contributed by David Mollitor. This adds operations in FileUtil to write text to a file via either a FileSystem or FileContext instance. Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410	2020-01-15 18:38:34 +00:00
Wei-Chiu Chuang	6b86a5110e	HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu.	2020-01-13 10:48:53 -08:00
Chen Liang	26a969ec73	HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.	2020-01-10 17:06:59 -08:00
Eric Yang	e589f6199c	HADOOP-16590. Update OS login modules for IBM JDK. Contributed by Nicholas Marion Close #1484	2020-01-10 14:33:12 -05:00
Daryn Sharp	c4fb43c942	HADOOP-16749. Configuration parsing of CDATA values are blank Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-01-10 10:31:39 -06:00
Steve Loughran	49df838995	HADOOP-16697. Tune/audit S3A authoritative mode. Contains: HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination dirirectory as authoritative on success. HADOOP-16684. S3guard bucket info to list a bit more about authoritative paths. HADOOP-16722. S3GuardTool to support FilterFileSystem. This patch improves the marking of newly created/import directory trees in S3Guard DynamoDB tables as authoritative. Specific changes: * Renamed directories are marked as authoritative if the entire operation succeeded (HADOOP-16474). * When updating parent table entries as part of any table write, there's no overwriting of their authoritative flag. s3guard import changes: * new -verbose flag to print out what is going on. * The "s3guard import" command lets you declare that a directory tree is to be marked as authoritative hadoop s3guard import -authoritative -verbose s3a://bucket/path When importing a listing and a file is found, the import tool queries the metastore and only updates the entry if the file is different from before, where different == new timestamp, etag, or length. S3Guard can get timestamp differences due to clock skew in PUT operations. As the recursive list performed by the import command doesn't retrieve the versionID, the existing entry may in fact be more complete. When updating an existing due to clock skew the existing version ID is propagated to the new entry (note: the etags must match; this is needed to deal with inconsistent listings). There is a new s3guard command to audit a s3guard bucket/path's authoritative state: hadoop s3guard authoritative -check-config s3a://bucket/path This is primarily for testing/auditing. The s3guard bucket-info command also provides some more details on the authoritative state of a store (HADOOP-16684). Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91	2020-01-10 11:11:56 +00:00
Steve Loughran	17aa8f6764	HADOOP-16785. Improve wasb and abfs resilience on double close() calls. This hardens the wasb and abfs output streams' resilience to being invoked in/after close(). wasb: Explicity raise IOEs on operations invoked after close, rather than implicitly raise NPEs. This ensures that invocations which catch and swallow IOEs will perform as expected. abfs: When rethrowing an IOException in the close() call, explicitly wrap it with a new instance of the same subclass. This is needed to handle failures in try-with-resources clauses, where any exception in closed() is added as a suppressed exception to the one thrown in the try {} clause and you cannot attach the same exception to itself Contributed by Steve Loughran. Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b	2020-01-08 11:46:54 +00:00
Steve Loughran	bb1aed475b	HADOOP-16751. Followup: move java import. (#1799 ) This moves the import of a java module to the preferred place in the import ordering. Change-Id: I1a594e3d954554a72c2b71c954eda0ae940a8f70	2020-01-08 11:32:31 +00:00
Hanisha Koneru	a7fccc1122	HADOOP-16727. KMS Jetty server does not startup if trust store password is null.	2020-01-07 15:46:14 -08:00
Akira Ajisaka	bc366d4ea7	HADOOP-16773. Fix duplicate assertj-core dependency in hadoop-common module. Contributed by Xieming Li.	2020-01-07 20:49:24 +09:00
Rajesh Balamohan	b19d87c2b7	HADOOP-16751. DurationInfo text parsing/formatting should be moved out of hotpath. Contributed by Rajesh Balamohan Change-Id: Icc3dcfa81aa69164f2c088f9b533d231138cbb8b	2020-01-02 17:03:07 +00:00
Steve Loughran	958764479d	HADOOP-16777. Add Tez to LimitedPrivate of ClusterStorageCapacityExceededException Contributed by Wang Yan. Change-Id: I92dfe7079ba8ebe89d70255bb845309be0603a8e	2020-01-02 15:49:42 +00:00
Rakesh Radhakrishnan	d79cce20ab	HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.	2020-01-02 11:44:00 +05:30
Ayush Saxena	926d0b48f0	HDFS-15081. Typo in RetryCache#waitForCompletion annotation. Contributed by Fei Hui.	2019-12-27 18:32:15 +05:30
Chandra Sanivarapu	d80d86264d	HADOOP-16042. Update the link to HadoopJavaVersion (#1784 )	2019-12-27 17:18:10 +09:00
Vinayakumar B	e2a5448d2b	HADOOP-16774. TestDiskChecker and TestReadWriteDiskValidator fails when run with -Pparallel-tests (#1776 ). Contributed by Vinayakumar B.	2019-12-21 00:08:15 +05:30
Ayush Saxena	ab32762f43	HADOOP-16755. Fix broken link in single node cluster setup documentation. Contributed by Denes Gerencser.	2019-12-16 08:44:28 +05:30
Kengo Seki	fd7de2b82a	HADOOP-16764. Rewrite Python example codes using Python3 (#1762 )	2019-12-16 11:04:20 +09:00
Szilard Nemeth	631dbbc6f2	YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal	2019-12-15 17:28:04 +01:00
Mingliang Liu	d12ad9e8ad	HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase (#1757 ) Contributed by Mingliang Liu. Signed-off-by: Steve Loughran <stevel@apache.org>	2019-12-13 08:19:27 -08:00
Erik Krogen	c174d50b30	HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen.	2019-12-11 09:39:37 -08:00
Mingliang Liu	19512b21e3	HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu This closes #1733	2019-12-05 17:37:17 -08:00
Ayush Saxena	83a14559e5	HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui.	2019-12-05 18:52:01 +05:30
Aravindan Vijayan	f1ab7f18c4	HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Siyao Meng <smeng@cloudera.com>	2019-12-03 15:47:16 -08:00
Yiqun Lin	2b452b4e60	HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.	2019-11-28 10:43:35 +08:00
Sahil Takiar	3161813482	HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists (#1695 )	2019-11-26 11:11:26 -08:00
Akira Ajisaka	da9435182a	HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li.	2019-11-25 14:36:27 +09:00
Siyao Meng	1a0c0e42e7	HADOOP-16276. Fix jsvc startup command in hadoop-functions.sh due to jsvc >= 1.0.11 changed default current working directory (#1272 )	2019-11-20 16:44:44 -08:00
David Mollitor	6f899e9030	HADOOP-15852. Refactor QuotaUsage. Contributed by David Mollitor. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-11-20 14:40:10 -08:00
Wei-Chiu Chuang	9664b9c7a6	HADOOP-15686. Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr. Contributed by Wei-Chiu Chuang. Reviewed-by: Xiaoyu Yao <xyao@apache.org>	2019-11-20 13:07:43 -08:00
Erik Krogen	be77231452	HADOOP-16700. Track response time separately in Server.Call to avoid negative RpcQueueTime values. Contributed by xuzq.	2019-11-20 07:56:33 -08:00
Steve Loughran	1415ad3800	HADOOP-16707. NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup. Contributed by Steve Loughran. Change-Id: I38fdba2fa70e534d78b15e61de19368912588b0c	2019-11-20 14:32:12 +00:00
Abhishek Modi	4050471b45	HADOOP-16657. Move remaining log4j APIs over to slf4j in hadoop-common. Contributed by Minni Mittal.	2019-11-19 08:01:17 +05:30
Szilard Nemeth	27642367ef	YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko	2019-11-18 16:29:39 +01:00
Ayush Saxena	7f811722b7	HADOOP-16712. Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml. Contributed by Xieming Li.	2019-11-18 19:00:25 +05:30
Hui Fei	67f2c491fe	HDFS-14802. The feature of protect directories should be used in RenameOp (#1669 )	2019-11-15 12:35:37 -08:00
He Xiaoqiao	c892a879dd	HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Istvan Fajth <pifta@cloudera.com>	2019-11-15 12:16:31 -08:00
Akira Ajisaka	92c28c100e	HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li.	2019-11-15 13:49:57 +09:00
belugabehr	d0302d3849	HADOOP-16691. Unify Logging in UserGroupInformation. Unify Logging in UserGroupInformation.	2019-11-14 10:28:26 -08:00
Lukas Majercak	c73334a924	HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log. MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.	2019-11-14 09:54:02 -08:00
Steve Loughran	a139a87610	HADOOP-16708. HadoopExecutors cleanup to only log at debug. Contributed by David Mollitor. Change-Id: Ib501a3ed6ed299ccd3c773c4dc086441a5728e73	2019-11-14 17:51:39 +00:00
Surendra Singh Lilhore	73a386ab76	HADOOP-16585. [Tool:NNloadGeneratorMR] Multiple threads are using same id for creating file LoadGenerator#write. Contributed by Ranith Sardar.	2019-11-14 01:45:46 +05:30
Xudong Cao	df6b3162c1	HADOOP-16677. Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. (#1687 )	2019-11-12 14:19:39 -08:00
Siyao Meng	3ab98eefa5	HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-11-12 11:34:24 -08:00
Steve Loughran	990063d2af	HADOOP-16665. Filesystems to be closed if they failed during initialize(). Contributed by Steve Loughran. This FileSystem instantiation so if an IOException or RuntimeException is raised in the invocation of FileSystem.initialize() then a best-effort attempt is made to close the FS instance; exceptions raised that there are swallowed. The S3AFileSystem is also modified to do its own cleanup if an IOException is raised during its initialize() process, it being the FS we know has the "potential" to leak threads, especially in extension points (e.g AWS Authenticators) which spawn threads. Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04	2019-11-12 18:17:21 +00:00
Szilard Nemeth	3d249301f4	HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal	2019-11-09 15:43:53 +01:00
Chen Liang	dd900259c4	HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko.	2019-11-06 09:56:19 -08:00
Wei-Chiu Chuang	ee8addbec4	Revert "HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng." This reverts commit `f9b99d2f24`.	2019-11-05 15:41:52 -08:00
belugabehr	6f0190d8e4	HADOOP-16678: Review of ArrayWritable (#1692 )	2019-11-04 14:24:44 -08:00
Gabor Bota	dca19fc3aa	HADOOP-16484. S3A to warn or fail if S3Guard is disabled (#1661 ). Contributed by Gabor Bota.	2019-11-04 12:55:36 +01:00
Jonathan Hung	ef9d12df24	Add 2.10.0 release notes for HDFS-12943	2019-10-31 20:33:59 -07:00
Siyao Meng	f9b99d2f24	HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-10-31 12:59:46 -07:00
Ayush Saxena	d9fbedc4ae	HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun.	2019-10-31 23:39:27 +05:30
Szilard Nemeth	8c9173c87c	HADOOP-16510. [hadoop-common] Fix order of actual and expected expression in assert statements. Contributed by Adam Antal	2019-10-31 14:35:20 +01:00
Jonathan Hung	e6137d0e50	Make upstream aware of 2.10.0 release	2019-10-30 20:58:32 -07:00
Steve Loughran	bd498bac86	HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library. Contributed by Steve Loughran. Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451	2019-10-29 17:09:26 +00:00
Fei Hui	7be5508d9b	HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-10-25 15:06:16 -07:00
hongdongdong	2eba262472	HADOOP-16615. Add password check for credential provider, Contributed by hongdongdong. Change-Id: Iaac01bc8594860064a80c822a0e47981243ab7e1	2019-10-24 18:48:27 +01:00
Ayush Saxena	2ae4b33d48	HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun.	2019-10-19 11:05:54 +05:30
Mate Szalay-Beko	6d92aa7c30	HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 (#1656 ). Contributed by Norbert Kalmár, Mate Szalay-Beko * HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5 - Add a static initializer for the unit tests using ZooKeeper to enable the four-letter-words diagnostic telnet commands. (this is an interface that become disabled by default, so to keep the ZooKeeper 3.4.x behavior we enabled it for the tests) - Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt records. The new ZooKeeper seems to respond quicker during the fail-over tests than the ZooKeeper, so we made sure to catch all the relevant records by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt(). Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>	2019-10-18 13:26:20 -07:00
Yuming Wang	3d41f33018	HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng. Co-authored-By: Siyao Meng <smeng@cloudera.com> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-10-18 08:25:02 -07:00
Szilard Nemeth	c79a5f2d99	HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal	2019-10-16 13:29:06 +02:00
Wei-Chiu Chuang	c39e9fc9a3	HADOOP-15169. "hadoop.ssl.enabled.protocols" should be considered in httpserver2. Contributed by Brahma Reddy Battula, Wei-Chiu Chuang. Reviewed-by: Xiaoyu Yao <xyao@apache.org> Co-authored-By: Brahma Reddy Battula <brahma@apache.org>	2019-10-15 13:55:23 -07:00
Steve Loughran	dee9e97075	Revert "HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos." This reverts commit `7a4b3d42c4`. The patch broke TestRouterWebHDFSContractSeek as it turns out that WebHDFSInputStream.available() is always 0.	2019-10-14 16:56:50 +01:00
Ayush Saxena	44bc095998	HADOOP-13907. Fix TestWebDelegationToken#testKerberosDelegationTokenAuthenticator on Windows. Contributed by Kitti Nanasi.	2019-10-12 13:16:51 +05:30
lqjacklee	7a4b3d42c4	HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos. Contributed by lqjacklee. Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c	2019-10-10 21:58:42 +01:00
Jonathan Hung	5462d2176f	YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena	2019-10-08 11:20:13 -07:00
Steve Loughran	1921e94292	HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3 Contributed by Steve Loughran. Includes -S3A glob scans don't bother trying to resolve symlinks -stack traces don't get lost in getFileStatuses() when exceptions are wrapped -debug level logging of what is up in Globber -Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest. -ITestRestrictedReadAccess tests incomplete read access to files. This adds a builder API for constructing globbers which other stores can use so that they too can skip symlink resolution when not needed. Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7	2019-10-01 18:11:05 +01:00
Sahil Takiar	13b427fc05	HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963 ) Contributed by Sahil Takiar. Reviewed-by: Siyao Meng <smeng@cloudera.com>	2019-09-27 12:25:03 -07:00
Zhankun Tang	2870668cfa	Make upstream aware of 3.1.3 release.	2019-09-27 16:44:07 +08:00
Gabor Bota	587a8eeec8	HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir (#1522 ). Contributed by Gabor Bota.	2019-09-26 10:15:13 +05:30
Steve Loughran	e346e3638c	HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. Contributed by Steve Loughran. This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client to offer a specific capability. This is intended to allow applications to determine * Whether a method is implemented before calling it and dealing with UnsupportedOperationException. * Whether a specific feature is believed to be available in the remote store. As well as a common set of capabilities defined in CommonPathCapabilities, file systems are free to add their own capabilities, prefixed with fs. + schema + . The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available. Note * The remote store is not expected to be checked for the feature; It is more a check of client API and the client's configuration/knowledge of the state of the remote system. * Permissions are not checked. Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5	2019-09-25 12:16:41 +01:00
Szilard Nemeth	c72457787d	YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko	2019-09-25 10:36:22 +02:00
Prabhu Joseph	3f89084ac7	HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs. Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-09-25 10:19:30 +09:00
Gabor Bota	aa664d7259	HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302 ). Contributed by Gabor Bota. Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023	2019-09-23 13:29:01 +02:00
Vinayakumar B	07c81e9bfc	HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494 ). Contributed by Vinayakumar B.	2019-09-23 12:37:47 +05:30
Rohith Sharma K S	4c0a7a9e13	Make upstream aware of 3.2.1 release.	2019-09-23 06:20:54 +00:00
Rakesh Radhakrishnan	659c88801d	HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.	2019-09-22 22:02:54 +05:30
Yuval Degani	dd0834696a	HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463 ) In the existing implementation, the ValueQueue::getAtMost() method will only trigger a refill on a key queue if it has gone empty, instead of triggering a refill when it has gone below the watermark. Revise the test suite to correctly verify this behavior.	2019-09-20 09:55:48 -07:00
Vinayakumar B	1654497f98	HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432 ) HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.	2019-09-20 16:08:30 +05:30
Takanobu Asanuma	298f07abce	HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao.	2019-09-20 13:55:34 +09:00
Malcolm Taylor	56248f9d87	HADOOP-16556. Fix some alerts raised by LGTM. Contributed by Malcolm Taylor. Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231	2019-09-19 16:00:05 +01:00
Kihwal Lee	d4205dce17	HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee	2019-09-19 08:23:35 -05:00
Weiwei Yang	f8c14326ee	YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu.	2019-09-18 09:12:28 +08:00
Sahil Takiar	3cf6e4272f	HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs Signed-off-by: Anu Engineer <aengineer@apache.org>	2019-09-17 15:51:39 -07:00
Sahil Takiar	55ce454ce4	HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8. Contributed by Sahil Takiar. This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common as the DelegatingSSLSocketFactory and binds the S3A connector to it so that it can avoid using those HTTPS algorithms which are underperformant on Java 8. Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd	2019-09-17 11:32:03 +01:00
Steve Loughran	9221704f85	HADOOP-16490. Avoid/handle cached 404s during S3A file creation. Contributed by Steve Loughran. This patch avoids issuing any HEAD path request when creating a file with overwrite=true, so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile in their own code. The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even after S3A is patched to not do it itself. Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy independently configurable from other retry policies; it is also exponential, but with different parameters. This is because every HEAD request will refresh any 404 cached in the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between attempts to (hopefully) ensure any cached entry wil be gone. The options and values are: fs.s3a.s3guard.consistency.retry.interval: 2s fs.s3a.s3guard.consistency.retry.limit: 7 The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated. Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way: it will fail fast with a different error message. However, because create(path, overwrite=false) no longer does HEAD path, we can at least be confident that S3A itself is not creating those cached 404 markers. Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d	2019-09-11 16:46:25 +01:00
Daisuke Kobayashi	bc2d3a71d6	HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2019-09-10 10:51:47 +08:00
Sneha Vijayarajan	147f98629c	HADOOP-16438. ADLS Gen1 OpenSSL config control. Contributed by Sneha Vijayarajan. Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d	2019-09-09 17:09:32 +01:00
Jungtaek Lim (HeartSaVioR)	bb0b922a71	HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) Contributed by Jungtaek Lim Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d	2019-09-06 21:53:00 +01:00
Erik Krogen	a23417533e	HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang.	2019-09-06 10:28:21 -07:00
Erik Krogen	c92a3e94d8	HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun.	2019-09-06 10:22:28 -07:00
Steve Loughran	511df1e837	HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions Contributed by Steve Loughran. This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables. It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent. For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath. Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e	2019-09-05 14:25:15 +01:00
Erik Krogen	337e9b794d	HADOOP-16268. Allow StandbyException to be thrown as CallQueueOverflowException when RPC call queue is filled. Contributed by CR Hota.	2019-09-04 08:22:02 -07:00
Surendra Singh Lilhore	5ff76cb8bc	HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina.	2019-09-03 12:37:09 +05:30
Stephen O'Donnell	915cbc91c0	HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-09-02 09:47:04 -07:00
Akira Ajisaka	dc0acceabb	YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko.	2019-08-30 10:13:10 +09:00
Surendra Singh Lilhore	29bd6f3fc3	HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun.	2019-08-28 23:58:23 +05:30
Akira Ajisaka	55cc115878	HADOOP-16527. Add a whitelist of endpoints to skip Kerberos authentication (#1336 ) Contributed by Akira Ajisaka.	2019-08-28 14:28:41 +09:00
Steve Loughran	61b2df2331	HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper. Contributed by Steve Loughran. Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider. Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8	2019-08-22 17:27:56 +01:00
Akira Ajisaka	30ce8546f1	HADOOP-16496. Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop common. This closes #1317 Reviewed-by: Bharat Viswanadham <bharat@apache.org>	2019-08-21 10:10:11 +09:00
Wei-Chiu Chuang	51b65370b9	HADOOP-14784. [KMS] Improve KeyAuthorizationKeyProvider#toString(). Contributed by Yeliang Cang. Reviewed-by: Dinesh Chitlangia <dchitlangia@cloudera.com>	2019-08-19 11:12:09 -07:00
David Mollitor	a707bb7c1b	HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-08-18 18:44:35 -07:00
Wei-Chiu Chuang	971a4c8e83	HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Chen Liang <cliang@apache.org>	2019-08-16 17:19:58 -07:00
Erik Krogen	e356e4f4b7	HADOOP-16391 Add a prefix to the metric names for MutableRatesWithAggregation used for deferred RPC metrics to avoid collision with non-deferred metrics. Contributed by Bilwa S T.	2019-08-16 09:01:44 -07:00
Wei-Chiu Chuang	5882cf94ea	HADOOP-16504. Increase ipc.server.listen.queue.size default from 128 to 256. Contributed by Lisheng Sun.	2019-08-15 15:20:54 -07:00
Akira Ajisaka	0f8add8a60	HADOOP-16495. Fix invalid metric types in PrometheusMetricsSink (#1244 )	2019-08-14 12:24:03 +09:00
Masatake Iwasaki	da0006fe04	HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>	2019-08-14 08:39:40 +09:00
Inigo Goiri	6b4564f1d5	HADOOP-16453. Update how exceptions are handled in NetUtils. Contributed by Lisheng Sun.	2019-08-11 20:34:36 -07:00
Steve Loughran	e25a5c2eab	HADOOP-16499. S3A retry policy to be exponential (#1246 ). Contributed by Steve Loughran.	2019-08-09 15:52:37 +02:00
Szilard Nemeth	df30d8ea09	YARN-9727: Allowed Origin pattern is discouraged if regex contains *. Contributed by Zoltan Siegl	2019-08-09 09:34:23 +02:00
Zsombor Gegesy	b0131bc265	HADOOP-15014. Addendum: KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-08-07 20:57:42 -07:00
Yiqun Lin	a5bb1e8ee8	HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.	2019-08-07 10:18:11 +08:00
Wei-Chiu Chuang	8cef9f89f4	HDFS-14652. Addendum: HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang.	2019-08-06 15:24:11 -07:00
Eric Yang	22430c10e2	HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager. Contributed by Prabhu Joseph	2019-08-06 17:04:17 -04:00
Jianfei Jiang	71aad60e51	HDFS-14691. Wrong usage hint for hadoop fs command "test". Contributed by Jianfei Jiang. Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e	2019-08-05 13:08:47 +01:00
Wei-Chiu Chuang	61180f4656	HADOOP-15942. Change the logging level form DEBUG to ERROR for RuntimeErrorException in JMXJsonServlet. Contributed by Anuhan Torgonshar.	2019-08-02 14:58:24 -07:00
Wei-Chiu Chuang	797d14e816	Revert "HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm." This reverts commit `076618677d`.	2019-08-02 08:25:41 -07:00
Wei-Chiu Chuang	e872ceb810	HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko.	2019-08-01 19:56:51 -07:00
Wei-Chiu Chuang	e20b19543b	HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat.	2019-08-01 17:35:31 -07:00
Wei-Chiu Chuang	d086d058d8	HDFS-14652. HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang.	2019-08-01 16:13:10 -07:00
Wei-Chiu Chuang	b94eba9f11	HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun.	2019-08-01 15:50:43 -07:00
He Xiaoqiao	f86de6f76a	HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-08-01 14:54:39 -07:00
Akira Ajisaka	8bda91d20a	HADOOP-16398. Exports Hadoop metrics to Prometheus (#1170 )	2019-07-31 10:11:36 -07:00
Wei-Chiu Chuang	0f2dad6679	HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina.	2019-07-30 16:52:42 -07:00
Siyao Meng	c75f16db79	HADOOP-16452. Increase ipc.maximum.data.length default from 64MB to 128MB. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-07-30 11:22:45 -07:00
Don Jeba	204a977f55	HADOOP-15910. Fix Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS Contributed by Don Jeba. Change-Id: I2755bfb1263fc659078a1af8f0bdfd739fd1ae40	2019-07-30 12:39:48 +01:00
Akira Ajisaka	cbfa3f3e98	HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.	2019-07-29 17:37:26 -07:00
Erik Krogen	62efb63006	HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.	2019-07-26 11:16:58 -07:00
Gopal V	aebac6d2d2	HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock. Contributed by Gopal V. Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4	2019-07-26 11:32:13 +01:00
Steve Loughran	07530314c2	HADOOP-9844. NPE when trying to create an error message response of SASL RPC This closes #55 Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-07-26 17:53:18 +09:00
sunlisheng	a1251addff	HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java. This closes #1091 Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-07-24 10:04:39 +09:00
Steve Loughran	4317d33232	HADOOP-16380. S3Guard to determine empty directory status for all non-root directories. Contributed by Steve Loughran and Gabor Bota. This * Asks S3Guard to determine the empty directory status. * Has S3A's root directory rm("/") command to always return false (as abfs does) * Documents that object stores MAY do this * Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test. It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty". Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108	2019-07-23 14:52:03 +01:00
S O'Donnell	eb36b09cb7	HADOOP-16443. Improve help text for setfacl --set option. Contributed by S O'Donnell. Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413	2019-07-23 10:24:07 +01:00
Sean Mackrory	7f1b76ca35	HADOOP-13868. [s3a] New default for S3A multi-part configuration (#1125 )	2019-07-19 09:49:59 -06:00
Josh Rosen	d545f9c290	HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise Contributed by: Josh Rosen	2019-07-18 23:19:38 +01:00
Gopal V	b4466a3b0a	HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 Contributed by Gopal V and Atilla Magyar. Change-Id: I066d5eece332a1673594de0f9b484443f95530ec	2019-07-17 13:50:02 +01:00
Steve Loughran	b15ef7dc3d	HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3. Contributed by Steve Loughran Contains - HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option. - HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently This patch doesn't fix the underlying problem but it * changes some tests to clean up better * does a lot more in logging operations in against DDB, if enabled * adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck) * adds a purge entry point to help clean up after a test run has got a store into a mess * s3guard prune command adds -tombstone option to only clear tombstones The outcome is that tests should pass consistently and if problems occur we have better diagnostics. Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb	2019-07-12 13:02:25 +01:00
Chen Liang	8fb5ca3f40	HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.	2019-07-11 13:23:25 -07:00
Steve Loughran	6a3433bffd	HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists. Contributed by Steve Loughran. This patch * changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer * adds a check for the dest path being a file not a dir * adds tests for this * Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present. Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53	2019-07-11 18:15:34 +01:00
Inigo Goiri	aa9f0850e8	HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.	2019-07-02 13:22:30 -07:00
Christopher Gregorian	129576f628	HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian.	2019-06-24 12:09:17 -07:00
Ajay Kumar	95c94dcca7	HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.	2019-06-24 11:59:07 -07:00
Ayush Saxena	b52fd05d42	HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma.	2019-06-24 22:03:04 +05:30
Ayush Saxena	ba221dc50e	HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma.	2019-06-24 22:03:04 +05:30
Takanobu Asanuma	559cb11551	HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.	2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore	f40e10b349	HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar.	2019-06-24 22:03:03 +05:30
Prabhu Joseph	371452e260	HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-06-21 17:05:13 -07:00
Wei-Chiu Chuang	840d02ca5b	HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-06-20 19:57:48 -07:00
Steve Loughran	e02eb24e0a	HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. Contributed by Steve Loughran. Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb	2019-06-20 09:56:40 +01:00
Inigo Goiri	3ab77d9bc9	HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.	2019-06-18 09:58:29 -07:00
Shweta Yakkali	dd4a7633ec	HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-06-17 23:47:57 -07:00
Adam Antal	56c621c68f	HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-06-17 22:56:00 -07:00
Wei-Chiu Chuang	f1c239c6a4	HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor.	2019-06-17 21:51:33 -07:00
Wei-Chiu Chuang	10311c30b0	HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong.	2019-06-17 19:55:56 -07:00
Wei-Chiu Chuang	1e92db5a1e	HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend.	2019-06-17 18:29:34 -07:00
Gabor Bota	f9cc9e1621	HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). Contributed by Gabor Bota. Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9	2019-06-16 17:05:01 +01:00
cxorm	076618677d	HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm.	2019-06-14 20:05:20 -07:00
dineshchitlangia	c7554ffd5c	HADOOP-16373. Fix typo in FileSystemShell#test documentation (#968 )	2019-06-14 14:36:04 -07:00
Takanobu Asanuma	54f9f75a44	HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.	2019-06-14 10:17:25 +09:00
Eric Yang	4ea6c2f457	HADOOP-16354. Enable AuthFilter as default for WebHDFS. Contributed by Prabhu Joseph	2019-06-11 18:41:08 -04:00
Wanqiang Ji	b417a4c854	MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours` Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-06-11 18:40:35 +09:00
Shweta Yakkali	d6d95d2686	HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-06-10 10:23:18 -07:00
Erik Krogen	76b94c274f	HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.	2019-06-07 14:20:44 -07:00
Steve Loughran	4e38dafde4	HADOOP-15563. S3Guard to support creating on-demand DDB tables. Contributed by Steve Loughran Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087	2019-06-07 18:26:10 +01:00
Huan-Ping Su	14552d19e3	HADOOP-16344. Make DurationInfo public unstable. Second attempt: imports fixed up. Contributed by Huan-Ping Su and Kai Xie. Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5	2019-06-07 12:55:17 +01:00
Akira Ajisaka	3ea4f41d9f	MAPREDUCE-6794. Remove unused properties from TTConfig.java	2019-06-07 10:27:41 +09:00
Steve Loughran	1f244b4227	Revert "HADOOP-16344. Make DurationInfo public unstable." This reverts commit `829848ba2e`. Change-Id: Ied91250e191b2ba701a8fc697c78b3756ce76be8	2019-06-06 17:49:31 +01:00
Huan-Ping Su	829848ba2e	HADOOP-16344. Make DurationInfo public unstable. Contributed by Huan-Ping Su Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c	2019-06-06 12:52:49 +01:00
Eric Yang	294695dd57	HADOOP-16314. Make sure all web end points are covered by the same authentication filter. Contributed by Prabhu Joseph	2019-06-05 18:55:13 -04:00
Sammi Chen	d1aad44490	HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.	2019-06-05 21:33:00 +08:00
Steve Loughran	309501c6fa	Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" This reverts commit `b067f8acaa`. Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8	2019-06-05 13:54:55 +01:00
Steve Loughran	7724d8031b	Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors" This reverts commit `5906268f0d`.	2019-06-05 13:54:42 +01:00
Steve Loughran	827a84778a	HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch. This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of stack traces because the timeout passed down to select was in nanos, whereas the API expected millis. Contributed by Erik Krogen. Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445	2019-06-04 15:56:47 +01:00
Shweta Yakkali	6f5a36c13c	HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-05-30 13:21:58 -07:00
Akira Ajisaka	afd844059c	HADOOP-16331. Fix ASF License check in pom.xml Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2019-05-29 17:25:13 +09:00
Akira Ajisaka	9f933e6446	HADOOP-16323. https everywhere in Maven settings.	2019-05-27 15:24:59 +09:00
Shweta	93d7630062	HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-05-24 18:42:46 +02:00
Christopher Gregorian	f96a2df38d	HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.	2019-05-23 10:28:37 -07:00
Eric Yang	ea0b1d8fba	HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation. Contributed by Prabhu Joseph	2019-05-23 11:36:32 -04:00
Akira Ajisaka	a771e2a638	HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang.	2019-05-23 13:59:42 +09:00
Sahil Takiar	5906268f0d	HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors	2019-05-21 11:30:45 -06:00
Ben Roling	a36274d699	HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. Contributed by Ben Roling. S3Guard will now track the etag of uploaded files and, if an S3 bucket is versioned, the object version. You can then control how to react to a mismatch between the data in the DynamoDB table and that in the store: warn, fail, or, when using versions, return the original value. This adds two new columns to the table: etag and version. This is transparent to older S3A clients -but when such clients add/update data to the S3Guard table, they will not add these values. As a result, the etag/version checks will not work with files uploaded by older clients. For a consistent experience, upgrade all clients to use the latest hadoop version.	2019-05-19 22:29:54 +01:00
Alexis Daboville	4cb3da6ac7	HADOOP-16248. MutableQuantiles leak memory under heavy load. Contributed by Alexis Daboville,	2019-05-17 15:15:22 +01:00
Sahil Takiar	b067f8acaa	HADOOP-16050: s3a SSL connections should use OpenSSL (cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)	2019-05-16 08:57:54 -06:00
David Mollitor	2713dcf6e9	HADOOP-16307. Intern User Name and Group Name in FileStatus. Author: David Mollitor	2019-05-16 16:02:07 +02:00
Bharat Viswanadham	d4c8858586	HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.	2019-05-15 17:41:36 -07:00
Inigo Goiri	389e640f0c	HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.	2019-05-13 11:46:16 -07:00
Akira Ajisaka	f257497b0f	HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11 Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2019-05-09 14:49:46 +09:00
Prabhu Joseph	96dc5cedfe	HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo. Author: Prabhu Joseph	2019-05-08 10:18:20 +01:00
Peter Bacsko	713e8a27ae	HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-05-07 17:48:27 -07:00
Siyao Meng	93f2283a69	HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-05-06 15:47:33 -07:00
Vinayakumar B	f1875b205e	HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena.	2019-05-03 11:22:14 +05:30
Giovanni Matteo Fumarola	7a3188d054	HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.	2019-05-02 12:58:42 -07:00
Sahil Takiar	4877f0aa51	HDFS-3246: pRead equivalent for direct read path (#597 ) HDFS-3246: pRead equivalent for direct read path Contributed by Sahil Takiar	2019-04-30 14:52:16 -07:00
Ben Roling	0af4011580	HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.	2019-04-30 11:53:26 +01:00
Sean Mackrory	a703dae25e	HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.	2019-04-24 10:39:00 -06:00
Anu Engineer	f4ab9370f5	HADOOP-16026:Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia.	2019-04-22 14:02:13 -07:00
Inigo Goiri	fb1c549139	HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota.	2019-04-22 13:32:08 -07:00
Erik Krogen	1ddb48872f	HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.	2019-04-22 08:16:57 -07:00
Zsombor Gegesy	008766c119	HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-04-16 05:28:18 -07:00
Kenneth Yang	b1120d27ab	HADOOP-16249. Make CallerContext LimitedPrivate scope to Public. Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-04-16 11:18:21 +09:00
Gabor Bota	1943db5571	HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre. Author: Gabor Bota <gabor.bota@cloudera.com>	2019-04-12 18:28:38 -07:00
Sahil Takiar	2382f63fc0	HADOOP-14747. S3AInputStream to implement CanUnbuffer. Author: Sahil Takiar <stakiar@cloudera.com>	2019-04-12 18:12:02 -07:00
Chen Liang	626fec652b	HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.	2019-04-12 17:37:51 -07:00
Arpit Agarwal	87407553ef	HADOOP-16243. Change Log Level to trace in NetUtils.java. Contributed by chencan.	2019-04-10 13:21:04 -07:00
Giovanni Matteo Fumarola	813cee1a18	HDFS-14420. Fix typo in KeyShell console. Contributed by Hu Xiaodong.	2019-04-10 11:18:40 -07:00
Todd Lipcon	65deb1ac42	HADOOP-16179. hadoop-common pom should not depend on kerb-simplekdc The hadoop-common pom currently has a dependency on kerb-simplekdc. In fact, the only classes used from Kerby are in kerb-core and kerb-util (which is a transitive dependency frmo kerb-core). Depending on kerb-simplekdc pulls a bunch of other unnecessary classes into the hadoop-common classpath. This changes the hadoop-common pom to depend only on kerb-core. hadoop-minikdc already had the appropriate dependency on kerb-simplekdc so it continues to pull in what it needs. Signed-off-by: Todd Lipcon <todd@apache.org>	2019-04-10 08:49:46 -07:00
Akira Ajisaka	bb8dda2bf9	HADOOP-12890. Fix typo in AbstractService. Contributed by Gabor Liptak.	2019-04-08 15:26:12 +09:00
Akira Ajisaka	ab645b3caa	HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.	2019-04-08 15:18:45 +09:00
Akira Ajisaka	72f4b9cd68	HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed by Ray Chiang.	2019-04-08 13:20:21 +09:00
Akira Ajisaka	0d47d283a6	HADOOP-10848. Cleanup calling of sun.security.krb5.Config.	2019-04-08 10:02:34 +09:00
David Mollitor	c90736350b	HADOOP-16208. Do Not Log InterruptedException in Client. Contributed by David Mollitor.	2019-04-04 16:15:57 +01:00
Inigo Goiri	7b5b783f66	HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.	2019-04-03 16:11:13 -07:00
Siyao Meng	e62cbcbc83	HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-04-03 14:29:52 -07:00
Steve Loughran	366186d999	HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685 ) This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start. Production code changes: * S3AFileStatus Adds "true" to the superclass's encrypted flag during construction. Tests * Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts? * Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads * And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up Docs: * Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch. Contributed by Steve Loughran. Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3	2019-04-03 21:23:40 +01:00
Gabor Bota	d7979079ea	HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.	2019-04-03 12:59:39 -06:00
Sahil Takiar	3b0c5016b2	HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags Signed-off-by: Todd Lipcon <todd@apache.org>	2019-04-03 10:56:33 -07:00
Akira Ajisaka	aaaf856f4b	HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str	2019-04-03 13:16:59 +09:00
Lokesh Jain	cf268114c9	HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-04-02 12:24:55 -07:00
Steve Loughran	61d19110d4	HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. (#655 ) Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8	2019-04-02 09:15:11 +01:00
Akira Ajisaka	ebd0d21538	HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.	2019-04-02 10:51:47 +09:00
Xiaoyu Yao	f41f938b2e	HADOOP-16199. KMSLoadBlanceClientProvider does not select token correctly. Contributed by Xiaoyu Yao. This closes #642.	2019-03-28 21:55:31 -07:00
Gabor Bota	b5db238383	HADOOP-15999. S3Guard: Better support for out-of-band operations. Author: Gabor Bota	2019-03-28 15:59:25 +00:00
David Mollitor	d18d0859eb	HADOOP-16181. HadoopExecutors shutdown Cleanup. Author: David Mollitor <david.mollitor@cloudera.com>	2019-03-22 10:29:27 +00:00
David Mollitor	246ab77f28	HADOOP-16196. Path Parameterize Comparable. Author: David Mollitor <david.mollitor@cloudera.com>	2019-03-22 10:26:24 +00:00
Eric Yang	5f6e225166	YARN-9363. Replaced debug logging with SLF4J parameterized log message. Contributed by Prabhu Joseph	2019-03-18 13:57:18 -04:00
Eric Yang	5446e3cb8a	HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18. Contributed by Daniel Templeton	2019-03-18 13:04:49 -04:00
Erik Krogen	8c95cb9d6b	HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.	2019-03-18 08:13:43 -07:00
Shweta Yakkali	2db38abffc	HDFS-14328. [Clean-up] Remove NULL check before instanceof in TestGSet (Contributed by Shweta Yakkali via Daniel Templeton) Change-Id: I5b9f0e66664714d7c5bbfa30492a09f770626711	2019-03-18 15:10:26 +01:00
Eric Yang	2064ca015d	YARN-9349. Changed logging to use slf4j api. Contributed by Prabhu Joseph	2019-03-15 19:20:59 -04:00
Ben Roling	6fa229891e	HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. Author: Ben Roling <ben.roling@gmail.com> Initial patch from Brahma Reddy Battula.	2019-03-13 20:37:11 +00:00
Erik Krogen	66357574ae	HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.	2019-03-13 13:15:56 -07:00
Matt Foley	f74159c8fc	HADOOP-16166. TestRawLocalFileSystemContract fails with build Docker container running on Mac. Also provided similar fix for Windows.	2019-03-13 09:33:24 -07:00
Konstantin V Shvachko	2a54feabb2	HDFS-14347. [SBN Read] Restore a comment line mistakenly removed in ProtobufRpcEngine. Contributed by Fengnan Li.	2019-03-11 18:59:15 -07:00
Konstantin V Shvachko	4ad295a4f1	HDFS-14270.[SBN Read] Add trace level logging for stateId in RPC Server. Contributed by Shweta Yakkali.	2019-03-11 13:48:06 -07:00
Steve Loughran	0cbe9ad8c2	HADOOP-16109. Parquet reading S3AFileSystem causes EOF Nobody gets seek right. No matter how many times they think they have. Reproducible test from: Dave Christianson Fixed seek() logic: Steve Loughran	2019-03-09 16:00:34 +00:00
Praveen Krishna	2b94e51a8f	HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. Author: Praveen Krishna <praveenkrishna@tutanota.com>	2019-03-07 11:06:34 +00:00
Vivek Ratnavel Subramanian	a55fc36299	HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values.	2019-03-06 17:43:57 -08:00
Sahil Takiar	618e009ac0	HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-03-06 15:04:06 -08:00
Wei-Chiu Chuang	6192c1fe3b	Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." This reverts commit `f5a4b43a49`.	2019-03-06 15:02:18 -08:00
Sahil Takiar	f5a4b43a49	HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-03-06 14:58:45 -08:00
Eric Yang	3c5b7136e2	HADOOP-16150. Added concat method to ChecksumFS as unsupported operation. Contributed by Steve Loughran (cherry picked from commit 8b517e7ad670aa8ee0b73ce5a572f36ce63eabee)	2019-03-05 13:32:00 -05:00
Stephen O'Donnell	686c0141ef	HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. Contributed by Stephen O'Donnell. Signed-off-by: Steve Loughran <stevel@apache.org>	2019-03-05 14:09:00 +00:00
Prabhu Joseph	e40e2d6ad5	YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-03-05 14:10:08 +09:00
coder-chenzhi	fe7551f21b	HADOOP-16162. Remove unused Job Summary Appender configurations from log4j.properties This closes #551 Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2019-03-05 13:41:33 +09:00
David Mollitor	9fcd89ab93	HADOOP-16148. Cleanup LineReader Unit Test. Contributed by David Mollitor. Signed-off-by: Steve Loughran <stevel@apache.org>	2019-03-04 23:08:12 +00:00
Ajay Kumar	0d61facd37	HADOOP-15889. Add hadoop.token configuration parameter to load tokens. Contributed by Íñigo Goiri	2019-02-28 10:34:28 -08:00
Márton Elek	84c4966a5a	HADOOP-16067. Incorrect Format Debug Statement KMSACLs. Contributed by Charan Hebri.	2019-02-28 12:15:47 +01:00
Eric Yang	feccd282fe	HADOOP-16107. Update ChecksumFileSystem createFile/openFile API to generate checksum. Contributed by Steve Loughran	2019-02-27 15:53:41 -05:00
Tsz Wo Nicholas Sze	9192f71e21	HADOOP-16127. In ipc.Client, put a new connection could happen after stop.	2019-02-26 15:14:21 -08:00
Abhishek Modi	52b2eab575	HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util. Contributed by Abhishek Modi	2019-02-26 17:10:41 +00:00
Surendra Singh Lilhore	59ba3552d3	HDFS-14299. ViewFs: Correct error message for read only operations. Contributed by hu xiaodong.	2019-02-26 12:40:32 +05:30
Konstantin V Shvachko	a6ab37192a	HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko.	2019-02-25 14:35:02 -08:00
Inigo Goiri	ba4e7bd192	HADOOP-16125. Support multiple bind users in LdapGroupsMapping. Contributed by Lukas Majercak.	2019-02-25 13:39:13 -08:00
Tsz Wo Nicholas Sze	0edb0c51dc	HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections.	2019-02-25 13:15:28 -08:00
Yongjun Zhang	f7a27cdee4	HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.	2019-02-23 09:35:36 -08:00
George Huang	9daf43c6fa	HADOOP-16129. Misc. bug fixes for KMS Benchmark. Contributed by George Huang. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-02-22 17:52:09 -08:00
Daryn Sharp	a87e458432	HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-02-20 18:17:04 -08:00
Shweta Yakkali	371a6db59a	HDFS-14273. Fix checkstyle issues in BlockLocation's method javadoc (Contributed by Shweta Yakkali via Daniel Templeton) Change-Id: I546aa4a0fe7f83b53735acd9925f366b2f1a00e2	2019-02-20 15:36:53 -08:00
George Huang	0525d85d57	HADOOP-15967. KMS Benchmark Tool. Contributed by George Huang. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-02-19 15:24:30 -08:00
Akira Ajisaka	dabfeab785	YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg.	2019-02-15 14:49:49 +09:00
Erik Krogen	64f28f9efa	HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.	2019-02-14 11:22:04 -08:00
Siyao Meng	fd026863b1	HDFS-14241. Provide feedback on successful renameSnapshot and deleteSnapshot. Contributed by Siyao Meng. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-02-13 19:48:02 -08:00
Chen Liang	024c87291c	HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang	2019-02-13 12:40:31 -08:00
Vinayakumar B	00c5ffaee2	HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena.	2019-02-13 16:44:32 +05:30
Yiqun Lin	7b11b404a3	HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen.	2019-02-13 11:16:04 +08:00
Xiaoyu Yao	ca4e46a05e	HDDS-1075. Fix CertificateUtil#parseRSAPublicKey charsetName. Contributed by Siddharth Wagle.	2019-02-11 12:09:14 -08:00
Steve Loughran	668817a6ce	Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." (accidentally mixed in two patches) This reverts commit `fa8cd1bf28`.	2019-02-07 21:57:22 +00:00
Wangda Tan	308f3168fa	Make upstream aware of 3.1.2 release Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365	2019-02-05 14:03:18 -08:00
Da Zhou	fa8cd1bf28	HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. Contributed by Da Zhou and Junhua Gu.	2019-02-05 19:23:15 +00:00
Steve Loughran	f365957c63	HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API. The new openFile() API is asynchronous, and implemented across FileSystem and FileContext. The MapReduce V2 inputs are moved to this API, and you can actually set must/may options to pass in. This is more useful for setting things like s3a seek policy than for S3 select, as the existing input format/record readers can't handle S3 select output where the stream is shorter than the file length, and splitting plain text is suboptimal. Future work is needed there. In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific configuration parameters which can be set in jobs and used to set filesystem input stream options (seek policy, retry, encryption secrets, etc). Contributed by Steve Loughran	2019-02-05 11:51:02 +00:00
Steve Loughran	7f46d13dac	HADOOP-16079. Token.toString faulting if any token listed can't load. Contributed by Steve Loughran.	2019-02-01 14:31:47 +00:00
Inigo Goiri	bcc3a79f58	HADOOP-16084. Fix the comment for getClass in Configuration. Contributed by Fengnan Li.	2019-01-31 10:06:05 -08:00
Akira Ajisaka	1129288cf5	HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.	2019-01-29 18:29:56 -08:00
Hanisha Koneru	b3bc94ebfd	HDFS-14236. Lazy persist copy/ put fails with ViewFs.	2019-01-29 16:45:44 -08:00
Wei-Chiu Chuang	d1714c20e9	Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." This reverts commit `1d523279da`.	2019-01-29 15:43:09 -08:00
Pranay Singh	1d523279da	HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-01-25 09:02:57 -08:00
Eric Yang	0dd35e218f	HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. Contributed by He Xiaoqiao	2019-01-22 18:59:36 -05:00
Akira Ajisaka	a463cf75a0	HADOOP-15787. [JDK11] TestIPC.testRTEDuringConnectionSetup fails. Contributed by Zsolt Venczel.	2019-01-22 10:19:05 +09:00
Sunil G	2e2508b8e3	Make 3.2.0 aware to other branches	2019-01-21 21:24:51 +05:30
Xiaoyu Yao	751bc62df7	Merge branch 'HDDS-4' into trunk	2019-01-18 14:44:24 -08:00
Surendra Singh Lilhore	dacc1a759e	HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena.	2019-01-18 13:55:12 +05:30
Xiaoyu Yao	570b503e3e	Fix merge conflicts	2019-01-15 22:54:50 -07:00
Xiaoyu Yao	ff61931f91	HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.	2019-01-15 22:16:58 -07:00
Akira Ajisaka	01405df87f	HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma.	2019-01-16 10:28:08 +09:00
luhuachao	ccc43621ab	HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. Contributed by luhuachao.	2019-01-15 13:15:18 +00:00
Steve Loughran	6d0bffe17e	HADOOP-14556. S3A to support Delegation Tokens. Contributed by Steve Loughran and Daryn Sharp.	2019-01-14 17:59:27 +00:00
Inigo Goiri	35fa3bd685	HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak.	2019-01-11 18:07:19 -08:00
Chen Liang	bf08f4abae	HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian.	2019-01-11 14:01:23 -08:00
Chen Liang	01cb958af4	HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen.	2019-01-11 12:51:07 -08:00
Giovanni Matteo Fumarola	fb8932a727	HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.	2019-01-11 10:54:49 -08:00
Akira Ajisaka	d4ca907da6	HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper	2019-01-11 11:24:05 +09:00
Kihwal Lee	33c009a45d	HADOOP-15909. KeyProvider class should implement Closeable. Contributed by Kuhu Shukla.	2019-01-10 13:21:37 -06:00
Jason Lowe	c634589ab2	Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." This reverts commit `ecdeaa7e6a`.	2019-01-09 17:24:58 -06:00
Takanobu Asanuma	709ddb10e5	HADOOP-15941. [JDK 11] Compilation failure: package com.sun.jndi.ldap is not visible.	2019-01-09 13:55:25 +09:00
shwetayakkali@cloudera.com	4ab5260b7e	HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped (Contributed by Shweta Yakkali via Daniel Templeton) Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8	2019-01-08 17:04:57 -08:00
Akira Ajisaka	7f78397036	Revert "HADOOP-14556. S3A to support Delegation Tokens." This reverts commit `d7152332b3`.	2019-01-08 14:51:30 +09:00
Eric Yang	802932ca0b	HADOOP-16031. Fixed TestSecureLogins unit test. Contributed by Akira Ajisaka	2019-01-07 13:21:14 -05:00
Steve Loughran	d7152332b3	HADOOP-14556. S3A to support Delegation Tokens. Contributed by Steve Loughran.	2019-01-07 13:18:03 +00:00
Eric Yang	d43af8b3db	HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. Contributed by Bolke de Bruin	2019-01-04 17:54:15 -05:00
Inigo Goiri	f4e18242bd	HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke.	2019-01-04 09:56:58 -08:00
Sunil G	51427cbdfb	HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang.	2019-01-04 21:49:06 +05:30
Pranay Singh	ecdeaa7e6a	HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2019-01-03 09:56:07 -08:00
Akira Ajisaka	57198bc511	HADOOP-15962. The buffer size is small when unpacking tar archives. Contributed by BELUGA BEHR.	2018-12-28 15:38:47 +09:00
Konstantin V Shvachko	bd2a59e508	HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.	2018-12-24 09:39:20 -08:00
Chao Sun	fa8550337d	HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.	2018-12-24 09:34:01 -08:00
Erik Krogen	ef3e1929dd	HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.	2018-12-24 09:34:01 -08:00
Chen Liang	cae279610b	HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng.	2018-12-24 09:34:01 -08:00
Erik Krogen	b73fb70f97	HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.	2018-12-24 09:34:01 -08:00
Chao Sun	cbc2f932eb	HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.	2018-12-24 09:34:01 -08:00
Chao Sun	8c49135078	HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.	2018-12-24 09:34:01 -08:00
Chen Liang	a65bb97f5d	HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.	2018-12-24 09:34:00 -08:00
Chen Liang	d2e85b0b6d	HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.	2018-12-24 09:34:00 -08:00
Chen Liang	66ec3a67e8	HDFS-13767. Add msync server implementation. Contributed by Chen Liang.	2018-12-24 09:34:00 -08:00
Plamen Jeliazkov	e27708c2da	HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov.	2018-12-24 09:34:00 -08:00
Chao Sun	64b7cf59bd	HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.	2018-12-24 09:33:59 -08:00
Erik Krogen	151c8ddbe4	HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.	2018-12-24 09:33:59 -08:00
Plamen Jeliazkov	e880660a20	HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.	2018-12-24 09:33:59 -08:00
Erik Krogen	091ad974cd	HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.	2018-12-24 09:33:59 -08:00
Erik Krogen	9bf0696c73	HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.	2018-12-24 09:33:59 -08:00
Plamen Jeliazkov	c9d73437e8	HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.	2018-12-24 09:33:59 -08:00
Yiqun Lin	feb2664ac4	HDFS-13946. Log longest FSN write/read lock held stack trace.	2018-12-22 23:09:59 +08:00
Surendra Singh Lilhore	763e96ea2b	HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan.	2018-12-22 12:22:24 +05:30
Bharat Viswanadham	2499435d9d	HADOOP-16014. Fix test, checkstyle and javadoc issues in TestKerberosAuthenticationHandler. Contributed by Dinesh Chitlangia.	2018-12-21 13:30:48 -08:00
Jason Lowe	3961690037	HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne	2018-12-20 09:29:48 -06:00
Takanobu Asanuma	d963575918	HDFS-13661. Ls command with e option fails when the filesystem is not HDFS.	2018-12-20 10:03:33 +09:00
Surendra Singh Lilhore	788e7473a4	HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.	2018-12-17 11:24:57 +05:30
Gabor Bota	1ea29b7385	HADOOP-16000. Remove TLSv1 and SSLv2Hello from the default value of hadoop.ssl.enabled.protocols Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2018-12-15 10:53:52 +09:00
Giovanni Matteo Fumarola	76efeacd5f	HADOOP-15995. Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping. Contributed by Lukas Majercak.	2018-12-12 11:10:24 -08:00
Steve Loughran	202926ac33	HADOOP-15808. Harden Token service loader use. Contributed by Steve Loughran.	2018-12-11 17:33:49 +00:00
Giovanni Matteo Fumarola	aa89492f29	Revert "HADOOP-15852. Refactor QuotaUsage. Contributed by Beluga Behr." This reverts commit `fb9deed41d`.	2018-12-04 12:57:28 -08:00
Steve Loughran	db2d8b01c6	HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. Contributed by Steve Loughran	2018-12-04 15:35:43 +00:00
Giovanni Matteo Fumarola	fb9deed41d	HADOOP-15852. Refactor QuotaUsage. Contributed by Beluga Behr.	2018-12-03 13:25:02 -08:00
Giovanni Matteo Fumarola	c9a3aa64dc	HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak.	2018-12-03 12:10:05 -08:00
Eric Yang	38ea3814bd	HADOOP-15922. Revert patch 004.	2018-11-30 12:49:46 -05:00
Akira Ajisaka	c9bfca217f	MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.	2018-11-30 13:13:23 +09:00
Steve Loughran	d0edd37269	HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" This reverts commit `829a2e4d27`.	2018-11-29 17:52:11 +00:00
Ewan Higgs	c1d24f8483	HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. Contributed by Ewan Higgs and Steve Loughran.	2018-11-29 15:12:17 +00:00
Eric Yang	a9d96948ed	HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. Contributed by He Xiaoqiao	2018-11-26 13:49:19 -05:00
Vinayakumar B	23b441c225	HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena.	2018-11-26 18:11:01 +05:30
Márton Elek	6934a65402	HADOOP-15358. SFTPConnectionPool connections leakage. Contributed by Mikhail Pryakhin.	2018-11-23 09:48:26 +01:00
Akira Ajisaka	1a00b4e325	Add 2.9.2 release notes and changes documents.	2018-11-20 14:25:18 +09:00
Inigo Goiri	460a94a10f	HDFS-14045. Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN. Contributed by Jiandan Yang.	2018-11-15 10:58:57 -08:00
Surendra Singh Lilhore	75291e6d53	HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore	2018-11-13 20:22:58 +05:30
Steve Loughran	3c9d97b8f7	HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. Contributed by LiXin Ge.	2018-11-12 12:58:05 +00:00
Giovanni Matteo Fumarola	89b49167a5	HADOOP-15903. Allow HttpServer2 to discover resources in /static when symlinks are used. Contributed by Inigo Goiri.	2018-11-08 14:52:24 -08:00
Jason Lowe	0b6625a973	MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan	2018-11-07 08:20:49 -06:00
Jason Lowe	ab6aa4c726	YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg	2018-11-06 08:40:59 -06:00
Lars Francke	cb8d679c95	HADOOP-15687. Credentials class should allow access to aliases. Author: Lars Francke <lars.francke@gmail.com>	2018-11-03 16:21:29 +00:00
Giovanni Matteo Fumarola	44e37b4fd9	HADOOP-15885. Add base64 (urlString) support to DTUtil. Contributed by Inigo Goiri.	2018-11-02 10:54:12 -07:00
Akira Ajisaka	f747f5b06c	HADOOP-15886. Fix findbugs warnings in RegistryDNS.java.	2018-10-31 10:01:31 +09:00
Steve Loughran	62d98ca92a	HADOOP-15855. Review hadoop credential doc, including object store details. Contributed by Steve Loughran.	2018-10-30 15:58:04 +00:00
Wei-Chiu Chuang	63e7134df6	Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." This reverts commit `fb2b72e6fc`.	2018-10-28 17:04:59 -07:00
Hrishikesh Gadre	bfb9adc2b9	HADOOP-9567. Provide auto-renewal for keytab based logins. Contributed by Hrishikesh Gadre, Gary Helmling and Harsh J. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2018-10-27 08:59:47 -07:00
Wei-Chiu Chuang	fb2b72e6fc	HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao.	2018-10-25 09:33:31 -07:00
Akira Ajisaka	8f97d6f2cd	HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe.	2018-10-24 13:49:28 +09:00
Haibo Chen	823bb5dda8	MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)	2018-10-23 15:28:13 -07:00
Chen Liang	635786a511	HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.	2018-10-23 14:53:45 -07:00
Inigo Goiri	dd268a64d3	Revert "HADOOP-15836. Review of AccessControlList. Contributed by BELUGA BEHR." This reverts commit `00254d7b8c`.	2018-10-23 09:23:03 -07:00
Steve Loughran	52cb766ad0	HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility. Contributed by Wei-Chiu Chuang.	2018-10-22 11:21:12 +01:00
Eric Yang	a043dfa216	HADOOP-15821. Addendum patch to fix Hadoop Registry parent project. Contributed by Íñigo Goiri	2018-10-21 12:50:10 -04:00
Eric Yang	e2a9fa8448	HADOOP-15821. Move YARN Registry to Hadoop Registry. Contributed by Íñigo Goiri	2018-10-19 19:46:48 -04:00
Inigo Goiri	00254d7b8c	HADOOP-15836. Review of AccessControlList. Contributed by BELUGA BEHR.	2018-10-19 13:57:43 -07:00
Wei-Chiu Chuang	cd2158456d	HADOOP-15418. Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException. Contributed by lqjack and Suma Shivaprasad	2018-10-18 17:28:28 -07:00
Jitendra Pandey	d93d515af5	HDDS-651. Rename o3 to o3fs for Filesystem.	2018-10-17 14:19:17 -07:00
Jason Lowe	9abda83947	HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe	2018-10-17 14:38:42 -05:00
Adam Antal	24dc068a36	HADOOP-11100. Support to configure ftpClient.setControlKeepAliveTimeout. Contributed by Adam Antal. Signed-off-by: Xiao Chen <xiao@apache.org>	2018-10-17 11:34:50 -07:00
Steve Loughran	41b3603b5b	HADOOP-15861. Move DelegationTokenIssuer to the right path. Contributed by Wei-Chiu Chuang.	2018-10-17 11:01:53 +01:00
Steve Loughran	b738cb148c	HADOOP-15854. AuthToken Use StringBuilder instead of StringBuffer. Contributed by Beluga Behr.	2018-10-17 10:29:09 +01:00
Haibo Chen	c2288ac45b	YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter)	2018-10-16 13:36:26 -07:00
Robert Kanter	7fe1a40a6b	HADOOP-15853. TestConfigurationDeprecation leaves behind a temp file, resulting in a license issue (ayushtkn via rkanter)	2018-10-15 15:02:37 -07:00
Xiao Chen	b6fc72a025	HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.	2018-10-15 10:51:55 -07:00
Anu Engineer	3c1fe073d2	HDDS-445. Create a logger to print out all of the incoming requests. Contributed by Bharat Viswanadham.	2018-10-12 16:27:54 -07:00
Xiao Chen	5ec86b445c	HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.	2018-10-12 09:35:52 -07:00
Robert Kanter	d91d47bc73	HADOOP-15717. TGT renewal thread does not log IOException (snemeth via rkanter)	2018-10-11 15:35:44 -07:00
Xiao Chen	64f2b32d57	HADOOP-15676. Cleanup TestSSLHttpServer. Contributed by Szilard Nemeth.	2018-10-11 15:12:36 -07:00
Takanobu Asanuma	7b57f2f71f	HADOOP-15785. [JDK10] Javadoc build fails on JDK 10 in hadoop-common. Contributed by Dinesh Chitlangia.	2018-10-11 13:51:51 +09:00
Robert Kanter	f261c31937	HADOOP-15708. Reading values from Configuration before adding deprecations make it impossible to read value with deprecated key (zsiegl via rkanter)	2018-10-10 18:51:37 -07:00
Steve Loughran	cdc4350718	HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys. Contributed by Steve Loughran.	2018-10-10 17:00:37 +01:00
Akira Ajisaka	6fa3feb577	HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.	2018-10-10 10:16:57 +09:00
Eric Yang	bf04f19456	YARN-8845. Removed unused hadoop.registry.rm reference. Contributed by Íñigo Goiri	2018-10-09 15:54:45 -04:00
Takanobu Asanuma	9bbeb52486	HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. Contributed by Akira Ajisaka.	2018-10-09 13:06:14 +09:00
Vidura Mudalige	a30b1d1824	HADOOP-15818. Fix deprecated maven-surefire-plugin configuration in hadoop-kms module This closes #425 Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2018-10-09 12:55:34 +09:00
Jason Lowe	f13e231025	HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe	2018-10-05 09:06:02 -05:00
Yiqun Lin	81f635f47f	HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.	2018-10-04 10:30:30 +08:00

... 7 8 9 10 11 ...

5748 Commits