big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,915 Commits 2 Branches 2 Tags 264 MiB
dd0834696a
Commit Graph

5206 Commits

Author SHA1 Message Date
Yuval Degani
dd0834696a HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark (#1463) 
In the existing implementation, the ValueQueue::getAtMost() method will only trigger a refill on a key queue if it has gone empty, instead of triggering a refill when it has gone below the watermark. Revise the test suite to correctly verify this behavior.
 2019-09-20 09:55:48 -07:00
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Takanobu Asanuma
298f07abce HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao. 2019-09-20 13:55:34 +09:00
Malcolm Taylor
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM. 
Contributed by Malcolm Taylor.

Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
 2019-09-19 16:00:05 +01:00
Kihwal Lee
d4205dce17 HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee 2019-09-19 08:23:35 -05:00
Weiwei Yang
f8c14326ee YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 2019-09-18 09:12:28 +08:00
Sahil Takiar
3cf6e4272f HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-17 15:51:39 -07:00
Sahil Takiar
55ce454ce4
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8. 
Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common
as the DelegatingSSLSocketFactory and binds the S3A connector to it so that
it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd
 2019-09-17 11:32:03 +01:00
Steve Loughran
9221704f85
HADOOP-16490. Avoid/handle cached 404s during S3A file creation. 
Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,
so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile
in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files
for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even
after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy
independently configurable from other retry policies; it is also exponential, but with
different parameters. This is because every HEAD request will refresh any 404 cached in
the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between
attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s
fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught
so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:
it will fail fast with a different error message. However, because create(path, overwrite=false) no
longer does HEAD path, we can at least be confident that S3A itself is not creating those cached
404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d
 2019-09-11 16:46:25 +01:00
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Sneha Vijayarajan
147f98629c
HADOOP-16438. ADLS Gen1 OpenSSL config control. 
Contributed by Sneha Vijayarajan.

Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d
 2019-09-09 17:09:32 +01:00
Jungtaek Lim (HeartSaVioR)
bb0b922a71
HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) 
Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
 2019-09-06 21:53:00 +01:00
Erik Krogen
a23417533e HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang. 2019-09-06 10:28:21 -07:00
Erik Krogen
c92a3e94d8 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2019-09-06 10:22:28 -07:00
Steve Loughran
511df1e837 HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions 
Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
 2019-09-05 14:25:15 +01:00
Erik Krogen
337e9b794d HADOOP-16268. Allow StandbyException to be thrown as CallQueueOverflowException when RPC call queue is filled. Contributed by CR Hota. 2019-09-04 08:22:02 -07:00
Surendra Singh Lilhore
5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
Stephen O'Donnell
915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-02 09:47:04 -07:00
Akira Ajisaka
dc0acceabb
YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko. 2019-08-30 10:13:10 +09:00
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
Akira Ajisaka
55cc115878 HADOOP-16527. Add a whitelist of endpoints to skip Kerberos authentication (#1336) Contributed by Akira Ajisaka. 2019-08-28 14:28:41 +09:00
Steve Loughran
61b2df2331
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper. 
Contributed by Steve Loughran.

Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.

Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8
 2019-08-22 17:27:56 +01:00
Akira Ajisaka
30ce8546f1
HADOOP-16496. Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop common. 
This closes #1317

Reviewed-by: Bharat Viswanadham <bharat@apache.org>
 2019-08-21 10:10:11 +09:00
Wei-Chiu Chuang
51b65370b9 HADOOP-14784. [KMS] Improve KeyAuthorizationKeyProvider#toString(). Contributed by Yeliang Cang. 
Reviewed-by: Dinesh Chitlangia <dchitlangia@cloudera.com>
 2019-08-19 11:12:09 -07:00
David Mollitor
a707bb7c1b HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 18:44:35 -07:00
Wei-Chiu Chuang
971a4c8e83 HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
 2019-08-16 17:19:58 -07:00
Erik Krogen
e356e4f4b7 HADOOP-16391 Add a prefix to the metric names for MutableRatesWithAggregation used for deferred RPC metrics to avoid collision with non-deferred metrics. Contributed by Bilwa S T. 2019-08-16 09:01:44 -07:00
Wei-Chiu Chuang
5882cf94ea HADOOP-16504. Increase ipc.server.listen.queue.size default from 128 to 256. Contributed by Lisheng Sun. 2019-08-15 15:20:54 -07:00
Akira Ajisaka
0f8add8a60
HADOOP-16495. Fix invalid metric types in PrometheusMetricsSink (#1244) 2019-08-14 12:24:03 +09:00
Masatake Iwasaki
da0006fe04 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-14 08:39:40 +09:00
Inigo Goiri
6b4564f1d5 HADOOP-16453. Update how exceptions are handled in NetUtils. Contributed by Lisheng Sun. 2019-08-11 20:34:36 -07:00
Steve Loughran
e25a5c2eab HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by Steve Loughran. 2019-08-09 15:52:37 +02:00
Szilard Nemeth
df30d8ea09 YARN-9727: Allowed Origin pattern is discouraged if regex contains *. Contributed by Zoltan Siegl 2019-08-09 09:34:23 +02:00
Zsombor Gegesy
b0131bc265 HADOOP-15014. Addendum: KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-07 20:57:42 -07:00
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Wei-Chiu Chuang
8cef9f89f4 HDFS-14652. Addendum: HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-06 15:24:11 -07:00
Eric Yang
22430c10e2 HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager. 
Contributed by Prabhu Joseph
 2019-08-06 17:04:17 -04:00
Jianfei Jiang
71aad60e51
HDFS-14691. Wrong usage hint for hadoop fs command "test". 
Contributed by Jianfei Jiang.

Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
 2019-08-05 13:08:47 +01:00
Wei-Chiu Chuang
61180f4656 HADOOP-15942. Change the logging level form DEBUG to ERROR for RuntimeErrorException in JMXJsonServlet. Contributed by Anuhan Torgonshar. 2019-08-02 14:58:24 -07:00
Wei-Chiu Chuang
797d14e816 Revert "HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm." 
This reverts commit 076618677d.
 2019-08-02 08:25:41 -07:00
Wei-Chiu Chuang
e872ceb810 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 2019-08-01 19:56:51 -07:00
Wei-Chiu Chuang
e20b19543b HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat. 2019-08-01 17:35:31 -07:00
Wei-Chiu Chuang
d086d058d8 HDFS-14652. HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-01 16:13:10 -07:00
Wei-Chiu Chuang
b94eba9f11 HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 2019-08-01 15:50:43 -07:00
He Xiaoqiao
f86de6f76a HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-01 14:54:39 -07:00
Akira Ajisaka
8bda91d20a
HADOOP-16398. Exports Hadoop metrics to Prometheus (#1170) 2019-07-31 10:11:36 -07:00
Wei-Chiu Chuang
0f2dad6679 HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina. 2019-07-30 16:52:42 -07:00
Siyao Meng
c75f16db79 HADOOP-16452. Increase ipc.maximum.data.length default from 64MB to 128MB. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 11:22:45 -07:00
Don Jeba
204a977f55
HADOOP-15910. Fix Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS 
Contributed by Don Jeba.

Change-Id: I2755bfb1263fc659078a1af8f0bdfd739fd1ae40
 2019-07-30 12:39:48 +01:00
Akira Ajisaka
cbfa3f3e98
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 2019-07-29 17:37:26 -07:00