big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,829 Commits 2 Branches 2 Tags 264 MiB
ec913398a9
Commit Graph

23829 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
avijayanhwx
4b099b8b89 HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM. (#819) 2019-05-23 15:34:55 -07:00
Giovanni Matteo Fumarola
6a0e7dd454 YARN-9482. DistributedShell job with localization fails in unsecure cluster. Contributed by Prabhu Joseph. 2019-05-23 13:27:52 -07:00
Arpit Agarwal
b4b9120e28
HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka (#832) 
* HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka

* Fixed checkstyle issues.

* verifyContainerData also does fixup, renamed. Added a Javadoc comment, both as per review discussion

* fixed merge error. adapted to new signature of BlockUtils getDB

* fixed checkstyle issue post merge
 2019-05-23 10:42:08 -07:00
Xiaoyu Yao
20a4ec351c HDDS-700. Support rack awared node placement policy based on network topology. Contributed by Sammi Chen. 2019-05-23 10:40:15 -07:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Igor Rudenko
e2b79912c3 YARN-9469. Fix typo in YarnConfiguration. 2019-05-23 19:00:38 +02:00
Eric Yang
7b03072fd4 YARN-9080. Added clean up of bucket directories. 
Contributed by Prabhu Joseph, Peter Bacsko, Szilard Nemeth
 2019-05-23 12:08:44 -04:00
Eric Yang
ea0b1d8fba HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation. 
Contributed by Prabhu Joseph
 2019-05-23 11:36:32 -04:00
sdeka
ca93760504 fixed checkstyle issue post merge 2019-05-23 20:49:52 +05:30
sdeka
72bef0f6cb fixed merge error. adapted to new signature of BlockUtils getDB 2019-05-23 11:36:40 +05:30
Akira Ajisaka
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang. 2019-05-23 13:59:42 +09:00
supratimdeka
9da62f33be
Merge branch 'trunk' into HDDS-1535 2019-05-23 09:51:33 +05:30
Zhankun Tang
03aa70fe19 SUBMARINE-83. Refine the documents of submarine targeting 0.2.0 release. Contributed by Zhankun Tang. 2019-05-23 10:02:00 +08:00
Akira Ajisaka
5565f2c532
MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work. 2019-05-23 10:21:11 +09:00
Ajay Yadav
9c61494c02 HDDS-1065. OM and DN should persist SCM certificate as the trust root. Contributed by Ajay Kumar. (#834) 2019-05-22 11:47:32 -07:00
Shashikant Banerjee
a315913c48 HDDS-1517. AllocateBlock call fails with ContainerNotFoundException (#826). Contributed by Shashikant Banerjee. 2019-05-22 17:24:27 +05:30
Mukul Kumar Singh
2fc6f8599a HDDS-1449. JVM Exit in datanode while committing a key. Contributed by Mukul Kumar Singh. (#825) 2019-05-22 17:18:40 +05:30
Wanqiang Ji
67f9a7b165
MAPREDUCE-7205. Treat container scheduler kill exit code as a task attempt killing event. 
This closes #821

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-05-22 14:23:13 +09:00
Siyao Meng
9dff6eff81 HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-22 13:22:26 +09:00
sdeka
456bb8a3e0 verifyContainerData also does fixup, renamed. Added a Javadoc comment, both as per review discussion 2019-05-22 09:34:21 +05:30
Bharat Viswanadham
77c49f2963
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. (#714) 2019-05-21 18:41:30 -07:00
Ajay Yadav
ef1cc725b8
HDDS-1538. Update ozone protobuf message for ACLs. Contributed by Ajay Kumar. (#828) 2019-05-21 15:53:40 -07:00
Giovanni Matteo Fumarola
a85451c8d2 YARN-9575. Fix TestYarnConfigurationFields testcase failing. Contributed by Prabhu Joseph. 2019-05-21 11:25:30 -07:00
Sahil Takiar
5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ajay Kumar
ff5691ca06
HDDS-1442. add spark container to ozonesecure-mr compose files 
Closes #746
 2019-05-21 17:10:46 +02:00
Lokesh Jain
c1d7d68c78
HDDS-1461. Optimize listStatus api in OzoneFileSystem (#782) 2019-05-21 14:47:48 +05:30
Inigo Goiri
1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Vivek Ratnavel Subramanian
05db2a598e HDDS-1487. Bootstrap React framework for Recon UI (#831) 
HDDS-1487. Bootstrap React framework for Recon UI.
Contributed by Vivek Ratnavel Subramanian.
 2019-05-20 22:09:58 +02:00
avijayanhwx
f63300228e HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not lock protected. (#799) 
* HDDS-1451 : SCMBlockManager findPipeline and createPipeline are not lock protected.

* HDDS-1451 : Address review comments.
 2019-05-20 22:02:56 +02:00
Sunil G
24c53e057a YARN-9546. Add configuration option for YARN Native services AM classpath. Contributed by Gergely Pollak. 2019-05-20 10:53:01 -04:00
sdeka
64c39856d2 Fixed checkstyle issues. 2019-05-20 08:13:37 +05:30
Bharat Viswanadham
0d1d7c86ec HDDS-1499. OzoneManager Cache. (#798) 2019-05-19 19:23:02 -07:00
Ben Roling
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. 
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
 2019-05-19 22:29:54 +01:00
Zhankun Tang
729ccb2cab SUBMARINE-58. Submarine client needs to generate fat jar. Contributed by Zac Zhou. 2019-05-19 21:18:33 +08:00
sdeka
869a1ab41a HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka 2019-05-18 16:16:33 +05:30
Arpit Agarwal
732133cb2a HDDS-1422. Exception during DataNode shutdown. Contributed by Arpit A… (#725) 
* HDDS-1422. Exception during DataNode shutdown. Contributed by Arpit Agarwal.

Change-Id: I6db6bdd19839a45e5341ed7e745cd38f68af8378

* Suppress spurious findbugs warning.

* Remove log file that got committed in error
 2019-05-18 11:47:19 +02:00
Giovanni Matteo Fumarola
12c81610e0 YARN-9505. Add container allocation latency for Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-05-17 12:03:21 -07:00
Srinivasu Majeti
3e5e5b028a HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-17 19:20:28 +02:00
Alexis Daboville
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,
 2019-05-17 15:15:22 +01:00
Siddharth
c183bd8e20 HDDS-1527. HDDS Datanode start fails due to datanode.id file read error. Contributed by Siddharth Wagle. 
This closes #822.
 2019-05-16 15:13:10 -07:00
Eric Yang
fab5b80a36 YARN-9554. Fixed TimelineEntity DAO serialization handling. 
Contributed by Prabhu Joseph
 2019-05-16 16:39:50 -04:00
Márton Elek
03ea8ea92e
HDDS-1297. Fix IllegalArgumentException thrown with MiniOzoneCluster Initialization. Contributed by Yiqun Lin. 2019-05-16 17:04:46 +02:00
Márton Elek
9248b7d58b
HDDS-1284. Adjust default values of pipline recovery for more resilient service restart. 2019-05-16 17:04:46 +02:00
Sahil Takiar
b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)
 2019-05-16 08:57:54 -06:00
Elek, Márton
b7de735159 HDDS-1522. Provide intellij runConfiguration for Ozone components. Contributed by Elek, Marton. 2019-05-16 16:38:50 +02:00
Andrew Olson
c15b3bca86
HADOOP-16294: Enable access to input options by DistCp subclasses. 
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
 2019-05-16 16:11:12 +02:00
David Mollitor
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor
 2019-05-16 16:02:07 +02:00
Shashikant Banerjee
e66ecc564a HDDS-1531. Disable the sync flag by default during chunk writes in Datanode (#820). Contributed by Shashikant Banerjee. 2019-05-16 18:38:52 +05:30
Sunil G
de01422c2e SUBMARINE-56. Update documentation to describe single-node PyTorch integration. Contributed by Szilard Nemeth. 2019-05-15 21:26:48 -07:00
Bharat Viswanadham
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 2019-05-15 17:41:36 -07:00