big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
24,408 Commits 2 Branches 2 Tags
Commit Graph

24408 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fbbcad8618f99abf7130e53f98879a33)
 2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f0004dd64a7143d1c400de1361b13f3)
 2021-02-18 20:32:32 +05:30
Inigo Goiri
5cb9657320 YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 96bbc3bc972619bd830b2f935c06a1585a5470c6)
 2021-02-18 20:29:57 +05:30
Inigo Goiri
5be3a1dc7b YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 9cbd0cd2a9268ff2e8fed0af335e9c4f91c5f601)
 2021-02-18 20:29:42 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edfe63355ec38ffdaacc0c27d14cad31c)
 2021-02-18 20:23:56 +05:30
Prabhu Joseph
72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0beb2748988b339a851a129b5e57f8c4)
 2021-02-18 14:25:16 +05:30
Prabhu Joseph
0c46ab51b5 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850449c51a13f3a15fe0d756fdf50b4b2)
 2021-02-18 13:59:50 +05:30
Steve Loughran
4423a7e736
HADOOP-16906. Abortable (#2684) 
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.

Change-Id: I7fcc25e9dd8c10ce6c29f383529f3a2642a201ae
 2021-02-17 11:29:19 +00:00
Takanobu Asanuma
73f9083f2a
YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 
(cherry picked from commit 658b1a6cde6834be41de8745684b9c40f8b9c12c)

 Conflicts:
	.gitignore
 2021-02-17 10:28:17 +09:00
Jim Brennan
457c8404b9 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit 3fcc7a55b6ee8f93ff01061aec0aefc81d98eaab)
 2021-02-16 16:47:46 +00:00
Akira Ajisaka
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b173381204bd8d2c984f19c52104a419322)
 2021-02-16 14:52:31 +09:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd27c69b86b837e8da320bad0031f7895)
 2021-02-15 17:32:14 +00:00
Masatake Iwasaki
4468378e4b YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 
(cherry picked from commit f9a073c6c186848e09e2ee04118fd996ea8ace59)
 2021-02-11 20:26:09 +00:00
Akira Ajisaka
98eec71218
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee1e9e2b9d78b5be31466d4d640768c5)
 2021-02-11 08:07:55 +09:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
Eric Badger
37971c71d1 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654bb37dc1d453bfe15c3e75c274bbfb02)
 2021-02-09 19:03:20 +00:00
Jim Brennan
62389a5a04 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 22:16:36 +00:00
Jim Brennan
d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af4368cd4fb1d630f84d0d0c5cc11b224f)
 2021-02-05 17:37:04 +00:00
Akira Ajisaka
bdd22b61c2
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546) 
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 52217fce3d2a553c8850b7a1886340ae5a092436)
 2021-02-05 16:15:06 +09:00
fermi
ae80fc2477 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c8932c54ced1f0d4b892a54d22d7cfaf3)
 2021-02-05 11:05:18 +08:00
bilaharith
35c93ef5f3
HADOOP-17475. ABFS : add high performance listStatusIterator (#2548) 
The ABFS connector now implements listStatusIterator() with
asynchronous prefetching of the next page(s) of results.
For listing large directories this can provide tangible speedups.

If for any reason this needs to be disabled, set
fs.azure.enable.abfslistiterator to false.

Contributed by Bilahari T H.

Change-Id: Ic9a52b80df1d0ffed4c81beae92c136e2a12698c
 2021-02-04 13:37:36 +00:00
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc02af81e89e9fb96691f6cdf19ceff5)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc73cac4764149fcc9c7d94c1541f89c)
 2021-02-03 08:58:10 -06:00
Steve Loughran
70411cb1f1
HADOOP-17337. S3A NetworkBinding has a runtime dependency on shaded httpclient. (#2599) 
Contributed by Steve Loughran.

Change-Id: I0471322fc88d8bc3896ac439aefb31e6a856936c
 2021-02-03 14:32:55 +00:00
Steve Loughran
99337a4dd0
HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException. (#2648) 
When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.

Change-Id: I519d50ccd657968fd8ee72d132518099de901e15
 2021-02-02 18:17:38 +00:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e204d105fb05807d33387a048e9ddb762)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
f97709beaa
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:52:08 +08:00
Gautham B A
58a9640f8e HADOOP-17508. Simplify deps install instructions (#2664) 
(cherry picked from commit a163aa739482d05a2e82f84d7b4f9b562ed1ab99)

 Conflicts:
	BUILDING.txt
 2021-02-02 05:37:19 +00:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Akira Ajisaka
98104537b2
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc054843234e9a3c4c27748f87582807f)
 2021-02-01 18:52:50 +09:00
Gautham B A
25677df019 HADOOP-17506. Fix typo in BUILDING.txt (#2662) 
(cherry picked from commit 9801b78edf25289d02194d649bb5f679bd35600c)
 2021-01-31 14:23:28 +00:00
He Xiaoqiao
f61e3f5599
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 19:46:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939332c2b17a0f470ac650ab55c6c0f64)
 2021-01-28 18:19:21 +00:00
Jim Brennan
2c70705575 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f670c181d1fda99e76729e920fff287e)
 2021-01-28 16:51:40 +00:00
Jungtaek Lim
ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624) 
Contributed by Jungtaek Lim.

Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
 2021-01-27 19:08:54 +00:00
Steve Loughran
2d124f2f5e HADOOP-17483. Magic committer is enabled by default. (#2656) 
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
 2021-01-27 19:05:07 +00:00
Steve Loughran
3e1eb16837
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649) 
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.

Change-Id: I16db6e788c9fd628d3295671d7c2861c249d5ef1
 2021-01-27 16:40:27 +00:00
Steve Loughran
fb603e81f0
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530) 
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.

Change-Id: I4fe75b057159e35aacc072da3cb7343467c0c3f1
 2021-01-26 19:42:16 +00:00
Steve Loughran
bd85f6acea
HADOOP-17480. Document that AWS S3 is consistent and that S3Guard is not needed (#2636) 
Contributed by Steve Loughran.

Change-Id: I775e3ee7b60665240ec621859c337b053f747a49
 2021-01-25 13:24:34 +00:00
Akira Ajisaka
886b245de6
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:57:22 +09:00
Akira Ajisaka
7106a789f4
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6baf2fa9071729cac57516d366e928a9f)
 2021-01-24 22:52:51 +09:00
Anton Kutuzov
dcf6d77279 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:09:57 -08:00
Mehakmeet Singh
d20b2deac3
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604) 
Contributed by Mehakmeet Singh.

Change-Id: I3445dec84b9b9e43bb1e41f709944ea05416bd74
 2021-01-22 14:21:31 +00:00
Sneha Vijayarajan
4865589bb4 HADOOP-17404. ABFS: Small write - Merge append and flush 
- Contributed by Sneha Vijayarajan

(cherry picked from commit b612c310c26394aa406c99d8598c9cb7621df052)
 2021-01-22 10:48:04 +00:00
bilaharith
cb6729224e HADOOP-17347. ABFS: Read optimizations 
- Contributed by Bilahari T H

(cherry picked from commit 1448add08fcd4a23e59eab5f75ef46fca6b1c3d1)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
f3a0ca66c2 HADOOP-17407. ABFS: Fix NPE on delete idempotency flow 
- Contributed by Sneha Vijayarajan

(cherry picked from commit 5ca1ea89b3f57017768ae4d8002f353e3d240e07)
 2021-01-22 10:48:04 +00:00
Sumangala
5f312a0d85 HADOOP-17422: ABFS: Set default ListMaxResults to max server limit (#2535) 
Contributed by Sumangala Patki

TEST RESULTS:

namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141

(cherry picked from commit a35fc3871b01d8a3a375f3ae0e330b55a1d9009f)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
d3caa1552b Hadoop-17413. Release elastic byte buffer pool at close 
- Contributed by Sneha Vijayarajan

(cherry picked from commit 5bf977e6b16287d7d140dd96dad66d0fce213954)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
a44890eb63 HADOOP-17296. ABFS: Force reads to be always of buffer size. 
Contributed by Sneha Vijayarajan.

(cherry picked from commit 142941b96e221fc1b4524476ce445714d7f6eec3)
 2021-01-22 10:48:04 +00:00