big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,214 Commits 2 Branches 2 Tags 264 MiB
ee0e32abdc
Commit Graph

24214 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
 2021-02-22 15:24:37 +08:00
Akira Ajisaka
ff8b79f890
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.3) (#2709) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-02-22 15:19:05 +09:00
zhaorenhai
c7bab342ee
HADOOP-17084 Update Dockerfile_aarch64 to use Bionic (#2103). Contributed by zhaorenhai. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 6c57be4897)
 2021-02-22 13:49:51 +09:00
Akira Ajisaka
118a5747f8
HADOOP-16054. Update Dockerfile to use Bionic (#1966) 
(cherry picked from commit 81d8b71534)
 2021-02-22 13:49:29 +09:00
Eric Badger
27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:25:25 +00:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a9)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fb)
 2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f)
 2021-02-18 20:32:32 +05:30
Inigo Goiri
5cb9657320 YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 96bbc3bc97)
 2021-02-18 20:29:57 +05:30
Inigo Goiri
5be3a1dc7b YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 
(cherry picked from commit 9cbd0cd2a9)
 2021-02-18 20:29:42 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edf)
 2021-02-18 20:23:56 +05:30
Prabhu Joseph
72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0b)
 2021-02-18 14:25:16 +05:30
Prabhu Joseph
0c46ab51b5 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850)
 2021-02-18 13:59:50 +05:30
Steve Loughran
4423a7e736
HADOOP-16906. Abortable (#2684) 
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.

Change-Id: I7fcc25e9dd8c10ce6c29f383529f3a2642a201ae
 2021-02-17 11:29:19 +00:00
Takanobu Asanuma
73f9083f2a
YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 
(cherry picked from commit 658b1a6cde)

 Conflicts:
	.gitignore
 2021-02-17 10:28:17 +09:00
Jim Brennan
457c8404b9 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit 3fcc7a55b6)
 2021-02-16 16:47:46 +00:00
Akira Ajisaka
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b1733)
 2021-02-16 14:52:31 +09:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:32:14 +00:00
Masatake Iwasaki
4468378e4b YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 
(cherry picked from commit f9a073c6c1)
 2021-02-11 20:26:09 +00:00
Akira Ajisaka
98eec71218
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee)
 2021-02-11 08:07:55 +09:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
Eric Badger
37971c71d1 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654b)
 2021-02-09 19:03:20 +00:00
Jim Brennan
62389a5a04 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 22:16:36 +00:00
Jim Brennan
d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af43)
 2021-02-05 17:37:04 +00:00
Akira Ajisaka
bdd22b61c2
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546) 
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 52217fce3d)
 2021-02-05 16:15:06 +09:00
fermi
ae80fc2477 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:05:18 +08:00
bilaharith
35c93ef5f3
HADOOP-17475. ABFS : add high performance listStatusIterator (#2548) 
The ABFS connector now implements listStatusIterator() with
asynchronous prefetching of the next page(s) of results.
For listing large directories this can provide tangible speedups.

If for any reason this needs to be disabled, set
fs.azure.enable.abfslistiterator to false.

Contributed by Bilahari T H.

Change-Id: Ic9a52b80df1d0ffed4c81beae92c136e2a12698c
 2021-02-04 13:37:36 +00:00
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc)
 2021-02-03 08:58:10 -06:00
Steve Loughran
70411cb1f1
HADOOP-17337. S3A NetworkBinding has a runtime dependency on shaded httpclient. (#2599) 
Contributed by Steve Loughran.

Change-Id: I0471322fc88d8bc3896ac439aefb31e6a856936c
 2021-02-03 14:32:55 +00:00
Steve Loughran
99337a4dd0
HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException. (#2648) 
When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.

Change-Id: I519d50ccd657968fd8ee72d132518099de901e15
 2021-02-02 18:17:38 +00:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
f97709beaa
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:52:08 +08:00
Gautham B A
58a9640f8e HADOOP-17508. Simplify deps install instructions (#2664) 
(cherry picked from commit a163aa7394)

 Conflicts:
	BUILDING.txt
 2021-02-02 05:37:19 +00:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Akira Ajisaka
98104537b2
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc0)
 2021-02-01 18:52:50 +09:00
Gautham B A
25677df019 HADOOP-17506. Fix typo in BUILDING.txt (#2662) 
(cherry picked from commit 9801b78edf)
 2021-01-31 14:23:28 +00:00
He Xiaoqiao
f61e3f5599
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 19:46:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:19:21 +00:00
Jim Brennan
2c70705575 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 16:51:40 +00:00
Jungtaek Lim
ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624) 
Contributed by Jungtaek Lim.

Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
 2021-01-27 19:08:54 +00:00
Steve Loughran
2d124f2f5e HADOOP-17483. Magic committer is enabled by default. (#2656) 
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
 2021-01-27 19:05:07 +00:00
Steve Loughran
3e1eb16837
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649) 
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.

Change-Id: I16db6e788c9fd628d3295671d7c2861c249d5ef1
 2021-01-27 16:40:27 +00:00
Steve Loughran
fb603e81f0
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530) 
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.

Change-Id: I4fe75b057159e35aacc072da3cb7343467c0c3f1
 2021-01-26 19:42:16 +00:00
Steve Loughran
bd85f6acea
HADOOP-17480. Document that AWS S3 is consistent and that S3Guard is not needed (#2636) 
Contributed by Steve Loughran.

Change-Id: I775e3ee7b60665240ec621859c337b053f747a49
 2021-01-25 13:24:34 +00:00
Akira Ajisaka
886b245de6
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a)
 2021-01-24 22:57:22 +09:00
Akira Ajisaka
7106a789f4
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6ba)
 2021-01-24 22:52:51 +09:00
Anton Kutuzov
dcf6d77279 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5)
 2021-01-22 18:09:57 -08:00
Mehakmeet Singh
d20b2deac3
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604) 
Contributed by Mehakmeet Singh.

Change-Id: I3445dec84b9b9e43bb1e41f709944ea05416bd74
 2021-01-22 14:21:31 +00:00