big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDFS-11067. DFS#listStatusIterator(..) should throw FileNotFoundException if the directory deleted before fetching next batch of entries. Contributed by Vinayakumar B.

Browse Source
This commit is contained in:
Vinayakumar B 2017-06-22 17:35:40 +05:30
parent b64951905e
commit 8dbd53ef9f
4 changed files with 31 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
View File

@ -1185,6 +1185,10 @@ on (possibly remote) filesystems. These filesystems are invariably accessed
concurrently; the state of the filesystem MAY change between a `hasNext()` concurrently; the state of the filesystem MAY change between a `hasNext()`
probe and the invocation of the `next()` call. probe and the invocation of the `next()` call.
During iteration through a `RemoteIterator`, if the directory is deleted on
remote filesystem, then `hasNext()` or `next()` call may throw
`FileNotFoundException`.
Accordingly, a robust iteration through a `RemoteIterator` would catch and Accordingly, a robust iteration through a `RemoteIterator` would catch and
discard `NoSuchElementException` exceptions raised during the process, which discard `NoSuchElementException` exceptions raised during the process, which
could be done through the `while(true)` iteration example above, or could be done through the `while(true)` iteration example above, or

2
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java
View File

@ -232,7 +232,7 @@ public boolean hasNext() throws IOException {
thisListing = dfs.listPaths(src, thisListing.getLastName(), thisListing = dfs.listPaths(src, thisListing.getLastName(),
needLocation); needLocation);
if (thisListing == null) { if (thisListing == null) {
return false; // the directory is deleted throw new FileNotFoundException("File " + src + " does not exist.");
} }
i = 0; i = 0;
} }

2
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
View File

@ -1168,7 +1168,7 @@ private boolean hasNextNoFilter() throws IOException {
needLocation); needLocation);
statistics.incrementReadOps(1); statistics.incrementReadOps(1);
if (thisListing == null) { if (thisListing == null) {
return false; throw new FileNotFoundException("File " + p + " does not exist.");
} }
i = 0; i = 0;
} }

27
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatus.java
View File

@ -317,8 +317,31 @@ public void testGetFileStatusOnDir() throws Exception {
assertEquals(file3.toString(), itor.next().getPath().toString()); assertEquals(file3.toString(), itor.next().getPath().toString());
assertFalse(itor.hasNext()); assertFalse(itor.hasNext());
fs.delete(dir, true); itor = fs.listStatusIterator(dir);
assertEquals(dir3.toString(), itor.next().getPath().toString());
assertEquals(dir4.toString(), itor.next().getPath().toString());
fs.delete(dir.getParent(), true);
try {
itor.hasNext();
fail("FileNotFoundException expected");
} catch (FileNotFoundException fnfe) {
}
fs.mkdirs(file2);
fs.mkdirs(dir3);
fs.mkdirs(dir4);
fs.mkdirs(dir5);
itor = fs.listStatusIterator(dir);
int count = 0;
try {
fs.delete(dir.getParent(), true);
while (itor.next() != null) {
count++;
}
fail("FileNotFoundException expected");
} catch (FileNotFoundException fnfe) {
}
assertEquals(2, count);
} }
} }