org.apache.hadoop.io.nativeio.NativeIO.copyFileUnbuffered()方法的使用及代码示例

x33g5p2x  于2022-01-24 转载在 其他  
字(3.2k)|赞(0)|评价(0)|浏览(159)

本文整理了Java中org.apache.hadoop.io.nativeio.NativeIO.copyFileUnbuffered()方法的一些代码示例,展示了NativeIO.copyFileUnbuffered()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。NativeIO.copyFileUnbuffered()方法的具体详情如下:
包路径:org.apache.hadoop.io.nativeio.NativeIO
类名称:NativeIO
方法名:copyFileUnbuffered

NativeIO.copyFileUnbuffered介绍

[英]Unbuffered file copy from src to dst without tainting OS buffer cache In POSIX platform: It uses FileChannel#transferTo() which internally attempts unbuffered IO on OS with native sendfile64() support and falls back to buffered IO otherwise. It minimizes the number of FileChannel#transferTo call by passing the the src file size directly instead of a smaller size as the 3rd parameter. This saves the number of sendfile64() system call when native sendfile64() is supported. In the two fall back cases where sendfile is not supported, FileChannle#transferTo already has its own batching of size 8 MB and 8 KB, respectively. In Windows Platform: It uses its own native wrapper of CopyFileEx with COPY_FILE_NO_BUFFERING flag, which is supported on Windows Server 2008 and above. Ideally, we should use FileChannel#transferTo() across both POSIX and Windows platform. Unfortunately, the wrapper(Java_sun_nio_ch_FileChannelImpl_transferTo0) used by FileChannel#transferTo for unbuffered IO is not implemented on Windows. Based on OpenJDK 6/7/8 source code, Java_sun_nio_ch_FileChannelImpl_transferTo0 on Windows simply returns IOS_UNSUPPORTED. Note: This simple native wrapper does minimal parameter checking before copy and consistency check (e.g., size) after copy. It is recommended to use wrapper function like the Storage#nativeCopyFileUnbuffered() function in hadoop-hdfs with pre/post copy checks.
[中]从src到dst的无缓冲文件拷贝不会污染POSIX平台中的OS缓冲区缓存:它使用FileChannel#transferTo()在内部尝试在支持原生sendfile64()的OS上进行无缓冲IO,否则会退回到缓冲IO。它通过直接传递src文件大小,而不是将较小的大小作为第三个参数,来最小化FileChannel#transferTo调用的数量。当支持本机sendfile64()时,这将保存sendfile64()系统调用的次数。在不支持sendfile的两种回退情况下,FileChannle#transferTo已经有了自己的批处理,大小分别为8MB和8KB。在Windows平台中:它使用自己的CopyFileEx本机包装器,带有COPY_FILE_NO_BUFFERING标志,Windows Server 2008及更高版本支持该标志。理想情况下,我们应该在POSIX和Windows平台上使用FileChannel#transferTo()。不幸的是,FileChannel#transferTo用于无缓冲IO的包装器(Java_sun_nio_Chu FileChannelImpl_transferTo0)没有在Windows上实现。基于OpenJDK 6/7/8源代码,Windows上的Java_sun_nio_ch_FileChannelImpl_transferTo0只返回不支持的IOS_。注意:这个简单的本机包装器在复制之前进行最小的参数检查,在复制之后进行一致性检查(例如大小)。建议使用包装函数,比如hadoop hdfs中的Storage#nativeCopyFileUnbuffered()函数,并进行复制前/复制后检查。

代码示例

代码示例来源:origin: org.apache.hadoop/hadoop-hdfs

NativeIO.copyFileUnbuffered(srcFile, destFile);
} catch (NativeIOException e) {
 throw new IOException("Failed to copy " + srcFile.getCanonicalPath()

代码示例来源:origin: ch.cern.hadoop/hadoop-hdfs

NativeIO.copyFileUnbuffered(srcFile, destFile);
} catch (NativeIOException e) {
 throw new IOException("Failed to copy " + srcFile.getCanonicalPath()

代码示例来源:origin: io.prestosql.hadoop/hadoop-apache

NativeIO.copyFileUnbuffered(srcFile, destFile);
} catch (NativeIOException e) {
 throw new IOException("Failed to copy " + srcFile.getCanonicalPath()

代码示例来源:origin: com.github.jiayuhan-it/hadoop-common

mapBuf.put(bytesToWrite);
 NativeIO.copyFileUnbuffered(srcFile, dstFile);
 Assert.assertEquals(srcFile.length(), dstFile.length());
} finally {

代码示例来源:origin: ch.cern.hadoop/hadoop-common

mapBuf.put(bytesToWrite);
 NativeIO.copyFileUnbuffered(srcFile, dstFile);
 Assert.assertEquals(srcFile.length(), dstFile.length());
} finally {

相关文章