org.kitesdk.data.Dataset.getNamespace()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(11.4k)|赞(0)|评价(0)|浏览(129)

本文整理了Java中org.kitesdk.data.Dataset.getNamespace()方法的一些代码示例，展示了Dataset.getNamespace()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Dataset.getNamespace()方法的具体详情如下：
包路径：org.kitesdk.data.Dataset
类名称：Dataset
方法名：getNamespace

Dataset.getNamespace介绍

[英]Get the namespace that contains this Dataset. A namespace is a string that identifies a logical group of datasets.
[中]获取包含此数据集的命名空间。命名空间是标识数据集逻辑组的字符串。

代码示例

代码示例来源：origin: org.kitesdk/kite-tools

private URI getLegacyRepoUri(Dataset<GenericRecord> dataset) {
 return FlumeConfigCommand.this.getLegacyRepoUri(dataset.getUri(), dataset.getNamespace());
}

代码示例来源：origin: kite-sdk/kite

private static DatasetRepository getDatasetRepository(JobContext jobContext) {
 Configuration conf = Hadoop.JobContext.getConfiguration.invoke(jobContext);
 DatasetRepository repo = DatasetRepositories.repositoryFor(conf.get(KITE_OUTPUT_URI));
 if (repo instanceof TemporaryDatasetRepositoryAccessor) {
  Dataset<Object> dataset = load(jobContext).getDataset();
  String namespace = dataset.getNamespace();
  repo = ((TemporaryDatasetRepositoryAccessor) repo)
    .getTemporaryRepository(namespace, getJobDatasetName(jobContext));
 }
 return repo;
}

代码示例来源：origin: org.kitesdk/kite-data-mapreduce

private static DatasetRepository getDatasetRepository(JobContext jobContext) {
 Configuration conf = Hadoop.JobContext.getConfiguration.invoke(jobContext);
 DatasetRepository repo = DatasetRepositories.repositoryFor(conf.get(KITE_OUTPUT_URI));
 if (repo instanceof TemporaryDatasetRepositoryAccessor) {
  Dataset<Object> dataset = load(jobContext).getDataset();
  String namespace = dataset.getNamespace();
  repo = ((TemporaryDatasetRepositoryAccessor) repo)
    .getTemporaryRepository(namespace, getJobDatasetName(jobContext));
 }
 return repo;
}

代码示例来源：origin: org.kitesdk/kite-tools

target.getDataset().getNamespace(),
UUID.randomUUID().toString());

代码示例来源：origin: org.kitesdk/kite-tools

target.getDataset().getNamespace(),
UUID.randomUUID().toString());

代码示例来源：origin: org.kitesdk/kite-tools

target.getDataset().getNamespace(),
UUID.randomUUID().toString());

代码示例来源：origin: kite-sdk/kite

boolean deleteAllUnsafe(boolean useTrash) {
 boolean deleted = false;
 if (dataset.getDescriptor().isPartitioned()) {
  for (StorageKey key : partitionIterator()) {
   deleted = (useTrash ? FileSystemUtil.cleanlyMoveToTrash(fs, root, key.getPath())
     : FileSystemUtil.cleanlyDelete(fs, root, key.getPath())) || deleted;
   if (listener != null) {
    // the relative path is the partition name, so we can simply delete it
    // in Hive
    listener.partitionDeleted(dataset.getNamespace(),
      dataset.getName(), key.getPath().toString());
   }
  }
 }
 else {
  for (Path path : pathIterator()) {
   deleted = (useTrash ? FileSystemUtil.cleanlyMoveToTrash(fs, root, path)
     : FileSystemUtil.cleanlyDelete(fs, root, path)) || deleted;
  }
 }
 return deleted;
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testRelative() {
 DatasetRepository repo = DatasetRepositories.repositoryFor("repo:file:target/data");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Record> ds = Datasets.<Record, Dataset<Record>>
   load("dataset:file:target/data/ns/test", Record.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Path cwd = localFS.makeQualified(new Path("."));
 Assert.assertEquals("Locations should match",
   new Path(cwd, "target/data/ns/test").toUri(), ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testRelative() {
 DatasetRepository repo = DatasetRepositories
   .repositoryFor("repo:hdfs://" + hdfsAuth + "/data?absolute=false");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Object> ds = Datasets.<Object, Dataset<Object>>
   load("dataset:hdfs://" + hdfsAuth + "/data/ns/test?absolute=false",
   Object.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Path cwd = getDFS().makeQualified(new Path("."));
 Assert.assertEquals("Locations should match",
   new Path(cwd, "data/ns/test").toUri(), ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testAbsolute() {
 DatasetRepository repo = DatasetRepositories.repositoryFor("repo:file:/tmp/data");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Record> ds = Datasets.<Record, Dataset<Record>>
   load("dataset:file:/tmp/data/ns/test", Record.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Assert.assertEquals("Locations should match",
   URI.create("file:/tmp/data/ns/test"),
   ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testAbsolute() {
 DatasetRepository repo = DatasetRepositories
   .repositoryFor("repo:hdfs://" + hdfsAuth + "/tmp/data");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Object> ds = Datasets.<Object, Dataset<Object>>
   load("dataset:hdfs://" + hdfsAuth + "/tmp/data/ns/test", Object.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Assert.assertEquals("Locations should match",
   URI.create("hdfs://" + hdfsAuth + "/tmp/data/ns/test"),
   ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testAbsoluteTrailingSlash() {
 DatasetRepository repo = DatasetRepositories
   .repositoryFor("repo:hdfs://" + hdfsAuth + "/tmp/data/");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Object> ds = Datasets.<Object, Dataset<Object>>
   load("dataset:hdfs://" + hdfsAuth + "/tmp/data/ns/test/", Object.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Assert.assertEquals("Locations should match",
   URI.create("hdfs://" + hdfsAuth + "/tmp/data/ns/test"),
   ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
 public void testAbsoluteWebHdfs() {
  Assume.assumeTrue(!Hadoop.isHadoop1());

  String webhdfsAuth = getConfiguration().get(
    DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_KEY);
  DatasetRepository repo = DatasetRepositories
    .repositoryFor("repo:webhdfs://" + webhdfsAuth + "/tmp/data");
  repo.delete("ns", "test");
  repo.create("ns", "test", descriptor);

  Dataset<Object> ds = Datasets.<Object, Dataset<Object>>
    load("dataset:webhdfs://" + webhdfsAuth + "/tmp/data/ns/test", Object.class);

  Assert.assertNotNull("Should load dataset", ds);
  Assert.assertTrue(ds instanceof FileSystemDataset);
  Assert.assertEquals("Locations should match",
    URI.create("webhdfs://" + webhdfsAuth + "/tmp/data/ns/test"),
    ds.getDescriptor().getLocation());
  Assert.assertEquals("Descriptors should match",
    repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
  Assert.assertEquals("Should report correct namespace",
    "ns", ds.getNamespace());
  Assert.assertEquals("Should report correct name",
    "test", ds.getName());

  repo.delete("ns", "test");
 }
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testAbsoluteRoot() {
 DatasetRepository repo = DatasetRepositories
   .repositoryFor("repo:hdfs://" + hdfsAuth + "/");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 Dataset<Object> ds = Datasets.<Object, Dataset<Object>>
   load("dataset:hdfs://" + hdfsAuth + "/ns/test",
   Object.class);
 Assert.assertNotNull("Should load dataset", ds);
 Assert.assertTrue(ds instanceof FileSystemDataset);
 Assert.assertEquals("Locations should match",
   URI.create("hdfs://" + hdfsAuth + "/ns/test"),
   ds.getDescriptor().getLocation());
 Assert.assertEquals("Descriptors should match",
   repo.load("ns", "test").getDescriptor(), ds.getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", ds.getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", ds.getName());
 repo.delete("ns", "test");
}

代码示例来源：origin: kite-sdk/kite

@Test
public void testViewConstraints() {
 DatasetRepository repo = DatasetRepositories.repositoryFor("repo:file:/tmp/data");
 repo.delete("ns", "test");
 repo.create("ns", "test", descriptor);
 RefinableView<Record> v = Datasets.<Record, RefinableView<Record>>
   load("view:file:/tmp/data/ns/test?username=user", Record.class);
 Assert.assertNotNull("Should load view", v);
 Assert.assertTrue(v instanceof FileSystemView);
 Assert.assertEquals("Locations should match",
   URI.create("file:/tmp/data/ns/test"),
   v.getDataset().getDescriptor().getLocation());
 DatasetDescriptor loaded = repo.load("ns", "test").getDescriptor();
 Assert.assertEquals("Descriptors should match",
   loaded, v.getDataset().getDescriptor());
 Assert.assertEquals("Should report correct namespace",
   "ns", v.getDataset().getNamespace());
 Assert.assertEquals("Should report correct name",
   "test", v.getDataset().getName());
 Constraints withUser = new Constraints(loaded.getSchema())
   .with("username", new Utf8("user"));
 Assert.assertEquals("Constraints should be username=user",
   withUser, ((FileSystemView) v).getConstraints());
 repo.delete("ns", "test");
}

内容来源于网络，如有侵权，请联系作者删除！

Dataset

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com