无法传输项目ioconfluent:kafka-connect-storage-common-parent:pom:6.0.0-snapshot from/to 合流(${confluent.maven.repo})

ibps3vxo  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(433)

我第一次尝试kafka connect,我想将sap s/4 hana连接到hive。我使用以下方法创建了sap s/4源kafka连接器:
https://github.com/sap/kafka-connect-sap
但是,我无法创建hdfs接收器连接器。这个问题与pom文件有关。
我试过了 mvn clean package . 但是,我有个错误:

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project io.confluent:kafka-connect-hdfs:[unknown-version] (/home/dev_dice_sa/test-dicestreaming/config/kafka-connect-hdfs/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13: Cannot access ${confluent.maven.repo} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

这是我的pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright 2018 Confluent Inc.
  ~
  ~ Licensed under the Confluent Community License (the "License"); you may not use
  ~ this file except in compliance with the License.  You may obtain a copy of the
  ~ License at
  ~
  ~ http://www.confluent.io/confluent-community-license
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
  ~ WARRANTIES OF ANY KIND, either express or implied.  See the License for the
  ~ specific language governing permissions and limitations under the License.
  -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-connect-storage-common-parent</artifactId>
        <version>6.0.0-SNAPSHOT</version>
    </parent>

    <artifactId>kafka-connect-hdfs</artifactId>
    <packaging>jar</packaging>
    <name>kafka-connect-hdfs</name>
    <organization>
        <name>Confluent, Inc.</name>
        <url>http://confluent.io</url>
    </organization>
    <url>http://confluent.io</url>
    <description>
        A Kafka Connect HDFS connector for copying data between Kafka and Hadoop HDFS.
    </description>

    <licenses>
        <license>
            <name>Confluent Community License</name>
            <url>http://www.confluent.io/confluent-community-license</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <scm>
        <connection>scm:git:git://github.com/confluentinc/kafka-connect-hdfs.git</connection>
        <developerConnection>scm:git:git@github.com:confluentinc/kafka-connect-hdfs.git</developerConnection>
        <url>https://github.com/confluentinc/kafka-connect-hdfs</url>
        <tag>HEAD</tag>
    </scm>

    <properties>
        <confluent.maven.repo>http://packages.confluent.io/maven/</confluent.maven.repo>
        <apacheds-jdbm1.version>2.0.0-M2</apacheds-jdbm1.version>
        <kafka.connect.maven.plugin.version>0.11.1</kafka.connect.maven.plugin.version>
    </properties>

    <repositories>
        <repository>
            <id>confluent</id>
            <name>Confluent</name>
            <url>${confluent.maven.repo}</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>connect-api</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>connect-json</artifactId>
            <version>${kafka.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-common</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-core</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-format</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-partitioner</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-wal</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-hive</artifactId>
            <version>${confluent.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty-all</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.github.spotbugs</groupId>
            <artifactId>spotbugs-annotations</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-minicluster</artifactId>
            <version>${hadoop.version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty-all</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-minikdc</artifactId>
            <version>${hadoop.version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <!-- exclude this from hadoop minikdc as the minikdc depends on
                    the apacheds-jdbm1 bundle, which is not available in maven central-->
                    <groupId>org.apache.directory.jdbm</groupId>
                    <artifactId>apacheds-jdbm1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <!-- add this to satisfy the dependency requirement of apacheds-jdbm1-->
            <groupId>org.apache.directory.jdbm</groupId>
            <artifactId>apacheds-jdbm1</artifactId>
            <version>${apacheds-jdbm1.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>io.confluent</groupId>
                <version>${kafka.connect.maven.plugin.version}</version>
                <artifactId>kafka-connect-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>kafka-connect</goal>
                        </goals>
                        <configuration>
                            <title>Kafka Connect HDFS</title>
                            <documentationUrl>https://docs.confluent.io/${project.version}/connect/connect-hdfs/docs/index.html</documentationUrl>
                            <description>
                                The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.

                                The connector periodically polls data from Kafka and writes them to HDFS. The data from each Kafka topic is partitioned by the provided partitioner and divided into chunks. Each chunk of data is represented as an HDFS file with topic, Kafka partition, start and end offsets of this data chunk in the filename. If no partitioner is specified in the configuration, the default partitioner which preserves the Kafka partitioning is used. The size of each data chunk is determined by the number of records written to HDFS, the time written to HDFS and schema compatibility.

                                The HDFS connector integrates with Hive and when it is enabled, the connector automatically creates an external Hive partitioned table for each Kafka topic and updates the table according to the available data in HDFS.
                            </description>

                            <supportProviderName>Confluent, Inc.</supportProviderName>
                            <supportSummary>Confluent supports the HDFS sink connector alongside community members as part of its Confluent Platform offering.</supportSummary>
                            <supportUrl>https://docs.confluent.io/current/</supportUrl>
                            <supportLogo>logos/confluent.png</supportLogo>

                            <ownerUsername>confluentinc</ownerUsername>
                            <ownerType>organization</ownerType>
                            <ownerName>Confluent, Inc.</ownerName>
                            <ownerUrl>https://confluent.io/</ownerUrl>
                            <ownerLogo>logos/confluent.png</ownerLogo>

                            <dockerNamespace>confluentinc</dockerNamespace>
                            <dockerName>cp-kafka-connect</dockerName>
                            <dockerTag>${project.version}</dockerTag>

                            <componentTypes>
                                <componentType>sink</componentType>
                            </componentTypes>

                            <tags>
                                <tag>hadoop</tag>
                                <tag>hdfs</tag>
                                <tag>hive</tag>
                            </tags>

                            <confluentControlCenterIntegration>true</confluentControlCenterIntegration>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <compilerArgs>
                        <arg>-Xlint:all,-deprecation,-processing</arg>
                        <arg>-Werror</arg>
                    </compilerArgs>
                    <showWarnings>true</showWarnings>
                    <showDeprecation>false</showDeprecation>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptors>
                        <descriptor>src/assembly/development.xml</descriptor>
                        <descriptor>src/assembly/package.xml</descriptor>
                    </descriptors>
                    <attach>false</attach>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <reuseForks>false</reuseForks>
                    <forkCount>1</forkCount>
                    <systemPropertyVariables>
                        <datanucleus.schema.autoCreateAll>true</datanucleus.schema.autoCreateAll>
                    </systemPropertyVariables>
                    <additionalClasspathElements>
                        <!--puts hive-site.xml on classpath - w/o HMS tables are not created-->
                        <additionalClasspathElement>src/test/resources/conf</additionalClasspathElement>
                    </additionalClasspathElements>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-checkstyle-plugin</artifactId>
                <executions>
                    <execution>
                        <id>validate</id>
                        <phase>validate</phase>
                        <configuration>
                            <suppressionsLocation>checkstyle/suppressions.xml</suppressionsLocation>
                        </configuration>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>.</directory>
                            <includes>
                                <include>derby.log</include>
                                <include>metastore_db/</include>
                            </includes>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
        </plugins>

        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
        </resources>
    </build>

    <profiles>
        <profile>
            <id>standalone</id>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-assembly-plugin</artifactId>
                        <configuration>
                            <descriptors>
                                <descriptor>src/assembly/standalone.xml</descriptor>
                            </descriptors>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>
</project>

我查看了以下链接:
https://github.com/confluentinc/kafka-connect-hdfs/wiki/faq
但是,我没能理解这个。请帮帮我。

cygmwpex

cygmwpex1#

我建议您下载现有的汇合平台,其中包括hdfs连接已经
否则, checkout 发布版本,而不是只 checkout 主分支来构建项目。

相关问题