Ufdjrw

I am trying to access a file in alluxio in a scala code in the IDE and i am getting this error Exception in thread "main" java.io.IOException: No FileSystem for scheme: alluxio

My code is as follows,

package com.example.sparkalliuxiodemo

import org.apache.spark.sql.SparkSession

object TestMain 
 def main(args: Array[String]): Unit = 
 val spark = SparkSession.builder().config("spark.master", "local[2]")
 .getOrCreate()

 var df = spark.read.parquet("alluxio://localhost:19998/sample-df")
 df.printSchema()

 

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.example</groupId>
 <artifactId>spark-alluxio-demo</artifactId>
 <version>0.0.1-SNAPSHOT</version>

 <dependencies>
 <dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-core_2.12</artifactId>
 <version>2.4.0</version>
 </dependency>
 <dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-sql_2.12</artifactId>
 <version>2.4.0</version>
 </dependency>

 <dependency>
 <groupId>org.alluxio</groupId>
 <artifactId>alluxio-core-client</artifactId>
 <version>1.8.1</version>
 <type>pom</type>
 </dependency>

 <dependency>
 <groupId>org.alluxio</groupId>
 <artifactId>alluxio-core-server</artifactId>
 <version>1.8.1</version>
 <type>pom</type>
 </dependency>


 <dependency>
 <groupId>org.alluxio</groupId>
 <artifactId>alluxio-core-common</artifactId>
 <version>1.8.1</version>
 </dependency>


 <dependency>
 <groupId>org.alluxio</groupId>
 <artifactId>alluxio-core-client-fs</artifactId>
 <version>1.8.1</version>
 </dependency>


 </dependencies>
</project>

I am able to access the alluxio file system in the Spark Installed directory via Shell. If I try to access the same via IDE, including the above alluxio dependencies in the POM.xml I am getting the above error.

If I place this jar `alluxio-1.8.1-client.jar' manually into my build path I am able to access the file in alluxio with out any error in IDE itself.

I need to access the file in alluxio directly in eclipse or scala-ide.

Can any one suggest me the right alluxio dependencies to include in the maven dependencies?

Try adding

 <dependency>
 <groupId>org.alluxio</groupId>
 <artifactId>alluxio-core-client-hdfs</artifactId>
 <version>1.8.1</version>
 </dependency>