Ufdjrw

Is 1 ppb equal to 1 μg/kg?

How does a Death Domain cleric's Touch of Death feature work with Touch-range spells delivered by familiars?

How can players work together to take actions that are otherwise impossible?

Center align columns in table ignoring minus signs?

Why aren't air breathing engines used as small first stages

Is there a service that would inform me whenever a new direct route is scheduled from a given airport?

Is the Standard Deduction better than Itemized when both are the same amount?

What are the pros and cons of Aerospike nosecones?

Proof involving the spectral radius and the Jordan canonical form

When to stop saving and start investing?

Right-skewed distribution with mean equals to mode?

Is there a documented rationale why the House Ways and Means chairman can demand tax info?

How discoverable are IPv6 addresses and AAAA names by potential attackers?

Is there a concise way to say "all of the X, one of each"?

How widely used is the term Treppenwitz? Is it something that most Germans know?

How much radiation do nuclear physics experiments expose researchers to nowadays?

What's the purpose of writing one's academic bio in 3rd person?

Did Kevin spill real chili?

How to bypass password on Windows XP account?

Diagram with tikz

Why did the IBM 650 use bi-quinary?

What makes black pepper strong or mild?

Disable hyphenation for an entire paragraph

Why does Python start at index -1 when indexing a list from the end?

spark structured stream read to hdfs files fails if data is read immediately

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

2

I'd like to load a Hive table (target_table) as a DataFrame after writing a new batch out to HDFS (target_table_dir) using Spark Structured Streaming as follows:

df.writeStream
 .trigger(processingTime='5 seconds')
 .foreachBatch(lambda df, partition_id:
 df.write
 .option("path", target_table_dir)
 .format("parquet")
 .mode("append")
 .saveAsTable(target_table))
 .start()

When we immediately read same data back from the Hive table we get a "partition not found exception". If we read with some delay, we have data correct.

It seems that Spark is still writing data to HDFS while execution has stopped and Hive Metastore is updated but data is still being written out to HDFS.

How to know when the writing of data to the Hive table (into the HDFS) is complete?

Note:
we have found that if we use processAllAvailable() after writing out,subsequent read works fine.but processAllAvailable() will block execution forever if we are dealing with continuous streams

apache-spark hive hdfs spark-structured-streaming

share|improve this question

edited Mar 15 at 10:59

Vish

asked Mar 8 at 16:08

VishVish

51961541

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  How do you do "when we immediately try to read same data back from hive table.we are getting partition not found exception."? How do you know when a table is appended (after a 5-sec trigger is executed)?
  
  – Jacek Laskowski
  Mar 13 at 17:47
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  we are not able to identify when table is appended.As part of workflow ,we are trying to read back data as next step and getting error.We need way to identify that table append is finished
  
  – Vish
  Mar 14 at 17:49
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JacekLaskowski ..we have found that if we use processAllAvailable() after writing out.subsequent read works fine..but processAllAvailable() will block execution forever if we are dealing with continuous streams.
  
  – Vish
  Mar 15 at 10:58
  
  

  
  
  
  

add a comment | 

2

I'd like to load a Hive table (target_table) as a DataFrame after writing a new batch out to HDFS (target_table_dir) using Spark Structured Streaming as follows:

df.writeStream
 .trigger(processingTime='5 seconds')
 .foreachBatch(lambda df, partition_id:
 df.write
 .option("path", target_table_dir)
 .format("parquet")
 .mode("append")
 .saveAsTable(target_table))
 .start()

When we immediately read same data back from the Hive table we get a "partition not found exception". If we read with some delay, we have data correct.

It seems that Spark is still writing data to HDFS while execution has stopped and Hive Metastore is updated but data is still being written out to HDFS.

How to know when the writing of data to the Hive table (into the HDFS) is complete?

Note:
we have found that if we use processAllAvailable() after writing out,subsequent read works fine.but processAllAvailable() will block execution forever if we are dealing with continuous streams

apache-spark hive hdfs spark-structured-streaming

share|improve this question

edited Mar 15 at 10:59

Vish

asked Mar 8 at 16:08

VishVish

51961541

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  How do you do "when we immediately try to read same data back from hive table.we are getting partition not found exception."? How do you know when a table is appended (after a 5-sec trigger is executed)?
  
  – Jacek Laskowski
  Mar 13 at 17:47
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  we are not able to identify when table is appended.As part of workflow ,we are trying to read back data as next step and getting error.We need way to identify that table append is finished
  
  – Vish
  Mar 14 at 17:49
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JacekLaskowski ..we have found that if we use processAllAvailable() after writing out.subsequent read works fine..but processAllAvailable() will block execution forever if we are dealing with continuous streams.
  
  – Vish
  Mar 15 at 10:58
  
  

  
  
  
  

add a comment | 

I'd like to load a Hive table (target_table) as a DataFrame after writing a new batch out to HDFS (target_table_dir) using Spark Structured Streaming as follows:

df.writeStream
 .trigger(processingTime='5 seconds')
 .foreachBatch(lambda df, partition_id:
 df.write
 .option("path", target_table_dir)
 .format("parquet")
 .mode("append")
 .saveAsTable(target_table))
 .start()

When we immediately read same data back from the Hive table we get a "partition not found exception". If we read with some delay, we have data correct.

It seems that Spark is still writing data to HDFS while execution has stopped and Hive Metastore is updated but data is still being written out to HDFS.

How to know when the writing of data to the Hive table (into the HDFS) is complete?

Note:
we have found that if we use processAllAvailable() after writing out,subsequent read works fine.but processAllAvailable() will block execution forever if we are dealing with continuous streams

apache-spark hive hdfs spark-structured-streaming

share|improve this question

edited Mar 15 at 10:59

Vish

asked Mar 8 at 16:08

VishVish

51961541

df.writeStream
 .trigger(processingTime='5 seconds')
 .foreachBatch(lambda df, partition_id:
 df.write
 .option("path", target_table_dir)
 .format("parquet")
 .mode("append")
 .saveAsTable(target_table))
 .start()

share|improve this question

edited Mar 15 at 10:59

Vish

asked Mar 8 at 16:08

VishVish

51961541

share|improve this question

edited Mar 15 at 10:59

Vish

asked Mar 8 at 16:08

VishVish

51961541

asked Mar 8 at 16:08

VishVish

51961541

asked Mar 8 at 16:08

VishVish

51961541