Ufdjrw

I have a class Indexer which is instantiated from the main thread, the instance of this class is stored in a variable, say, indexer. watchdog.observers.Observer() watches directories for changes and these happen in another thread. I tried passing this indexer variable from main thread through my handler Vigilante which was passed to ob.schedule(Vigilante(indexer)) alongside some other variables from main thread. I can't access the indexer variable in the Vigilante class, because of being in different threads. I know I could use a Queue but I don't know how I'd pass the Queue to watchdog's thread.

Here is the code from main thread:

if watch:
 import watchdog.observers
 from .utils import vigilante
 class callbacks:
 def __init__(self):
 pass
 @staticmethod
 def build(filename, response):
 return _build(filename, response)
 @staticmethod
 def renderer(src, mode):
 return render(src, mode)
 handler = vigilante.Vigilante(_filter, ignore, Indexer, callbacks, Mode)
 path_to_watch = os.path.normpath(os.path.join(workspace, '..'))
 ob = watchdog.observers.Observer()
 ob.schedule(handler, path=path_to_watch, recursive=True)
 ob.start()
 import time
 try:
 while True:
 time.sleep(1)
 except KeyboardInterrupt:
 ob.stop()
 Indexer.close()
 ob.join()

The Indexer class is meant to write to a database from another part of the code where the Indexer was instantiated.

Here is the code from Vigilante class running in watchdog's thread:

class Vigilante(PatternMatchingEventHandler):
 """Helps to watch files, directories for changes"""
 def __init__(self, pattern, ignore, indexer, callback, mode):
 pattern.append("*.yml")
 self.Callback = callback
 self.Mode = mode
 self.Indexer = indexer
 super(Vigilante, self).__init__(patterns=pattern, ignore_directories=ignore)

 def vigil(self, event):
 print(event.src_path, 'modified')
 IndexReader = self.Indexer.get_index_on(event.src_path)
 dep = IndexReader.read_index()
 print(dep.next(), 'dependency')
 feedout = self.Callback.build(
 os.path.basename(event.src_path)
 ,self.Callback.renderer(event.src_path, self.Mode.FILE_MODE)
 )

 def on_modified(self, event):
 self.vigil(event)

 def on_created(self, event):
 self.vigil(event)

All I need is a way to pass those variables from the main thread to watchdog's thread, through the Vigilante class

I finally found a way to do it without crossing threads too much as before, with an idea derived from @EvertW 's answer. I passed a Queue from main thread to the Vigilante class which was in another thread, so every modified file would be put in the Queue and then, from the main thread, I got the file modified from the queue, read from the Indexer database, and every other task which the Vigilante.vigil method needed to perform was moved to the main thread since those tasks depends on the modified file and what is read from the Indexer database.

This error disappeared:

SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 9788 and this is thread id 4288.

Here is a snippet from what I did:

....
q = Queue.LifoQueue(10)
handler = vigilante.Vigilante(q, _filter, ignore)
path_to_watch = os.path.normpath(os.path.join(workspace, '..'))
ob = watchdog.observers.Observer()
ob.schedule(handler, path=path_to_watch, recursive=True)
ob.start()
import time
try:
 while True:
 if not q.empty():
 modified = q.get()
 IndexReader = Indexer.get_index_on(modified)
 deps = IndexReader.read_index()
 print(deps.next(), 'dependency')
 # TODO
 else:
 print('q is empty')
 time.sleep(1)
except KeyboardInterrupt:
 ob.stop()
 Indexer.close()
ob.join()

Vigilante class:

class Vigilante(PatternMatchingEventHandler):
 """Helps to watch files, directories for changes"""
 def __init__(self, q, pattern, ignore):
 self.q = q
 super(Vigilante, self).__init__(
 patterns=pattern, 
 ignore_patterns=ignore, 
 ignore_directories=True
 )

 def vigil(self, event):
 print(event.src_path, 'modified')
 self.q.put(event.src_path)

 def on_modified(self, event):
 self.vigil(event)

 def on_created(self, event):
 self.vigil(event)
....

PS: A word of advice : my word of advice to whoever comes across this kind of problem with threading in watchdog is; "Don't trust the watchdog's thread to do tasks on modified files, just get the modified files out and do whatever you like with them, except the task is a simple one."

You could try the Observer pattern (no pun intended), i.e. let the Observer class have a list of listeners that it will inform of any changes it sees. Then let the indexer announce its interest to the Observer.

In my example, the Observer expects subscribers to be callables that receive the changes. Then you can do:

from queue import Queue
class Observable:
 def __init__(self):
 self.listeners = []
 def subscribe(listener):
 self.listeners.append(listener)
 def onNotify(change):
 for listener in self.listeners:
 listener(change)

class Indexer(Thread):
 def __init__(self, observer):
 Thread.__init__(self)
 self.q = Queue()
 observer.subscribe(self.q.put)
 def run(self):
 while True:
 change = self.q.get()

Because the standard Queue is completely thread-safe, this will work fine.