So, I've looked further into this bug and what seems to cause it. Apparently, the issue is with the workers not having the needed raw material assigned to them.
As far as I can tell, the root cause of this bug is that one of those buildings get assigned all available materials in the area. E.g. by having tow stone refineries next to each other, one will get assigned all the stone of the next warehouse/mine while the other one will get none. What is more, stone that is further away (not yet assigned to the first refinery) won't be brought to the second refinery, since stone is available at closer locations (although already being assigned), leading to a deadlock situation for the second refinery.
Sadly, even the courier available later in the game does not help to solve this issue. However, a stone mine and refinery built far away works just fine.
What is more, laborer also do not seem to care about buildings not having any raw materials while having nothing to do. In the picture below, you can see a bunch of laborers in the center "Doing Nothing" while clearly the restaurant in the background has no raw food (marked with blue flag). At the same time raw food would be available in the warehouse next to them. However, neither the restaurant’s cooks nor the available laborers seemed to do anything about that, probably because there is another restaurant not far away to the left of the camera.
FYI except for delivering resources to buildings, there was literally nothing to do for the laborers.
I'd be great if anyone could confirm this bug/theory, i.e. is anyone experiencing similar issues?