Information and our models of reality

So I am going back to our business intelligence and systems theme and getting a little more philosophical here. I wanted to explore information at a more fundamental level. Some of this is probably way over my head, but hey these are just the things I think about at night (and yet I wonder why my family thinks I am weird!).
 
I was thinking about how information is related to our mental models. At the core, most information is really an abstract model of reality that we use to understand things. We use the information that we have to construct a mental model of how the world works. Once we have that model in our heads we take actions based on what our particular mental models say will lead to a desired outcome. The problems and/or opportunities arise when we get information that contradicts how our mental model tells us things should be working. What should we do with the fact that there is a difference between what our mental model predicts for the outcome and the data that we are looking at?

I would argue that one of the first things most people do is assume that there was an error in how the data as abstracted up to the level that we are looking at. An example here might help. Assume that you are looking at an accounting report and you see that one of your main divisions reported only five dollars in revenue for a month. Last year that same division reported five million dollars of revenue. Your first assumption is probably to assume that there is an error in how the data was put into the system and moved up through the abstraction chain to your report. The first and usually logical step we want to perform when we see a difference is to trace back through the levels of abstraction to ensure there were no errors. In other words we (or someone we delegate to) drill down through the details to ensure that there are no errors from an overall system perspective.
 
Let's assume that we fail to find any errors in how the data was put into the system or in any of the calculations used to roll the information up to our report. So what does the difference in the number in the report imply at this point? The key is the difference in the numbers - it implies that something in our mental model of how that division is supposed to work is out of alignment compared to reality. So the next step then is to look at our mental model and make adjustments so that it better fits reality.

My thought is that most people today don't have a good understanding of their mental models that they are using and so don't understand where the best point is to make adjustments. From a business intelligence perspective I see two challenges. The first lies in understanding how BI tools and processes can be used to help people validate information that flows through the systems so that they know the data is accurate. The second challenge is how to we help users understand the mental models that are in use when they are looking at data and where the best leverage is for making changes.