Would it really matter?

So i mentioned earlier how we could never know if someone had changed the course of time because we would unconciously have different lives. But if we would never know about it then would it really matter or affect us at all if this were to occur? You would think that it was simply your normal life and nothing had ever been any different. So why would it matter then if it would never "affect" you? Well perhaps your life would have been better before a time traveler changed it and it is unfair that your life had to be made worse. Or vis versa. Someones life could have been much worse but then it was made much better. The cycle balances itself out. So do you really care if your life was altered because of a time traveler but you would never be aware of the change? Feel free to answer the poll given concerning this question.