There are many things the Bible does not outright forbid, yet we know they are better off abstained from. Even though we know we aren't sinning, it could cause others to sin.
Am I My Brother's Keeper???
Well no... but if you cause another to stumble even though you know you are in the right, according to Paul, you have destroyed your brother with your knowledge.