Can you be charged for NOT using a part-time bus lane outside it's hours of operation?

There is a stretch of road near me like that , people often queue on the right and give me dirty looks when I sail p in the temporary bus lane, but there is a large sign saying the bus lane is only in use from 7 til 10 & 4 til 7 pm , temporary lanes always have signs showing time restrictions so there is no excuse really.