Average time on site is flawed. TOTAL time on site (TTS) is more robust. It’s the total time spent on the site by all (target) users.
With this metric: some users spending more time on the site is always good; more users is always good.
Other issues are detailed here. For example, “bounces” count as zero seconds on site. (Some people may argue that penalising bounces is a good thing).
If your web analytics don’t support TTS then you can calculate an approximation: (Average Time on Site) x (Unique Users)
As for all metrics, it’s good to focus on your target audience. If you use Ucom or comScore then you could calculate a multiplier. For example, if you reach 25% more of your target audience then multiply TTS by 1.25