Compression is a key component in many imaging systems in order to accommodate limited resources such as power and bandwidth. Image compression is often done independent of the specific tasks that the systems are designed for, such as target detection, classification, diagnosis, etc. Standard compression techniques are designed based on quality metrics such as mean-squared error (MSE) or peak signal to noise ratio (PSNR). Recently, a metric based on task-specific information (TSI) was proposed and successfully incorporated into JPEG2000 encoding. It has been shown that the proposed TSI metric can optimize the task performance. In this work, a joint metric is proposed to provide a seamless transition between the conventional quality metric MSE and the recently proposed TSI. We demonstrate the effectiveness and flexibility of the proposed joint TSI metric for target detection tasks. Furthermore, it is extended to video tracking applications to demonstrate the robustness of the proposed metric. Experimental results show that although the metric is not directly designed for the applied task, better tracking performance can still be achieved when the joint metric is used, compared to results obtained with the traditional MSE metric.