In order to detect the object in video efficiently, an automatic and real time video segmentation algorithmbased on background model and color clustering is proposed. This algorithm consists of four phases:backgroundrestoration, moving objects extract,moving objects region clustering and post processing. The threshold of thebackground restoration is not given in advanced. It can be gotten automatically.And a new object region cluster