The Holographic masking system was a type of stealth technology used to create a false image of a starship for purposes of stealth and deception.
Conceptually, the process was quite simply. The vessel selects a preferred identity and then uses its array of holographic emitters to create the image of that ship around its self. Presumably, the technology was somewhat more sophisticated so that the user might take on the appropriate warp field frequency and other markers that would give away its true origins.
In the 24th century, the Federation implemented a version of the system which was deployed on the USS Incursion, a Starfleet Intelligence vessel which used the system to go undercover into enemy territory. The ability to hide amongst their enemies was a crucial factor in the Incursion's successful defeat of the Wardens in 2377 and later they were a crucial cog in the war with Species 8472, taking on the identity of 8472 vessels to infiltrate their space and locate installations.