The compute and control for adaptive optics (cacao) package is an open-source modular software environment for real-time control of modern adaptive optics system. By leveraging many-core CPU and GPU hardware, it can scale up to meet the demanding computing requirements of current and future high frame rate, high actuator count adaptive optics (AO) systems. cacao's modular design enables both simple/barebone operation, and complex full-featured AO control systems. cacao's design is centered on data streams that hold real-time data in shared memory along with a synchronization mechanism for computing processes. Users and programmers can add additional features by coding modules that interact with cacao's data stream format. We describe cacao's architecture and its design approach. We show that accurate timing knowledge is key to many of cacao's advanced operation modes. We discuss current and future development priorities, including support for machine learning to provide real-time optimization of complex AO systems.