Your board and your community (which you represent) are too deeply invested to allow for a lack of proper oversight sidelining your CEO – and potentially your organization’s mission and vision. Remember, your role with the CEO is a partnership!
As a nonprofit consultant, I'm particularly cognizant of the issues raised by this article. I've heard that among the membership of just one national nonprofit association, in the last year there have been about 12 member organizations that have gone through tumultuous leadership shifts, with this being the primary issue.
Developing, protecting, and reconciling the relationship with their sole employee, the CEO, is their primary job.
In light of these concerns, one of the first assessments I recommend is a board survey to highlight any governance and policy issues or disconnect with the CEO. In a majority of my cases so far there have been indicators of a need for board/CEO training in monitoring and policy creation. Every effort at deepening the relationship and trust with your CEO will pay dividends.