About CivicWatchBot

If you are reading this because you saw CivicWatchBot in a server log: hello. This page explains exactly what it is doing and how to make it stop.

What it does

CivicWatchBot visits the public websites of local government bodies in Shelby, Moultrie and Effingham counties, Illinois, and keeps a copy of the documents they publish — agendas, minutes, packets, ordinances, resolutions and notices. It records the time each copy was taken and fingerprints it, so that a document can later be shown to be the one that was published on a given day.

It reads only pages that are already public. It does not attempt to log in, does not submit forms, does not follow links behind an authentication prompt, and does not collect anything about the people who visit your site.

How it behaves

• It obeys robots.txt. If your robots.txt disallows a path, CivicWatchBot does not fetch it. If robots.txt cannot be read at all, it treats that as a refusal and fetches nothing.
• It is slow on purpose. One request at a time per site, with a pause between them. It is not trying to mirror your site quickly.
• It prefers off-hours. Crawls are paced to avoid the working day where practical.
• It does not re-download what has not changed. Content is stored by fingerprint, so an unchanged file is fetched once and not again.
• It identifies itself honestly. The User-Agent above is the one it always sends. It does not impersonate a browser.

How to make it stop

Add this to your robots.txt and it will stop on its next visit:

User-agent: CivicWatchBot
Disallow: /

Or write to us and we will remove your site by hand, without argument and without asking why. We would rather you tell us than block us silently, because a body we cannot read is recorded here as a gap in our coverage — not as a body that published nothing. Those are very different claims and we are careful never to confuse them.

If something looks wrong

If CivicWatchBot is hitting your site too hard, fetching something it should not, or appearing to misbehave in any way, tell us and we will stop it that day. Please include the timestamps and paths from your log if you have them.

Contact a human

Email libermediasites@gmail.com. A person reads it.

More detail on what we crawl and what a timestamp does and does not prove is on the about page.