You might still be able to brew with them as others have said but would you want to?
There are more than just the bugs themselves in your grain. Bugs defecate and their poop is called "frass". Like all critters their lifespan comes to an end meaning you have bug carcasses in your grain as well. Probably a lot of them since your second infestation is likely the offspring of the now dead first generation(s). So when you raise a glass of that beer try not to think about all the bug s*** and insect cadavers that went into it.
If you only knew how many insect parts, frass and mold are in foods that you eat every day: info below from CNN health . . .
The FDA has established standards to keep food defects to a minimum.
Peanut butter is one of the most controlled foods in the FDA list; an average of one or more rodent hairs and 30 (or so) insect fragments are allowed for every 100 grams, which is 3.5 ounces. Apple butter can also contain up to 12% mold, which is better than cherry jam, which can be 30% moldy, or black currant jam, which can be 75% moldy.
And the list goes on and on.
Fortunately AFAIK, e coli and norovirus can't grow in beer and unlike eating at certain restaurants that made marketing campaigns touting their organic ingredients, but whose patrons fell ill from eating food contaminated with the above organisms, beer is safe from microbes that will make you sick.