BTOP Reviewer

It's pretty clear by now that my services won't be needed as a BTOP reviewer so I think I'll bookend this little mini-blog with a final post and call it quits.  This has been a weird ride so I thought I'd conclude with a few observations.

I hope the Obama Administration learned a lot from this cluster-fuck and don't repeat their mistakes the next time around.  Here are some things to avoid and lessons learned (not that they're going to listen to me, after all they haven't communicated with me at all up 'til now so why should things change?).

• Don't use volunteer reviewers in the future. Period.  I've gotten scuttlebutt that BTOP switched over to contracted reviewers, maybe from Booz Allen, in mid-stream and I think that's a great idea.  Using volunteers was a bad idea for several reasons.  How on earth are you going to assure quality?  What about impartiality and conflict of interest?  How are you going to manage the 1000+ people you need, but don't know and haven't vetted?
• For cripes sake do some planning. This mess was easy to see coming.  The next round will have the same logistical problems unless they get ahead of the game.
• Live by the "transparency" credo. The Obama Administration talks a lot about open/transparent processes.  This one wasn't.  Reviewers were confronted with a lot of "non-public" information that didn't need to be non-public.  It got ridiculous after a while.  They treated the training materials as non-public!
• Don't use Mushroom Management the next time. You know about Mushroom Management?  Mushroom Management is "keep 'em in the dark and feed them bullshit" whether you're growing mushrooms or managing processes.  This was a classic case.  Why couldn't they send the occasional email to volunteers letting them know what's up?  Hell, if you don't want me that's fine.  But send me a freekin' email telling me, don't just sit there silent and paralyzed in your Washington DC ivory tower.
• Treat the other reviewers better. My state, Minnesota, got sucked into a PR disaster when the BTOP folks suddenly punted all the applications --un-ranked, un-reviewed -- to the State and told the State folks they had to rank them in a super-short period of time.  The State folks were expecting reviewed applications, they got an un-reviewed pile.  So instead of having applications that had been vetted in depth by experts, they got this pile that they had to wade through on their own with virtually no assistance or guidance from BTOP.  There's a rumpus now because the State fell back on the rules that they apply to RFPs and kept their recommendations and process private until the grants are awarded.  People have been on me to raise my voice on that one -- but I'm on the State's side at this point.  This is BTOP's cluster-fuck not Minnesota's, you handed my state's people a steaming pile of crap and let them take the PR hit.  I talked to a friend at a carrier who's dealing with the same kind of problem.

I could go on, but you get the drift.  Did I mention cluster-fuck?  Oh, I did?  Several times?  Good.  This one is fucked up by the numbers.  I sure hope they do better the next time.

Signing off...

Thanks to all who've posted comments -- it's my only source of information about what's going on with my role as a BTOP reviewer.  Haven't heard a peep from the NTIA folks, so it's nice to have some companions.  Ron and Stacy commented that they're in pretty much the same boat -- no news from BTOP.  Harold dashed my hopes but then revived them when he realized that he was going through the same training I had already attended.

Reiner has actually been assigned credentials to get into the system to review grants! Here's the full comment (from Sept 14th);

My panels / apps along with the Easygrant login credentials came in on the 11th, but no word from the panel coordinator yet (so I'm not clear on deadlines). If however my assignments were representive of the average, and assuming that it's going to be a 2-week process as NTIA initially indicated, it boils down to approximately 230 proposal pages per day.

Sept 14th is an interesting date -- I think that was the original deadline by which us reviewers were supposed to be done.  As I have been saying, I haven't heard anything from Central Command, but I think it's a safe bet we're going to miss that deadline. 

Meanwhile, BTOP has punted all the applications to the States for their review.  I'd heard about that from my pals in state government here in Minnesota.  They, like pretty much all states, are scrambling to meet the deadline that they've been handed.  Click HERE to read a comprehensive piece by Geoff Daily over at App-Rising.

Pretty soon here I'm going to have to drop off the "reviewer-candidate" list because other commitments are coming due.  I volunteered thinking that the review-work would fit nicely in my schedule because it was going to be done by mid-September.  But if a pile of apps land on my desk now, I'm in a pickle.  I bet other reviewers are in the same boat.

Pretty soon I'll write a post about how I'd spin this if I were BTOP.  The short-preview version is to get out there loud and proud instead of emitting silence.  I'd be saying "wow! We're overwhelmed with success!  We've got way more great applications than we ever dreamed we'd get, and a boatload of reviewers."  I think it's too bad that they've let themselves get behind the PR curve -- but I agree with Geoff that it's still recoverable, and too important to let fail.

Geoff Daily (over at App-Rising) has been blogging about this BTOP process and I agree with him.  Here's the lead from his latest post (click HERE to read the whole thing):

Over the weekend I learned something about NTIA's handling of BTOP that shook me to the core: last week they dumped all of their applications onto states without conducting any initial review to weed out ineligible projects or including any guidelines to help frame how states should go about making their recommendations.

While there's a chance they're going to announce an initial round of cuts today, I'm not optimistic given that the volunteer reviewers I know haven't been asked to do anything yet.

And while they're supposed to give guidelines to states this week, I'm not sure how those guidelines can overcome the many problems with relying on states to play a primary role in deciding who gets what. For one, most states don't have any formal decision-making process or the in-house tech or business expertise to enable them to pick the right projects. For two, few if any states have thoroughly read and tried to understand the NOFA, so they're not ready to pick the best projects relative to the original rules that were set out. Three, there's no end to the potential conflicts of interest that could negatively influence a state's recommendations, including the fact that some states have put in their own applications.

If I'm being brutally honest then I can't help but admit feeling like NTIA dumping these applications on states is a sign that they're punting, that they'd rather someone else make the tough decisions, and that they're positioning themselves to be able to deflect blame to the states if any bad projects end up getting funded.

I'm starting to get the same feeling, especially given I haven't heard a peep from the BTOP folks since I went through my initial reviewer-training webinar a week or so ago.  Geoff and I had breakfast together a few weeks ago and he included some of my ideas in an earlier post on the same subject (click HERE to read the whole post).  Here's a bit of what we talked about that rainy morning in Minneapolis:

My first concern about this review process doesn't stem from a blog post of Mike's but something he told me during a face-to-face meeting, namely how they're vetting reviewers to avoid conflicts of interest. Apparently all that Mike knows they've done so far is have him sign a piece of paper claiming he has no conflicts. There doesn't appear to be any other serious vetting going on of him.

Even if there is, I can't help be concerned that a reviewer would be totally unaware that it's going on, plus there's the matter of thoroughly vetting the thousand people who signed up to volunteer taking some significant time, and yet it appears as though they're either not doing anything or haven't started doing anything, both of which are troubling.

Many have been critical of using an all-volunteer army from the get go as it seems wide open to attempts to game the system, plus it can be a challenge to make sure that not only do the volunteers not have conflicts of interest, but also that they're competent.

But there are other issues with using volunteers that Mike brings up. Like how many hours in how short of a window of time are volunteers expected to invest in this process? He's heard 30-40 hours, but will he have a month to put that time in or a week? I don't think we can expect volunteers to devote full-time work to this, and yet stretching it out means the review process will drag on. Also, what support will reviewers have during odd hours like nights and weekends, which is likely when many of the will be working through these applications?

But none of this is what has me worried the most. What worries me more than anything is that after Mike went through an introductory orientation webinar on Aug 28th, he has not heard from NTIA since then. He has no idea if the review process has started yet. If he's made the final cut of reviewers. If a pile of applications is in the mail on its way to him as I type this. Nothing.

And yet a quick glance at the calendar shows that NTIA's supposed to have some initial recommendations to pass on to states for their input on Monday. That's four days from now.

So there you have it.  My status?  Silence.  I haven't a clue.

Harold just posted a comment on my "no news" post saying that he's got webinars scheduled for this Monday and the following one.  He says that about 600 reviewers have made it to that step.  Thanks for putting me out of my misery Harold. 

Since I haven't been invited to join those, I guess I'm out of it.  But I think I'll keep blogging away for a while.  I have some observations to share about the process to date, now that my brief run is over.

UPDATE;

Harold elaborated a bit in a comment to this post -- it turns out he's attending the initial webinar, which I've already attended.  So maybe I'm still in the game.  T'would be nice to have a sense of what's going on. 

Hm.  Things may be pretty lively back at the BTOP mother-ship.  "The Commerce Department's Inspector General (IG) is conducting a review of its handling of the Broadband Technology Opportunities Program (BTOP) portion of the economic stimulus package, according to the IG's office."

Click HERE to read the article.

Hi gang, sorry to be so quiet.  I've sortof been waiting for something to happen, but...  um...  nothing has.  So there wasn't much to write about.  But with the deadline growing super-close, maybe that "nothing" is noteworthy.

My sense is that the Next Step will be the arrival of a gaggle of applications to look at.  Or not (if it's decided that I'm not a good fit with any applications to review).

I decided to post today just to let you know that I'm still here, and still waiting to get started.  Are there any other reviewers out there that have started the review process?  Just curious.  Feel free to post in the comments if you have.

Dear BTOP-review leader type people -- an update on where things are at would be helpful.

I was glad to see that we're going to have a help desk available to us while we're working on these reviews.  But I wonder if it's going to be staffed when most of us are going to be doing the work -- i.e. "nights and weekends."   I'm the exception in that I'll be doing a fair amount of this during the normal work week -- that's the beauty of being retired.  But even I tend to do a lot of stuff outside normal working hours and having to wait until the next workday is going to be a hassle.  I just fired off a note to the leader type people saying this.  Fingers crossed.

I've never done this kind of reviewer type thing before -- so I just fired off a question to the leader type people about "non-public" information.  Our non-disclosure agreement prohibits us from sharing non-public info.  I get that -- these applications contain a lot of sensitive information that shouldn't get into the wild.  But what about the reviewer process itself?  I may have already strayed outside the lines by mentioning the "1000 resumes" thingy in the last post -- that statement came from the webinar presenter and, since they didn't mention anything about the webinar being "non-public" I just parroted it out to this blog.

I guess I'm really pondering two questions.

• Which info is non-public?
• Which info should be non-public?

My bias is towards transparency unless the information is truly confidential.  But the Reviewer Guidance document has "not for public disclosure" across the top of every page so maybe I'm in violation of even mentioning that there is a reviewer guidance document, never mind the little bit of content that says that it's not public.  I think it would be useful if the document could be segregated so that the purely process stuff could be public, and the sensitive stuff (like the criteria we're supposed to be using in our evaluations) curtained off in a narrower document.  Throwing a really wide non-public blanket over our training and support materials a) makes us less transparent and b) makes us less effective because we can't talk amongst ourselves very easily.

I decided to post about my experience as a BTOP grant reviewer on this site after sitting through the initial webinar today.  It may just wind up being tips and techniques, or it may evolve into something more.  Any other reviewers out there?  I'd love to hear from you.  Post your "howdy Mike" message in the comments to this post, or send me email.  I'll build a "contact me" page in a minute...

A few tidbits from the webinar today.

• We are 1000 strong (that's roughly the number of resumes the NTIA folks received)
• There are 2200 applications
• There will be at least 3 reviewers per application
• So that means we each will be reviewing about 7 applications (6600 application/reviewer slots, 1000 reviewers)
• We'll be getting our assignments pretty soon and the goal is to start making recommendations in mid-September

I captured most of the slides from the webinar and asked the question "are these public information, something I can post to the Internet?"  Didn't get an answer so I guess I'll hold my fire for a while.  I would presume that most of the reviewer-instructions info we're going to receive will be public.

Anyway, here we go!